记录租户间网络不通且出不了公网

问题

平台上,组应用无法启动,初步检查A应用与B应用网络不通,且B应用无法出公网.

解决思路

查看应用状态

平台应用A(调度在c1),B(调度在c2) A 可以出公网 B 不可以出公网 同租户间网络不通

检查是否物理机问题

选择某一租户

调度在c1机器上的应用a1,a2 查看应用a1,a2是否可以互通,是否可以出公网. 经测试c1机器上应用正常

调度在c2机器上的应用b1,b2 查看应用b1,b2是否可以互通,是否可以出公网. 经测试c2机器上应用存在上述问题

检查c2机器网络问题

由于网络使用的是midolman,查看midolman日志 /var/log/midolman/midolman.log

2017.08.23 09:40:16.557 WARN  [midolman-akka.actor.default-dispatcher-6]
datapath-control -  Failed to create port vif6519ded4f34d: java.io.IOException:
failed to connect to socket: Bad file descriptor
2017.08.23 09:40:16.557 WARN  [midolman-akka.actor.default-dispatcher-6]
datapath-control -  Failed to create port vif4ab5ef7d1ca5: java.io.IOException:
failed to connect to socket: Bad file descriptor
2017.08.23 09:40:16.557 WARN  [midolman-akka.actor.default-dispatcher-6]
datapath-control -  Failed to create port vif984e5ee724ed: java.io.IOException:
failed to connect to socket: Bad file descriptor
2017.08.23 09:40:16.557 WARN  [midolman-akka.actor.default-dispatcher-6]
datapath-control -  Failed to create port vif5f83afd01ddf: java.io.IOException:
failed to connect to socket: Bad file descriptor

大佬一看可能与socket连接数有关.

ps -ef | grep midolman | grep -v 'grep' | awk '{print $2}'
7351
7390
lsof -p 7351 | wc -l
43
lsof -p 7390 | wc -l
4295
cat /proc/7351/limits
cat /proc/7390/limits

发现Max open files最大才4096, 而此时状态不对。 google获得新姿势 ```

Centos/Ubuntu 16.04(未测试ubuntu 16.04)

vi /lib/systemd/system/midolman.service 新增 LimitNOFILE=60000 systemctl daemon-reload systemctl restart midolman

ubuntu 14.04

vi /etc/init/midolman.conf script ulimit -SHn 60000 exec /usr/share/midolman/midolman-start ``` 到这里基本已经ok了`

*****
Written by ysicing on 23 August 2017