自己写的GO服务在跑了几天后会出现所有携程无法服务, 表现在业务携程TCP建立后无法接收TCP recv_q的数据,处理信号的协成无法处理信号,pprof端口在的进程也无法响应。TCP 接受队列堆积
![屏幕快照 2017-02-15 下午2.40.30.png](http://studygolang.qiniudn.com/170215/455a32217fd999b47ecd874c2f4ded84.png)
但是 进程还显示100%左右的CPU的占用,
有什么方法进行定位么?
https://github.com/xuewindy/dbatman 这个不过不太好看,是个mysql proxy,出问题的环境有几十个库再跑,不知道怎么飞到那里去的 ,有兴趣可以看看。。
#8
更多评论
服务停掉,再启动,过段时间还会这样?试试 strace,也可以用 [gops](https://github.com/google/gops) 试试
#1
现在的感觉是在调度器部分死锁了 gdb进去调试
```
(gdb) bt
#0 runtime.futex () at /home/xuegeng/go/src/runtime/sys_linux_amd64.s:307
#1 0x000000000042c4e0 in runtime.futexsleep (addr=0x11d3020 <runtime.sched+192>, val=0, ns=100000) at /home/xuegeng/go/src/runtime/os1_linux.go:57
#2 0x000000000041103b in runtime.notetsleep_internal (n=0x11d3020 <runtime.sched+192>, ns=100000, ~r2=false) at /home/xuegeng/go/src/runtime/lock_futex.go:174
#3 0x000000000041112b in runtime.notetsleep (n=0x11d3020 <runtime.sched+192>, ns=100000, ~r2=false) at /home/xuegeng/go/src/runtime/lock_futex.go:194
#4 0x00000000004334ad in runtime.stopTheWorldWithSema () at /home/xuegeng/go/src/runtime/proc.go:963
#5 0x000000000045f419 in runtime.systemstack () at /home/xuegeng/go/src/runtime/asm_amd64.s:291
#6 0x0000000000433870 in runtime.startTheWorldWithSema () at /home/xuegeng/go/src/runtime/proc.go:986
#7 0x000000c82001a000 in ?? ()
#8 0x0000000000000000 in ?? ()
```
然后strace 时 显示这样 目前GO 版本是1.6 ,debian
```
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
[pid 231942] futex(0x11d3020, FUTEX_WAIT, 0, {0, 100000}) = -1 ETIMEDOUT (Connection timed out)
```
我试试gops ,感觉是不行的。。
#2