Egg.js 应用经常会出现大量的 socket 连接
当前的一个 Egg.js 应用,之前是正常的,最近几天常常会出现大量的 socket 连接,频率不固定,大概1、2天左右就会出现,表现就是请求响应速度特别慢,但是 CPU 正常,SQL 也没有慢查询。使用命令 lsof -i:7001
就会发现大量的 socket 连接。这种情况该怎么解决呢?
以下是 nginx 的错误日志。
2020/07/24 08:03:44 [error] 20926#0: *131353 connect() failed (111: Connection refused) while connecting to upstream, client: 203.168.21.214, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://xxx.cn/charge"
2020/07/24 08:03:44 [error] 20926#0: *131487 connect() failed (111: Connection refused) while connecting to upstream, client: 27.195.150.150, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://www.cn/platform/agent"
2020/07/24 08:03:44 [error] 20926#0: *131487 connect() failed (111: Connection refused) while connecting to upstream, client: 27.195.150.150, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://www.cn/platform/agent"
2020/07/24 08:03:44 [error] 20926#0: *131353 connect() failed (111: Connection refused) while connecting to upstream, client: 203.168.21.214, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://xxx.cn/charge"
2020/07/24 08:03:44 [error] 20926#0: *131535 connect() failed (111: Connection refused) while connecting to upstream, client: 117.136.56.82, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://xxx.cn/login"
2020/07/24 08:03:44 [error] 20926#0: *131534 connect() failed (111: Connection refused) while connecting to upstream, client: 223.104.16.16, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://xxx.cn/charge"
2020/07/24 08:03:45 [error] 20926#0: *131534 connect() failed (111: Connection refused) while connecting to upstream, client: 223.104.16.16, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://xxx.cn/charge"
2020/07/24 08:05:18 [error] 20926#0: *131726 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 112.97.59.250, server: xxx.cn, request: "POST /graphql HTTP/1.1", upstream: "http://127.0.0.1:7001/graphql", host: "xxx.cn", referrer: "http://xxx.cn/packages"
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 25611 root 27u IPv4 349980918 0t0 TCP *:afs3-callback (LISTEN)
nginx 25622 nginx 23u IPv4 350837668 0t0 TCP localhost:46782->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 24u IPv4 350835359 0t0 TCP localhost:46234->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 27u IPv4 350837451 0t0 TCP localhost:46320->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 28u IPv4 350836524 0t0 TCP localhost:46460->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 34u IPv4 350835463 0t0 TCP localhost:46484->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 35u IPv4 350838859 0t0 TCP localhost:46944->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 38u IPv4 350837405 0t0 TCP localhost:46254->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 39u IPv4 350836513 0t0 TCP localhost:46426->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 43u IPv4 350835409 0t0 TCP localhost:46358->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 55u IPv4 350836498 0t0 TCP localhost:46396->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 57u IPv4 350837654 0t0 TCP localhost:46752->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 63u IPv4 350836536 0t0 TCP localhost:46492->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 65u IPv4 350836538 0t0 TCP localhost:46500->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 66u IPv4 350839848 0t0 TCP localhost:47026->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 74u IPv4 350836709 0t0 TCP localhost:46792->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 79u IPv4 350836617 0t0 TCP localhost:46644->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 81u IPv4 350837758 0t0 TCP localhost:46916->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 82u IPv4 350839809 0t0 TCP localhost:46938->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 84u IPv4 350837702 0t0 TCP localhost:46828->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 85u IPv4 350836647 0t0 TCP localhost:46704->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 88u IPv4 350835569 0t0 TCP localhost:46716->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 92u IPv4 350837714 0t0 TCP localhost:46854->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 98u IPv4 350838817 0t0 TCP localhost:46872->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 101u IPv4 350837738 0t0 TCP localhost:46880->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 105u IPv4 350839815 0t0 TCP localhost:46948->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 106u IPv4 350839854 0t0 TCP localhost:47048->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 107u IPv4 350838902 0t0 TCP localhost:47010->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 109u IPv4 350838867 0t0 TCP localhost:46968->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 110u IPv4 350838087 0t0 TCP localhost:46986->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 113u IPv4 350839832 0t0 TCP localhost:spremotetablet->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 115u IPv4 350835303 0t0 TCP localhost:46036->localhost:afs3-callback (ESTABLISHED)
nginx 25622 nginx 129u IPv4 350836359 0t0 TCP localhost:46046->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 23u IPv4 350838919 0t0 TCP localhost:47040->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 31u IPv4 350837624 0t0 TCP localhost:46700->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 38u IPv4 350837561 0t0 TCP localhost:46566->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 41u IPv4 350835052 0t0 TCP localhost:45560->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 42u IPv4 350837552 0t0 TCP localhost:46544->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 52u IPv4 350837978 0t0 TCP localhost:46588->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 54u IPv4 350838058 0t0 TCP localhost:46928->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 58u IPv4 350837994 0t0 TCP localhost:46632->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 61u IPv4 350837681 0t0 TCP localhost:46802->localhost:afs3-callback (ESTABLISHED)
nginx 25624 nginx 62u IPv4 350836628 0t0 TCP localhost:46660->localhost:afs3-callback (ESTABLISHED)
node 25644 root 55u IPv4 350837387 0t0 TCP localhost:afs3-callback->localhost:46234 (ESTABLISHED)
node 25644 root 59u IPv4 350837284 0t0 TCP localhost:afs3-callback->localhost:46046 (ESTABLISHED)
node 25644 root 155u IPv4 350836438 0t0 TCP localhost:afs3-callback->localhost:46254 (ESTABLISHED)
node 25644 root 156u IPv4 350835560 0t0 TCP localhost:afs3-callback->localhost:46704 (ESTABLISHED)
node 25644 root 157u IPv4 350838847 0t0 TCP localhost:afs3-callback->localhost:46916 (ESTABLISHED)
node 25644 root 158u IPv4 350834506 0t0 TCP localhost:afs3-callback->localhost:45560 (ESTABLISHED)
node 25644 root 159u IPv4 350835643 0t0 TCP localhost:afs3-callback->localhost:46944 (ESTABLISHED)
node 25644 root 160u IPv4 350836719 0t0 TCP localhost:afs3-callback->localhost:46802 (ESTABLISHED)
node 25644 root 161u IPv4 350836653 0t0 TCP localhost:afs3-callback->localhost:46716 (ESTABLISHED)
node 25644 root 162u IPv4 350838795 0t0 TCP localhost:afs3-callback->localhost:46828 (ESTABLISHED)
node 25644 root 163u IPv4 350839839 0t0 TCP localhost:afs3-callback->localhost:47010 (ESTABLISHED)
node 25644 root 164u IPv4 350838879 0t0 TCP localhost:afs3-callback->localhost:46986 (ESTABLISHED)
node 25644 root 165u IPv4 350837470 0t0 TCP localhost:afs3-callback->localhost:46358 (ESTABLISHED)
node 25644 root 166u IPv4 350835435 0t0 TCP localhost:afs3-callback->localhost:46426 (ESTABLISHED)
node 25644 root 167u IPv4 350837934 0t0 TCP localhost:afs3-callback->localhost:46460 (ESTABLISHED)
node 25644 root 168u IPv4 350836558 0t0 TCP localhost:afs3-callback->localhost:46544 (ESTABLISHED)
node 25644 root 169u IPv4 350835470 0t0 TCP localhost:afs3-callback->localhost:46500 (ESTABLISHED)
node 25644 root 170u IPv4 350836701 0t0 TCP localhost:afs3-callback->localhost:46782 (ESTABLISHED)
node 25644 root 171u IPv4 350838888 0t0 TCP localhost:afs3-callback->localhost:spremotetablet (ESTABLISHED)
node 25644 root 172u IPv4 350835528 0t0 TCP localhost:afs3-callback->localhost:46660 (ESTABLISHED)
node 25644 root 173u IPv4 350835679 0t0 TCP localhost:afs3-callback->localhost:47040 (ESTABLISHED)
node 25644 root 179u IPv4 350837815 0t0 TCP localhost:afs3-callback->localhost:46036 (ESTABLISHED)
node 25645 root 56u IPv4 350836475 0t0 TCP localhost:afs3-callback->localhost:46320 (ESTABLISHED)
node 25645 root 60u IPv4 350837673 0t0 TCP localhost:afs3-callback->localhost:46792 (ESTABLISHED)
node 25645 root 156u IPv4 350837523 0t0 TCP localhost:afs3-callback->localhost:46484 (ESTABLISHED)
node 25645 root 157u IPv4 350837917 0t0 TCP localhost:afs3-callback->localhost:46396 (ESTABLISHED)
node 25645 root 158u IPv4 350835675 0t0 TCP localhost:afs3-callback->localhost:47026 (ESTABLISHED)
node 25645 root 159u IPv4 350835508 0t0 TCP localhost:afs3-callback->localhost:46588 (ESTABLISHED)
node 25645 root 160u IPv4 350837531 0t0 TCP localhost:afs3-callback->localhost:46492 (ESTABLISHED)
node 25645 root 161u IPv4 350835628 0t0 TCP localhost:afs3-callback->localhost:46928 (ESTABLISHED)
node 25645 root 162u IPv4 350835639 0t0 TCP localhost:afs3-callback->localhost:46938 (ESTABLISHED)
node 25645 root 163u IPv4 350836574 0t0 TCP localhost:afs3-callback->localhost:46566 (ESTABLISHED)
node 25645 root 164u IPv4 350838073 0t0 TCP localhost:afs3-callback->localhost:46968 (ESTABLISHED)
node 25645 root 165u IPv4 350837589 0t0 TCP localhost:afs3-callback->localhost:46632 (ESTABLISHED)
node 25645 root 166u IPv4 350837600 0t0 TCP localhost:afs3-callback->localhost:46644 (ESTABLISHED)
node 25645 root 167u IPv4 350835647 0t0 TCP localhost:afs3-callback->localhost:46948 (ESTABLISHED)
node 25645 root 168u IPv4 350836681 0t0 TCP localhost:afs3-callback->localhost:46752 (ESTABLISHED)
node 25645 root 169u IPv4 350835557 0t0 TCP localhost:afs3-callback->localhost:46700 (ESTABLISHED)
node 25645 root 170u IPv4 350837729 0t0 TCP localhost:afs3-callback->localhost:46872 (ESTABLISHED)
node 25645 root 171u IPv4 350835683 0t0 TCP localhost:afs3-callback->localhost:47048 (ESTABLISHED)
node 25645 root 172u IPv4 350838805 0t0 TCP localhost:afs3-callback->localhost:46854 (ESTABLISHED)
node 25645 root 173u IPv4 350838824 0t0 TCP localhost:afs3-callback->localhost:46880 (ESTABLISHED)
2 回复
nginx 和 egg 应用建立的连接?你这还有很多 root 用户建立的,看看是什么应用,nginx 用户的应该是正常的
@hyj1991 应该是和nginx建立的正向和反向的连接吧,你看端口号都是成对出现的,nginx->node,node->nginx