Vulnerability in connection negotation leads to server not accepting further new connections

[Txema Ynzenga]

I have discovered a vulnerability in how the server negotiates new connections with clients that, if exploited, leaves the server in a state where it doesn't respond to new connection requests.

I discovered this while doing some nmap work around my VPN. Running nmap without version scanning results in no problems (tcpdump follows)

22:50:49.266912 IP localhost.33702 > localhost.42420: Flags [S], seq 1600030875, win 65495, options [mss 65495,sackOK,TS val 1406686149 ecr 0,nop,wscale 7], length 0
22:50:49.266919 IP localhost.42420 > localhost.33702: Flags [S.], seq 789782222, ack 1600030876, win 65483, options [mss 65495,sackOK,TS val 1406686149 ecr 1406686149,nop,wscale 7], length 0
22:50:49.266925 IP localhost.33702 > localhost.42420: Flags [.], ack 1, win 512, options [nop,nop,TS val 1406686149 ecr 1406686149], length 0
22:50:49.266941 IP localhost.33702 > localhost.42420: Flags [R.], seq 1, ack 1, win 512, options [nop,nop,TS val 1406686149 ecr 1406686149], length 0

while doing an nmap with version scanning enabled (option -sV)

21:55:51.660760 IP localhost.33684 > localhost.42420: Flags [S], seq 390754457, win 65495, options [mss 65495,sackOK,TS val 1403388543 ecr 0,nop,wscale 7], length 0
21:55:51.660767 IP localhost.42420 > localhost.33684: Flags [S.], seq 1576880969, ack 390754458, win 65483, options [mss 65495,sackOK,TS val 1403388543 ecr 1403388543,nop,wscale 7], length 0
21:55:51.660773 IP localhost.33684 > localhost.42420: Flags [.], ack 1, win 512, options [nop,nop,TS val 1403388543 ecr 1403388543], length 0
21:55:51.660786 IP localhost.33684 > localhost.42420: Flags [R.], seq 1, ack 1, win 512, options [nop,nop,TS val 1403388543 ecr 1403388543], length 0
21:55:51.714520 IP localhost.33686 > localhost.42420: Flags [S], seq 1229067138, win 65495, options [mss 65495,sackOK,TS val 1403388596 ecr 0,nop,wscale 7], length 0
21:55:51.714528 IP localhost.42420 > localhost.33686: Flags [S.], seq 1163438190, ack 1229067139, win 65483, options [mss 65495,sackOK,TS val 1403388596 ecr 1403388596,nop,wscale 7], length 0
21:55:51.714534 IP localhost.33686 > localhost.42420: Flags [.], ack 1, win 512, options [nop,nop,TS val 1403388596 ecr 1403388596], length 0
21:55:57.720588 IP localhost.33686 > localhost.42420: Flags [P.], seq 1:5, ack 1, win 512, options [nop,nop,TS val 1403394602 ecr 1403388596], length 4
21:55:57.720592 IP localhost.42420 > localhost.33686: Flags [.], ack 5, win 512, options [nop,nop,TS val 1403394602 ecr 1403394602], length 0
21:56:02.725633 IP localhost.33686 > localhost.42420: Flags [F.], seq 5, ack 1, win 512, options [nop,nop,TS val 1403399608 ecr 1403394602], length 0
21:56:02.725671 IP localhost.33688 > localhost.42420: Flags [S], seq 2544262819, win 65495, options [mss 65495,sackOK,TS val 1403399608 ecr 0,nop,wscale 7], length 0
21:56:02.767116 IP localhost.42420 > localhost.33686: Flags [.], ack 6, win 512, options [nop,nop,TS val 1403399649 ecr 1403399608], length 0
21:56:03.727121 IP localhost.33688 > localhost.42420: Flags [S], seq 2544262819, win 65495, options [mss 65495,sackOK,TS val 1403400609 ecr 0,nop,wscale 7], length 0
21:56:05.913793 IP localhost.33688 > localhost.42420: Flags [S], seq 2544262819, win 65495, options [mss 65495,sackOK,TS val 1403402796 ecr 0,nop,wscale 7], length 0
21:56:07.731934 IP localhost.33690 > localhost.42420: Flags [S], seq 3097561542, win 65495, options [mss 65495,sackOK,TS val 1403404614 ecr 0,nop,wscale 7], length 0
21:56:08.734782 IP localhost.33692 > localhost.42420: Flags [S], seq 3320616648, win 65495, options [mss 65495,sackOK,TS val 1403405617 ecr 0,nop,wscale 7], length 0

leaves the server in a status where attempting to connect to the server results in endless waiting client-side for the server to respond until a timeout happens.

The only difference I can see between both tcpdumps is that nmap sends some data over (PSH packets) when doing version scanning, while the default nmap doesn't.

I logged a dump of a benign connection to the server

22:07:30.469310 IP localhost.33698 > localhost.42420: Flags [S], seq 3009253364, win 65495, options [mss 65495,sackOK,TS val 1404087351 ecr 0,nop,wscale 7], length 0
22:07:30.469321 IP localhost.42420 > localhost.33698: Flags [S.], seq 1869281750, ack 3009253365, win 65483, options [mss 65495,sackOK,TS val 1404087351 ecr 1404087351,nop,wscale 7], length 0
22:07:30.469331 IP localhost.33698 > localhost.42420: Flags [.], ack 1, win 512, options [nop,nop,TS val 1404087351 ecr 1404087351], length 0
22:07:30.470114 IP localhost.33698 > localhost.42420: Flags [P.], seq 1:133, ack 1, win 512, options [nop,nop,TS val 1404087352 ecr 1404087351], length 132
22:07:30.470122 IP localhost.42420 > localhost.33698: Flags [.], ack 133, win 511, options [nop,nop,TS val 1404087352 ecr 1404087352], length 0
22:07:30.828485 IP localhost.42420 > localhost.33698: Flags [P.], seq 1:13, ack 133, win 512, options [nop,nop,TS val 1404087710 ecr 1404087352], length 12
22:07:30.828498 IP localhost.33698 > localhost.42420: Flags [.], ack 13, win 512, options [nop,nop,TS val 1404087710 ecr 1404087710], length 0
22:07:30.855322 IP localhost.33698 > localhost.42420: Flags [P.], seq 133:141, ack 13, win 512, options [nop,nop,TS val 1404087737 ecr 1404087710], length 8
22:07:30.855332 IP localhost.42420 > localhost.33698: Flags [.], ack 141, win 512, options [nop,nop,TS val 1404087737 ecr 1404087737], length 0
22:07:31.329368 IP localhost.42420 > localhost.33698: Flags [P.], seq 13:81, ack 141, win 512, options [nop,nop,TS val 1404088211 ecr 1404087737], length 68
<dump continues with normal connection>

The server waits for 132 bytes to be sent by the client before doing anything else. nmap with version control sends only 4, and I assume the server waits for the other 128 to be produced, even though the client has finalized the connection (which I quite strange indeed). My guess is implementing a timeout in the connection negotiation code would circumvent this issue, if my diagnosis of the bug in the code is correct (which is wishful thinking without having access to the code itself, but one can hope).

After the "aborted negotiation" the server doesn't answer to any TCP SYN packets, or anything else for that matter, because I guess it's stuck busy looping waiting for the previous negotiation to end, which leads to no new connections being handled. I am unaware as to whether players in-game are able to keep on playing, but that is, as it stands secondary.

This is, of course, a possible attack vector to effectively disable servers, including those in the public server browser, since one can just try to connect to one, get the IP during the connection attempt (this obviously works even if the server is password protected), and then just nuke the server via the nmap -sV method described earlier.

Most amusingly, the server doesn't even seem to complain at all, no error messages in the log related to any of this can be found, which could prove to be a head-scratcher for the server admin trying to debug their newly setup VPN with port-forwading (this is precisely how I discovered it, and I thought my setup had just broken again, except for the fact that my Minecraft server was still reachable).