C#: Understanding CLOSE_WAIT and FIN_WAIT_2

Here’s a short C# program which can be used to better understand what the TCP states CLOSE_WAIT and FIN_WAIT_2 are and why you sometimes see connections stuck in these states:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;

namespace TcpTester
{
    internal static class Program
    {
        private const int Port = 15000;
        private const string Hostname = "127.0.0.1";

        private static void Main(string[] args)
        {
            if (args.Length > 0 && args[0] == "client")
            {
                // Started in client mode
                var tcpClient = new TcpClient();
                tcpClient.Connect(Hostname, Port);
                Console.WriteLine("Connected to {0}:{1}", Hostname, Port);
                PrintState();
                Console.WriteLine("Press any key to close the connection from this side.");
                Console.ReadKey();
                tcpClient.Close();
                PrintState();
            }
            else
            {
                // Started in server mode
                var tcpListener = new TcpListener(IPAddress.Parse(Hostname), Port);
                tcpListener.Start();
                Console.WriteLine("Listening on {0}:{1}", Hostname, Port);
                TcpClient tcpClient = tcpListener.AcceptTcpClient();
                tcpListener.Stop();
                Console.WriteLine("Client connected on {0}:{1}", Hostname, Port);
                PrintState();
                Console.WriteLine("Press any key to close the connection from this side.");
                Console.ReadKey();
                tcpClient.Close();
                PrintState();
            }
        }

        private static void PrintState()
        {
            IEnumerable<TcpConnectionInformation> activeTcpConnections =
                IPGlobalProperties.GetIPGlobalProperties().GetActiveTcpConnections()
                    .Where(c => c.LocalEndPoint.Port == Port || c.RemoteEndPoint.Port == Port);
            foreach (TcpConnectionInformation connection in activeTcpConnections)
            {
                Console.WriteLine("{0} {1} {2}", connection.LocalEndPoint, connection.RemoteEndPoint, connection.State);
            }
        }
    }
}

You can start the program without parameters to start a server and with the parameter “client” to start a client (guess it was kind of obvious…).

The server listens to 127.0.01:15000 and the clients connects to it. First start the server. The following will be written to the console:

Listening on 127.0.0.1:15000

Then start the client in another window. The following will appear in the client window:

Connected to 127.0.0.1:15000
127.0.0.1:15000 127.0.0.1:57663 Established
127.0.0.1:57663 127.0.0.1:15000 Established

This tells you that the client is connected from port 57663 (this will change every time you run this test) to port 15000 (where the server is listening).

In the server window, you will see that it got a client connection and the same information regarding port and connection states.

Then press any key on the server console and the following will be displayed:

127.0.0.1:15000 127.0.0.1:57663 FinWait2
127.0.0.1:57663 127.0.0.1:15000 CloseWait

So once the server closed the connection, the connection on the server side went to FIN_WAIT_2 and the one on the client side went to CLOSE_WAIT.

Then press any key in the client console to get the following displayed:

127.0.0.1:15000 127.0.0.1:57663 TimeWait

The connection will stay in TIME_WAIT state for some time. If you really wait a long time before pressing a key in the client console, this last line will not be displayed at all.

So, this should make it easier to understand what the TCP states CLOSE_WAIT and FIN_WAIT_2 are: When the connection has been closed locally but not yet remotely, the local connection is the state FIN_WAIT and the remote one in CLOSE_WAIT.

For more details about the different TCP states, please refer to TCP: About FIN_WAIT_2, TIME_WAIT and CLOSE_WAIT.

Windows: Network connections timing out too quickly on temporary connectivity loss

If you have a rather unstable network where you tend to loose connectivity for a short time frequently, you might notice that established connections (e.g. ssh connections using putty) will get lost. You can then immediately reconnect but it’s still a pain.

The issue is not really with the software loosing the connection (e.g. putty) but rather with the Windows network configuration. A single application cannot set the network settings for the whole application or a specific session to prevent this problem. To solve this problem, you will need to tweak a few Windows network settings.

Basically tweaking these settings means increasing the TCP timeout in Windows. This can be done in the registry.

The relevant TCP/IP settings are:

  • KeepAliveTime
  • KeepAliveInterval
  • TcpMaxDataRetransmissions

These parameters are all located at the following registry location: \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Tcpip\Parameters.

On Windows versions which are not based on Windows NT (i.e. Windows 95, Windows 98 and Windows ME), these parameters are located under: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VxD\MSTCP.

KeepAliveTime

The KeepAliveTime parameters controls how long the TCP driver waits until the a keep-alive packet is sent over an idle TCP connection. A TCP keep-alive packet is simply an ACK packet sent over the connection with the sequence number set to one less than the current sequence number for the connection. When the other end receives this packet, it will send an ACK as a response with the current sequence number. These communication is used to make sure that the remote host at the other end of the connection is still available and make sure the connection is kept open.

Since TCP keep-alives are disabled by default, the application opening the connection needs to specifically enable them.

The value is the number of milliseconds of inactivity before a keep-alive packet is sent. The default is 7,200,000 milliseconds (ms) i.e. 2 hours.

Note that the default of 2 hours might be to high in some cases. Having a high KeepAliveTime brings two problems:

  1. it may cause a delay before the machine at one end of the connection detects that the remote machine is no longer available
  2. many firewalls drop the session if no traffic occurs for a given amount of time

In the first case, if your application can handle reconnect scenario, it will take a very long time until it notices the connection is dead and it would have been able to handle it properly if it failed fast.

In the second case, it’s the opposite, the connection is articially closed by the firewall inbetween.

If you encounter one of these cases on a regular basis, you should consider reducing the KeepAliveTime from 2 hours to 10 or 15 minutes (i.e. 600,000 or 900,000 milliseconds).

But also keep in mind that lowering the value for the KeepAliveTime:

  • increases network activity on idle connections
  • can cause active working connections to terminate because of latency issues.

Setting it to less than 10 seconds, is not a good idea except if you have a network environment with with a very low latency.

KeepAliveInterval

If the remote host at the other end of the connection does not respond to the keep-alive packet, it is repeated. This is where the KeepAliveInterval is used. This parameter determines how often this retry mechanism will be triggered. This is basically the wait time before another keep-alive packet is sent. If at some point in time the remote hosts responds to the keep-alive packet, the next keep-alive packet will be again sent based on the KeepAliveTime parameter (assuming the connection is still idle).

The value is the number of milliseconds before a keep-alive packet is resent. The default is 1,000 milliseconds (ms) i.e. 1 second. If the network connectivity losses sometimes last a few minutes, it’d make sense increasing this parameter to 60,000 milliseconds i.e. 1 minute.

TcpMaxDataRetransmissions

Of course this retry process cannot go on for ever. If the connection is not only temporarily lost but lost for good, then the connection needs to be closed. This is where the parameter TcpMaxDataRetransmissions is used. This parameter defines the number of keep-alive retries to be performed before the connection is aborted.

The default value is to perform 5 TCP keep-alive retransmits. If you experience network instability and lose connections too often, you should consider increasing this value to 10 or 15.

Note that starting with Windows Vista, this parameter doesn’t exist anymore and is replaced by a hard-coded value of 10. After 10 unanswered retransmissions, the connection will be aborted. But you can still control the time frame which a connection could survive a temporary connectivity loss by adapting the KeepAliveInterval parameter.

Also note that this parameter only exists in Windows NT based versions of Windows. On old systems running Windows 95, Windows 98 or Windows ME, the corresponding parameter is HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\VxD\MSTCP\MaxDataRetries.

Summary

Tweaking the parameters above, one can configure the Windows TCP driver so that connections can survive small connectivity losses. Remember that after changing these settings, you’ll need to reboot the machine (it’s Windows after all…).

If you cannot modify TcpMaxDataRetransmissions because you have a newer version of Windows, you can still reach the same results by increasing KeepAliveInterval instead.

Also note that issues with lost connections in unstable networks seems to especially affect Windows Vista and later. So if you move from Windows XP to let’s say Windows 7 and you experience such issues, you should first add the KeepAliveTime  and KeepAliveInterval parameters to the registry, reboot, check whether it’s better and possibly increase the value of KeepAliveInterval if required.

All parameters above should be stored in the registry as DWORD (32bit value).

New wireless router

My wireless router seems to feel ready for retirement. As long as I sit 2 meters away from it, the connection is good. At about 3 meters I loose the connection every 5 to 10 minutes. I have installer repeaters at the stairs in all floors. It used to work fine. But now with the router problems, there is almost no connectivity in the basement and bedrooms upstairs. In the beginning I thought the repeaters had problems so I reset them all reconfigured them, got mad, gave up, tried again… Until I notice I already had problem on the first floor 3 meters away from the router.

So I had a look at which router I should buy to replace it. My main requirement was a good range. Ideally I’d be able to be connected throughout the house without needing the repeaters. The minimum I need is that the range is good enough that my repeaters actually get some signal from the router.

I ended up looking at these two routers (I didn’t look at AC routers since I have no AC device and when it comes to hardware I’m not really an early adopter):

After reading many comments on Amazon I finally decided to go for the Asus router which seems to have a better range. The Netgear router looks better but this doesn’t really help me right now. Let’s see whether my whole house gets connected again when I get the new router…

Mail and domain check tools

I’ve been working on a few tools over the past few days. I’m personally using them whenever I create new mailboxes, move mailboxes from one server to another, create new web pages… My goal is to end up having a tool suite which provides all checks that I need in order to figure out whether the configuration of the web server, the mail server and dns server are fine.

I currently do not yet have a central page for all the tools but most of the tools have a navigation area where you can get access to the other tools. The following tools are currently available (more tools are on the way but not yet tested):

Check HTTP headers

In this tool, you can give a URL and the tool will show you the HTTP headers for this URL. It will also follow redirects and show you the headers for the redirected URLs. If you look up “http://www.benohead.com”, you’ll see the following:

HTTP Response Code: 301
HTTP Response Code: 200

HTTP/1.0 301 Moved Permanently
Date: Mon, 02 Sep 2013 08:38:16 GMT
Server: Apache
X-Pingback: http://benohead.com/xmlrpc.php
Location: http://benohead.com/
Cache-Control: max-age=3600
Expires: Mon, 02 Sep 2013 09:38:16 GMT
Vary: Accept-Encoding,User-Agent
X-Powered-By: PleskLin
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8

HTTP/1.0 200 OK
Date: Mon, 02 Sep 2013 08:38:17 GMT
Server: Apache
X-Pingback: http://benohead.com/xmlrpc.php
Pragma: public
Cache-Control: max-age=0, no-cache
Vary: Accept-Encoding
X-Mod-Pagespeed: 1.4.26.4-3396
Content-Length: 53605
Connection: close
Content-Type: text/html; charset=UTF-8

The first line is displayed in blue to indicate a redirect and the second one is displayed in green. If we would end up getting a 4xx or 5xx return code, it’d be shown in red.

Check SMTP server

This tool will check the DNS settings for the mail server, connect to it and perform a few checks and show you a summary as well as the log of what was done. Here’s an example when you check the domain benohead.com:

DNS MX entries for benohead.com

Priority Mail Server IP Address TTL
10 mail.benohead.com 94.102.209.13 24975

Testing mail server: mail.benohead.com

IP address: 94.102.209.13

220 1a-8627.antagusserver.de ESMTP

Server Identity: 1a-8627.antagusserver.de
Server IP address: 109.235.59.67
Reverse Banner: 1a-8627.antagusserver.de
Reverse DNS: 1A-7583.antagus.de

>>> EHLO benohead.com

250-1a-8627.antagusserver.de

250-AUTH=LOGIN CRAM-MD5 PLAIN

250-AUTH LOGIN CRAM-MD5 PLAIN

250-STARTTLS

250-PIPELINING

250 8BITMIME

TLS support available.

>>> STARTTLS

220 ready for tls

TLS could be started.
Switched to TLS.

>>> MAIL FROM: <example@benohead.com>

250 ok

>>> RCPT TO: <example@relay.check>

553 sorry, that domain isn’t in my list of allowed rcpthosts; no valid cert for gatewaying (#5.7.1)

Not an open relay

>>> QUIT

Summary

Check Results
Reverse lookup of SMTP Banner 1A-7583.antagus.de – 1a-8627.antagusserver.de
SMTP Reverse DNS Mismatch 1a-8627.antagusserver.de – 1a-8627.antagusserver.de
TLS Support Supported
TLS Start Successful
Switch to TLS communication Successful
Open relay Not an open relay

 

I’ll also add in the future a description of what exactly needs to be fixed when something is marked as not OK but I guess it’s a good start (and it’s already enough for me since I know how to fix it ;-)).

Blacklist check

This tool will check whether the specified domain name or IP address is contained in anti-spam databases. This check will take much longer than the other ones since it will check 67 anti-spam databases. This tool uses Ajax to check multiple black lists in parallel for better performance. It checks for black list entries at over 60 anti-spam databases.

The information displayed should be pretty self-explanatory so I won’t go into details in this post.

Whois

This tool is basically just an online version of the whois tool you know from the command line. It will show the domain registration information for the specified IP address or a host name.

The information displayed should be pretty self-explanatory so I won’t go into details in this post.

Check Alexa Rank

This tool uses the script described in my previous post: PHP: Displaying the Alexa ranking of a web sit. If you enter benohead.com as domain, you’ll get something like this:

Domain: benohead.com
Alexa Rank: 444,068
Number of links: 67

 

HTML Escape Tool

The HTML Escape Tool is already described in this post. So I won’t add anything to that.

Other tools

I’m currently working on the following tools which will hopefully be online within the next weeks:

  • An HTML code cleaner to make sure that all tags are properly closed, indented and capitalized.
  • A check for all DNS entries for a specified domain.

I guess I’ll have other ideas as I finish implementing these tools.

If you encounter any problem working with any of the tools please drop me a line.

Qmail: 30 to 60 seconds connection delay

I’ve noticed that when connection to port 25 of one of our servers, it took quite a long time until commands can be sent. To test it, zou connect using telnet>

telnet xxx.xxx.xxx.xxx 25

Where xxx.xxx.xxx.xxx is the IP address of the server on which the mail server is running.

I saw that the connection was immediately established but it took quite some time for the server to send a 220 response. Before this response is received no commands (e.g. HELO, QUIT…) can be processed. Here’s the sequence in which things happen:

  1. Client attempts to open an SMTP connection to port 25 of the server.
  2. Client waits for the socket connection to be established.
  3. Connection established.
  4. Client waits for protocol to start (i.e. waits for a 220 message).
  5. Server sends a 220 code saying it is ready for action.
  6. Now the client can send commands to the server.

The delay occurs in step 4. It takes a long time until the server sends a 220 code.

What happens in the background is that the server validates that the client is trustworthy. This involves performing a reverse lookup of the IP address and checking for known spammers. This can take some time especially if the reverse lookup results in a timeout.

Here are a few things you need to do to get rid of this timeout:

First, make sure that the name servers that are listed in /etc/resolv.conf are working properly.

You can check whether the IP from where the client is accessing the server can be reverse resolved. You can check it with:

# host 192.168.1.1
Host 1.1.168.192.in-addr.arpa. not found: 3(NXDOMAIN)

If you get a “not found” as shown above then it means the name resolution will not work. You might also get the following:

# host 192.168.1.1
;; connection timed out; no servers could be reached

In this case, the reverse lookup times out. You can prevent the qmail from performing a reverse lookup by changing the xinetd script used to start qmail. The script is located in /etc/xinetd.d. It’s usually called smtp or smtp_psa if you’re using Plesk. It looks like this:

service smtp
{
       socket_type     = stream
       protocol        = tcp
       wait            = no
       disable         = no
       user            = root
       instances       = UNLIMITED
       server          = /var/qmail/bin/tcp-env
       server_args     = /usr/sbin/rblsmtpd  -r sbl-xbl.spamhaus.org /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
}

tcp-env supports the following options:

-r (Default.) Attempt to obtain TCPREMOTEINFO from the
remote host.

-R Do not attempt to obtain TCPREMOTEINFO from the remote
host.

-t
Give up on the TCPREMOTEINFO connection attempt after
timeout seconds. Default: 30.

So we need to use the -R option and could also set the timeout to 0 seconds with the -t0 option:

service smtp
{
       socket_type     = stream
       protocol        = tcp
       wait            = no
       disable         = no
       user            = root
       instances       = UNLIMITED
       server          = /var/qmail/bin/tcp-env
       server_args     = -Rt0 /usr/sbin/rblsmtpd  -r sbl-xbl.spamhaus.org /var/qmail/bin/relaylock /var/qmail/bin/qmail-smtpd /var/qmail/bin/smtp_auth /var/qmail/bin/true /var/qmail/bin/cmd5checkpw /var/qmail/bin/true
}

Actually -t0 is redundant when using -R. So you can try with -R and with -Rt0 and see whether it makes any difference on your system.

After changing this, you’ll need to restart xinetd:

/etc/init.d/xinetd restart

If you use inet instead of xinetd then you need to update /etc/inetd.conf instead and restart inetd by using:

/etc/init.d/inetd restart

or:

killall -HUP inetd

Please note that Plesk seems to overwrite this file once in a while. So you might loose this setting.

Even after setting this, I’ve seen that from most servers the delay was gone but I still had a remaining 30 seconds delay when connection to the mail server from my laptop.

The mail server uses to auth protocol to contact the client machine. Unfortunately, many firewalls are configured to just ignore these requests without answering. In this case, you’ll also have to wait for a timeout to occur before the mail server goes to the next step.

In order to get rid of it, you can configure a rule using iptables to actively reject connections from the server to a remote port 113 so that you do not have to wait for 30 seconds in case the remote firewall just silently ignores your request. The processing of the request on port 113 is not critical for the functioning of the mail server, it’s safe to prevent it. Execute the following on the server to have these connection rejected by the local firewall on the server:

iptables -A OUTPUT -p TCP --dport 113 -j REJECT --reject-with tcp-reset

Making these few changes made sure that the connection from our other servers was done without delays. And from my laptop I do not get a fix 60 seconds or 30 seconds delay anymore but it’s mostly taking between 1 and 15 seconds.

If after doing all this, it’s still slow, you should check the list of servers used by rblsmtpd. rblsmtpd blocks mail from RBL-listed sites. It uses a list of sources. In my case only sbl-xbl.spamhaus.org. First make sure that all source are still up and running, since this could also cause an additional timeout (the default timeout is 60 seconds). Additionally, checking whether the client is blacklisted also costs time. So you might want to remove rblstmpd and its parameters (basically what’s between -Rt0 and /var/qmail/bin/relaylock above and see whether it’s faster.

Mac OS X: List listening ports and programs using netstat

When checking the listening ports on my Linux machine I put netstat some pants on:

# netstat -pant | grep LISTEN
tcp        0      0 0.0.0.0:1024            0.0.0.0:*               LISTEN      20797/sasdwl
tcp        0      0 192.168.230.236:3873    0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:4999            0.0.0.0:*               LISTEN      28475/pcp
tcp        0      0 192.168.230.236:2055    0.0.0.0:*               LISTEN      20037/dataserver
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:1098            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:9002            0.0.0.0:*               LISTEN      20731/sasfrompacs
tcp        0      0 192.168.230.236:2058    0.0.0.0:*               LISTEN      20475/backupserver
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      28306/smbd
tcp        0      0 0.0.0.0:1099            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:33518           0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:4847            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:33519           0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:4848            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      19532/httpd
tcp        0      0 0.0.0.0:4849            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:33521           0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:8083            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      5342/master
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      19532/httpd
tcp        0      0 0.0.0.0:4444            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      28306/smbd
tcp        0      0 0.0.0.0:8093            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:4445            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 0.0.0.0:4446            0.0.0.0:*               LISTEN      21201/java
tcp        0      0 127.0.0.1:2054          :::*                    LISTEN      20542/java
tcp        0      0 :::22                   :::*                    LISTEN      5025/sshd
tcp        0      0 ::1:25                  :::*                    LISTEN      5342/master

I want to see the ports and the programs listening on these ports. The netstat options used mean:

  • -p: show the program name / PID owning the socket
  • -a: show all connections
  • -n: show numerical addresses
  • -t: show only TCP connections

Somehow I’ve only noticed now that netstat on Mac OS X cannot show the program name. Actually on Mac OS X, the -p parameter of netstat doesn’t mean program or process but protocol. Also there is no -t parameter but it can be done using -ptcp.

$ netstat -an -ptcp | grep LISTEN
tcp4       0      0  127.0.0.1.10000        *.*                    LISTEN
tcp6       0      0  *.43611                *.*                    LISTEN
tcp4       0      0  *.43611                *.*                    LISTEN
tcp46      0      0  *.80                   *.*                    LISTEN
tcp4       0      0  127.0.0.1.3306         *.*                    LISTEN
tcp4       0      0  127.0.0.1.631          *.*                    LISTEN
tcp6       0      0  ::1.631                *.*                    LISTEN

There seems to be no way to get the same kind of info using netstat on Mac OS X. But everything is not lost. A tcp socket is just another type of file descriptor in Unix derivatives so we can use lsof to get the same info on Mac OS X:

$ lsof -i -P | grep -i "listen"
mysqld    31797 henribenoit   10u  IPv4 0xe1e3f49df503def7      0t0  TCP localhost:3306 (LISTEN)

This shows you the listening ports for programs running under your user name. If you want to see it for all users, you’ll have to use sudo.

$ sudo lsof -i -P | grep -i "listen"
launchd       1           root   28u  IPv6 0xe1e3f49dedc794b7      0t0    TCP localhost:631 (LISTEN)
launchd       1           root   29u  IPv4 0xe1e3f49dedc7f49f      0t0    TCP localhost:631 (LISTEN)
mysqld    31797    henribenoit   10u  IPv4 0xe1e3f49df503def7      0t0    TCP localhost:3306 (LISTEN)

TCP: About FIN_WAIT_2, TIME_WAIT and CLOSE_WAIT

TCP states

Most of the 11 TCP states are pretty easy to understand and most programmers know what they mean:

  • CLOSED: There is no connection.
  • LISTEN: The local end-point is waiting for a connection request from a remote end-point i.e. a passive open was performed.
  • SYN-SENT: The first step of the three-way connection handshake was performed. A connection request has been sent to a remote end-point i.e. an active open was performed.
  • SYN-RECEIVED: The second step of the three-way connection handshake was performed. An acknowledgement for the received connection request as well as a connection request has been sent to the remote end-point.
  • ESTABLISHED: The third step of the three-way connection handshake was performed. The connection is open.
  • FIN-WAIT-1: The first step of an active close (four-way handshake) was performed. The local end-point has sent a connection termination request to the remote end-point.
  • CLOSE-WAIT: The local end-point has received a connection termination request and acknowledged it e.g. a passive close has been performed and the local end-point needs to perform an active close to leave this state.
  • FIN-WAIT-2: The remote end-point has sent an acknowledgement for the previously sent connection termination request. The local end-point waits for an active connection termination request from the remote end-point.
  • LAST-ACK: The local end-point has performed a passive close and has initiated an active close by sending a connection termination request to the remote end-point.
  • CLOSING: The local end-point is waiting for an acknowledgement for a connection termination request before going to the TIME-WAIT state.
  • TIME-WAIT: The local end-point waits for twice the maximum segment lifetime (MSL) to pass before going to CLOSED to be sure that the remote end-point received the acknowledgement.

Most people working with high-level programming languages actually only really know the states CLOSED, LISTEN and ESTABLISHED. Using netstat the chances are that you will not see connections in the SYN_SENT, SYN_RECV, FIN_WAIT_1, LAST_ACK or CLOSING states. A TCP end-point usually stays in these states for only a very short period of time and if many connections get stuck for a longer time in these states, something really bad happened.

FIN_WAIT_2, TIME_WAIT and CLOSE_WAIT are more common. They are all related to the connection termination four-way handshake. Here is a short overview of the states involved:

TCP CLOSE_WAIT

The upper part shows the states on the end-point initiating the termination. The lower part the states on the other end-point.

So the initiating end-point (i.e. the client) sends a termination request to the server and waits for an acknowledgement in state FIN-WAIT-1. The server sends an acknowledgement and goes in state CLOSE_WAIT. The client goes into FIN-WAIT-2 when the acknowledgement is received and waits for an active close. When the server actively sends its own termination request, it goes into LAST-ACK and waits for an acknowledgement from the client. When the client receives the termination request from the server, it sends an acknowledgement and goes into TIME_WAIT and after some time into CLOSED. The server goes into CLOSED state once it receives the acknowledgement from the client.

FIN_WAIT_2

If many sockets which were connected to a specific remote application end up stuck in this state, it usually indicates that the remote application either always dies unexpectedly when in the CLOSE_WAIT state or just fails to perform an active close after the passive close.

The timeout for sockets in the FIN-WAIT-2 state is defined with the parameter tcp_fin_timeout. You should set it to value high enough so that if the remote end-point is going to perform an active close, it will have time to do it. On the other hand sockets in this state do use some memory (even though not much) and this could lead to a memory overflow if too many sockets are stuck in this state for too long.

TIME_WAIT

The TIME-WAIT state means that from the local end-point point of view, the connection is closed but we’re still waiting before accepting a new connection in order to prevent delayed duplicate packets from the previous connection from being accepted by the new connection.

In this state, TCP blocks any second connection between these address/port pairs until the TIME_WAIT state is exited after waiting for twice the maximum segment lifetime (MSL).

In most cases, seeing many TIME_WAIT connection doesn’t show any issue. You only have to start worrying when the number of TIME_WAIT connections cause performance problems or a memory overflow.

CLOSE_WAIT

If you see that connections related to a given process tend to always end up in the CLOSE_WAIT state, it means that this process does not perform an active close after the passive close. When you write a program communicating over TCP, you should detect when the connection was closed by the remote host and close the socket appropriately. If you fail to do this the socket will stay in the CLOSE_WAIT until the process itself disappears.

So basically, CLOSE_WAIT means the operating system knows that the remote application has closed the connection and waits for the local application to also do so. So you shouldn’t try and tune the TCP parameters to solve this but check the application owning the connection on the local host. Since there is no CLOSE_WAIT timeout, a connection can stay in this state forever (or at least until the program does eventually close the connection or the process exists or is killed).

If you cannot fix the application or have it fixed, the solution is to kill the process holding the connection open. Of course, there is still a risk of losing data since the local end-point may still send data it has in a buffer. Also, if many applications run in the same process (as it is the case for Java Enterprise applications), killing the owning process is not always an option.

I haven’t ever tried to force closing of a CLOSE_WAIT connection using tcpkill, killcx or cutter but if you can’t kill or restart the process holding the connection, it might be an option.

In order to better understand how CLOSE_WAIT and FIN_WAIT_2 work, you might want to have a look at C#: Understanding CLOSE_WAIT and FIN_WAIT_2.