Wireshark showing RST packets when unable to access website

I was unable to access a website on a webserver I frequently access. When I examined the network traffic to and from the server with Wireshark network analyzer software, I could see a repeating sequence of SYN packets from the IP address of the Windows laptop from which I was attempting to access the web server followed immediately by RST, ACK packets from the IP address of the server. That sequence repeated over and over again. E.g.:

130	2.766497	TCP	74	44970?80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1029178963 TSecr=0 WS=16
131	2.768022	TCP	60	80?44970 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
132	2.800200	TCP	74	27779?80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1029178972 TSecr=0 WS=16
133	2.801724	TCP	60	80?27779 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

Wireshark Crash Course
Wireshark Crash Course
1x1px






Wireshark showing Syn packets followed by RSTs

That could indicate the port is closed on the destination IP address, but I didn't understand why that would be the case, since the web server has been accessible for a long time and I had accessed it a short time before the problem occurred. I logged into the web server and from a shell prompt on the server attempted to connect to the HTTP port, i.e., well-known port 80 using the the Telnet program, but was unsuccesful, so I checked to see if the Apache web server software, httpd, was even running and found it was not running.

$ telnet 127.0.0.1 80
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
$ ps aux | grep httpd
$ ps aux | grep httpd | grep -v grep
$

So I attempted to start the Apache webserver, but that attempt failed.

$ apachectl start
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: root
Password:
==== AUTHENTICATION COMPLETE ===
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
$ systemctl status httpd.service
â httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-01-05 20:43:28 EST; 24s ago
  Process: 7491 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 7489 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 7489 (code=exited, status=1/FAILURE)
$

So I logged into the root account and tried again and then checked for further details on the problem after Apache failed to start with the systemctl status httpd.service command.

# apachectl start
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
# systemctl status httpd.service
â httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2017-01-05 20:44:42 EST; 12s ago
  Process: 7579 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
  Process: 7577 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 7577 (code=exited, status=1/FAILURE)

Jan 05 20:44:42 example01.com systemd[1]: Starting The Apache HTTP Server...
Jan 05 20:44:42 example01.com httpd[7577]: AH00526: Syntax error on line 425...:
Jan 05 20:44:42 example01.com httpd[7577]: Invalid command 'SSLEngine', perh...n
Jan 05 20:44:42 example01.com systemd[1]: httpd.service: main process exited...E
Jan 05 20:44:42 example01.com kill[7579]: kill: cannot find process ""
Jan 05 20:44:42 example01.com systemd[1]: httpd.service: control process exi...1
Jan 05 20:44:42 example01.com systemd[1]: Failed to start The Apache HTTP Se....
Jan 05 20:44:42 example01.com systemd[1]: Unit httpd.service entered failed ....
Jan 05 20:44:42 example01.com systemd[1]: httpd.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

This time I saw that the problem was due to an error on line 425 of Apache's configuration file, /etc/httpd/conf/httpd.conf. The referenced line was SSLEngine On. I commented it out and was able to restart the Apache webserver. I believe I had entered that line when setting up HTTPS access for the website, but hadn't completed configuring that access at that time and hadn't returned to complete the work. After I commented out that line by putting a hash sign at the beginning of the line, I was able to start the web server software and saw that the httpd process was running afterwards. I was also able to telnet to port 80 and enter HEAD / HTTP/1.1 and then hit Enter followed by Host: and the domain name of the site followed by Enter, which resulted in Apache returning information about its version number, etc., to verify that the server was functioning normally and the site was accessible.



Web Performance Testing using Jmeter
Web Performance Tesing using Jmeter
1x1px

# apachectl start
[root@example01 ~]# ps aux | grep httpd | grep -v grep
root      7644  1.1  0.8 462000 15844 ?        Ss   20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7654  0.3  0.6 464308 11640 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7655  0.2  0.6 464308 11580 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7656  0.0  0.5 464236 10544 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7657  0.0  0.6 464216 11428 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7658  0.0  0.4 464216  9264 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7659  0.0  0.4 464216  9264 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7662  0.0  0.4 464084  9324 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7663  0.0  0.6 464216 11428 ?        S    20:46   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7720  0.0  0.4 464084  8532 ?        S    20:47   0:00 /usr/sbin/httpd -DFOREGROUND
apache    7724  0.0  0.4 464084  8532 ?        S    20:47   0:00 /usr/sbin/httpd -DFOREGROUND
[root@example01 ~]# telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HEAD / HTTP/1.1
Host: example01.com

HTTP/1.1 200 OK
Date: Fri, 06 Jan 2017 01:47:47 GMT
Server: Apache/2.4.6 (CentOS) PHP/5.4.16
X-Powered-By: PHP/5.4.16
Content-Type: text/html; charset=UTF-8

Connection closed by foreign host.
#

I then entered the command last reboot | tail -n 1, which showed me the server recently rebooted, which explained why it had stopped working. Apparently, I hadn't tried restarting the webserver after last editing /etc/httpd/httpd.conf, so hadn't realized I had left the configuration file with an error in it.

After I did that, when I observed network traffic to/from the server again, I saw the normal TCP connection establishment process occurring, i.e., a SYN packet was sent to the server from the Windows laptop, the server responded with a SYN-ACK packet, and then the laptop sent an ACK packet. Those packets set up the connection with both sides acknowledging the connection. So then requests from the client could flow to the server and its responses flowed back to the client, i.e., the web browser on the laptop. The 3-way handshake of SYN, SYN-ACK, and ACK is detailed below:

  1. SYN: The active open is performed by the client sending a SYN to the server. The client sets the segment's sequence number to a random value A.
  2. SYN-ACK: In response, the server replies with a SYN-ACK. The acknowledgment number is set to one more than the received sequence number i.e. A+1, and the sequence number that the server chooses for the packet is another random number, B.
  3. ACK: Finally, the client sends an ACK back to the server. The sequence number is set to the received acknowledgement value i.e. A+1, and the acknowledgement number is set to one more than the received sequence number i.e. B+1.

E.g., I saw the following packets sent to/from the webserver:

30	0.466054	TCP	74	9468?80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1030171820 TSecr=0 WS=16
31	0.467601	TCP	74	80?9468 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=5866076 TSecr=1030171820 WS=128
31	0.467601	TCP	74	80?9468 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=5866076 TSecr=1030171820 WS=128
45	0.533160	HTTP	878	GET /cometchat/cometchat_receive.php?callback=jqcc171027928900874533813_1463341509817&buddylist=1&initial

After the 3-way handshake established the connection, an HTTP GET request was issued to the web server to retrieve data from the sever.

Aiseesoft Video Converter Ultimate

TCP 3-way handshake