I was unable to access a website on a webserver I frequently access. When I examined the network traffic to and from the server with Wireshark network analyzer software, I could see a repeating sequence of SYN packets from the IP address of the Windows laptop from which I was attempting to access the web server followed immediately by RST, ACK packets from the IP address of the server. That sequence repeated over and over again. E.g.:
130 2.766497 TCP 74 44970?80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1029178963 TSecr=0 WS=16 131 2.768022 TCP 60 80?44970 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0 132 2.800200 TCP 74 27779?80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1029178972 TSecr=0 WS=16 133 2.801724 TCP 60 80?27779 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
That could indicate the port is closed on the destination IP address, but I didn't understand why that would be the case, since the web server has been accessible for a long time and I had accessed it a short time before the problem occurred. I logged into the web server and from a shell prompt on the server attempted to connect to the HTTP port, i.e., well-known port 80 using the the Telnet program, but was unsuccesful, so I checked to see if the Apache web server software, httpd, was even running and found it was not running.
$ telnet 127.0.0.1 80 Trying 127.0.0.1... telnet: connect to address 127.0.0.1: Connection refused $ ps aux | grep httpd $ ps aux | grep httpd | grep -v grep $
So I attempted to start the Apache webserver, but that attempt failed.
$ apachectl start ==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Authenticating as: root Password: ==== AUTHENTICATION COMPLETE === Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. $ systemctl status httpd.service â httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-01-05 20:43:28 EST; 24s ago Process: 7491 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE) Process: 7489 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) Main PID: 7489 (code=exited, status=1/FAILURE) $
So I logged into the root account and tried again and then checked for
further details on the problem after Apache failed to start with the
systemctl status httpd.service command.
# apachectl start Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details. # systemctl status httpd.service â httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2017-01-05 20:44:42 EST; 12s ago Process: 7579 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE) Process: 7577 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE) Main PID: 7577 (code=exited, status=1/FAILURE) Jan 05 20:44:42 example01.com systemd[1]: Starting The Apache HTTP Server... Jan 05 20:44:42 example01.com httpd[7577]: AH00526: Syntax error on line 425...: Jan 05 20:44:42 example01.com httpd[7577]: Invalid command 'SSLEngine', perh...n Jan 05 20:44:42 example01.com systemd[1]: httpd.service: main process exited...E Jan 05 20:44:42 example01.com kill[7579]: kill: cannot find process "" Jan 05 20:44:42 example01.com systemd[1]: httpd.service: control process exi...1 Jan 05 20:44:42 example01.com systemd[1]: Failed to start The Apache HTTP Se.... Jan 05 20:44:42 example01.com systemd[1]: Unit httpd.service entered failed .... Jan 05 20:44:42 example01.com systemd[1]: httpd.service failed. Hint: Some lines were ellipsized, use -l to show in full.
This time I saw that the problem was due to an error on line 425 of
Apache's configuration file, /etc/httpd/conf/httpd.conf. The
referenced line was SSLEngine On. I commented it out and was
able to restart the Apache webserver. I believe I had entered that line when
setting up HTTPS access for the website, but hadn't completed configuring that
access at that time and hadn't returned to complete the work. After I commented
out that line by putting a hash sign at the beginning of the line, I was able to
start the web server software and saw that the httpd process was running
afterwards. I was also able to telnet to port 80 and enter
HEAD / HTTP/1.1 and then hit Enter followed by
Host: and the domain name of the site followed by Enter,
which resulted in Apache returning information about its version number, etc.,
to verify that the server was functioning normally and the site was accessible.
# apachectl start [root@example01 ~]# ps aux | grep httpd | grep -v grep root 7644 1.1 0.8 462000 15844 ? Ss 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7654 0.3 0.6 464308 11640 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7655 0.2 0.6 464308 11580 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7656 0.0 0.5 464236 10544 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7657 0.0 0.6 464216 11428 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7658 0.0 0.4 464216 9264 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7659 0.0 0.4 464216 9264 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7662 0.0 0.4 464084 9324 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7663 0.0 0.6 464216 11428 ? S 20:46 0:00 /usr/sbin/httpd -DFOREGROUND apache 7720 0.0 0.4 464084 8532 ? S 20:47 0:00 /usr/sbin/httpd -DFOREGROUND apache 7724 0.0 0.4 464084 8532 ? S 20:47 0:00 /usr/sbin/httpd -DFOREGROUND [root@example01 ~]# telnet 127.0.0.1 80 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. HEAD / HTTP/1.1 Host: example01.com HTTP/1.1 200 OK Date: Fri, 06 Jan 2017 01:47:47 GMT Server: Apache/2.4.6 (CentOS) PHP/5.4.16 X-Powered-By: PHP/5.4.16 Content-Type: text/html; charset=UTF-8 Connection closed by foreign host. #
I then entered the command last reboot | tail -n 1, which
showed me the server recently rebooted, which explained why it had stopped
working. Apparently, I hadn't tried restarting the webserver after last editing
/etc/httpd/httpd.conf, so hadn't realized I had left the
configuration file with an error in it.
After I did that, when I observed network traffic to/from the server again, I saw the normal TCP connection establishment process occurring, i.e., a SYN packet was sent to the server from the Windows laptop, the server responded with a SYN-ACK packet, and then the laptop sent an ACK packet. Those packets set up the connection with both sides acknowledging the connection. So then requests from the client could flow to the server and its responses flowed back to the client, i.e., the web browser on the laptop. The 3-way handshake of SYN, SYN-ACK, and ACK is detailed below:
E.g., I saw the following packets sent to/from the webserver:
30 0.466054 TCP 74 9468?80 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1030171820 TSecr=0 WS=16 31 0.467601 TCP 74 80?9468 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=5866076 TSecr=1030171820 WS=128 31 0.467601 TCP 74 80?9468 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=5866076 TSecr=1030171820 WS=128 45 0.533160 HTTP 878 GET /cometchat/cometchat_receive.php?callback=jqcc171027928900874533813_1463341509817&buddylist=1&initial
After the 3-way handshake established the connection, an HTTP GET request was issued to the web server to retrieve data from the sever.
