Using netstat to determine the process that is using a network port under Linux

While troubleshooting an isuue on a CentOS server, which functions as a web server, I used the tcpdump utility to monitor network traffic to and from the web server. I used the tcpdump command tcpdump -i enp1s4 -vvv port 80 to observe traffic on network interface enp1s4, which was the Local Area Network (LAN) interface, and only on port 80, the well-known port for HTTP traffic. Amidst the expected traffic I also saw HTTP connectivity from the server on which I was performing the troublehshooting to another web server, which seemed odd, since it wasn't immediately apparent to me why the server I was troubleshooting was connecting to that other web server at IP address 8.247.90.236.

Aiseesoft Video Converter Ultimate 
15:12:46.491073 IP (tos 0x0, ttl 64, id 21907, offset 0, flags [DF], proto TCP (
6), length 52)
moonpoint.com.33309 > 8.247.90.236.http: Flags [F.], cksum 0x26b7 (incorrect
-> 0x2738), seq 3599572683, ack 3802137359, win 115, options [nop,nop,TS val 28
33407685 ecr 423340583], length 0
15:12:46.515987 IP (tos 0x0, ttl 54, id 31318, offset 0, flags [none], proto TCP
(6), length 52)
8.247.90.236.http > moonpoint.com.33309: Flags [F.], cksum 0x13c6 (correct),
seq 1, ack 1, win 114, options [nop,nop,TS val 423345561 ecr 2833407685], lengt
h 0
15:12:46.516052 IP (tos 0x0, ttl 64, id 21908, offset 0, flags [DF], proto TCP (
6), length 52)
moonpoint.com.33309 > 8.247.90.236.http: Flags [.], cksum 0x26b7 (incorre
ct -> 0x13ac), seq 1, ack 2, win 115, options [nop,nop,TS val 2833407710 ecr 423
345561], length 0

When I looked up the IP address of the other system at the American Registry for Internet Numbers (ARIN) website at www.arin.net, I found the address of the other server was assigned to Level 3 Communications, Inc., an Internet Service Provider (ISP). When I used tcpdump -i enp1s4 -vvv -X port 80 and host 8.247.90.236, to look at HTTP traffic just to/from that IP address and to also see the contents of packets in hexadecimal and ASCII with the -X option, I didn't see anything that pointed to the source of the traffic.

But when I also looked again at all HTTP traffic, not just traffic to/from this particular remote host, I saw the following:



DJI Phantom 3 Drone 
# tcpdump -i enp1s4 -X -vvv port 80
tcpdump: listening on enp1s4, link-type EN10MB (Ethernet), capture size 65535 by
tes
15:55:17.659624 IP (tos 0x0, ttl 64, id 26053, offset 0, flags [DF], proto TCP (
6), length 414)
moonpoint.com.33276 > ec2-52-207-47-14.compute-1.amazonaws.com.http: Flags [
P.], cksum 0x281b (incorrect -> 0xfd75), seq 1404522575:1404522937, ack 34836983
97, win 331, options [nop,nop,TS val 2835958854 ecr 88952910], length 362
0x0000:  4500 019e 65c5 4000 4006 ad0a c0a8 0205  E...e.@.@.......
0x0010:  34cf 2f0e 81fc 0050 53b7 504f cfa5 04dd  4./....PS.PO....
0x0020:  8018 014b 281b 0000 0101 080a a909 4c46  ...K(.........LF
0x0030:  054d 504e 4745 5420 2f70 623f 6969 643d  .MPNGET./pb?iid=
0x0040:  3137 3631 342d 3330 302d 3630 302d 6978  17614-300-600-ix
0x0050:  6470 6a32 6c39 3863 3675 6776 3532 6264  dpj2l98c6ugv52bd
0x0060:  6670 6326 6362 3d32 3536 3233 3734 3431  fpc&cb=256237441
0x0070:  2048 5454 502f 312e 310d 0a48 6f73 743a  .HTTP/1.1..Host:
0x0080:  2061 6473 2e69 7373 6967 7065 6e2e 636f  .ads.issigpen.co
0x0090:  6d0d 0a55 7365 722d 4167 656e 743a 204d  m..User-Agent:.M
0x00a0:  6f7a 696c 6c61 2f35 2e30 2028 5769 6e64  ozilla/5.0.(Wind
0x00b0:  6f77 7320 4e54 2036 2e32 3b20 574f 5736  ows.NT.6.2;.WOW6
0x00c0:  343b 2072 763a 3331 2e30 2920 4765 636b  4;.rv:31.0).Geck
0x00d0:  6f2f 3230 3130 3031 3031 2046 6972 6566  o/20100101.Firef
0x00e0:  6f78 2f33 312e 3020 4b2d 4d65 6c65 6f6e  ox/31.0.K-Meleon
0x00f0:  2f37 352e 310d 0a41 6363 6570 743a 202a  /75.1..Accept:.*
0x0100:  2f2a 0d0a 4163 6365 7074 2d4c 616e 6775  /*..Accept-Langu
0x0110:  6167 653a 2065 6e2d 5553 2c65 6e3b 713d  age:.en-US,en;q=
0x0120:  302e 350d 0a41 6363 6570 742d 456e 636f  0.5..Accept-Enco
0x0130:  6469 6e67 3a20 677a 6970 2c20 6465 666c  ding:.gzip,.defl
0x0140:  6174 650d 0a52 6566 6572 6572 3a20 6874  ate..Referer:.ht
0x0150:  7470 3a2f 2f76 696d 2e77 696b 6961 2e63  tp://vim.wikia.c
0x0160:  6f6d 2f77 696b 692f 486f 775f 746f 5f73  om/wiki/How_to_s
0x0170:  746f 705f 6175 746f 5f69 6e64 656e 7469  top_auto_indenti
0x0180:  6e67 0d0a 436f 6e6e 6563 7469 6f6e 3a20  ng..Connection:.
0x0190:  6b65 6570 2d61 6c69 7665 0d0a 0d0a       keep-alive....
15:55:17.659831 IP (tos 0x0, ttl 64, id 24771, offset 0, flags [DF], proto TCP (
6), length 621)

In the ASCII text version of the packets data contents I could see that HTTP request was made by a web browser using the user agent string "K-Meleon" and the browser version was 75.1. K-Meleon is a web browser that runs on Microsoft Windows systems. Seeing that information reminded me that K-Meleon was open on a system where the browser was configured to use a SOCKS proxy via an SSH connection from PuTTY to the server on which I was working - see Enabling SOCKS Proxy support in the K-Meleon Browser. That explained why I was seeing traffic from the server to other webservers.

The port that was being used as the source network port on the local system I was on for connecting to this particular remote web serer was port 33276, which I could see from the tcpdump output. On a Linux system, such as a CentOS system, you can use the netstat command to view network ports that are in use. E.g.:

# netstat -an | grep 33276
tcp        0      0 192.168.2.5:33276       52.207.47.14:80         ESTABLISHED
#

You can include the -p option to also view information on the process that has the port open. E.g., in this case:

# netstat -pn | grep 33276
tcp        0      0 192.168.2.5:33276       52.207.47.14:80         ESTABLISHED
920/sshd: ann@pts/0
#

The process associated with the port was "sshd" because the SSH server process, sshd, was being used by an SSH client that had established a Socket Secure (SOCKS) proxy server connection.