Network connectivity troubleshooting
Networking is sophisticated, and debugging application errors caused by them can be a painful process.
Here is what I’ve used to root cause server issues - feel free to add others in a comment if you know of a helpful utility I haven’t listed.
Gather full hostname, and ip address of source, and destination hosts with nslookup
Section titled “Gather full hostname, and ip address of source, and destination hosts with nslookup”nslookup <host> - will show ip address
From the source computer - confirm you’re able to resolve the destination ip address
Section titled “From the source computer - confirm you’re able to resolve the destination ip address”nslookup rhino# Server: 192.168.1.1# Address: 192.168.1.1#53## Name: rhino.localdomain# Address: 192.168.1.168
Confirm server is online reachable
Section titled “Confirm server is online reachable”ping <host>ssh <host>
Confirm server responding to curl commands
Section titled “Confirm server responding to curl commands”curl -v https://example.com:443
Check if port is listening
Section titled “Check if port is listening”ss -tuln | grep :PORTlsof -i :PORTnmap -p- TARGET
Confirm ports are available and not blocked
Section titled “Confirm ports are available and not blocked”iperf3 - allows testing host to host on a specific portiperf3 -s -p 5201 # On serveriperf3 -c <server-ip> -p 5201 # On Clientiperf3 -c rhino -p 5201# Connecting to host rhino, port 5201# [ ID] Interval Transfer Bitrate# [ 7] 0.00-1.00 sec 20.6 MBytes 172 Mbits/sec
iperf3 -c <server-ip> -p 5201 -u # udpfor port in {5201..5210}; do iperf3 -c <server-ip> -p $port; done # multiple ports
Check route server is using
Section titled “Check route server is using”traceroute <host> - shows jumps made to get to host
traceroute rhino# traceroute to rhino.localdomain (192.168.1.168), 64 hops max, 40 byte packets# 1 rhino.localdomain (192.168.1.168) 3.722 ms 5.200 ms 3.759 ms
Confirm DNS Resolution
Section titled “Confirm DNS Resolution”dig - allows troubleshooting DNSdig google.comdig @8.8.8.8 example.com - uses specific dns server to resolvedig +trace example.com - adds tracing
Find what process is using a port
Section titled “Find what process is using a port”lsof -ilsof -i TCP:22
Show Route Address
Section titled “Show Route Address”routeip link show # shows mac address
curl -H "Authorization: Bearer <Token>" -T myfile.tar.gz "https://artifactory/artifactory/<reponame>/path/to/file.tar.gz
Tracing traffic through firewalls
Section titled “Tracing traffic through firewalls”ip route# example output# DEFAULT GATEWAY# default via 10.0.0.1 dev eth0 proto dhcp metric 100
# LOCAL LAN ROUTE# 10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.10 metric 100
# Traffic to 172.16.x.x is routed through 10.0.0.254 - likely firewall or router# 172.16.0.0/16 via 10.0.0.254 dev eth0
traceroute 172.16.0.1 # shows hops to destinationping 10.0.0.254nslookup 10.0.0.254# check arp tablearp -n | grep 10.0.0.254
# scan devicenmap -sV 10.0.0.254# can tell you if its a firewall
Check DNS Resolution
Section titled “Check DNS Resolution”less /etc/sssd/sssd.conf # look at avilable ipas#Confirm you can hit itnslookup url.ipa.net # lets say this returned 10.0.0.0# Try resolving to url from ip in dig# Trying to confirm artifactory.internal.net resolves to ip addressdig @10.0.0.0 artifactory.internal.net