network services troubleshooting

Overview

Here are some tips & tricks on troubleshooting network services.

General Steps

Narrow down the problem

The more you know about the problem, and the more you manage to narrow it down, the easier it will be to fix. For network services, the crucial question is to know if the problem is located on one end, the other, or in between.

Server Side

Here are the basics to perform server-side if a network service isn't responding at all :

# Check what services are listening
netstat -lnp
# Example for port 25 (smtp)
netstat -lnp | grep :25
# Dump firewall rules to check them
iptables-save | more
# Check tcp wrappers
cat /etc/hosts.*
# Check kernel messages
dmesg | tail

If the service is answering but not fully working :

# Check system messages
less /var/log/messages
less /var/log/secure
# Check the application's logs (varies)
less /var/log/<daemon name>
# To get the above location if you know the configuration file, use something like this :
rpm -qlf /etc/httpd/conf/httpd.conf | grep log
# Check kernel messages (selinux denials)
dmesg | tail

Client side

1. Use telnet to low-level test remote TCP ports : `telnet servername 22` will check that the ssh service is answering. To disconnect, just use Ctrl+AltGr+]. 1. Use curl (or wget) to low-level test HTTP answers : `curl -I http://servername/` will show you the returned headers. 1. Use openssl to low-level test any SSL enabled services : `openssl s_client -connect servername:443` 1. Speak the daemon's language to test the service. Example for SMTP :

telnet server 25
EHLO foo
MAIL FROM:<me@mydomain>
RCPT TO:<me@mydomain>
DATA
Subject: Test email...
.

QUIT