1-888-310-4540 (main) / 1-888-707-6150 (support) info@spkaa.com
Select Page

Troubleshoot Common Application & Network Issues with tcpdump

Here’s a common situation:  you manage an application;  you haven’t gone through a release; no patches have been applied (to your knowledge); and the infrastructure guys say that they haven’t touched the network in ages.  But a user calls, and to your surprise, they’re reporting that a certain piece of functionality within your application is no longer working as of this morning.  What do you do?  Where do you start?  What makes application management such a challenge is that there are tons of moving parts to this puzzle, and if any one piece malfunctions, you could have a major issue on your hands.

Despite the pressure, you keep your cool and immediately put your problem-solving hat on.  You run through your basic application sanity checks, and they all pass.  What next?  Well, in my arsenal of troubleshooting tools, I find that the most useful of them all is tcpdump.  After all, most applications follow the client/server model, and being able to visualize this communication often leads to the answer you’re looking for.  If not, it can at least help you rule out certain suspects.

Before I started using tcpdump on a regular basis, I was pretty overwhelmed with all of its different options and how I just might be able to leverage the tool.  If you’re a little bit tentative as well, read further, and I’ll break down for you its basic usage.  I’ll also explain some common situations where it may come in handy.

In its most basic form, you run tcpdump on your server, and you simply specify the interface that you want to listen to:

 tcpdump -ni eth0 not port 22

The -n says that we shouldn’t bother doing DNS lookups for all addresses.  The “not port 22” is quite helpful as well, since we don’t want to be flooded with our own ssh session traffic.  What you’ll immediately see is all of the traffic to and from your server.  You’ll see others attempting to ping you, traffic to your database server, DNS lookups — perhaps too much traffic.  The key is to isolate the traffic we’re interested in, and then make sure that traffic is as we expect it to be.

In order for us to isolate the traffic we want to see, we need to create filters.  Similar to the example “not port 22” that we used above, we can build expressions that help us sift through all of the extraneous packets.  Here is a quick cheat sheet for your reference:

  • port 22  (show all packets where the source or destination port is 22)
  • host  (show all packets where is either the source or the destination)
  • src port 22 / src host  (show all packets where only the source is either port 22 or the host
  • multicast  (show all packets that are multicast packets)
  • udp / tcp / icmp / arp  (show all packets only of this type of protocol)

We can also build more complex expressions if needed:

  • not src port 22 and multicast and udp
  • host and port 22
  • ‘(host and not dst port 22) and (not src port 55555)’

Once we’re able to build our expressions, we can easily see how tcpdump can be used to verify some facts in our troubleshooting process that we otherwise may not be able to know for sure.  For instance, is our database server really communicating with our application server?; or is it sending a request and not getting a response?  Likewise, is our http server getting a request from a specific client but not sending a response?  Is my server getting flooded by traffic from a specific host?

Tcpdump can also be used to troubleshoot more advanced issues:

  • Duplicate IP / ARP issue?  A new server that comes online with the same IP address as your webserver would wreak immediate havoc.  What are the symptoms of this?  Well, with tcpdump we can look for ARP responses, tie them to a Mac address, and then in turn, we can trace the Mac address to a switchport to find the culprit.
  • Packets tagged when they shouldn’t be?  Is your network interface misconfigured with a VLAN tag when it shouldn’t be?  Determine the traffic type for sure by using tcpdump to look at the ethernet header.
  • Asymmetricly routed?  In a multi-homed server, this is quite a common issue.  Do we see responses coming back from an IP that isn’t our default gateway or that isn’t the intended interface?
  • Rogue DHCP server?  Do we see a DHCP response from more than one server?
  • Application performance / timing:  Using tcpdump, we can calculate round trip times using packet timestamps.  This can be useful for benchmarking hardware or application latency.
  • Extract payload messages:  Using tcpdump, we can make correlations between actions that occur in our applications, versus what gets sent to the client across the network.  For instance, when I click the “submit” button in my application, why is it sending the client X when it really should be sending Y?
  • Compliance or forensics:  tcpdump can be used on a machine hanging off of a switch span port for logging purposes.

I’ve only scratched the surface with what tcpdump is capable of doing.  It’s a powerful tool, but ultimately, you do need to understand what you’re looking for in the stream of packets.  What tcpdump provides is clear visibility into what otherwise might be a black box.

Do you have an interesting situation where tcpdump helped you get out of a rut?  Let me know!

Michael Solinap

Sr. Systems Integrator

Latest White Papers

6 Secrets To A Successful Atlassian Migration At Scale

6 Secrets To A Successful Atlassian Migration At Scale

With large scale migrations, large user bases, multiple Atlassian tools, plenty of apps, and lots of data, moving to Atlassian Cloud may feel like a steep mountain to climb. But, it doesn't have to be. In fact, we've already helped many customers make the move. Plus,...

Related Resources

Storytelling with Data

Storytelling with Data

Telling a story and telling a story with data are similar, but also different. And many of the differences are points that people don’t consider.  In this blog, we’ll try to break down the components of storytelling with data in an effort to share main points to...

Data Engineering Supports Digital Transformation

Data Engineering Supports Digital Transformation

Data engineering supports digital transformation. Fact. But, how do companies move away from more paper processes and towards digital transformation? It’s not easy. And, the larger the organization, the more difficult it is. Companies can take solace in knowing there...

What Is Atlassian?

What Is Atlassian?

Distributed workforces are commonplace in today’s manufacturing and IT worlds. More and more colleagues work remotely. Additionally, we use more data to drive decisions as the market competition increases. Alongside this, our businesses are expected to meet...