One of the biggest misconceptions of all time in networking is the use of a traceroute to determine that your communication with a server has high latency. On windows, traceroute is the same command as tracert.
Many people beleive that when they see high latency such as 250ms+ in a single hop of a trace-route that it means that that device in the transit path is responsible for the degraded network performance when in fact it could not be more further from the truth.
First lets look at how ping works.
PING, is an application based on the ICMP protocol which is used to send echo packets to a destination and expecting to receive an echo response and it calculates the RTT (Round Trip Time) from when the packet was sent to when the echo response was received. Generally when using PING on a LAN network you can trust that what it is saying is accurate unless you have foreknowledge of network devices in the transit path that prioritize ICMP over mission critical TCP/UDP Traffic. This however is very common in networks that utilize unified communications, meaning voice and data on the same network. This is because QoS Policies are put in place to ensure voice traffic and other mission critical traffic is prioritized over ICMP thus indirectly affecting the RTT time of an ICMP ping test.
Trace-route is another method commonly used by technicians and engineers to diagnosis latency in the transit path however any engineer that has studied how trace-route works would know that its results are nearly always misleading.
Trace-route works in a manner similar to ping however it uses the TTL feature to make each successive hop in the transit path respond with an ICMP TTL Expired packet. Thus gives you the ability to determine which network devices the ICMP packet is traversing.
When you dig deeper into the operation of traceroute you will see that traceroute utilizes 3 probe packets for each successive hop by default unless you specify other wise. Each probe packet indirectly measures the latency between the source and the device where the TTL is declared expired. This latency calculation is a by product of its true intended purpose. Keep in mind even if you send probes to a device that is five hops away, random latency spikes in any four devices prior to the fifth hop can result in the fifth hop looking like it has high latency.
Also note that any Control Plane Policing policy enforced on any device in the transit path could result in ICMP being prioritized to the control plane of the transit device. ICMP is processed switched by most devices whereas TCP/UDP is express forwarded.
An example below of a traceroute on Windows 7;
You can see from the trace route shown above that there is 3 probes per hop between the source and destination and that it does not appear to have latency until traffic traverses 64.69.97.217
The whole point of this blog is to teach you how to interpret such data. Just because you see a spike in latency on the 5th hop does not mean that the 5th hop is causing latency. It can easily mean that the control plane in the device on the fifth hop is under marginal load and that the processor does not respond to the ICMP immediately due to other processes with priority.
Just because you see potential latency with trace-route, you should never expect that to be an accurate representation of latency for TCP/UDP traffic because ICMP and TCP/UDP traffic is treated completely different when it comes to the routers control/forwarding planes.
Most ISP’s use control-plane policing (CoPP) to prevent overwhelming ICMP flooding to a devices control plane. This type of flood prevention mechanism can also result in skewed data in trace routes.
Shown below is a simple CoPP Policy which can result in skewed trace route data.
If you examine the CoPP policy in detail you will notice that all ICMP destined to the control plane is limited to 50000bps as shown below. It can bust up to 5000bps and if it conforms to the policy it is transmited, if it exceeds the policy the ICMP is dropped.
With this in mind you should always use trace route for its intended purpose which is to determine the route traffic takes when traversing the transit path and that latency shown on the per hop probe basis is to be taken with at grain of salt when traversing public devices.
The intended purpose of the 3 probe count is to determine if the traffic traverses multiple routed paths due to route engineering, not to determine the latency 3 times.
I will conclude this blog with the pathping command. This command found on windows is a command similar to traceroute but it combines traceroute with ping to give you a better understanding of latency in the transit path.
Pathping works first by doing a traceroute to the destination then it uses ICMP to ping each hop in the transit path 100 times. This is used to verify latency between the source and destination via icmp echo per each hop. But remember what I said earlier, you cannot rely ICMP when public devices are involved. So you can run into cases where you see ICMP pings destined to one hop in the transit drop 40% of the traffic whereas the next hop has 100% success rate. This is due to CoPP.
Pathping in general is a much better tool to diagnosis latency from a specific source to destination with arelative degree of accuracy. Note that I said Relative, this is because latency is ALWAYS relative to your location on the network.
Shown below is an example of pathping in the works;
As you can see from the pathping shown above there are some hops in the transit path that completely drop ICMP. You can also notice that the latency to hop 5 is higher then the latency is to hop 6. This shows that either Control Plane Policing is used on 64.69.99.238 or the process utilization on hop 5 is relatively higher.
You should know that there are other tools out there that are extremely useful when trying to diagnosis latency related problems. Most of these tools rely on ICMP and your decision to trust them is based on your understanding the transit path. One of these tools being Ping Plotter. There are several useful tools included in the Solarwinds Engineers Toolset however this toolset is extremely expensive. You can download a trial and check it out at Solarwinds Engineers Toolset
The most accurate tools depend on TCP however since TCP is a connection oriented protocol, both the source and destination must be willing to participate in the testing. Some tools are hardware based such as the Fluke Network EtherScope which cost several thousand dollars.
So in conclusion, your decision to trust and use data from ICMP based troubleshooting should be based on your relative understanding of the transit path. You should never take a traceroute that has high latency on it and say its a network issue just because hope 7 has latency greater then 250ms. This is no different the a doctor telling you your spleen is the result of your headaches without factual basis.
If you do not have clear factual data when diagnosing a problem and you blame the network because of a traceroute, you may very well be completely missing the root cause of the problem. Think of it as getting tunneled vision when sh!t hits the fan and management is expecting answers and the first thing you notice is high latency on a traceroute. With out completely understanding traceroute you may be fixating on an issue that is really not an issue at all.
No comments:
Post a Comment