Duplicate packets are one of those things that happen in a busy network. Although using different network equipment and optimizing your network will minimize duplicates, they’ll still crop up on a fairly regular basis. This can be a problem for your monitoring tools, because analyzing duplicate packets can take up their processing resources while also throwing off your analytics.
Network packet brokers (NPBs) can remediate this issue by finding and removing duplicate packets before they can reach your analytics tools. Here are some of the best ways to use this feature to optimize your network and improve monitoring throughput.
No matter what happens, you’ll encounter duplicate packets in your network from time to time—but if your network uses Switch Port Analyzers (SPAN ports), you’ll find them a lot more often. When SPAN ports are used to either mirror ingress and egress ports, or to mirror traffic onto a VLAN, your monitoring network might experience a 50% increase in traffic from duplicates alone.
One simple fix is to utilize a network TAP instead of a SPAN port, which will slash the number of duplicate packets you need to deal with. We recognize that this is sometimes easier said than done—that you might have legacy equipment or may have to rely on the SPAN for network access. Additionally, switching from SPAN to TAP won’t get rid of duplicate packets altogether. For example, some hosts will commonly transmit duplicate packets in order to guard against packet loss.
If your monitoring needs are sensitive enough that any number of duplicate packets is too many, or if you can’t switch away from SPAN ports, then duplicate packets can cause you real problems.
First, data center traffic has already reached enormous throughput levels and is now approaching the 100G range. Monitoring tools are already undersized for the amount of traffic that they need to deal with. With duplicate packets added to the mix, you’ll find your monitoring tools stretched past the breaking point.
Second, your monitoring and analytics depend on having consistent data—and duplicate packets can skew this data in several ways. Let’s say that your network is starting to become unstable, but your monitoring tools are chewing through a queue of “good” packets—that are in fact all duplicates. You won’t catch a hint of anything wrong until there’s a performance issue. Conversely, a stream of duplicated bad packets can display a false positive for a cyberattack or an unexplained slowdown.
Again, there are ways to mitigate this problem, some tools may have deduplication capabilities but if you want to quash it permanently and not transfer the processing burden to the tools, use a network packet broker with deduplication functionality.
When a packet comes in, the network packet broker uses one-way encryption to turn the packet into a series of letters and numbers called a hash. Its hash is compared to the hash of every packet that came before it—and comparing two hashes is much faster than comparing two packets line by line. Since the same content will always produce the same hash, any two hashes that match are automatically duplicates. The duplicate hash is discarded before it can be sent on to your network monitoring or security tools.
This explanation is simple, but it also obscures a few edge cases. For example, what happens if two packets have the same header and the same content, but different IP type of service (TOS) numbers. Are they different or are they duplicates? What about time series? If you get two virtually identical packets a few minutes apart, is the second packet a duplicate, or is the same user just logging in a second time?
We take the approach of providing solutions that are scalable for future on-demand growth and ROI. Garland’s PacketMAXTM line of deconstructed packet brokers includes an Advanced Features Deduplication device purpose-built to extend the feature set of existing infrastructure and reduce the processing load to security or monitoring tools. Allowing you to deploy what you need, when you need it.
Looking to add a deduplication to your deployment, but not sure where to start? Join us for a brief network Design-IT consultation or demo. No obligation - it’s what we love to do.