Visibility Solutions

Garland Technology is committed to educating the benefits of having a strong foundation of network visibility and access. By providing this insight we protect the security of data across your network and beyond.

Resources

Garland Technology's resource library offers free use of white papers, eBooks, use cases, infographics, data sheets, video demos and more.

Blog

The TAP into Technology blog provides the latest news and insights on network access and visibility, including: network security, network monitoring and appliance connectivity and guest blogs from Industry experts and technology partners

Partners

Our extensive technology partnership ecosystem solves critical problems when it comes to network security, monitoring, application analysis, forensics and packet inspection.

Company

Garland Technology is dedicated to high standards in quality and reliability, while delivering the greatest economical solutions for enterprise, service providers, and government agencies worldwide.

Contact

Whether you are ready to make a network TAP your foundation of visibility or just have questions, please contact us. Ask us about the Garland Difference!

Packet Deduplication Explained

Duplicate packets are one of those things that happen in a busy network. Although using different network equipment and optimizing your network will minimize duplicates, they’ll still crop up on a fairly regular basis. This can be a problem for your monitoring tools, because analyzing duplicate packets can take up their processing resources while also throwing off your analytics.

Network packet brokers (NPBs) can remediate this issue by finding and removing duplicate packets before they can reach your analytics tools. Here are some of the best ways to use this feature to optimize your network and improve monitoring throughput.

Why do Duplicate Packets Appear—And Why are they Bad for Monitoring?

No matter what happens, you’ll encounter duplicate packets in your network from time to time—but if your network uses Switch Port Analyzers (SPAN ports), you’ll find them a lot more often. When SPAN ports are used to either mirror ingress and egress ports, or to mirror traffic onto a VLAN, your monitoring network might experience a 50% increase in traffic from duplicates alone.

One simple fix is to utilize a network TAP instead of a SPAN port, which will slash the number of duplicate packets you need to deal with. We recognize that this is sometimes easier said than done—that you might have legacy equipment or may have to rely on the SPAN for network access. Additionally, switching from SPAN to TAP won’t get rid of duplicate packets altogether. For example, some hosts will commonly transmit duplicate packets in order to guard against packet loss.

If your monitoring needs are sensitive enough that any number of duplicate packets is too many, or if you can’t switch away from SPAN ports, then duplicate packets can cause you real problems.

First, data center traffic has already reached enormous throughput levels and is now approaching the 100G range. Monitoring tools are already undersized for the amount of traffic that they need to deal with. With duplicate packets added to the mix, you’ll find your monitoring tools stretched past the breaking point.

Second, your monitoring and analytics depend on having consistent data—and duplicate packets can skew this data in several ways. Let’s say that your network is starting to become unstable, but your monitoring tools are chewing through a queue of “good” packets—that are in fact all duplicates. You won’t catch a hint of anything wrong until there’s a performance issue. Conversely, a stream of duplicated bad packets can display a false positive for a cyberattack or an unexplained slowdown.

Again, there are ways to mitigate this problem, some tools may have deduplication capabilities but if you want to quash it permanently and not transfer the processing burden to the tools, use a network packet broker with deduplication functionality.

>> Download now: EMA Security Best Practices [Whitepaper]

How Does Deduplication Work?

When a packet comes in, the network packet broker uses one-way encryption to turn the packet into a series of letters and numbers called a hash. Its hash is compared to the hash of every packet that came before it—and comparing two hashes is much faster than comparing two packets line by line. Since the same content will always produce the same hash, any two hashes that match are automatically duplicates. The duplicate hash is discarded before it can be sent on to your network monitoring or security tools.

This explanation is simple, but it also obscures a few edge cases. For example, what happens if two packets have the same header and the same content, but different IP type of service (TOS) numbers. Are they different or are they duplicates? What about time series? If you get two virtually identical packets a few minutes apart, is the second packet a duplicate, or is the same user just logging in a second time?

Deduplication2
Generally, these are all variables that must be defined by the user and which depend on the data center context. If two packets are generally the same, but have different IP TOS or TCP sequence numbers, they may or may not be duplicates—it depends on the data center. If you’ve bought the right kind of NPB, you’ll be able to specify whether they are or not. If specified that they’re duplicates, then the NPB will hash every part of your packet except the IP TOS or TCP sequence numbers, which means that those numbers won’t be taken into consideration when the hashes are compared.

Meanwhile, time series information is something that is also up to the discretion of the administrator. Given the right kind of granular control, you’ll be able to specify how long the network packet broker keeps hashes before it discards them. This allows you to say that packets containing the same or similar information which are received in a short amount of time—say a 500ms window—are likely to be duplicates, but that apparent duplicates received over a longer interval may not be duplicates at all.

With granular controls, network administrators can optimize their control over deduplication at a highly individualized level. This means that they’ll be able to achieve the highest possible throughput for their monitoring equipment while retaining the highest level of accuracy for their analytics. This in turn will help them maintain a stable and more secure network without being forced to purchase, install, and configure large amounts of new equipment.

Adding Deduplication To Existing Infrastructure

In today’s environment with significant investment in security and monitoring tools, Garland recognizes the need for a cost effective solution that allows the flexibility and high speeds that are required for the networks of the future. 

TAP-to-Agg-Dedup

We take the approach of providing solutions that are scalable for future on-demand growth and ROI. Garland’s PacketMAXTM line of deconstructed packet brokers includes an Advanced Features Deduplication device purpose-built to extend the feature set of existing infrastructure and reduce the processing load to security or monitoring tools. Allowing you to deploy what you need, when you need it.


Looking to add a deduplication to your deployment, but not sure where to start? Join us for a brief network Design-IT consultation or demo. No obligation - it’s what we love to do.

EMA Security visibility in your network

Written by Neil Wilkins

Neil is a Systems Engineer at Garland Technology focusing on customer challenges with network visibility such as resilience, interoperability, and integration into data center topology. Wilkins is a seasoned network professional with 30 years of experience globally within the computing industry, in product marketing and technical support, for both the commercial and public sectors