Packets don’t lie: where network taps fit in modern observability

Most of what we call observability is a system describing itself. A server writes a log. An application reports a metric. A service emits a trace of the request it just handled. All of it is useful, and all of it is testimony: the system’s own account of what happened.

Packets are different. A packet is what actually crossed the wire. When the application team says the network is slow and the network team says the application is, the packets settle it. When a security team needs to know what left the building during an incident, the packets are the record. That is why, in a period when nearly everything else about infrastructure has moved into software, organizations still install hardware whose only job is to copy traffic.

Why not just mirror a port?

Every managed switch can copy traffic from one port to another. It costs nothing, and for occasional troubleshooting it is fine.

As a permanent source of truth it has limits. A switch treats mirroring as a lower priority than forwarding, so when the switch is busy, the copy is the first thing to suffer. Packets go missing from the mirror that were delivered perfectly well on the network. Timing can shift. Malformed frames, often the ones you most want to see, may be dropped before they are copied.

A tap is a separate device placed in the link itself. A passive optical tap contains no electronics at all: it splits the light in the fiber, sending most of it on its way and a fraction to a monitoring port. It cannot be misconfigured, cannot be attacked over the network, and keeps working with the power off.

The broker in the middle

A single tap is simple. A hundred of them are not. Each produces two streams, one for each direction of traffic, and each tool that wants to see traffic would otherwise need its own connection to every one.

The network packet broker solves that. It collects the output of every tap and mirror, then does the unglamorous work that makes the traffic usable: removing duplicates where the same packet was captured at two points, filtering out what a given tool does not need, trimming or masking sensitive contents such as card numbers, stamping each packet with a precise time, and spreading the load across several tools. Without it, expensive analysis tools spend much of their capacity on traffic they should never have received.

Gigamon is one of the best-known vendors in this category. It is far from alone: Keysight, NETSCOUT, Arista and Cisco all offer brokers, and specialists such as Garland Technology, Profitap, Niagara Networks, Cubro and APCON focus on taps and brokers specifically.

The part nobody draws on the whiteboard

Architecture diagrams show a tap as a small box with a line coming out of it. The reality is a cabling project.

Every tapped link has to be cut and re-joined through the tap, which means a planned outage for each one. Each tap’s two outputs need two ports on the broker, so fifty links become a hundred monitoring cables. Those cables carry light in one direction only, which makes them easy to patch incorrectly, and nothing works until they are right.

Then there is the light itself. A tap that sends, for example, 30 percent of the signal to monitoring leaves 70 percent for the production link. Someone has to work out, link by link, that both sides still receive enough. Get it wrong and the monitoring project degrades the network it was meant to observe. Add matching fiber types, breakout panels for higher-speed links, a separate management network and accurate labeling, and it becomes clear why these projects succeed or fail on planning rather than on product choice.

What has changed

Three things, mainly.

Traffic moved. Much of it now runs between virtual machines and containers, or inside a cloud provider, where there is no cable to tap. The cloud platforms offer traffic mirroring as a service, and the broker vendors offer virtual versions of their products to collect it.

Traffic is encrypted. A large share of what crosses a network today cannot be read without the keys. Vendors have responded in different ways: decrypting at the broker, observing traffic before it is encrypted, or extracting useful summaries from the parts that remain visible.

Software can see more. A newer technique observes network activity from inside each server’s operating system, with no tap at all. It reaches places hardware cannot. It also depends on the very systems it is observing, which is exactly the dependence a tap avoids.

None of these replaced the tap. They extended the same idea, an independent copy of the traffic, into places a splitter cannot go.

Where it earns its place

The case is strongest wherever being wrong is expensive and the answer has to be provable. Financial firms measure the time between a market data update arriving and an order leaving, in millionths of a second, and need timestamps accurate enough to stand behind. Security teams need to see traffic moving inside the network, not only at its edge. Any organization that has had to reconstruct an incident from incomplete logs understands the value of a record that does not depend on what each system chose to write down.

It is not the first thing most organizations should buy. It is infrastructure for a monitoring practice that already knows what questions it is asking. Decide what you need to see, and which tools will look at it, before deciding where to cut the first fiber.