The pipeline has seven stages, left to right. Here are the first two:
1 Raw events
Telemetry: metrics, logs, traps, traces, alarms
What it does. Every system sends out signals as it runs: metrics, log lines, SNMP traps, traces and alarms. Collecting them in one place is the starting point for everything after.
If it’s missing. Parts of the estate are blind. You find out about a failure there when a user calls.
Signs you need it.
- Outages that none of your tools saw coming.
- Each team watches a different set of consoles.
2 Normalize
One format, one time zone, one severity scale
What it does. Each source describes events in its own way, with its own field names, time zone and severity words. Normalization converts them all to one format, one time zone and one severity scale.
If it’s missing. The same fault looks different depending on which tool reported it, so rules cannot match events from different sources and timelines do not line up.
Normalize: one language for every event
Every tool reports events in its own format. Normalization translates them into one common shape, so the rest of the pipeline can compare, correlate and route them.
- Field mapping: each source’s fields mapped to one shared schema (host, service, severity, message)
- Severity and time alignment: every source’s severity scale and timestamp converted to one standard
- Non-standard data: syslog, SNMP traps and custom logs parsed into structured fields
- IoT and device telemetry: sensor and device readings translated into consistent units and names
Signs you need it.
- “Critical” means something different in each tool.
- Events from two tools about the same moment show up hours apart.