Event-Driven Pipeline
AuroraSOC processes security events through a multi-stage pipeline using three complementary messaging systems.
Pipeline overview
The ingest plane uses Vector to parse and normalise raw feeds into OCSF 1.3 events and publishes them to NATS JetStream, the primary event bus between planes. The Rust detection workers consume from NATS, evaluate Sigma-compiled rules, and emit alerts. Redis holds ephemeral state and the agent task queue; it is not the primary event bus. The agent plane enriches IOCs against external threat-intel sources and issues queued containment actions, gated by human approval for L3 tiers, back to the EDR fleet through the collector bridge.
Why three messaging systems
| System | Purpose | Pattern | Durability |
|---|---|---|---|
| NATS JetStream | Primary event bus, cross-site federation | Pub/Sub plus persistence | Disk-backed |
| Redis Streams | Ephemeral state, agent task queue | Consumer groups | In-memory plus AOF |
| MQTT | IoT device communication | Pub/Sub plus QoS levels | Broker-dependent |
NATS JetStream: the primary event bus
NATS JetStream carries normalised OCSF events between the ingest, detection, and agent planes and is the primary event bus within and across AuroraSOC sites. Cross-site, it provides persistent delivery with exactly-once semantics through consumer acknowledgment and deduplication, and NATS clusters span data centres with geographic distribution. Subjects include AURORA.alerts for alert federation and AURORA.ioc_sharing for IOC dissemination.
Redis: ephemeral state and the agent task queue
Redis is not the primary event bus. It holds ephemeral hot state and the agent task queue, exposed as Redis Streams with consumer groups:
aurora:agent:tasksfor agent task queueaurora:agent:resultsfor agent resultsaurora:agent:dead_letterfor dead-letter streamaurora:auditfor audit log
Why Redis Streams for the queue: consumer groups share workload with each message processed once, failed messages can be re-processed, automatic stream trimming cleans up old messages, sub-millisecond latency, and Redis is already deployed for caching so no additional infrastructure is needed.
Agent task worker
A dedicated worker process at aurorasoc/workers/agent_task_worker.py executes queued tasks and publishes correlated results. Tasks from aurora:agent:tasks are consumed by the aurora-agents consumer group. Results are published to aurora:agent:results. Failed tasks retry up to 3 times before dead-lettering.
MQTT: IoT edge communication
MQTT connects resource-constrained IoT devices with minimal overhead (2-byte header). Topics follow aurora/sensors/+/telemetry, aurora/sensors/+/alerts, and aurora/sensors/+/attestation. QoS levels guarantee delivery even with intermittent connectivity. TLS encrypts device-to-broker communication.
Event processing guarantees
| Guarantee | Implementation |
|---|---|
| At-least-once delivery | Consumer group ack plus re-delivery on timeout |
| Ordering | Per-stream ordering via Redis Streams |
| Deduplication | SHA-256 hash-based dedup in scheduler, 5-min window |
| Backpressure | Consumer groups naturally provide backpressure |
| Dead letter | Failed tasks after 3 retries moved to dead letter stream |
AuroraSOC follows the "smart endpoints, dumb pipes" philosophy. The messaging systems are simple transport. All business logic lives in the consuming services.