Event-Driven Pipeline

AuroraSOC processes security events through a multi-stage, event-driven pipeline using three complementary messaging systems. This architecture ensures events are processed reliably, at scale, and with proper ordering.

Pipeline Overview

Why Three Messaging Systems?

Each messaging system serves a distinct purpose:

System	Purpose	Pattern	Durability
Redis Streams	Internal event bus	Consumer groups	In-memory + AOF
NATS JetStream	Cross-site federation	Pub/Sub + persistence	Disk-backed
MQTT	IoT device communication	Pub/Sub + QoS levels	Broker-dependent

Redis Streams — The Internal Bus

Redis Streams is the primary event bus within a single AuroraSOC deployment:

Why Redis Streams?

• Consumer groups — Multiple consumers share the workload, each message processed once
• Message acknowledgment — Failed messages can be re-processed
• Stream trimming — Automatic cleanup of old messages (configurable max length)
• Sub-millisecond latency — Critical for real-time alerting
• Already deployed — Redis is used for caching, so no additional infrastructure

NATS JetStream — Cross-Site Federation

For organizations with multiple AuroraSOC instances:

The AURORA JetStream stream has subjects:

• AURORA.alerts — Alert federation
• AURORA.ioc_sharing — IOC dissemination
• AURORA.agent_results — Investigation sharing
• AURORA.audit — Cross-site audit trail

Why NATS JetStream?

• Persistent delivery — Messages survive broker restarts
• Exactly-once semantics — Via consumer acknowledgment + deduplication
• Geographic distribution — NATS clusters span data centers
• Lightweight — Single binary, minimal resource usage

MQTT — IoT Edge Communication

MQTT connects resource-constrained IoT devices:

Why MQTT?

• Minimal overhead — 2-byte header, ideal for constrained devices (nRF52840 with 256KB RAM)
• QoS levels — Guaranteed delivery even with intermittent connectivity
• Topic wildcards — aurora/sensors/+/* captures all device telemetry
• Industry standard — Every IoT platform speaks MQTT
• TLS support — Encrypted device-to-broker communication

Event Processing Guarantees

Guarantee	Implementation
At-least-once delivery	Consumer group ack + re-delivery on timeout
Ordering	Per-stream ordering via Redis Streams
Deduplication	SHA-256 hash-based dedup in scheduler (5-min window)
Backpressure	Consumer groups naturally provide backpressure
Dead letter	Failed messages after 3 retries logged to audit stream

Design Principle

AuroraSOC follows the "smart endpoints, dumb pipes" philosophy. The messaging systems are simple transport — all business logic lives in the consuming services (API, agents, scheduler).

Event Flow Example: Alert Processing

Here's the complete journey of a security event: