Skip to main content

Technology Stack — Every Tool Explained

This page lists every framework, library, language, and service used in AuroraSOC. For each one, you'll learn: what it is, what it does in this project, why it was chosen, and how it connects to other components.

Programming Languages​

Python 3.12+​

  • What it is: A high-level, general-purpose programming language widely used in AI/ML, web development, and automation.
  • Role in AuroraSOC: The primary language. All AI agents, the FastAPI REST API, the MCP tool servers, the event consumers, the training pipeline, and the tests are Python.
  • Why chosen: Best ecosystem for AI/ML (PyTorch, Transformers, BeeAI), enormous library selection, async/await support. Python 3.12+ is required for performance improvements and better typing.

Rust​

  • What it is: A systems programming language focused on performance, memory safety, and concurrency — without a garbage collector.
  • Role in AuroraSOC: Powers the Rust Core Engine — a high-performance service that normalizes incoming security alerts from various formats (CEF, Syslog, JSON) into a unified format, and verifies hardware attestation certificates from IoT devices.
  • Why chosen: Alert normalization is the hottest path in the system (every alert passes through it). Rust gives C-level performance with memory safety guarantees. The attestation verification uses ECDSA P-256 cryptography that benefits from native speed.

TypeScript / JavaScript​

  • What it is: TypeScript is a typed superset of JavaScript used for web development.
  • Role in AuroraSOC: The Next.js dashboard (browser UI) and the Docusaurus documentation site are written in TypeScript.
  • Why chosen: Industry standard for frontend development. TypeScript adds type safety on top of JavaScript.

Ada SPARK​

  • What it is: A subset of the Ada programming language designed for formal verification — mathematically proving that code is free of certain classes of bugs.
  • Role in AuroraSOC: Firmware for the STM32 microcontroller. Monitors hardware security registers, detects Read Data Protection regression attacks, and reports attestation data.
  • Why chosen: When a sensor monitors physical security (doors, industrial equipment), firmware bugs can have physical safety consequences. SPARK allows proving the firmware is correct at the mathematical level.

C (Zephyr RTOS)​

  • What it is: A low-level programming language for systems and embedded development.
  • Role in AuroraSOC: Firmware for the ESP32-S3 microcontroller using the Zephyr RTOS (Real-Time Operating System).
  • Why chosen: Zephyr's C API is the most mature and well-supported path for ESP32-S3 development with WiFi and BLE.

AI / LLM Layer​

IBM BeeAI Agent Framework​

  • What it is: An open-source framework from IBM for building AI agents that can use tools, communicate with other agents, and follow structured reasoning patterns.
  • Website: https://framework.beeai.dev/introduction/welcome
  • Role in AuroraSOC: The foundation of the entire agent system. Every one of the 16 AI agents is a BeeAI RequirementAgent — a class that can enforce rules like "always think before acting." BeeAI provides:
    • ACP (Agent Communication Protocol) — How agents talk to each other. The Orchestrator uses HandoffTool to delegate tasks to specialists.
    • MCP (Model Context Protocol) — How agents discover and use tools. Each agent connects to MCP tool servers to get security-specific tools like SearchLogs or IsolateEndpoint.
    • ThinkTool — A special tool forced at step 1 of every agent interaction, requiring the agent to write a structured reasoning plan before taking any action.
    • GlobalTrajectoryMiddleware — Records every step an agent takes for observability and audit.
  • Why chosen over LangChain/LangGraph: Purpose-built for production multi-agent systems with standardized inter-agent communication (ACP), standardized tool access (MCP), and built-in observability. LangChain is more of a general LLM toolkit; BeeAI is specifically designed for the agent use case AuroraSOC needs.

IBM Granite 4​

  • What it is: A family of open-source large language models (LLMs) developed by IBM, available in multiple sizes.
  • Website: https://www.ibm.com/granite/docs/models/granite
  • Role in AuroraSOC: The "brain" of every AI agent. When an agent needs to reason about a security alert, decide on next steps, or generate a report, it sends a prompt to a Granite model and receives a completion. Specifically:
    • granite4:8b — Used by most specialist agents (8 billion parameters — a good balance of capability and speed)
    • granite4:dense — Used by the Orchestrator (larger model for complex coordination and synthesis)
    • Per-agent fine-tuned variants — Each agent can optionally use a model fine-tuned on its specific domain data
  • Why chosen: Open source (runs fully locally — no data ever leaves your infrastructure), fine-tunable with domain-specific security data, efficient enough to run on consumer GPUs with quantization.

Ollama​

  • What it is: A tool for running LLMs locally. It downloads, manages, and serves LLM models on your machine.
  • Role in AuroraSOC: The default model serving backend. All agents send their prompts to Ollama, which runs the Granite model and returns completions. Ollama manages model downloads, GPU allocation, and inference.
  • Why chosen: Simplest path to running LLMs locally. Single binary, no Python dependencies for serving. Supports GGUF format (quantized models that fit in less GPU memory).

vLLM (Optional)​

  • What it is: A high-throughput LLM serving engine designed for production deployments, supporting batched inference and continuous batching.
  • Role in AuroraSOC: An alternative serving backend for when you need higher throughput than Ollama — for example, serving many agents simultaneously in production.
  • Why chosen: Continuous batching can serve 10-50x more requests per second than naive one-at-a-time inference.

Unsloth​

  • What it is: A library that makes LLM fine-tuning 2x faster and uses 70% less GPU memory compared to standard training.
  • Role in AuroraSOC: Used in the training pipeline to fine-tune Granite models on security domain data. Makes it feasible to fine-tune on consumer GPUs (single RTX 3090/4090) rather than requiring expensive multi-GPU clusters.
  • Why chosen: Dramatically lowers the hardware barrier for fine-tuning.

LoRA (Low-Rank Adaptation)​

  • What it is: A technique for fine-tuning LLMs by training only a small number of additional parameters (adapters) instead of modifying the entire model. This is much cheaper and faster.
  • Role in AuroraSOC: The fine-tuning strategy. Instead of retraining all 8 billion parameters of Granite, LoRA adds ~0.5% extra parameters that specialize the model for security operations. Configuration: rank=64, alpha=64.

Sentence Transformers​

  • What it is: A Python library for generating embeddings — fixed-size numerical vectors that capture the semantic meaning of text.
  • Role in AuroraSOC: Generates embeddings for the episodic memory system. When an agent completes an investigation, the case summary is embedded into a vector and stored in Qdrant. Later, agents can search for "cases similar to this one" using vector similarity.

Web Framework Layer​

FastAPI​

  • What it is: A modern, high-performance Python web framework for building REST APIs, built on Starlette and Pydantic.
  • Role in AuroraSOC: The central API server (port 8000). Provides:
    • 40+ REST endpoints for managing alerts, cases, devices, playbooks, agents
    • 3 WebSocket channels for real-time streaming (alerts, agent thoughts, system events)
    • Background tasks for consuming Redis Streams and pushing updates to connected clients
    • Health checks, CORS configuration, OpenTelemetry auto-instrumentation
  • Why chosen: Async-native (critical for I/O-heavy security operations), automatic API documentation (OpenAPI/Swagger), type validation via Pydantic.

Next.js 15​

  • What it is: A React framework for building web applications with server-side rendering, file-based routing, and modern React features.
  • Role in AuroraSOC: The dashboard (port 3000) — the browser-based UI that security analysts use to view alerts, track cases, monitor AI agents, manage playbooks, and view device status.
  • Why chosen: App Router provides modern React 19 support, server components for performance, and file-based routing for simplicity.

Docusaurus​

  • What it is: A static site generator from Meta designed for documentation websites.
  • Role in AuroraSOC: Generates this documentation site. Supports multiple sidebars (User Guide, Developer Guide, Contributor Guide), i18n (English + Arabic), Mermaid diagrams, and versioning.

UI Libraries (Dashboard)​

shadcn/ui​

  • What it is: A collection of reusable, accessible UI components built on Radix UI primitives and styled with Tailwind CSS.
  • Role: All dashboard UI components (buttons, dialogs, dropdowns, data tables, tabs, tooltips, etc.).

Zustand​

  • What it is: A small, fast state management library for React.
  • Role: Manages client-side state in the dashboard (which alerts are selected, filter settings, agent status, etc.).
  • Why over Redux: Much simpler API, less boilerplate, excellent TypeScript support.

Recharts​

  • What it is: A React charting library built on D3.
  • Role: Renders time-series charts and analytics visualizations (alert trends, severity distributions, agent activity).

Tailwind CSS​

  • What it is: A utility-first CSS framework where you style elements by composing small utility classes in HTML.
  • Role: All dashboard styling.

Data Layer​

PostgreSQL 16​

  • What it is: An enterprise-grade open-source relational database.
  • Role in AuroraSOC: The primary database. Stores 13 tables: alerts, investigation cases, case timeline events, CPS device records, attestation results, playbooks, playbook executions, IOCs (indicators of compromise), agent audit logs, human approval requests, and reports.
  • Driver: asyncpg — an async PostgreSQL driver for Python, used via SQLAlchemy's async engine.

SQLAlchemy 2.0 (Async)​

  • What it is: Python's most popular ORM (Object-Relational Mapper) — it lets you interact with database tables using Python classes instead of writing raw SQL.
  • Role: Defines all 13 database models as Python classes. Uses the async engine (create_async_engine) with the asyncpg driver so database queries never block the event loop.

Alembic​

  • What it is: A database migration tool for SQLAlchemy. It tracks changes to your database schema over time.
  • Role: Manages schema changes. When you add a new column or table, you create an Alembic migration file. Running alembic upgrade head applies all pending migrations.

Qdrant​

  • What it is: An open-source vector database designed for storing and searching high-dimensional embeddings.
  • Role in AuroraSOC: Stores the episodic memory of AI agents. When an investigation is completed, its summary is embedded into a vector and stored in Qdrant. When a new alert arrives, agents can search Qdrant for "past cases similar to this alert" using vector similarity (cosine distance).
  • Why a separate database: PostgreSQL is great for structured queries ("give me all critical alerts from today") but terrible at semantic similarity search. Qdrant is purpose-built for "find the 5 most similar vectors."

Redis 7​

  • What it is: An in-memory data store used as a cache, message broker, and rate limiter.
  • Role in AuroraSOC: Three responsibilities:
    1. Event bus — Redis Streams delivers alerts, agent tasks, and results between components
    2. Cache — Threat intelligence lookups are cached in Redis (TTL: 1 hour) so agents don't re-query external feeds for the same IOC
    3. Rate limiting — Sliding-window rate limiters (e.g., 10 investigations/min per user) use Redis sorted sets for distributed counting

Event / Messaging Layer​

Redis Streams​

  • What it is: A data structure in Redis designed for event streaming — like a log file that multiple consumers can read from independently.
  • Role in AuroraSOC: The internal event bus with 5 streams:
    • aurora:alerts — Security alerts from the Rust normalizer
    • aurora:cps:alerts — IoT/CPS device alerts from MQTT
    • aurora:agent:tasks — Orchestrator dispatching tasks to specialist agents
    • aurora:agent:results — Agents returning their findings
    • aurora:audit — Immutable audit trail of every action
  • Consumer groups allow multiple workers to share the load of processing a stream.

NATS JetStream​

  • What it is: A messaging system with persistent, exactly-once delivery. JetStream is NATS's persistence layer.
  • Role in AuroraSOC: Cross-site federation. If you run AuroraSOC at multiple locations, NATS syncs IOCs (indicators of compromise), alerts, and attestation results between sites. Subjects:
    • aurora.intel.ioc.* — shared threat indicators
    • aurora.alerts.federation — federated alerts
    • aurora.cps.attestation — device attestation distribution

MQTT (Mosquitto)​

  • What it is: A lightweight messaging protocol designed for IoT devices with limited bandwidth and computing power. Mosquitto is the most widely used MQTT broker.
  • Role in AuroraSOC: The bridge between edge devices (physical sensors/controllers) and the system. Devices publish telemetry, alerts, and attestation responses via MQTT. The Rust Core Engine subscribes to MQTT topics and relays events to Redis Streams.
  • Security: Uses mTLS (mutual TLS) — both the device and broker verify each other's certificates. QoS 2 (exactly-once delivery) for security-critical messages like attestation.

Security Libraries​

PyJWT​

  • What it is: A Python library for creating and verifying JSON Web Tokens (JWTs).
  • Role: Authentication. When a user logs in, the API creates a JWT containing their username, role, and permissions. The JWT is sent with every subsequent request to prove identity.

bcrypt​

  • What it is: A password hashing algorithm designed to be intentionally slow, making brute-force attacks impractical.
  • Role: Passwords are hashed with bcrypt before storage. The system never stores plaintext passwords.

cryptography​

  • What it is: A comprehensive Python cryptography library.
  • Role: Provides the ECDSA P-256 primitives used for verifying hardware attestation certificates from IoT devices.

HashiCorp Vault​

  • What it is: A secrets management system that securely stores and controls access to sensitive data (API keys, certificates, database credentials).
  • Role in AuroraSOC: Manages the PKI (Public Key Infrastructure) for CPS device certificates. Issues, rotates, and revokes certificates for IoT devices.

Observability Stack​

OpenTelemetry​

  • What it is: An open-source standard for collecting traces, metrics, and logs from applications.
  • Role in AuroraSOC: Distributed tracing. Every API request, agent action, and tool call generates a trace span. These spans are exported via OTLP gRPC to the OpenTelemetry Collector, which routes them to monitoring backends.
  • Components used:
    • opentelemetry-api + opentelemetry-sdk — core tracing API
    • opentelemetry-instrumentation-fastapi — auto-instruments every HTTP request
    • opentelemetry-exporter-otlp-proto-grpc — exports spans to the collector
    • BatchSpanProcessor — batches spans for efficient export

Prometheus​

  • What it is: A time-series database and monitoring system that scrapes metrics from applications.
  • Role: Collects system metrics — alert counts by severity, case statistics, device counts, agent response times — every 60 seconds. These metrics feed Grafana dashboards.

Grafana​

  • What it is: A visualization platform for metrics and logs.
  • Role: Provides dashboards showing real-time system health, alert trends, agent performance, and device status.

structlog​

  • What it is: A structured logging library for Python that outputs machine-readable JSON logs instead of plain text.
  • Role: All application logging. Each log entry includes key-value pairs (timestamp, service, trace_id, span_id, event details) that can be searched and analyzed by log aggregation tools.

Build / Dev Tools​

Docker + Docker Compose​

  • What it is: Docker packages applications into containers (lightweight, portable units). Docker Compose orchestrates multiple containers together.
  • Role: Runs the entire AuroraSOC stack — all 16 agents, Redis, PostgreSQL, Qdrant, NATS, Mosquitto, Ollama, Vault, OTel Collector, Prometheus, Grafana, 15 MCP tool servers, the API, and the dashboard — in one docker compose up command.

pytest + pytest-asyncio​

  • What it is: Python's most popular testing framework. pytest-asyncio adds support for testing async code.
  • Role: All Python tests. Tests use SQLite in-memory databases (no external dependencies needed) with compatibility shims for PostgreSQL-specific types.

Ruff​

  • What it is: An extremely fast Python linter and formatter written in Rust.
  • Role: Enforces code style and catches common bugs. Run with make lint and make format.

mypy​

  • What it is: A static type checker for Python.
  • Role: Validates type annotations. Run with make type-check.

Cargo (Rust)​

  • What it is: Rust's package manager and build system.
  • Role: Builds and tests the Rust Core Engine. Key dependencies: tokio (async runtime), axum (web framework), redis (Streams publisher), p256 + ecdsa (cryptography), tracing (logging).

How Technologies Connect​

Next: System Architecture → — How all these technologies work together end-to-end.