Repository Structure

This page gives you a complete map of the AuroraSOC monorepo. Every folder and key file is explained so you know exactly where to look when reading or modifying the project.

---

Top-Level Overview

AuroraSOC/
├── aurorasoc/          # Python backend — agents, API, tools, events, memory
├── rust_core/          # Rust edge engine — MQTT consumer, alert normaliser, attestation
├── dashboard/          # Next.js 15 web UI
├── firmware/           # Embedded device firmware (STM32, nRF52840, ESP32-S3)
├── training/           # Granite LLM fine-tuning pipeline
├── docs/               # This Docusaurus documentation site
├── tests/              # Python test suite (pytest)
├── infrastructure/     # Canonical infra configs (Docker, Mosquitto, NATS, Vault, etc.)
├── scripts/            # Setup and helper scripts
├── alembic/            # Database migration scripts
├── .env.example               # Required environment variables (copy to .env)
│
├── Makefile            # 70+ build/dev/test/deploy targets (start here)
├── pyproject.toml      # Python project metadata and dependencies
├── pytest.ini          # Pytest configuration
├── alembic.ini         # Alembic migration configuration
├── docker-compose.yml          # Production Docker stack
├── docker-compose.dev.yml      # Development Docker stack
├── docker-compose.training.yml # GPU training Docker stack
├── Dockerfile.python   # Python backend image
├── Dockerfile.rust     # Rust core engine image
├── Dockerfile.dashboard # Next.js dashboard image
├── Dockerfile.training  # Training environment image
└── README.md           # Project README

---

aurorasoc/ — Python Backend

This is the heart of the project. All AI agents, the REST API, event processing, tools, memory, and business logic live here.

aurorasoc/
├── __init__.py
├── config/
│   ├── __init__.py
│   └── settings.py            # Pydantic BaseSettings — all env-var configuration
│
├── core/
│   ├── __init__.py
│   ├── auth.py                # JWT token creation, verification, RBAC permission checks
│   ├── database.py            # Async SQLAlchemy engine, session factory, get_db()
│   ├── logging.py             # structlog config with OpenTelemetry trace correlation
│   ├── models.py              # 13 SQLAlchemy ORM models (Alert, Case, Agent, etc.)
│   ├── rate_limit.py          # Redis-backed sliding-window rate limiter
│   └── tracing.py             # OpenTelemetry tracer/meter/provider setup
│
├── api/
│   ├── __init__.py
│   └── main.py                # FastAPI app — lifespan, routes, WebSocket managers, auth
│
├── agents/
│   ├── __init__.py
│   ├── factory.py             # AuroraAgentFactory + AGENT_SPECS dict (16 specialists)
│   ├── generic_server.py      # Generic A2AServer bootstrapper for any agent
│   ├── mcp_agent_loader.py    # AGENT_MCP_BINDINGS + load_agent_tools() for MCP tools
│   ├── prompts.py             # System prompts for all 17 agents
│   ├── server_builder.py      # Helpers for building A2A agent servers
│   │
│   ├── orchestrator/          # The Orchestrator agent
│   │   ├── __init__.py
│   │   ├── server.py          # deploy_orchestrator(), HandoffTool creation
│   │   └── dispatch.py        # CircuitBreaker (CLOSED→OPEN→HALF_OPEN)
│   │
│   ├── security_analyst/      # Per-specialist agent directories
│   ├── threat_hunter/         #   Each contains __init__.py and server.py
│   ├── malware_analyst/       #   that deploys the specialist on its port
│   ├── incident_responder/
│   ├── network_security/
│   ├── web_security/
│   ├── cloud_security/
│   ├── cps_security/
│   ├── threat_intel/
│   ├── endpoint_security/
│   ├── ueba_analyst/
│   ├── forensic_analyst/
│   ├── compliance_analyst/
│   ├── vulnerability_manager/
│   ├── report_generator/
│   └── network_analyzer/
│
├── tools/
│   ├── __init__.py
│   ├── base.py                # Base tool class and shared tool utilities
│   ├── mcp_domain_registry.py # Registry mapping MCP domain names to ports
│   ├── mcp_launcher.py        # Launches MCP tool servers
│   │
│   ├── siem/                  # SIEM tools (port 8101)
│   ├── soar/                  # SOAR playbook tools (port 8102)
│   ├── edr/                   # Endpoint detection tools (port 8103)
│   ├── network/               # Network analysis tools (port 8104)
│   ├── malware/               # Malware sandbox tools (port 8105)
│   ├── threat_intel/          # Threat intelligence tools (port 8106)
│   ├── cps/                   # CPS/IoT device tools (port 8107)
│   ├── ueba/                  # User behavior analytics tools (port 8108)
│   ├── forensics/             # Forensic analysis tools (port 8109)
│   ├── osint/                 # Open-source intelligence tools (port 8110)
│   ├── network_capture/       # Packet capture tools (port 8111)
│   ├── document/              # Document/report generation tools (port 8112)
│   ├── malware_intel/         # Malware intelligence tools (port 8113)
│   ├── cloud_provider/        # Cloud security tools (port 8114)
│   ├── vuln_intel/            # Vulnerability intelligence tools (port 8115)
│   └── registry/              # Tool registry server
│
├── events/
│   ├── __init__.py
│   ├── redis_streams.py       # RedisStreamPublisher — 5 internal streams
│   ├── nats_jetstream.py      # NATS JetStream — cross-site federation
│   └── mqtt_consumer.py       # MQTT v5 consumer — IoT edge integration
│
├── engine/
│   ├── __init__.py
│   └── playbook.py            # SOAR playbook execution engine
│
├── memory/
│   ├── __init__.py
│   ├── tiered.py              # TieredAgentMemory — 3-tier memory with 6 presets
│   └── stores.py              # Backend stores (Redis cache, Qdrant vector, PostgreSQL)
│
├── models/
│   ├── __init__.py
│   └── domain.py              # Pydantic domain models (DTOs, request/response schemas)
│
├── services/
│   ├── __init__.py
│   └── scheduler.py           # APScheduler background tasks (stale cases, metrics, etc.)
│
├── granite/
│   ├── __init__.py
│   └── registry.py            # Model registry — maps agents to Granite model variants
│
└── workflows/
    ├── __init__.py
    └── investigation.py       # End-to-end investigation workflow orchestration

Key Relationships

You want to…	Look in
Change an agent's personality	agents/prompts.py
Add a new specialist agent	agents/factory.py + new agents/<name>/ dir
Give an agent access to new tools	agents/mcp_agent_loader.py
Add a new MCP tool domain	tools/<domain>/ + tools/mcp_domain_registry.py
Modify the REST API	api/main.py
Change database schema	core/models.py + create migration
Modify event processing	events/redis_streams.py or events/nats_jetstream.py
Tune agent memory	memory/tiered.py
Change configuration defaults	config/settings.py

---

rust_core/ — Rust Edge Engine

High-performance alert ingestion and normalisation engine, written in Rust with the Tokio async runtime.

rust_core/
├── Cargo.toml             # Rust dependencies (tokio, axum, rumqttc, redis, etc.)
├── build.rs               # Proto/build script
├── Dockerfile             # Standalone Rust image
├── proto/                 # Protobuf definitions (if used for gRPC)
│
└── src/
    ├── main.rs            # Entry point — starts Axum HTTP server + MQTT consumer
    ├── config.rs          # Configuration (env vars, MQTT/Redis connection strings)
    ├── mqtt.rs            # MQTT v5 client — connects to Mosquitto, subscribes to topics
    ├── normalizer.rs      # Alert normaliser — parses CEF, Syslog, JSON into unified format
    ├── publisher.rs       # Publishes normalised alerts to Redis Streams
    ├── events.rs          # Event type definitions
    ├── health.rs          # Health check endpoint (/health)
    └── attestation.rs     # ECDSA P-256 firmware attestation signature verification

Data Flow Through Rust Core

---

dashboard/ — Next.js Web UI

The analyst-facing dashboard built with Next.js 15, React 19, and shadcn/ui.

dashboard/
├── package.json               # Dependencies (next, react, shadcn, zustand, recharts)
├── next.config.ts             # Next.js configuration
├── tailwind.config.ts         # Tailwind CSS theme
├── tsconfig.json              # TypeScript configuration
├── postcss.config.js          # PostCSS plugin config
│
└── src/
    ├── app/                   # Next.js App Router pages
    │   ├── layout.tsx         # Root layout (sidebar, auth provider)
    │   ├── page.tsx           # Dashboard home page
    │   ├── globals.css        # Global styles
    │   ├── login/             # Login page
    │   ├── alerts/            # Alert management page
    │   ├── cases/             # Case management page
    │   ├── agents/            # Agent fleet monitoring page
    │   ├── chat/              # Interactive agent chat page
    │   ├── cps/               # CPS/IoT device monitoring page
    │   ├── edr/               # EDR endpoint page
    │   ├── firmware/          # Firmware status page
    │   ├── network-analyzer/  # Network analysis page
    │   ├── siem/              # SIEM integration page
    │   ├── soar/              # SOAR playbook page
    │   ├── threat-intel/      # Threat intelligence page
    │   └── settings/          # System settings page
    │
    ├── components/
    │   ├── sidebar.tsx        # Navigation sidebar
    │   ├── client-layout.tsx  # Client-side layout wrapper
    │   ├── auth-guard.tsx     # Authentication guard component
    │   ├── dashboard/         # Dashboard-specific components
    │   └── ui/                # shadcn/ui reusable components
    │
    └── lib/
        ├── api.ts             # HTTP client for FastAPI backend
        ├── auth.tsx           # Auth context and hooks
        ├── store.ts           # Zustand global state store
        └── utils.ts           # Utility functions

---

firmware/ — Embedded Device Firmware

Three firmware targets for hardware-rooted CPS/IoT security:

firmware/
├── stm32/          # STM32 microcontroller — Ada SPARK (formal verification)
├── nrf52840/       # Nordic nRF52840 BLE — Rust Embassy (async embedded Rust)
└── esp32s3/        # Espressif ESP32-S3 WiFi — Zephyr RTOS (C)

Each firmware performs:

1. Sensor data collection from physical processes
2. ECDSA P-256 signature of attestation reports
3. MQTT QoS 2 publish to the Rust core engine over mTLS

---

training/ — LLM Fine-Tuning Pipeline

Everything needed to fine-tune IBM Granite 4 models for SOC-specific tasks.

training/
├── configs/
│   ├── granite_soc_finetune.yaml   # Training hyperparameters and agent configs
│   └── Modelfile.granite-soc       # Ollama Modelfile template
│
├── scripts/
│   ├── __init__.py
│   ├── prepare_datasets.py     # Downloads MITRE ATT&CK, Sigma, NVD, creates datasets
│   ├── finetune_granite.py     # Unsloth + LoRA fine-tuning (r=64, alpha=64)
│   ├── evaluate_model.py       # Evaluation against SOC benchmarks
│   ├── serve_model.py          # Export to Ollama GGUF or vLLM
│   └── train_all_agents.py     # Batch fine-tuning for all 16 specialists
│
├── data/                       # Generated/downloaded training datasets
└── notebooks/                  # Jupyter notebooks for experiments

---

tests/ — Python Test Suite

All Python tests use pytest and live in a single flat directory.

tests/
├── conftest.py            # Shared fixtures (async DB sessions, mock Redis, etc.)
├── test_auth.py           # JWT creation/verification, RBAC permission tests
├── test_dispatch.py       # CircuitBreaker state machine tests
├── test_models.py         # ORM model creation and relationship tests
├── test_normalizer.py     # Alert normalisation (CEF, Syslog, JSON) tests
├── test_rate_limit.py     # Sliding-window rate limiter tests
├── test_scheduler.py      # APScheduler task tests
├── test_settings.py       # Settings loading and validation tests
└── test_tiered_memory.py  # 3-tier memory system tests

Run all tests:

make test        # Quick run
make test-cov    # With coverage report

---

alembic/ — Database Migrations

PostgreSQL schema migrations managed by Alembic.

alembic/
├── env.py                         # Migration environment (async SQLAlchemy setup)
├── script.py.mako                 # Template for new migration files
└── versions/
    └── 001_initial_schema.py      # Initial schema — creates all 13 tables

Usage:

make migrate           # Apply all pending migrations
make migrate-new MSG="add new column"   # Generate a new migration
make migrate-down      # Roll back one migration

---

infrastructure/ — Supporting Services

Docker configs, monitoring, and infrastructure for all supporting services.

infrastructure/
├── docker/          # Shared Docker utilities and scripts
├── postgres/        # PostgreSQL init scripts and config
├── grafana/         # Grafana dashboard JSON definitions
├── prometheus/      # prometheus.yml scrape config
├── otel/            # OpenTelemetry Collector config (otel-collector-config.yaml)
├── mosquitto/       # Mosquitto MQTT broker config (mosquitto.conf, ACLs, TLS certs)
├── nats/            # NATS JetStream server config (nats-server.conf)
├── vault/           # HashiCorp Vault config (secrets management)
└── vllm/            # vLLM serving config (FP16 model serving)

---

docs/ — Documentation Site

This Docusaurus site you are reading now.

docs/
├── docusaurus.config.ts   # Site config (Mermaid, i18n, themes)
├── sidebars.ts            # Sidebar navigation definitions
├── package.json           # Docusaurus dependencies
├── tsconfig.json          # TypeScript config
│
├── docs/                  # Markdown documentation pages
│   ├── user/              # User-facing guide (dashboard, concepts, security)
│   ├── developer/         # Developer guide (architecture, agents, API, tools)
│   └── contributor/       # Contributor guide (this section — you are here)
│
├── src/
│   ├── pages/index.tsx    # Custom homepage
│   └── css/custom.css     # Theme customisation
│
├── static/                # Static assets (images, favicon)
└── i18n/                  # Translations (Arabic)

---

scripts/ — Helper Scripts

scripts/
└── setup_local.sh    # One-command local setup — installs Ollama, pulls Granite models,
                      # configures .env, verifies dependencies

---

Root Configuration Files

File	Purpose
Makefile	Start here — 70+ targets for every development task. Run make help to list them all.
pyproject.toml	Python project metadata, dependencies, and tool config (ruff, mypy, pytest)
pytest.ini	Pytest settings (asyncio mode, test paths, markers)
alembic.ini	Alembic database migration config (connection string, script location)
docker-compose.yml	Production stack — all services (API, Rust core, agents, Redis, PostgreSQL, etc.)
docker-compose.dev.yml	Development overrides (hot-reload, debug ports)
docker-compose.training.yml	GPU training stack (Unsloth, vLLM, Ollama import)
Dockerfile.python	Python backend image (FastAPI + agents)
Dockerfile.rust	Rust core engine image
Dockerfile.dashboard	Next.js dashboard image
Dockerfile.training	Training environment with CUDA support

---

Quick Navigation by Task

I want to…	Go to
Understand the full system	aurorasoc/ (start with api/main.py)
Read or modify an agent's behaviour	aurorasoc/agents/prompts.py
Add a new API endpoint	aurorasoc/api/main.py
Work on the web dashboard	dashboard/src/
Fix a bug in alert parsing	rust_core/src/normalizer.rs
Add a new MCP tool	aurorasoc/tools/<domain>/
Change database schema	aurorasoc/core/models.py → make migrate-new
Write or run tests	tests/
Set up infrastructure	infrastructure/ + docker-compose.yml
Fine-tune the LLM	training/
Work on firmware	firmware/<target>/
Update documentation	docs/docs/