Skip to main content

Per-site continuous learning

What this page is​

The opt-in learning loop that sharpens triage scoring, per-rule false-positive rates, CPS baselines, and automation-tier calibration from real outcomes inside a deployment, and the automated canary promotion and rollback that keeps a regressing model from ever affecting live scoring. It complements the eval harness on the LLM evaluation page.

Why it exists this way​

Detection and triage decisions were static: the pre-LLM triage filter (ADR 037) scored alerts with a hand-tuned heuristic, per-rule false-positive strength was a constant, CPS anomaly thresholds were fixed, and automation tiers came from a fixed cutoff. Every closed case, analyst override, and quiet auto-resolution is ground truth about that specific environment. ADR 043 makes live operation sharpen the system per deployment without raw telemetry leaving a site and without a bad model ever regressing live scoring.

How it works​

Foundation (aurorasoc/learning/)​

  • A label store fuses four ground-truth sources (analyst disposition, agent override, threat intel, downstream outcome) with confidence weights (human > intel > silence).
  • A point-in-time feature snapshot is written at scoring time to prevent train/serve skew.
  • A model registry carries a shadow / active / retired lifecycle.
  • An enable / force-deterministic switch (AURORA_LEARNING_ENABLED, AURORA_FORCE_DETERMINISTIC) gates whether learned models influence scoring at all. This complements the ADR 038 automation kill-switch, which gates how much agents act.

Tenant maps to site (site_id, enforced by DataResidencyMiddleware); models and training stay within a site, with a global base for cold-start.

Per-surface learners​

  • Triage scorer. A per-site LightGBM model emits calibrated P(true-positive), fed to score_alert through a pure model_proba parameter. The retrain job (python -m aurorasoc.learning.retrain) holds out the recent time slice; a promotion gate admits a candidate only if it beats both the deterministic baseline and the incumbent by an AUC margin on a sufficient, two-class holdout.
  • Per-rule false-positive learning. A transparent, recency-weighted, support-shrunk FP-rate per (rule_id, site_id) feeds fp_strength and surfaces suppression suggestions.
  • CPS baselines. The per-(site, zone, metric) online baseline is persisted across restarts, and the cohort divergence threshold adapts to a zone's learned spread, never below the fixed floor.
  • Automation-tier calibration. Learns where autonomy is less safe, but is downgrade-only: it may lower the tier the filter chose (L2_ASSIST -> L1_PROPOSE), never raise it past policy.

Every learned lookup is best-effort: on any failure, absent model, or disabled learning, the existing deterministic path runs unchanged.

Automated canary promotion and rollback​

ADR 034 adds eval/promotion.py, the executor that acts on the canary gate verdict the eval harness produces. A strict state machine (shadowing -> canary -> promoted) forces rolled_back and reverts to the baseline on a breach at any stage. CanaryController.apply is a pure function, so the auto-rollback guarantee is unit-tested without a DB or live model, and BackgroundScheduler._canary_promotion_loop runs the pass every 600s. This is the guarantee that makes raising the automation tier defensible: a regressing model self-heals.

What goes wrong​

  • Learning seems to do nothing: it is off unless AURORA_LEARNING_ENABLED is set, and any failure falls back to the deterministic path by design.
  • A candidate never promotes: the promotion gate requires a sufficient, two-class holdout and a real margin over both baseline and incumbent. A marginal or small-sample candidate correctly changes nothing.
  • Candidate registration is the remaining wiring for canary promotion; the shared store and loop are in place.