OSINT Tools

The osint tool server provides web search, domain WHOIS, and related open-source intelligence gathering capabilities.

Search Backend Configuration

The OSINT search tool supports two modes:

1. Dedicated Search API (recommended for production)

Set OSINT_SEARCH_API_URL to point at a self-hosted or commercial search API (e.g., SearXNG, Brave Search API, Serper):

OSINT_SEARCH_API_URL=http://searxng:8080/search

This is the preferred approach because it avoids rate-limiting and provides structured JSON responses.

2. DuckDuckGo HTML Fallback

When no search API is configured, the tool falls back to scraping DuckDuckGo's HTML endpoint (https://html.duckduckgo.com/html/).

Limitations:

Concern	Detail
Rate limiting	DuckDuckGo may throttle or block high-frequency requests
Fragility	HTML structure can change without notice, breaking parsing
No pagination	Only the first page of results is available
Legal	Automated scraping may violate DuckDuckGo ToS in some jurisdictions

Controls:

Variable	Default	Description
OSINT_ENABLE_PUBLIC_SEARCH_FALLBACK	true	Enable/disable the DuckDuckGo fallback
OSINT_PUBLIC_SEARCH_URL	https://html.duckduckgo.com/html/	Override the fallback search endpoint

To disable the fallback entirely and require an explicit search API:

OSINT_ENABLE_PUBLIC_SEARCH_FALLBACK=false

Recommended Production Setup

Deploy SearXNG as a sidecar container:

services:
  searxng:
    image: searxng/searxng:latest
    environment:
      SEARXNG_URL: http://searxng:8080
    volumes:
      - ./infrastructure/searxng/settings.yml:/etc/searxng/settings.yml:ro

Then configure AuroraSOC to use it:

OSINT_SEARCH_API_URL=http://searxng:8080/search
OSINT_ENABLE_PUBLIC_SEARCH_FALLBACK=false