HTTP API

crawlsign serve (the api extra) runs the engine as a local HTTP service, so crawlers in any language, other processes, or several workers can share one detector. It runs on your infrastructure; there is no hosted CrawlSign API.

crawlsign serve                       # http://127.0.0.1:8765
curl -s http://127.0.0.1:8765/v1/analyze/url \
  -H 'content-type: application/json' \
  -d '{"url": "https://example.com/"}'

127.0.0.1:8765 is the server you run, not an Apotropic endpoint: page content and results never leave your infrastructure. If the server runs on another host in your network, use its address in place of 127.0.0.1:8765 throughout this page.

The running server also serves its OpenAPI schema at /openapi.json and interactive docs at /docs; start it with --no-docs to turn both off.

Choosing an endpoint

• POST /v1/analyze/url when the server can do the fetch. robots.txt is enforced before the request, and private, loopback, link-local and internal targets are refused.
• POST /v1/analyze/html when your crawler already fetched the page. Send the response headers too, or X-Robots-Tag refusals are invisible. robots.txt is not checked here, so check it in your crawler.

Gate on result.action: ingest only continue and log.

Binding and authentication

The server binds to loopback by default and needs no token there. To expose it on another interface, pass --host with --allow-public and set CRAWLSIGN_API_TOKEN; the server refuses to start publicly without one. With a token set, every endpoint except /health requires Authorization: Bearer <token>, and a wrong token returns 401.

Shared state

One server process keeps one detector and fetcher per config file for its lifetime. Domain-stop escalation therefore works across requests and across both analyze endpoints, and robots.txt is cached per host. Editing a config file starts fresh state for it; restarting the server clears everything. Per-host rate limits and budgets in a config accumulate for the server's lifetime.

The fixture endpoints analyze the demo pages from the examples bundle and return an empty list when the server runs without it.

Endpoints

POST /v1/analyze/url

Fetch and analyze a URL. Fetches through SafeFetcher, preserves robots behavior, and validates URL targets.

Request body: UrlAnalyzeRequest

POST /v1/analyze/html

Analyze submitted HTML. Analyzes caller-supplied HTML without fetching network resources.

Request body: HtmlAnalyzeRequest

GET /health

Check local API health. Returns a small deterministic health payload for local service checks.

GET /v1/fixtures

List demo fixtures. Returns checked-in deterministic MVP fixtures. Arbitrary fixture paths are not accepted.

POST /v1/analyze/fixture

Analyze a checked-in fixture. Analyzes one fixture from /v1/fixtures using the existing CrawlSign engine.

Request body: FixtureAnalyzeRequest

Schemas

AnalyzeResponse

DetectionResultResponse

EvidenceItem

EvidenceSection

FixtureAnalyzeRequest

FixtureCatalogEntry

HTTPValidationError

HealthResponse

HtmlAnalyzeRequest

ResultView

UrlAnalyzeRequest

ValidationError

ViewSummary