CrawlSign / Crawler safety for AI ingestion
CrawlSign runs inside your crawler and checks every page before it enters a dataset, an index, or an agent's context. It honors opt-outs, catches tarpits, poison sources, and hidden-link traps, and returns one decision with an audit trail. Engine and data stay on your infrastructure.
{
"url": "https://example.com/noai",
"should_stop": true,
"score": 80,
"action": "stop",
"reasons": [
"ai_refusal_signal_detected"
],
"threshold": 70
}Three risks
Poisoned content enters at index time. Re-indexing is the only remedy once it is in a dataset.
noai and X-Robots-Tag are cited in opt-out frameworks. Ignoring them carries risk beyond a policy question.
Tarpit pages inflate crawl depth and quota with no useful signal. Hidden-link clusters compound the problem quietly.
Three transports
The same detector core, three transports. Start with the CLI or SDK; run the local HTTP API when crawlers in other languages or processes need the same decisions.
Scan live URLs, URL lists, or saved HTML from a terminal. Output is JSON or Markdown, stop decisions exit with code 2 for CI, and reason codes are stable across releases.
crawlsign scan https://example.com
crawlsign scan-file urls.txt --compact
crawlsign analyze-file page.html --url https://example.com/page --format markdown
crawlsign analyze-warc crawl.warc.gz --output report.jsonlUse scan_url or SafeFetcher for live URLs so robots.txt is checked first. Use PoisonDetector on bytes you already have, with no network hop on the critical path.
from crawlsign import PoisonDetector, scan_url
# Live URL: robots.txt, then a bounded fetch, then analysis
live = scan_url("https://example.com")
# HTML you already have: no network I/O
result = PoisonDetector().analyze(
url=url,
html=html,
headers=headers,
)
if result.should_stop:
print(result.action, result.reasons)Expose the same detector over HTTP for crawlers in any language. Python callers get the typed CrawlSignClient.
crawlsign serve
# http://127.0.0.1:8765, OpenAPI schema at /openapi.json
curl -X POST http://127.0.0.1:8765/v1/analyze/html \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com","html":"<page html here>"}'Framework adapters
Built-in loaders and drop-in guards for the stacks AI and RAG teams already use. Each one gates on the same action and never hands a withheld page to your pipeline.
CrawlSignWebLoader fetches through one shared SafeFetcher, so robots.txt is checked first, and only safe pages become Document objects. Withheld pages are listed, without content, for the audit trail.
from crawlsign.integrations.langchain import CrawlSignWebLoader
loader = CrawlSignWebLoader(urls)
docs = loader.load() # metadata: crawlsign_action, crawlsign_reasons, ...
audit = loader.skipped # what was withheld, and whyCrawlSignWebReader is the equivalent reader. CrawlSign metadata is stored with each document but kept out of embeddings and prompts.
from crawlsign.integrations.llamaindex import CrawlSignWebReader
reader = CrawlSignWebReader()
documents = reader.load_data(urls)
audit = reader.skippedDownloader middleware that analyzes every HTML response, tracks per-reason crawler stats, and raises IgnoreRequest on a stop or quarantine verdict. Scrapy's own ROBOTSTXT_OBEY stays on.
# settings.py
ROBOTSTXT_OBEY = True
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.CrawlSignDownloaderMiddleware": 543,
}
# stats keys: crawlsign/action/{stop|quarantine|log|continue}
# crawlsign/reason/{reason_code}Checks robots.txt before the browser navigates, then inspects the rendered DOM with the real response headers, so traps injected late in the page lifecycle still get caught.
fetcher = SafeFetcher() # one per crawl
outcome = await guarded_goto(page, url, fetcher)
if outcome["safe"]:
pages.append(outcome["html"])Five patterns
Five patterns the detector flags, drawn from the bundled fixture library and the mocked gzip relay in the offline demo.
Page markup (strict-enterprise config)
<div style="display:none">
<a href="/poison/aaa...">hidden</a>
<a href="/poison/bbb...">hidden</a>
<!-- 8 more hidden anchors -->
</div>hidden_link_detectedhidden_internal_link_cluster_detectedPage markup
<meta name="robots"
content="noai,noimageai">ai_refusal_signal_detectedEndpoint path
GET /nepenthes
User-Agent: ResearchCrawler/1.0known_tarpit_reference_detectedPage markup
<h1>Hello</h1>
<p>A normal article.</p>
<a href="/about">About</a>Poison Fountain-style gzip relay (live fetch)
GET /hidden/relay
Content-Encoding: gzip
85 bytes on the wire, 71 bytes decoded:
This page relays rnsaffn.com/poison2 content.known_poison_source_detectedDecision path
Live URL scans check robots.txt first, then fetch with size, time, and redirect limits and decode gzip before analysis.
The detector checks refusal tags, known poison markers, tarpit endpoints, and hidden links. Signature matches take the highest score, not the sum.
Every run emits the same schema: score, action, reasons, threshold, and metadata, bucketed into four default bands.
On the live fetch path, stopped and quarantined pages are written to a local quarantine folder with their evidence, and their content is never returned to the caller.
Default score bands
Every edge is set in config. Monitor mode logs what it would withhold without blocking, strict-enterprise quarantines on a single hidden same-origin link, and repeated stops on one host escalate to a domain-wide stop.
Detection surface
| Detector | Signals | Typical action |
|---|---|---|
| Robots and refusal | robots.txt, X-Robots-Tag, noai, noimageai |
stop |
| Poison signatures | Curated signatures for known poison sources and tarpit tools, matched by URL, endpoint path, and page marker, plus your own YAML list | quarantine or stop |
| Hidden links | Anchors hidden by inline CSS, the hidden or aria-hidden attributes, zero size, visually-hidden classes, or local <style> rules, including hiding inherited from parent elements. Same-origin targets are listed so the crawler can drop them from its frontier. |
log to quarantine |
| Safe fetching | HTTP/HTTPS only; connect, read, and total deadlines plus a minimum-throughput floor against slow-drip tarpits; size caps; redirect validation and loop detection; non-HTML skipped; per-host rate limits and page, byte, and redirect budgets. The local API also rejects loopback, private, link-local, and .internal targets |
reject, stop, or analyze |
| Compressed relays | Content-Encoding: gzip decoded before signature and marker analysis; compressed and decompressed sizes both capped by max_response_bytes |
stop, content withheld |
Design-partner pilot
Pilots run on your infrastructure. In monitor mode CrawlSign still honors refusals, but only logs what it would withhold. The monitor report shows exactly what enforcement would change, by host and reason code, with a column for marking false positives. Enforce when the report matches what you expect.
robots.txt and noai are honored throughout# 1. Run your crawl, or an existing WARC, in monitor mode
crawlsign analyze-warc crawl.warc.gz \
--config configs/audit-only.yaml --output run.jsonl
# 2. Review what enforcement would have withheld
crawlsign monitor-report run.jsonl \
--format html --output monitor-report.html
# 3. Enforce: same engine, default config
crawlsign serveCommon questions
Standard robots parsers only read robots.txt. CrawlSign also inspects
X-Robots-Tag headers, <meta name="robots"> with
noai and noimageai tokens, known poison signatures, and
hidden-link traps in page markup. All signals are scored and bucketed into a single
action you can act on without writing decision logic yourself.
LangChain and LlamaIndex get built-in loaders; Scrapy, Playwright, Crawl4AI,
Firecrawl, httpx, and Node crawlers get drop-in guards. The
integration guide states what each one checks,
including where robots.txt can and cannot be enforced. The offline path
(PoisonDetector.analyze) works on any bytes you already have, with no
network I/O added to the critical path.
Apotropic curates the signature list and ships updates with each release. Every signature is verified against the tool or source it targets and regression-tested against ordinary pages that share its words, so a page about a plant called Nepenthes is not treated as a tarpit. Every result records the signature list version it was checked against. You can add your own list in YAML.
No. The HTTP API runs on your infrastructure and binds to loopback (127.0.0.1:8765) by default, so pages and results never leave your machines. Binding to another interface requires --allow-public and a CRAWLSIGN_API_TOKEN. Every endpoint except /health then requires that bearer token.
No. The product is designed for crawler self-protection and responsible ingestion. It stops crawlers from ingesting harmful content and respects refusal signals by default.
The live URL endpoint blocks localhost, private IP ranges, link-local hosts, multicast, and internal schemes by default. The offline analyzer accepts any bytes you supply.
The HTTP API reference covers every endpoint and schema. A running server also serves its OpenAPI schema at /openapi.json and interactive docs at /docs.
CrawlSign is licensed to design partners during the pre-release. Request access and tell us what you crawl and where the pages end up; we will set up a pilot that starts in monitor mode.
Yes. In monitor mode nothing is withheld on risk evidence alone: those pages are logged with what enforcement would have done. Refusals such as robots.txt and noai are still honored, because ignoring them is never part of a trial.
CrawlSign is licensed to a small group of design partners. Tell us what you crawl and where the pages end up, and we will set up a pilot that starts in monitor mode.
adi@apotropic.com