CrawlSign docs

Results and monitor mode

Every interface returns the same result: the CLI prints it, the SDK returns it as a DetectionResult (or a dict from SafeFetcher.fetch), and the HTTP API nests it under result.

The result

Field Type Meaning
url string The analyzed URL.
action string continue, log, quarantine or stop. The field to gate on.
should_stop boolean The crawl decision: stop requesting this URL, path or domain.
score integer The summed evidence score behind the action.
reasons list of strings Stable reason codes, sorted.
threshold integer The configured stop threshold.
metadata object Evidence details; see below. Open-ended: new keys may be added.

Actions

Action Ingest? Meaning
continue Yes No risk signal worth recording.
log Yes A soft signal, recorded in reasons for review.
quarantine No Risky enough to set aside for review; you may keep crawling the host.
stop No Refused or high-confidence risk: stop requesting the URL, path or domain.

The action comes from the score and the configured thresholds. With the defaults, an explicit AI refusal (noai, noimageai) or a known poison source stops on its own, a known tarpit endpoint quarantines, and circumstantial evidence such as hidden links needs corroboration before it does more than log. Known-signature matches count once, at the highest matching score, so overlapping signatures never add up.

Stop scope

A stop result carries metadata.stop_scope:

  • url_or_path: stop this URL or path.
  • domain: the same fetcher or detector has seen at least two stop-worthy URLs on this host at or above the domain threshold. Stop crawling the whole host. A single page never stops a domain.

Metadata keys

Key Type Produced by
signature_list_version integer Every analyzed page: the signature list in use.
matched_signatures list Known signatures: id, type, reason and score of each match.
hidden_link_count integer Hidden-link detector.
hidden_internal_link_count integer Hidden-link detector: hidden links to the same host.
hidden_internal_link_targets list Same-host hidden link targets, to drop from your crawl frontier.
stop_scope string url_or_path or domain, on stop.
ignored_refusal_signals list Refusal signals seen while a respect setting was turned off.
redirect_count, final_url, cross_domain_redirect, redirect_chain mixed Fetch-time checks only.
content_quarantined boolean Fetch time: the page was written to the quarantine folder.
monitor_action, monitor_stop_scope string Monitor mode only; see below.

ignored_refusal_signals is empty unless an operator turned a respect setting off. Any entry means the crawl chose to ignore a refusal, and is kept in the result so it can be reviewed.

Fetch results

SafeFetcher.fetch(url) wraps the result with what happened on the wire:

Key Meaning
safe true only for continue and log.
content The decoded HTML when safe, otherwise null, so withheld pages cannot be ingested by accident.
status_code HTTP status, or null when no request was made (for example a robots.txt disallow).
quarantine_path Where the quarantined page and its evidence were written, if anywhere.
result The result described above.

Monitor mode

Set mode: monitor in your configuration to run CrawlSign on real crawls without it withholding anything on risk evidence alone. Use it for a pilot, or before turning on enforcement for a new crawl.

  • Pages flagged only by the risk detectors (signatures, hidden links, link maze, content anomaly) come back as log and are not quarantined.
  • metadata.monitor_action records what enforce mode would have done with the same evidence, and metadata.monitor_stop_scope the scope when that would have been a stop.
  • Scores and reason codes are identical to enforce mode; only the action taken differs.
  • Refusals (robots.txt, noai, X-Robots-Tag, meta robots) and crawler-safety stops (oversized responses, redirect loops, slow-drip tarpits, budgets) are still enforced.
  • Would-be domain escalation is tracked separately, so risk evidence never widens a real stop.

The monitor report

crawlsign monitor-report turns a monitor-mode run into a report you can read and share: what enforcement would have withheld, by host and reason code, with a review column for marking false positives. It contains URLs, actions, scores and reason codes only, never page content.

crawlsign analyze-warc crawl.warc.gz --config crawlsign-monitor.yaml --output run.jsonl
crawlsign monitor-report run.jsonl --format html --output monitor-report.html

It also reads the JSON array printed by crawlsign scan-file. Results from an enforce-mode run are rejected rather than reported as "nothing found".

Quarantine records

On the fetch path, quarantine and stop pages are written to the quarantine folder (when quarantine.enabled) as the page plus a metadata.json holding the result. Each record has a result_schema_version (currently 1).