Factbot crawler policy

Factbot is the crawler FactBox uses to read public official
gazettes and turn them into clear, multilingual articles. Our goal is to
give this content more reach: each article summarises the original
document, cites its source and links back to the official gazette, so
that more people — not fewer — end up at the source.

We download only documents that are already publicly available — the
indexes and PDFs of official bulletins — and we never republish the full
text: we summarise it, add context and always point back to the original.

How we identify ourselves

Every Factbot request sends this User-Agent:

Factbot/1.0 (+https://factbox.net/bot; bot@factbox.net)

We never impersonate a browser. If you see this identifier in your
logs, the traffic is ours and you can attribute it with confidence.

What we download

• Only publicly accessible official bulletins and gazettes.
• Only the published document: the index, PDF, HTML or feed that the
  site itself offers.
• We never fill in forms, never log in and never touch private areas.
• We honour each site’s robots.txt, including its Crawl-delay
  directive.

Rate and good practice

• One active connection per host, with a default pause of at least
  one second between requests. Each source has its own limit, tuned to
  the server’s capacity: large bodies such as Spain’s BOE handle more
  traffic; small provincial bulletins handle far less.
• Conditional requests (If-Modified-Since / ETag): gazettes are
  append-only, so anything we already hold is not re-downloaded.
• Exponential backoff on 429 or 503, and we pause a host
  entirely after repeated 403s.
• Small municipal sites are visited off-peak (nights and weekends in
  their local time).

Contact

If you run a site and want to:

• confirm that some traffic is coming from Factbot,
• ask us to slow down further or exclude a path,
• or simply tell us about a structural change,

email us at bot@factbox.net. If something
breaks, we would rather hear it in an email than through a block.

Technical contact details are also in
.well-known/security.txt, and the sources we cover
are listed at /sources.