SimpleSmartHeadCrawler

A change-detection crawler of TheCrawlerCompany.

SimpleSmartHeadCrawler checks public web pages of Dutch financial service providers (advisers, insurance brokers and similar firms) for changes. It does not log in, does not submit forms and does not collect anything behind a login or paywall.

Nederlands: SimpleSmartHeadCrawler controleert openbare pagina's van Nederlandse financiële dienstverleners op wijzigingen. Hij volgt robots.txt, vraagt pagina's zo zuinig mogelijk op en is hieronder te blokkeren.

How it identifies itself

User-Agent:

SimpleSmartHeadCrawler/0.1 (TheCrawlerCompany)

Requests only come from the addresses listed below. Each has a reverse DNS name under thesimplecrawlercompany.org that resolves back to the same address (forward-confirmed reverse DNS), so a request claiming to be this crawler can be verified.

How it behaves

• It honours robots.txt, including Crawl-delay.
• It uses conditional requests (If-None-Match, If-Modified-Since): when a page has not changed, your server answers 304 Not Modified and sends no page.
• At most one request every few seconds per site, over a single connection, and a limit per server so that sites sharing a hosting server are not visited at the same time.
• It backs off on 429 (honouring Retry-After), on 503 and on any block or challenge, and does not retry from another address.

How to block or slow it down

Add to your robots.txt:

User-agent: SimpleSmartHeadCrawler
Disallow: /

or, to slow it down instead:

User-agent: SimpleSmartHeadCrawler
Crawl-delay: 30

Changes to robots.txt are picked up on the next visit.

Addresses

Machine-readable list: /bot/ips.json

Any change to these addresses is published here and in the list before the new address sends a request.

Contact

Questions, complaints or a request to be excluded: the contact address of TheCrawlerCompany is added here shortly. Until then, robots.txt above takes effect on the next visit.