Q4RLBot#
Q4RLBot is a web crawler operated by Q4 Reach Labs GmbH, Munich, Germany.
Purpose#
It reads publicly available pages of online shops for market research. It does not log in, fill in forms or place orders.
How to recognise it#
- The User-Agent contains
Q4RLBotand a link to this page. - Requests are signed with Web Bot Auth (HTTP Message Signatures).
The public key is published at
/.well-known/http-message-signatures-directoryon this site.
robots.txt#
Q4RLBot follows robots.txt, including Crawl-delay.
It obeys the rules for Q4RLBot and, where there are none, the rules for *.
To keep it out of a site:
User-agent: Q4RLBot
Disallow: /Request rate#
Without a Crawl-delay, it sends at most one request every 5 seconds to a site.
When a server answers with HTTP 429, it pauses its requests to that site.