Collection policy
What we read, at what rate, and the requests we decline. Published because every legal team asks first, and our own addresses depend on the answer.
This page is not boilerplate. It describes what the software actually does, and the items listed under "What we decline to do" are enforced in code rather than promised in prose. What we keep to as an operating policy, without code enforcing it, is listed separately under "Operating policy".
What we read
Public listing pages — the same pages a shopper sees without signing in. Prices, titles, availability, and the identity of the seller who published them.
What we check before reading
robots.txt, each time we load a page by URL (extraction, monitoring sweeps, and navigating with the Browser API). The checks, pacing and 429 waits in this section apply to those page loads. A path that is disallowed is not fetched. Where a site publishes a crawl-delay, that delay is the floor; where it publishes none, we choose one and it is documented per source.
When robots.txt cannot be fetched. If it answers with a 5xx, cannot be reached, or does not answer in time, we proceed as if the site published no rules (our minimum interval still applies). We remember the failure for five minutes and then try to fetch it again. A 4xx such as 404 is treated as a site that publishes no rules.
The response. When the page itself returns 403, 429 or 503, we record it as a refusal, with its date. After one or two refusals we try once more on the next cycle. A source refused three times in a row is stopped, and the customer is told once, in the next morning's digest. It stays unread unless the customer chooses to resume it. We do not look for a way around a refusal.
After a 429 (too many requests). For the time given in Retry-After (one hour if none is given, kept between 60 seconds and 24 hours), we do not request pages from that host, from any address or for any customer.
What we decline to do
These are not aspirations. Each is absent from the code, and a customer asking for one is a customer we turn down.
- No fingerprint spoofing. We do not disguise what the client is.
- No CAPTCHA bypass. A challenge is an answer, and the answer is no.
- No authentication bypass. We read what a visitor can read without an account.
- No exceeding a stated limit. Rotating addresses to get around a rate limit is evasion, not capacity, and we do not sell it.
There is a commercial reason as well as a principled one. A source that refuses us once is a source nobody gets back, and the loss falls on every customer who needed it. The incentive and the ethics point the same way.
Operating policy
Unlike the list above, the following is not enforced in code. We keep to it as an operating policy.
- No collection of personal data. We read prices and product listings. Where a seller is a named business we record the business; where a listing carries an individual's details we do not retain them. There is no automated step that tells a business from an individual or removes an individual's details.
What we record
Every observation carries the source, the listing URL, the moment it was read, the price as published, the price with consumption tax removed, and the exchange rate used with the rate's own publication date. That is what makes a figure checkable rather than merely stated.
Removal
How our crawler identifies itself (WebIntelligence) and how to stop it are on one page: about our crawler.
If you operate a site and would prefer we did not read it, say so in your robots.txt. We honour it from the next pass onward and leave the source unread.
Last updated