We are counting the websites of businesses.
Nobody publishes a full count of them. Not the online maps, not the business lists people sell. So we are building one. Our crawler reads each website the same way, adding a few checks for some kinds of business and for Florida, and we publish what we find.
Did our crawler visit your site? Here is what it did, and how to stop it.
One reading from each of eight recent days
We never show which business a reading belongs to. A reading appears here only when we have read at least 30 sites of its kind in its state.
| Day (UTC, the world standard clock) | Kind of business | State | What happened |
|---|---|---|---|
| Oct 16 | Restaurant | California | graded C |
| Oct 15 | Business | Massachusetts | graded C |
| Oct 14 | Business | Ohio | graded C |
| Oct 13 | Business | Rhode Island | graded B |
| Oct 12 | Salon or spa | Pennsylvania | graded C |
| Oct 11 | Restaurant | New Jersey | graded B |
| Oct 10 | Business | Connecticut | graded C |
| Oct 9 | Law practice | Florida | graded D |
What the crawler does on a website
- 9requests at most, plus one each time the site forwards us to another address. Each gets one page or file: robots.txt first, the file where a site says which crawlers are welcome. Then come the home page and small files like the sitemap (a list of the site's pages) and the site icon.
- 12seconds, and it gives up on a page. It reads no more than 3 MB of it.
- 2seconds at least between the starts of two visits to the same website, longer if its robots.txt asks for a slower pace. A visit is the requests above.
- Nomeans no. If the robots.txt at the address we were given tells every crawler, or ours by name, to stay out, it stays out.
- NameIt names itself in every request, and that name links to a page that explains it.
- 0requests so far to put a site on our do-not-measure list. Anyone can ask, and neither the census nor the free inspection reads a listed site after that.
What the count shows
83 The middle website we measured scores 83 out of 100 on our checks.
8% get a D or an F. Most land in the B and C range.
36% of the 78,807 sites this count graded have no single clear main headline. On the benchmark list, a different and much smaller group of 87 graded sites, the share without one is about the same: 33%.
Read again, not just once
A website that answers us is not done. It joins the line to be read again. Of the 85,095 that have given us a page, 82,454 have been read on two or more different days, and nine in ten were looked at within the last 6 days.
That is more than the 78,807 in the count, which leaves out:
- 4,160 chains and franchises
- 470 sites outside the United States
- 1,658 addresses a later reading found were no longer the business, 8 of them outside the United States
One reading is a photograph. Reading the same sites again is what will show what changed, and a reading nobody took last month cannot be taken later.