FrontlatchBot

The crawler behind the business index.

What it sends, how it reads robots.txt, how fast it asks, and how to keep your site out.

Last updated 8 October 2026.

01User agents

The exact strings we send.

The crawler that builds the business index:

FrontlatchBot/0.1 (builds the Sydney agent-action index, read-only; +https://frontlatch.com/docs/agents)

The jobs that seed the index from public datasets:

FrontlatchBot/0.1 (+https://frontlatch.com/bot; index seed from OpenStreetMap)
FrontlatchBot/0.1 (+https://frontlatch.com/bot; Common Crawl host graph)
FrontlatchBot/0.1 (+https://frontlatch.com/bot; AU index ingest)

The scanner, when someone asks for a scan or an AI assistant reads a page through the gateway, and the browser that maps a site’s forms (it never submits them):

FrontlatchScanner/0.1 (read-only agent-readiness scan; +contact: <address>)
<browser user agent> FrontlatchMapper/0.1 (renders pages, fills forms to trigger client-side validation, NEVER submits; +contact: <address>)

02robots.txt

We read robots.txt before every site.

Each agent looks for a group naming it (FrontlatchBot, FrontlatchScanner or FrontlatchMapper) and otherwise follows the * group. A group named Frontlatch applies to all three. The longest matching rule wins, and an Allow beats a Disallow of the same length. If robots.txt disallows a page, the crawler does not fetch it. Crawl-delay is honoured.

03Crawl rate

How fast we ask any one host.

  • FrontlatchBot: at most one request per second to a host, or slower if your Crawl-delay says so, and at most 8 pages of a business’s site per visit.
  • FrontlatchScanner: at least half a second between requests to a host, or your Crawl-delay if longer, and at most 15 pages per scan.
  • FrontlatchMapper: at least 2.5 seconds between page loads on a host.
  • Every agent is read-only. None of them signs in, buys, or submits a form.

04Opting out

How to keep a site out.

To stop all three agents, add this to your site’s robots.txt:

User-agent: Frontlatch
Disallow: /

To remove a listing that is already in the index, use the remove link on the business’s own page in the business index: we email a code to an address at that domain, and once it is entered the listing is hidden and never added again. Or write to [email protected] with the domain.