What this bot does
It checks whether a website is readable by search engines and AI search bots, using public information only. It looks at robots.txt, sitemaps and public page HTML — the same things anyone can open in a browser.
How it identifies itself
Every request carries the User-Agent below, verbatim. It never pretends to be a browser.
Full User-Agent
ConnectAI-VisibilityBot/1.0 (+https://connectai.kr/bot)robots.txt product token
ConnectAI-VisibilityBotThis is the name to use when you address us in robots.txt. It contains no digits, per the product-token grammar in RFC 9309 §2.2.1.
How to block it
Add these two lines to your robots.txt. From the next run onward we will not request anything from this domain.
User-agent: ConnectAI-VisibilityBot
Disallow: /The name is matched case-insensitively. To block only part of the site, put that path in place of `Disallow: /`.
Blocking us costs you nothing. Your domain is not scored down and does not stay on any list — we fetch none of your pages. To know that you blocked us we still have to read robots.txt itself, so that one request and a DNS lookup for your domain name will continue.
What it fetches, and why
This is the complete list. Each item is the evidence for a specific check.
- robots.txt
- To see what crawlers are allowed or disallowed, and whether a sitemap is declared.
- sitemap.xml and any child sitemaps it points to
- To see which pages the site advertises to search engines. This list is the denominator of the check.
- /llms.txt
- To see whether a summary is offered to AI systems, and whether the site controls that wording.
- The HTML and response headers of the public pages listed in the sitemap
- To read the machine-facing markup: title, description, canonical, structured data, hreflang, image alt text.
- The home URL response and redirects, plus one request to a non-existent path
- To check the http→https redirect, apex/www consolidation, and whether a missing path returns 404.
- Public DNS records for the domain (A, AAAA, TXT)
- To see whether search-tool ownership verification is done at the DNS level.
What it never does
- It does not access anything behind a login. It does not create accounts or attempt to sign in.
- It does not submit forms. It sends no request that changes state — no enquiries, sign-ups or carts.
- It does not store or send back cookies.
- It does not send a Referer header.
- It does not use the fetched content to train AI models.
- It does not sell or redistribute the fetched content to third parties.
- It does not probe for vulnerabilities. It issues ordinary GET requests to public URLs — if a connection fails, drops or times out it retries once, and no more. It does not re-request error responses (the only exception is a 5xx from robots.txt).
How it follows robots.txt
- It follows RFC 9309 (Robots Exclusion Protocol).
- It looks for a group naming our product token first, and falls back to the `User-agent: *` group.
- If `Crawl-delay` is larger than our default interval, it slows down to match — up to 30 seconds per request. If you declare a larger value, we wait 30 seconds.
- If robots.txt returns 5xx, it checks once more after a short wait; if it still returns 5xx, it does **not crawl the site at all.** The specification reads that as a full disallow, so when we cannot read the rules we do not read the site.
- If it receives `429 Too Many Requests` even once, it stops the whole run immediately, records `Retry-After`, and stays off that domain for 24 hours.
- If we are disallowed, we do not check the site. The only exception is a site operator giving written permission for their own domain.
Request rate and limits
These are the upper bounds one check can place on a single domain.
| Concurrent requests (per domain) | 1 |
|---|---|
| Minimum interval between requests | 1000 ms |
| Requests per run | 320 |
| Download per run | 200 MB |
| Wall-clock per run | 30 min |
| Pages read per run | 300 |
| Cooldown before re-checking a domain | 24 h |
If `Crawl-delay` is declared, the minimum interval rises to match it, up to 30 seconds. If your last response was slow, the bot widens the interval on its own.
What is stored, and how it is deleted
- Response status codes and response headers
- The raw HTML of the pages we fetched
- The facts extracted from that HTML (title, description, types of structured data, and so on) and the resulting verdicts
The retention period for raw HTML is not settled yet. Once it is, the number of days will be printed on this page. In the meantime, ask us at the address below and we will delete it.
Send deletion requests to the address below. We confirm ownership of the domain and then delete what we fetched from it.
Contact
Questions about a request from this bot, block requests and deletion requests all go to the address below. It is a role address, not an individual.
Contact address support@connectai.kr
The values on this page are bound in code to the constants the crawler actually uses. If what you observe differs from what is written here, we have broken our promise — please tell us at the address above.
Last updated · 2026-09-15