crawlerpeek

by Community · TypeScript

View source

Overview

Check whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot can access a URL: robots.txt rules per bot (RFC 9309) plus the live HTTP status fetched with each bot's user agent.

Set up

Setup

The project docs have the install command. This catalog does not yet list a package name. Most clients expect an entry shaped like this:

mcp.json
{
  "mcpServers": {
    "crawlerpeek": {
      "command": "<command from the docs>",
      "args": [
        "<args>"
      ]
    }
  }
}
Open README