skip to content
superbot.gg

MCP fetch server error

The sites robots.txt (<robots.txt-url>), specifies that autonomous fetching of this page is not allowed

The server raises this as an MCP error. It goes on with the user agent, the url and the whole robots.txt, then tells the assistant to let you know.

The site's robots.txt disallows the fetch server.
The server stopped before it read the page.

Get the app other errors

What it means

The fetch server downloads the site's robots.txt before it fetches a page, and checks it with its own user agent, ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers).

If the file disallows that agent for the page, the server raises this error and does not fetch.

It checks only when the request comes from the model. A request you start yourself with the fetch prompt in your client is not checked.

If robots.txt itself answers 401 or 403, a sibling error says the server assumes autonomous fetching is not allowed.

Fix it

  1. Read the robots.txt the message prints. A Disallow line that covers your path, for your agent or for all agents, is the cause.
  2. If the site allows it, as your own site does, add --ignore-robots-txt to the server's args: "args": ["mcp-server-fetch", "--ignore-robots-txt"].
  3. Or fetch that one page yourself with the fetch prompt in your client. It is a user-initiated request, so the server does not check it.
  4. If the site does not want automated readers, leave it. robots.txt is the site's own rule.

With superbot

superbot is another way to read a page, not a fix for this server's check.

superbot's read of a page you name does not fetch robots.txt first.
It reads the page you pointed at, the way a browser follows a link.

A crawl does honour robots.txt on the links it finds.
The page it starts from is exempt.

Whether you may read the page is your call, so check the site's terms first.

ask

Read https://httpbin.org/deny and tell me what the page says. The robots.txt of that site disallows /deny, so also tell me whether superbot checked robots.txt before reading it.

reply
Short answer on robots: no, it wasn't checked before the read. My fetch tool sent a plain HTTP request straight to /deny and never looked at robots.txt. There is a crawl tool that reads and honours robots.txt, but it is for walking sites, not for reading one page, so it wasn't in play here.

The site's robots.txt, which I fetched separately and only after the page, says:

User-agent: *
Disallow: /deny
A real answer on 2026-10-18, trimmed to the robots answer. The test site disallows that path on purpose. The opening line and the page's own text are left out.

This needs an account.
An agent with none signs you in through /auth.md.
You read back one emailed code.

Source: modelcontextprotocol/servers: src/fetch server.py and its README. Read on 2026-10-18.