---
title: "The sites robots.txt (<robots.txt-url>), specifies that autonomous fetching of this page is not allowed (fix)"
canonical_url: "https://superbot.gg/fix/mcp-fetch-robots-txt-autonomous-fetching-not-allowed"
summary: "The MCP fetch server refused a page because of robots.txt. What it checks, the server flag that turns the check off where the site allows it, and what superbot does."
---

> Markdown of https://superbot.gg/fix/mcp-fetch-robots-txt-autonomous-fetching-not-allowed. Site index: /llms.txt.

MCP fetch server error

# The sites robots.txt (<robots.txt-url>), specifies that autonomous fetching of this page is not allowed

The server raises this as an MCP error. It goes on with the user agent, the url and the whole robots.txt, then tells the assistant to let you know.

The site's robots.txt disallows the fetch server.
The server stopped before it read the page.

[Get the app](https://superbot.gg/download?cta%5Fid=page%5Finstall) [other errors](https://superbot.gg/fix)

## What it means

The fetch server downloads the site's robots.txt before it fetches a page, and checks it with its own user agent, `ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)`.

If the file disallows that agent for the page, the server raises this error and does not fetch.

It checks only when the request comes from the model. A request you start yourself with the fetch prompt in your client is not checked.

If robots.txt itself answers 401 or 403, a sibling error says the server assumes autonomous fetching is not allowed.

## Fix it

1. Read the robots.txt the message prints. A `Disallow` line that covers your path, for your agent or for all agents, is the cause.
2. If the site allows it, as your own site does, add `--ignore-robots-txt` to the server's `args`: `"args": ["mcp-server-fetch", "--ignore-robots-txt"]`.
3. Or fetch that one page yourself with the fetch prompt in your client. It is a user-initiated request, so the server does not check it.
4. If the site does not want automated readers, leave it. robots.txt is the site's own rule.

## With superbot

superbot is another way to read a page, not a fix for this server's check.

superbot's read of a page you name does not fetch robots.txt first.
It reads the page you pointed at, the way a browser follows a link.

A crawl does honour robots.txt on the links it finds.
The page it starts from is exempt.

Whether you may read the page is your call, so check the site's terms first.

ask

Read https://httpbin.org/deny and tell me what the page says. The robots.txt of that site disallows /deny, so also tell me whether superbot checked robots.txt before reading it.

reply

```
Short answer on robots: no, it wasn't checked before the read. My fetch tool sent a plain HTTP request straight to /deny and never looked at robots.txt. There is a crawl tool that reads and honours robots.txt, but it is for walking sites, not for reading one page, so it wasn't in play here.

The site's robots.txt, which I fetched separately and only after the page, says:

User-agent: *
Disallow: /deny
```

A real answer on 2026-10-18, trimmed to the robots answer. The test site disallows that path on purpose. The opening line and the page's own text are left out.

This needs an account.
An agent with none signs you in through [/auth.md](https://superbot.gg/auth.md).
You read back one emailed code.

## Related

- [spawn npx ENOENT](https://superbot.gg/fix/spawn-npx-enoent)
- [✘ Failed to connect](https://superbot.gg/fix/mcp-failed-to-connect)
- [Read a web page as clean text](https://superbot.gg/use-cases/read-a-web-page-as-clean-text)

Source: [modelcontextprotocol/servers: src/fetch server.py](https://github.com/modelcontextprotocol/servers/blob/main/src/fetch/src/mcp%5Fserver%5Ffetch/server.py) and its [README](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch). Read on 2026-10-18.