skip to content
superbot.gg

Crawl a site and list its URLs

Point your agent at a site.
It lists the URLs it finds.

Get the app see the real call

What you get

Ask your agent

Connect superbot once.
Other clients are on the install page.

claude mcp add --transport http superbot https://api.superbot.gg/mcp -s user

With no browser, post to the install route instead.

curl -fsS -X POST https://api.superbot.gg/mcp/install -d client=claude-code

Then ask in plain words.

superbot({"intent": "Map https://modelcontextprotocol.io and list the URLs you find."})

This needs an account.
An agent with none signs you in through /auth.md.
You read back one emailed code.

One real call

A real call, run on 2026-10-18 from a test account.
The reply is the agent's own, with its heading and sources line cut.

call
superbot({"intent": "Map the site https://quotes.toscrape.com with a site map and list the first ten URLs it returns."})
reply
No sitemap exists there (robots.txt and sitemap.xml both returned 404), so the map came from harvesting links on the homepage: **47 URLs found** on [quotes.toscrape.com](https://quotes.toscrape.com).

The first ten, in the order returned:

1. https://quotes.toscrape.com/
2. https://quotes.toscrape.com/login
3. https://quotes.toscrape.com/author/Albert-Einstein
4. https://quotes.toscrape.com/tag/change/page/1/
5. https://quotes.toscrape.com/tag/deep-thoughts/page/1/
6. https://quotes.toscrape.com/tag/thinking/page/1/
7. https://quotes.toscrape.com/tag/world/page/1/
8. https://quotes.toscrape.com/author/J-K-Rowling
9. https://quotes.toscrape.com/tag/abilities/page/1/
10. https://quotes.toscrape.com/tag/choices/page/1/

The rest are the same three shapes: author pages, tag pages, and `page/2/` pagination, plus ten bare tag roots (`/tag/love/`, `/tag/reading/`, and so on) closing the list.
A site mapped, run live. Ten of the URLs it found.

Limits

Pages
Public pages only.
Crawl
Walking a few same-site pages is a separate ask. A crawl is bounded and honours robots.txt on discovered links. The call above used the site map and did not run one.
Needs
A superbot account.

The site mapped is a public scraping sandbox. Read on 2026-10-18.