---
title: "Crawl a site and list its URLs, from an AI agent – superbot.gg"
canonical_url: "https://superbot.gg/use-cases/crawl-a-site-and-list-its-urls"
summary: "Your agent maps a site and lists the URLs it finds, with a count."
---

> Markdown of https://superbot.gg/use-cases/crawl-a-site-and-list-its-urls. Site index: /llms.txt.

# Crawl a site and list its URLs

Point your agent at a site.
It lists the URLs it finds.

[Get the app](https://superbot.gg/download?cta%5Fid=page%5Finstall) [see the real call](https://superbot.gg/use-cases/crawl-a-site-and-list-its-urls#call)

## What you get

- A list of a site's URLs, with a count of how many it found.
- When a site has no sitemap, the list comes from the links on its home page.

## Ask your agent

Connect superbot once.
Other clients are on [the install page](https://superbot.gg/add).

```sh
claude mcp add --transport http superbot https://api.superbot.gg/mcp -s user
```

With no browser, post to the install route instead.

```sh
curl -fsS -X POST https://api.superbot.gg/mcp/install -d client=claude-code
```

Then ask in plain words.

```
superbot({"intent": "Map https://modelcontextprotocol.io and list the URLs you find."})
```

This needs an account.
An agent with none signs you in through [/auth.md](https://superbot.gg/auth.md).
You read back one emailed code.

## One real call

A real call, run on 2026-10-18 from a test account.
The reply is the agent's own, with its heading and sources line cut.

call

```
superbot({"intent": "Map the site https://quotes.toscrape.com with a site map and list the first ten URLs it returns."})
```

reply

```
No sitemap exists there (robots.txt and sitemap.xml both returned 404), so the map came from harvesting links on the homepage: **47 URLs found** on [quotes.toscrape.com](https://quotes.toscrape.com).

The first ten, in the order returned:

1. https://quotes.toscrape.com/
2. https://quotes.toscrape.com/login
3. https://quotes.toscrape.com/author/Albert-Einstein
4. https://quotes.toscrape.com/tag/change/page/1/
5. https://quotes.toscrape.com/tag/deep-thoughts/page/1/
6. https://quotes.toscrape.com/tag/thinking/page/1/
7. https://quotes.toscrape.com/tag/world/page/1/
8. https://quotes.toscrape.com/author/J-K-Rowling
9. https://quotes.toscrape.com/tag/abilities/page/1/
10. https://quotes.toscrape.com/tag/choices/page/1/

The rest are the same three shapes: author pages, tag pages, and `page/2/` pagination, plus ten bare tag roots (`/tag/love/`, `/tag/reading/`, and so on) closing the list.
```

A site mapped, run live. Ten of the URLs it found.

## Limits

Pages

Public pages only.

Crawl

Walking a few same-site pages is a separate ask. A crawl is bounded and honours robots.txt on discovered links. The call above used the site map and did not run one.

Needs

A superbot account.

## Related

- [Read a web page as clean text](https://superbot.gg/use-cases/read-a-web-page-as-clean-text)
- [Fetch a page that needs JavaScript](https://superbot.gg/use-cases/read-a-page-that-needs-javascript)
- [Web search for an agent, with no API key](https://superbot.gg/use-cases/web-search-for-agents-without-an-api-key)
- [All use cases](https://superbot.gg/use-cases)

The site mapped is a public scraping sandbox. Read on 2026-10-18.