skip to content
superbot.gg

Python requests error

403 Client Error: Forbidden for url

raise_for_status() raises this when a site answers 403. The url after it is the page you asked for.

The site answered 403.
It will not serve your request.

Get the app other errors

What it means

403 means the server understood the request and refuses it. In a scraper the usual cause is that the site spotted a plain HTTP client: the default python-requests User-Agent, no cookies, no JavaScript.

A sign-in wall, a region block or a bot challenge sends the same status. Print res.text before you call raise_for_status(). A Cloudflare ray ID or a CAPTCHA line points to a bot wall. A login form points to a sign-in wall.

Fix it

  1. Read the body of the 403 first, as above. The fix depends on which wall it is.
  2. Send a browser's User-Agent. On the Stack Overflow thread, headers = {'User-Agent': 'Mozilla/5.0'} fixed it for the site in question.
  3. If the wall comes back after many requests, slow down and reuse one requests.Session() so the cookies carry over.
  4. If the page needs JavaScript or serves a challenge, a plain HTTP client cannot read it. Use a real browser such as Playwright or Selenium, or read the same facts from another source.
  5. Check the site's terms and its robots.txt, and use its API or feed when it has one.

With superbot

superbot's read tool names a refusal instead of raising. A page that answers 403 comes back as page_refused, with this message:

page returned 403: the site refused to serve this page to a reader (a bot, sign-in or region wall). Read the same facts from another source, or open the page in a browser.

superbot can open a page that refused a plain fetch in a real browser. A browser can be refused too. Then the answer says why, and the next step is another source for the same facts.

ask

Read https://httpbin.org/status/403 and tell me why it did not open.

reply
It opened fine: httpbin.org/status/403 answers 403 by design, with an empty body.

…

That endpoint exists to echo whatever code you put in the path, so a request for 403 gets a 403 and nothing else.
A real answer on 2026-10-18, trimmed. The test address returns 403 on purpose, so the reply reads the status and says what it means.

This needs an account.
An agent with none signs you in through /auth.md.
You read back one emailed code.

Source: Stack Overflow: Python scraping HTTPError: 403 Client Error: Forbidden for url. Read on 2026-10-18.