> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pnbr.io/llms.txt
> Use this file to discover all available pages before exploring further.

# The site's rules keep a page out

> A site can publish rules that keep automated readers away from some pages, so Penumbra refuses those pages and captures nothing for them.

Troubleshooting · files

> **Use when**
>
> A save refuses a page with `url_blocked_by_robots`, or a dry run or a crawl counts pages left out by the site's rules.

> **Looking for something else?**
>
> An address that is not public, which is refused with `url_not_allowed`. A page that did not load. A page the project already holds (problem [The project already holds the page](https://docs.pnbr.io/troubleshooting/page-already-in-project.md)).

## Signs

* `penumbra web save` prints `refused` for a page, with `url_blocked_by_robots`, and exits with code 3. The other pages of the same save land.
* A dry run of `penumbra web crawl` shows a count on the `left out:` line before "by the site's rules".
* `penumbra web crawls get <crawl>` shows the same count on its `found:` line.

## Why it happens

A site publishes a file of rules, called robots.txt, that says which pages
automated readers may take. Penumbra reads the rules of a site before it
captures a page, for every save and every crawl. A page that the rules
disallow is not captured.

The rules belong to the site, not to the person who asked or to Penumbra. A
different command, option or address of the same page does not change them.

## What Penumbra does

By itself:

* Reads the rules of a site before it captures any page, for a save and for a crawl.
* Refuses a page that the rules disallow with `url_blocked_by_robots`, and captures nothing for it.
* Charges nothing for a page that it refuses.
* Counts a disallowed page of a crawl as left out by the site's rules, and does not capture it.
* Keeps the other pages of a save when one page is refused.
* Allows every page of a site that answers that it has no rules.
* Keeps the rules of a site for 24 hours, so a later request in that time makes no new read.
* Treats every page of a site as disallowed for one run when the rules cannot be read because the site errs or cannot be reached.

You must:

* Tell the person which pages the site keeps out.
* Name each refused page in your report.

Not handled:

* No option overrides the rules of a site.

## Do

1. Read the output. List the addresses that show `url_blocked_by_robots`.
2. If every page of a site is refused, the site may have been unreachable when
   Penumbra read its rules. Run the command again later with a new
   `--request-id`, because the same id returns the same answer. Run it once, not
   in a loop.
3. Tell the person which pages the site keeps out. Ask if they need them. The
   site's owner can allow them. If the person holds a copy of the page that they
   may use, upload it with `penumbra files upload`.
   If the owner changes the rules, run the command again with a new
   `--request-id`. Penumbra keeps the rules of a site for 24 hours, so a retry in
   that time can give the same answer.
4. Go on with the pages that landed. Name the refused pages in your report.

## Don't

* Don't run the command again, with the same or a new `--request-id`, to get a
  page the rules still disallow. The same id returns the same answer, and a new
  one meets the same rules.
* Don't save the same page from another address of the site to get around the
  rules.
* Don't tell the person that the page is in the project. No File exists for it.

## Check

* The refused pages show `refused` in the output of `penumbra web save`, and
  no File exists for them.
* The person knows which pages the site keeps out.

## Related

Card [file](https://docs.pnbr.io/concepts/file.md). Recipe [bring-in-a-website](https://docs.pnbr.io/guides/bring-in-a-website.md). Problems [crawl-found-fewer-pages](https://docs.pnbr.io/troubleshooting/crawl-found-fewer-pages.md) and
[page-already-in-project](https://docs.pnbr.io/troubleshooting/page-already-in-project.md).

## Keep going

[File](https://docs.pnbr.io/concepts/file.md) · [Website](https://docs.pnbr.io/concepts/website.md) · [Save web pages or crawl a website into the project](https://docs.pnbr.io/guides/bring-in-a-website.md) · [A crawl keeps fewer pages than the site has](https://docs.pnbr.io/troubleshooting/crawl-found-fewer-pages.md) · [The project already holds the page](https://docs.pnbr.io/troubleshooting/page-already-in-project.md)

Penumbra Grammar 0.6.26


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.