Skip to main content
Troubleshooting · files

Use when

A save refuses a page with url_blocked_by_robots, or a dry run or a crawl counts pages left out by the site’s rules.

Looking for something else?

An address that is not public, which is refused with url_not_allowed. A page that did not load. A page the project already holds (problem The project already holds the page).

​
Signs

  • penumbra web save prints refused for a page, with url_blocked_by_robots, and exits with code 3. The other pages of the same save land.
  • A dry run of penumbra web crawl shows a count on the left out: line before “by the site’s rules”.
  • penumbra web crawls get <crawl> shows the same count on its found: line.

​
Why it happens

A site publishes a file of rules, called robots.txt, that says which pages automated readers may take. Penumbra reads the rules of a site before it captures a page, for every save and every crawl. A page that the rules disallow is not captured. The rules belong to the site, not to the person who asked or to Penumbra. A different command, option or address of the same page does not change them.

​
What Penumbra does

By itself:
  • Reads the rules of a site before it captures any page, for a save and for a crawl.
  • Refuses a page that the rules disallow with url_blocked_by_robots, and captures nothing for it.
  • Charges nothing for a page that it refuses.
  • Counts a disallowed page of a crawl as left out by the site’s rules, and does not capture it.
  • Keeps the other pages of a save when one page is refused.
  • Allows every page of a site that answers that it has no rules.
  • Keeps the rules of a site for 24 hours, so a later request in that time makes no new read.
  • Treats every page of a site as disallowed for one run when the rules cannot be read because the site errs or cannot be reached.
You must:
  • Tell the person which pages the site keeps out.
  • Name each refused page in your report.
Not handled:
  • No option overrides the rules of a site.

​
Do

  1. Read the output. List the addresses that show url_blocked_by_robots.
  2. If every page of a site is refused, the site may have been unreachable when Penumbra read its rules. Run the command again later with a new --request-id, because the same id returns the same answer. Run it once, not in a loop.
  3. Tell the person which pages the site keeps out. Ask if they need them. The site’s owner can allow them. If the person holds a copy of the page that they may use, upload it with penumbra files upload. If the owner changes the rules, run the command again with a new --request-id. Penumbra keeps the rules of a site for 24 hours, so a retry in that time can give the same answer.
  4. Go on with the pages that landed. Name the refused pages in your report.

​
Don’t

  • Don’t run the command again, with the same or a new --request-id, to get a page the rules still disallow. The same id returns the same answer, and a new one meets the same rules.
  • Don’t save the same page from another address of the site to get around the rules.
  • Don’t tell the person that the page is in the project. No File exists for it.

​
Check

  • The refused pages show refused in the output of penumbra web save, and no File exists for them.
  • The person knows which pages the site keeps out.
Card file. Recipe bring-in-a-website. Problems crawl-found-fewer-pages and page-already-in-project.

​
Keep going

File · Website · Save web pages or crawl a website into the project · A crawl keeps fewer pages than the site has · The project already holds the page