Skip to main content
Troubleshooting · files

Use when

A save prints already for a page, or a crawl counts pages as already in the project.

Looking for something else?

Identical bytes uploaded from a computer (problem Identical bytes make no new version). A page that was refused or failed.

​
Signs

  • penumbra web save prints already and already in the project for a page, with the id of a File. When a capture of the page is still running, it prints already being saved and the id of that operation instead.
  • With --json, the page has the status already_in_project.
  • A dry run or a crawl shows a count on the left out: line before “already in the project”.
  • No new File appears in penumbra files list.

​
Why it happens

Penumbra identifies a page by its web address. It compares addresses after it lowercases the host and reads http and https as one. It also drops the fragment, a trailing slash and tracking parameters such as utm_source. Two addresses that differ only in those parts are the same page. When a live File, or a capture that has not finished, holds the address in the project, Penumbra reports the page and captures nothing. A changed page is not captured again.

​
What Penumbra does

By itself:
  • Compares addresses after it lowercases the host, reads http and https as one, and drops the fragment, a trailing slash and tracking parameters.
  • Reports a page that a live File or an unfinished capture holds as already_in_project, with the id of the File or of the operation.
  • Does not capture the page again, makes no new version and charges nothing for it.
  • Counts such a page of a crawl as already in the project and leaves it out.
  • Keeps the other pages of the save.
You must:
  • Tell the person which pages the project already held, with their Files.
Not handled:
  • No command refreshes a captured page from the web.

​
Do

  1. Read the File id in the output. Read the File: penumbra files show <file>. If the output says already being saved, no File exists yet. Follow the operation: penumbra files wait <operation-id>.
  2. Tell the person that the page was already in the project, and give the File.
  3. If the person needs the page as it is now, tell them that no command refreshes a page. The File shows the page as it was when it was saved (card rule website.5).
  4. Go on with the pages that landed.

​
Don’t

  • Don’t save the page again, or change its address, to force a new File.
  • Don’t tell the person that the save added the page. No new File exists.

​
Check

  • penumbra files show <file> shows the File from the output and its web address.
  • The person knows which pages were already in the project.
Cards file and website. Recipe bring-in-a-website. Problems same-bytes-no-new-version and page-blocked-by-robots.

​
Keep going

File · Website · Save web pages or crawl a website into the project · Identical bytes make no new version · The site’s rules keep a page out