Skip to main content
Troubleshooting · files

Use when

A dry run or a crawl keeps far fewer pages than the person expects from the site.

Looking for something else?

A page that the site’s rules keep out (problem The site’s rules keep a page out). A page the project already holds (problem The project already holds the page).

​
Signs

  • The dry run, or the dry run in a crawl_needs_preview refusal, says found: 12 pages for a site that has hundreds.
  • The sample lists pages from one section only.
  • The left out: line shows a large count outside the address you gave, or not web pages, or by your patterns.
  • A page that the person named is not in the sample.

​
Why it happens

Penumbra finds the pages of a site from its sitemap and from the pages it can reach. It does not follow links to a set depth. A site without a sitemap, or a site whose pages appear only after a script runs, can show fewer pages than it has. Your options narrow the result too. A crawl keeps pages on the host of the start address, and under its path, unless --include names others. Subdomains need --include-subdomains. A crawl also leaves out addresses that are not web pages, such as scripts, styles, pictures, fonts and data files. The --limit cuts the list last.

​
What Penumbra does

By itself:
  • Finds the pages of a site from its sitemap and from the pages it can reach.
  • Keeps only pages on the host of the start address, and on its subdomains only with --include-subdomains.
  • Keeps only pages under the path of the start address, unless --include names others.
  • Leaves out addresses that are not web pages (scripts, styles, pictures, fonts, audio, video, archives and data files). It keeps PDF files.
  • Applies --include and --exclude, drops pages that the site’s rules disallow and pages that the project holds, and then cuts to --limit.
  • Reports the found, kept and left-out counts for every dry run and every crawl.
You must:
  • Read the counts and the sample before you run a crawl.
Not handled:
  • Penumbra does not follow links to a set depth, so it cannot find a page that the sitemap and the reachable pages do not show.

​
Do

  1. Read the dry run again: found, kept, the three left-out counts and the sample.
  2. If the first left out count is large, read what it names. When it names your patterns, loosen --include or drop --exclude to what the person asked for. Run the dry run again. When it names the address you gave, see steps 3 and 4.
  3. If the start address is a deep page, start from the index page of the section, or from the root of the site with --include.
  4. If the docs are on a subdomain, add --include-subdomains.
  5. If a short list of pages is missing, save them by address with penumbra web save.
  6. Tell the person the kept count and that it is lower than the site’s size.

​
Don’t

  • Don’t raise --limit to see more pages. The limit only cuts the list, and it never adds pages that were not found.
  • Don’t widen the patterns beyond the request to reach a count.
  • Don’t run the crawl and hope that it finds more than the dry run showed.

​
Check

  • The new dry run shows the pages the person expects in its sample.
  • The person knows the kept count, and that the site may have more pages.
Cards file and website. Recipe bring-in-a-website. Problems page-blocked-by-robots and page-already-in-project.

​
Keep going

File · Website · Save web pages or crawl a website into the project · The site’s rules keep a page out · The project already holds the page