> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pnbr.io/llms.txt
> Use this file to discover all available pages before exploring further.

# A crawl keeps fewer pages than the site has

> A crawl keeps only the pages it finds and your options allow, so it can keep far fewer than the site has.

Troubleshooting · files

> **Use when**
>
> A dry run or a crawl keeps far fewer pages than the person expects from the site.

> **Looking for something else?**
>
> A page that the site's rules keep out (problem [The site's rules keep a page out](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md)). A page the project already holds (problem [The project already holds the page](https://docs.pnbr.io/troubleshooting/page-already-in-project.md)).

## Signs

* The dry run, or the dry run in a `crawl_needs_preview` refusal, says `found: 12 pages` for a site that has hundreds.
* The `sample` lists pages from one section only.
* The `left out:` line shows a large count outside the address you gave, or not web pages, or by your patterns.
* A page that the person named is not in the sample.

## Why it happens

Penumbra finds the pages of a site from its sitemap and from the pages it can
reach. It does not follow links to a set depth. A site without a sitemap, or a
site whose pages appear only after a script runs, can show fewer pages than it
has.

Your options narrow the result too. A crawl keeps pages on the host of the start
address, and under its path, unless `--include` names others. Subdomains need
`--include-subdomains`. A crawl also leaves out addresses that are not web pages,
such as scripts, styles, pictures, fonts and data files. The `--limit` cuts the
list last.

## What Penumbra does

By itself:

* Finds the pages of a site from its sitemap and from the pages it can reach.
* Keeps only pages on the host of the start address, and on its subdomains only with `--include-subdomains`.
* Keeps only pages under the path of the start address, unless `--include` names others.
* Leaves out addresses that are not web pages (scripts, styles, pictures, fonts, audio, video, archives and data files). It keeps PDF files.
* Applies `--include` and `--exclude`, drops pages that the site's rules disallow and pages that the project holds, and then cuts to `--limit`.
* Reports the found, kept and left-out counts for every dry run and every crawl.

You must:

* Read the counts and the sample before you run a crawl.

Not handled:

* Penumbra does not follow links to a set depth, so it cannot find a page that the sitemap and the reachable pages do not show.

## Do

1. Read the dry run again: `found`, `kept`, the three left-out counts and the
   sample.
2. If the first `left out` count is large, read what it names. When it names
   your patterns, loosen `--include` or drop `--exclude` to what the person
   asked for. Run the dry run again. When it names the address you gave, see
   steps 3 and 4.
3. If the start address is a deep page, start from the index page of the
   section, or from the root of the site with `--include`.
4. If the docs are on a subdomain, add `--include-subdomains`.
5. If a short list of pages is missing, save them by address with
   `penumbra web save`.
6. Tell the person the kept count and that it is lower than the site's size.

## Don't

* Don't raise `--limit` to see more pages. The limit only cuts the list, and it
  never adds pages that were not found.
* Don't widen the patterns beyond the request to reach a count.
* Don't run the crawl and hope that it finds more than the dry run showed.

## Check

* The new dry run shows the pages the person expects in its sample.
* The person knows the kept count, and that the site may have more pages.

## Related

Cards [file](https://docs.pnbr.io/concepts/file.md) and [website](https://docs.pnbr.io/concepts/website.md). Recipe [bring-in-a-website](https://docs.pnbr.io/guides/bring-in-a-website.md). Problems [page-blocked-by-robots](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md) and
[page-already-in-project](https://docs.pnbr.io/troubleshooting/page-already-in-project.md).

## Keep going

[File](https://docs.pnbr.io/concepts/file.md) · [Website](https://docs.pnbr.io/concepts/website.md) · [Save web pages or crawl a website into the project](https://docs.pnbr.io/guides/bring-in-a-website.md) · [The site's rules keep a page out](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md) · [The project already holds the page](https://docs.pnbr.io/troubleshooting/page-already-in-project.md)

Penumbra Grammar 0.6.26


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.