> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pnbr.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Save web pages or crawl a website into the project

> The pages a person asked for are in the project as Files, in one collection, searchable, and no more pages than they asked for.

Guide · files

> **Use when**
>
> A person asks you to save web pages, or to crawl a docs site, a blog or part of a website, into the project.

> **Looking for something else?**
>
> Documents on a computer or in a folder (recipe [Bring a set of documents into the project](https://docs.pnbr.io/guides/bring-in-a-corpus.md)). A page that needs a sign-in. Keeping a site up to date over time.

## Outcome

Each page the person asked for is one File in one collection. The File keeps its
web address, its text is searchable and its pictures show the page. No pages
outside the request were added.

## Before you start

* Ask what the person needs from the site, not only its address. "Their pricing
  and their docs" is two sections, not the whole domain. Write it down as a
  start address and include patterns.
* Read card [website](https://docs.pnbr.io/concepts/website.md): it says what a saved page is and where it is placed.
* If the person named a few pages, use `penumbra web save <url>…`, not a
  crawl.
* Run `penumbra usage`. Each page is charged, so check that the allowance covers
  the pages you plan to add.

## Steps

1. For one page or a short list, choose one collection name and create it with
   `penumbra files collections create "<name>"` (card rule [collection.1](https://docs.pnbr.io/concepts/collection.md#rule-collection-1)). Then
   run `penumbra web save <url>… --collection "<name>" --wait`, and go to
   step 8.
2. For a site, choose the bounds. `--include` patterns name the sections the
   person wants. `--limit` sets the most pages: 100 unless you raise it, and
   1,000 at most. Do not raise `--limit` to make a large site fit.
3. Look before you start a crawl. `penumbra web crawl <url> --include "<pattern>" --dry-run`
   adds nothing and prints four things: the kept count, the left-out counts, the
   sample addresses and the estimate. Run it when the person has a spend limit
   or you do not know how big the site is. A crawl that would keep more than 100
   pages gives you this look. The command is refused with `crawl_needs_preview`
   and prints the dry run it made, with the exact command that starts the crawl.
   Do not run a second dry run for it.
4. If the kept count is far above what the person expects, or the sample shows
   other pages, narrow `--include` or add `--exclude`, then look again.
5. Tell the person the kept count and the estimate. If more than 100 pages would
   be kept, or the estimate is more than they agreed to spend, wait for their
   yes.
6. Start the crawl without `--wait`: run the command that the refusal printed,
   or your dry run's command with `--preview <id>`:
   `penumbra web crawl <url> --include "<pattern>" --preview <id>`.
   Keep the crawl id it prints. A crawl with `--wait` holds the command until
   the crawl ends, so you cannot stop it on the way.
7. Read the crawl about once a minute with `penumbra web crawls get <crawl>`,
   until its state is `succeeded`, `partial`, `failed` or `cancelled`. After each
   read, compare its `kept` count with the dry run. If a `problem:` line shows
   `crawl_grew_since_preview`, or `kept` is above the dry run, cancel the crawl
   with `penumbra web crawls cancel <crawl>` and tell the person. Only the person
   or key that started the crawl, or a project admin, can cancel it.
8. Run `penumbra files collections readiness "<collection>"`. A crawl's
   collection is on the `collection:` line of `penumbra web crawls get`. Wait
   until every File is searchable.

## Check

* After a save, each page's line says `placed` or says why it did not land.
* After a crawl, `penumbra web crawls get <crawl> --pages` shows each page's outcome.
  The added count is close to the kept count of the dry run.
* `penumbra files search "<a phrase from one page>" --collection-id <collection>`
  finds that page.

## If it goes wrong

| You see | Read |
| - | - |
| A page is refused because the site does not allow it | problem [page-blocked-by-robots](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md) |
| The dry run keeps far fewer pages than the site has | problem [crawl-found-fewer-pages](https://docs.pnbr.io/troubleshooting/crawl-found-fewer-pages.md) |
| A page's pictures stop partway down | problem [page-pictures-truncated](https://docs.pnbr.io/troubleshooting/page-pictures-truncated.md) |
| A page is reported as already in the project | problem [page-already-in-project](https://docs.pnbr.io/troubleshooting/page-already-in-project.md) |

A refusal that says `web_capture_busy` or `web_capture_unavailable` kept
nothing. Wait the time it names, a few seconds or about a minute, then run the
same command again with the same `--request-id`. A refusal `crawl_bound_exceeded`
names the bound it crossed: narrow `--include`, or wait for a running crawl to
end. A refusal `permission_denied` on a crawl means your credential lacks the
right to add Files or to use collections: ask a project owner or admin.

Do not run a whole crawl again to retry a few pages. Save only the failed
addresses into the crawl's own folder and collection:
`penumbra web save <urls> --to /<crawl folder> --collection <crawl collection>`.
`penumbra web crawls get <crawl>` prints the folder and the collection, and
`--pages` lists the failed addresses. A page the site does not allow will fail
again, so tell the person instead.

## Worth offering

* Offer to create a Shape from the new collection, so the site's ideas become a
  lens for the project (recipe [create-a-shape](https://docs.pnbr.io/guides/create-a-shape.md)).
* Offer to extract entities from the collection with a Shape that fits it
  (recipe [extract-files](https://docs.pnbr.io/guides/extract-files.md)). Show the estimate first with
  `penumbra extractions plan`, and start with `--level fast`.

## Report

Say how many pages arrived, the collection name and what the pages cost
(`penumbra usage calls`). Say how many were left out and why: patterns, the
site's rules or already in the project. Name any page that failed and why.

## Keep going

[Website](https://docs.pnbr.io/concepts/website.md) · [File](https://docs.pnbr.io/concepts/file.md) · [Collection](https://docs.pnbr.io/concepts/collection.md) · [Search](https://docs.pnbr.io/concepts/search.md) · [Project](https://docs.pnbr.io/concepts/project.md) · [Answer from the project's documents, with citations](https://docs.pnbr.io/guides/answer-with-citations.md) · [The site's rules keep a page out](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md) · [A crawl keeps fewer pages than the site has](https://docs.pnbr.io/troubleshooting/crawl-found-fewer-pages.md) · [A long page keeps pictures for its top only](https://docs.pnbr.io/troubleshooting/page-pictures-truncated.md) · [The project already holds the page](https://docs.pnbr.io/troubleshooting/page-already-in-project.md)

Penumbra Grammar 0.6.26


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.