Save web pages or crawl a website into the project

​

Outcome

Each page the person asked for is one File in one collection. The File keeps its web address, its text is searchable and its pictures show the page. No pages outside the request were added.

​

Before you start

• Ask what the person needs from the site, not only its address. “Their pricing and their docs” is two sections, not the whole domain. Write it down as a start address and include patterns.
• Read card website: it says what a saved page is and where it is placed.
• If the person named a few pages, use penumbra web save <url>…, not a crawl.
• Run penumbra usage. Each page is charged, so check that the allowance covers the pages you plan to add.

​

Steps

1. For one page or a short list, choose one collection name and create it with penumbra files collections create "<name>" (card rule collection.1). Then run penumbra web save <url>… --collection "<name>" --wait, and go to step 8.
2. For a site, choose the bounds. --include patterns name the sections the person wants. --limit sets the most pages: 100 unless you raise it, and 1,000 at most. Do not raise --limit to make a large site fit.
3. Look before you start a crawl. penumbra web crawl <url> --include "<pattern>" --dry-run adds nothing and prints four things: the kept count, the left-out counts, the sample addresses and the estimate. Run it when the person has a spend limit or you do not know how big the site is. A crawl that would keep more than 100 pages gives you this look. The command is refused with crawl_needs_preview and prints the dry run it made, with the exact command that starts the crawl. Do not run a second dry run for it.
4. If the kept count is far above what the person expects, or the sample shows other pages, narrow --include or add --exclude, then look again.
5. Tell the person the kept count and the estimate. If more than 100 pages would be kept, or the estimate is more than they agreed to spend, wait for their yes.
6. Start the crawl without --wait: run the command that the refusal printed, or your dry run’s command with --preview <id>: penumbra web crawl <url> --include "<pattern>" --preview <id>. Keep the crawl id it prints. A crawl with --wait holds the command until the crawl ends, so you cannot stop it on the way.
7. Read the crawl about once a minute with penumbra web crawls get <crawl>, until its state is succeeded, partial, failed or cancelled. After each read, compare its kept count with the dry run. If a problem: line shows crawl_grew_since_preview, or kept is above the dry run, cancel the crawl with penumbra web crawls cancel <crawl> and tell the person. Only the person or key that started the crawl, or a project admin, can cancel it.
8. Run penumbra files collections readiness "<collection>". A crawl’s collection is on the collection: line of penumbra web crawls get. Wait until every File is searchable.

​

Check

• After a save, each page’s line says placed or says why it did not land.
• After a crawl, penumbra web crawls get <crawl> --pages shows each page’s outcome. The added count is close to the kept count of the dry run.
• penumbra files search "<a phrase from one page>" --collection-id <collection> finds that page.

​

If it goes wrong

A refusal that says web_capture_busy or web_capture_unavailable kept nothing. Wait the time it names, a few seconds or about a minute, then run the same command again with the same --request-id. A refusal crawl_bound_exceeded names the bound it crossed: narrow --include, or wait for a running crawl to end. A refusal permission_denied on a crawl means your credential lacks the right to add Files or to use collections: ask a project owner or admin. Do not run a whole crawl again to retry a few pages. Save only the failed addresses into the crawl’s own folder and collection: penumbra web save <urls> --to /<crawl folder> --collection <crawl collection>. penumbra web crawls get <crawl> prints the folder and the collection, and --pages lists the failed addresses. A page the site does not allow will fail again, so tell the person instead.

​

Worth offering

• Offer to create a Shape from the new collection, so the site’s ideas become a lens for the project (recipe create-a-shape).
• Offer to extract entities from the collection with a Shape that fits it (recipe extract-files). Show the estimate first with penumbra extractions plan, and start with --level fast.

​

Report

Say how many pages arrived, the collection name and what the pages cost (penumbra usage calls). Say how many were left out and why: patterns, the site’s rules or already in the project. Name any page that failed and why.

​

Keep going

Website · File · Collection · Search · Project · Answer from the project’s documents, with citations · The site’s rules keep a page out · A crawl keeps fewer pages than the site has · A long page keeps pictures for its top only · The project already holds the page