Website

​

For

​

Use when

• A person names a page, a docs site, a blog or a section of a site.
• You must find the Files that a crawl made.
• You must tell the person what a saved page is and where it is.

​

Not this

• Documents on a computer or in a folder: use recipe bring-in-a-corpus.
• A page that needs a sign-in: save it as a file and upload it (card file).
• A folder that Penumbra keeps current: use a synced folder (card source).

​

Rules

1. website.1 Saving a page makes a File, and nothing else. Enforced: penumbra web save and penumbra web crawl add Files. No command opens, renames or deletes a website. Your job: work with the Files after a save: search them, group them in a collection and extract from them.
2. website.2 A page is one File, however its address is spelled. Enforced: Penumbra reads an address in one form. It treats http and https as one and lowercases the host. It drops the default port, the fragment, a trailing slash and tracking parameters (utm_*, gclid, fbclid, mc_cid and mc_eid). It orders the other parameters by name. A page the project holds is reported as already_in_project and is not saved again.
3. website.3 A saved page has one place in the Workspace. Enforced: both commands place a page at <folder>/<host>/<path>/index, and the root page at <folder>/<host>/index. The default folder is /web. A page whose address has a query ends in index~ and eight characters of a hash of the query. So a page and the pages below it never take the same name, and pages that differ only by query stay apart. A path segment named index, with or without an extension, becomes the folder index~, so /docs/index is at <folder>/<host>/docs/index~/index. Your job: find a page by its address: https://example.com/docs/intro is at /web/example.com/docs/intro/index.
4. website.4 A crawl is one job with one collection. Enforced: penumbra web crawl answers at once with a crawl id. Penumbra finds the pages, saves each one and adds it to the crawl’s collection. That is a new collection named for the site, unless --collection names one. A crawl that would keep more than 100 pages is refused with crawl_needs_preview. The refusal carries the dry run Penumbra made while it checked: its counts, a sample and an estimate. The CLI prints them and the exact command that starts the crawl, with --preview <id>. Only the person or key that started a crawl, or a project admin, can cancel it (penumbra web crawls cancel). Your job: read the counts and the estimate with the person, and run the printed command if they agree. Read the crawl with penumbra web crawls get <crawl>. Do not search for its pages before it ends.
5. website.5 A saved page shows the site as it was on the day it was saved. Enforced: saving a page the project holds changes nothing, and a crawl leaves such a page out. No command refreshes a saved page. Your job: tell the person the date of the save, and do not promise a newer copy.

​

Failure signs

• A page is refused with url_blocked_by_robots: problem page-blocked-by-robots.
• A crawl keeps far fewer pages than the site has: problem crawl-found-fewer-pages.
• A page’s pictures stop partway down: problem page-pictures-truncated.
• A save reports a page as already in the project: problem page-already-in-project.
• An address is refused with url_not_allowed: it is not a public web page. Fix the address.
• A command is refused with web_capture_busy or web_capture_unavailable (status 503): nothing was saved or kept. Wait the time the refusal names, then send the same request again with the same key.
• A crawl is refused with crawl_bound_exceeded: it crossed the bound named in the refusal. Narrow --include or --limit, or wait for a running crawl to end.
• A crawl is refused with permission_denied: the credential lacks the right to add Files or to use collections. Ask a project owner or admin.
• A cancel is refused with permission_denied: only the person who started the crawl, or a project admin, can cancel it.

​

Doors

​

Keep going