> ## Documentation Index > Fetch the complete documentation index at: https://docs.pnbr.io/llms.txt > Use this file to discover all available pages before exploring further. # Website > A website is a web page, or a whole site, saved into a project. Each page becomes an ordinary File, and a crawl gathers the pages of a site in one collection. Concept · files > **Use when** > > A person asks you to add a web page, a docs site, a blog or part of a site. Or you must find what a saved page became. > **Looking for something else?** > > Documents on a computer or in a folder (recipe [bring-in-a-corpus](https://docs.pnbr.io/guides/bring-in-a-corpus.md)). A page behind a sign-in. A folder kept current from a source (card [source](https://docs.pnbr.io/concepts/source.md)). ## For A person points at public web pages, and Penumbra saves them into the project. A saved page is a File that keeps its web address. Its text is searchable and its pictures show the page. There is no website object: the pages are Files, and everything Files do applies (card [file](https://docs.pnbr.io/concepts/file.md)). `penumbra web save` saves up to 50 pages you name. `penumbra web crawl` finds the pages of a site, saves them as Files and puts them in one collection (card [collection](https://docs.pnbr.io/concepts/collection.md)). A crawl runs on its own after the command returns. Keeping a saved site current from the web is not available yet. When it is, a site will be a linked source, beside the synced folders (card [source](https://docs.pnbr.io/concepts/source.md)). ## Use when * A person names a page, a docs site, a blog or a section of a site. * You must find the Files that a crawl made. * You must tell the person what a saved page is and where it is. ## Not this * Documents on a computer or in a folder: use recipe [bring-in-a-corpus](https://docs.pnbr.io/guides/bring-in-a-corpus.md). * A page that needs a sign-in: save it as a file and upload it (card [file](https://docs.pnbr.io/concepts/file.md)). * A folder that Penumbra keeps current: use a synced folder (card [source](https://docs.pnbr.io/concepts/source.md)). ## Rules 1. `website.1` Saving a page makes a File, and nothing else. **Enforced:** `penumbra web save` and `penumbra web crawl` add Files. No command opens, renames or deletes a website. **Your job:** work with the Files after a save: search them, group them in a collection and extract from them. 2. `website.2` A page is one File, however its address is spelled. **Enforced:** Penumbra reads an address in one form. It treats `http` and `https` as one and lowercases the host. It drops the default port, the fragment, a trailing slash and tracking parameters (`utm_*`, `gclid`, `fbclid`, `mc_cid` and `mc_eid`). It orders the other parameters by name. A page the project holds is reported as `already_in_project` and is not saved again. 3. `website.3` A saved page has one place in the Workspace. **Enforced:** both commands place a page at `///index`, and the root page at `//index`. The default folder is `/web`. A page whose address has a query ends in `index~` and eight characters of a hash of the query. So a page and the pages below it never take the same name, and pages that differ only by query stay apart. A path segment named `index`, with or without an extension, becomes the folder `index~`, so `/docs/index` is at `//docs/index~/index`. **Your job:** find a page by its address: `https://example.com/docs/intro` is at `/web/example.com/docs/intro/index`. 4. `website.4` A crawl is one job with one collection. **Enforced:** `penumbra web crawl` answers at once with a crawl id. Penumbra finds the pages, saves each one and adds it to the crawl's collection. That is a new collection named for the site, unless `--collection` names one. A crawl that would keep more than 100 pages is refused with `crawl_needs_preview`. The refusal carries the dry run Penumbra made while it checked: its counts, a sample and an estimate. The CLI prints them and the exact command that starts the crawl, with `--preview `. Only the person or key that started a crawl, or a project admin, can cancel it (`penumbra web crawls cancel`). **Your job:** read the counts and the estimate with the person, and run the printed command if they agree. Read the crawl with `penumbra web crawls get `. Do not search for its pages before it ends. 5. `website.5` A saved page shows the site as it was on the day it was saved. **Enforced:** saving a page the project holds changes nothing, and a crawl leaves such a page out. No command refreshes a saved page. **Your job:** tell the person the date of the save, and do not promise a newer copy. ## Failure signs * A page is refused with `url_blocked_by_robots`: problem [page-blocked-by-robots](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md). * A crawl keeps far fewer pages than the site has: problem [crawl-found-fewer-pages](https://docs.pnbr.io/troubleshooting/crawl-found-fewer-pages.md). * A page's pictures stop partway down: problem [page-pictures-truncated](https://docs.pnbr.io/troubleshooting/page-pictures-truncated.md). * A save reports a page as already in the project: problem [page-already-in-project](https://docs.pnbr.io/troubleshooting/page-already-in-project.md). * An address is refused with `url_not_allowed`: it is not a public web page. Fix the address. * A command is refused with `web_capture_busy` or `web_capture_unavailable` (status 503): nothing was saved or kept. Wait the time the refusal names, then send the same request again with the same key. * A crawl is refused with `crawl_bound_exceeded`: it crossed the bound named in the refusal. Narrow `--include` or `--limit`, or wait for a running crawl to end. * A crawl is refused with `permission_denied`: the credential lacks the right to add Files or to use collections. Ask a project owner or admin. * A cancel is refused with `permission_denied`: only the person who started the crawl, or a project admin, can cancel it. ## Doors | Door | Call | | - | - | | CLI | `penumbra web save`, `penumbra web crawl`, `penumbra web crawls get`, `penumbra web crawls cancel` | | SDK | `project.web`, with `web.save` for pages and `web.crawls` (`preview`, `create`, `get`, `pages`, `cancel`) for a site | No MCP tool saves a web page. Use the CLI, the SDK or HTTP. An agent connected only through MCP can read the saved pages as Files, but cannot save new ones. ## Keep going [File](https://docs.pnbr.io/concepts/file.md) · [Collection](https://docs.pnbr.io/concepts/collection.md) · [Synced folder](https://docs.pnbr.io/concepts/source.md) · [Save web pages or crawl a website into the project](https://docs.pnbr.io/guides/bring-in-a-website.md) · [The site's rules keep a page out](https://docs.pnbr.io/troubleshooting/page-blocked-by-robots.md) · [A crawl keeps fewer pages than the site has](https://docs.pnbr.io/troubleshooting/crawl-found-fewer-pages.md) · [A long page keeps pictures for its top only](https://docs.pnbr.io/troubleshooting/page-pictures-truncated.md) · [The project already holds the page](https://docs.pnbr.io/troubleshooting/page-already-in-project.md) Penumbra Grammar 0.6.26 This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.