> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pnbr.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Extract entities from Files

> Entities from a set of Files wait in the new workspace that the run opens, and you know what it read, cost and missed.

Guide · extraction

> **Use when**
>
> A person asks for the knowledge in some Files or a collection, in the terms of one Shape.

> **Looking for something else?**
>
> Facts that a person tells you (recipe [capture-by-hand](https://docs.pnbr.io/guides/capture-by-hand.md)). Files whose text you have not checked (recipe [check-file-text](https://docs.pnbr.io/guides/check-file-text.md)).

## Outcome

A workspace holds the items that the run found, as entities and relationships
that nobody has approved yet. You know which Files and versions it read, how
much it cost, and which Files it did not read to the end. Nothing is in the
Graph until the workspace is submitted.

## Before you start

* Orient first (recipe [orient](https://docs.pnbr.io/guides/orient.md)). A key or a connected agent waits for approval
  to start a run (card rule [extraction.5](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-5)). A whole collection needs more for an
  API key, a connected agent or a Kit. An owner must turn on unattended
  collection extraction (card rule [collection.7](https://docs.pnbr.io/concepts/collection.md#rule-collection-7)). Check it with
  `penumbra usage unattended`.
* Check the text of each File first (recipe [check-file-text](https://docs.pnbr.io/guides/check-file-text.md)). A run reads the
  extracted text. A File whose text is partial gives partial entities, and you
  pay for the run.
* Choose one Shape. A run takes one Shape or one typeset, never several (card
  rule [extraction.2](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-2)). List them with `penumbra shapes`.
* Choose the Files by ID, or one collection, never both. A run takes 1 to 256
  Files (card rule [extraction.3](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-3)).
* Do not open a workspace for the run. The run opens its own and names it (card
  rule [extraction.15](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-15)). Leave that workspace alone until the run ends (card
  rule [extraction.8](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-8)).

## Steps

1. Plan the run. This holds nothing and charges nothing (card rule
   [extraction.1](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-1)).
   * CLI: `penumbra extractions plan --file-id <id> --file-id <id> --shape "<Shape>"`,
     or `penumbra extractions plan --collection <collection-id> --shape "<Shape>"`.
     Add `--level standard` or `--level fast` to choose how thorough it is.
   * Expect these lines, in order. A `Plan: read …` line counts the Files and
     their text. One line follows for each File, with its version. Then come
     `Shape: <name>`, `Types by route:`, `Expected cost:` and
     `Most a run can be charged:`. A `plan:` digest and the command to start the
     run end the answer.
   * MCP: none. No MCP tool plans a run. Use the CLI.
2. Read the plan. Check that the list holds the Files and versions you meant. A
   File you replace after the start is not read by that run (card rule
   [extraction.4](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-4)). More than 256 Files is refused with `collection_too_large`
   (card rule [collection.5](https://docs.pnbr.io/concepts/collection.md#rule-collection-5)).
3. Give the person the credit range and the Shape. Wait for a clear yes. The
   high number is an estimate, not a cap.
4. Start the run once, with a request id that you keep (card rule
   [extraction.5](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-5)).
   * CLI: `penumbra graph extract --file-id <id> --shape "<Shape>" --level <level> --request-id <key>`,
     or with `--collection <collection-id>`.
   * Expect `Extraction started` or `Extraction is queued`, the number of
     Files, the name of the new workspace that it opened, `Expected cost:`, and
     a line `extraction: <run-id>`. Use the run id and the workspace name exactly
     as they are printed.
   * A key or a connected agent gets exit code 4 and `outcome: pending`. Give
     the person the approval address and wait (problem
     [change-waits-for-approval](https://docs.pnbr.io/troubleshooting/change-waits-for-approval.md)). Do not start it again. The run and its
     workspace do not exist until the owner approves (problem
     [cannot-follow-approved-extraction](https://docs.pnbr.io/troubleshooting/cannot-follow-approved-extraction.md)).
   * MCP: `extraction_start` in the full catalog or in a Kit that selects it.
     The default catalog has none. It takes no workspace; the answer names the
     one it opened.
5. Follow the run.
   * CLI: `penumbra graph extractions "<workspace>" --run <run-id> --watch`.
     Expect `Extracting… N items so far` and a count for each type. The count
     can fall as well as rise. Ctrl-C stops the watch and not the run (card rule
     [extraction.10](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-10)).
   * MCP: `extraction_progress` and `extraction_findings` in the same catalog.
6. Stop only if the person asks. `penumbra graph extractions stop "<workspace>" --run <run-id>`.
   The items so far stay, the rest is not read, and you are charged for the
   work done. A run that is still queued costs nothing (card rule
   [extraction.6](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-6)).
7. Read how it ended.
   * `Extraction finished (<level>) — ready to review. N items in the review "<name>".`
     is a whole run. The level is `Standard` or `Fast`. The CLI's `<name>` is
     the workspace name.
   * `Extraction partly finished (<level>) — ready to review.`, then
     `Partly finished: N Files weren't fully read`, is a partial run. What was
     read is in the workspace, and you pay only for that. Decide on Try again
     now, before you read or edit the workspace, because the first read can
     change it (card rule [extraction.13](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-13)). Run
     `penumbra extractions try-again <run-id>` to read the rest (card rule
     [extraction.12](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-12); problem [extraction-incomplete](https://docs.pnbr.io/troubleshooting/extraction-incomplete.md)). It opens a new
     workspace, so follow the name and the run id that it prints.
   * `Stopped early` after a stop means the rest was not read. Try again is not
     offered. Start a new extraction for the rest (card rule [extraction.11](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-11)).
   * `Extraction didn't finish` with a reason is a failed run. Nothing was
     added and nothing was charged.
   * `These items weren't double-checked` appears after a Fast run. Tell the
     person.
8. Compare what the run read with what it should have read (problem
   [extraction-incomplete](https://docs.pnbr.io/troubleshooting/extraction-incomplete.md)). The plan listed the Files and versions.
   `penumbra graph extractions "<workspace>" --run <run-id> --json` names the
   Files that were not fully read.
9. Check the items against the Graph. Extraction matches an item to the one
   entity with the same Shape type and name (card rule [extraction.9](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-9)).
   * CLI: `penumbra graph workspace show "<workspace>"` lists those items as
     `updates existing <Type> "<name>"`. Check each one. Run
     `penumbra graph workspace keep-new "<workspace>" <handle>` for one that is
     a different thing. For each other item, search:
     `penumbra graph search "<name>"` and `penumbra graph nodes --type <Type>`
     (problem [extraction-duplicates-existing-entity](https://docs.pnbr.io/troubleshooting/extraction-duplicates-existing-entity.md)).
10. Review and submit (recipe [review-and-submit](https://docs.pnbr.io/guides/review-and-submit.md)).

## Check

* `penumbra graph extractions "<workspace>"` shows the run as `Ready to review`
  with its item count, or as partly finished with the Files that were not read.
* `penumbra graph workspace show "<workspace>"` lists the items. Open some of
  them and confirm that each cites a File version that the plan named.
* You can state the Files and versions the run read, its cost, and what is left
  to read.
* Nothing is in the Graph yet. You said so to the person (card rule
  [workspace.2](https://docs.pnbr.io/concepts/workspace.md#rule-workspace-2)).

## If it goes wrong

| You see | Read |
| - | - |
| The start returned `pending` or exit code 4 | problem [change-waits-for-approval](https://docs.pnbr.io/troubleshooting/change-waits-for-approval.md) |
| The start is refused with `collection_too_large` | card rule [collection.5](https://docs.pnbr.io/concepts/collection.md#rule-collection-5) |
| The start is refused with `review_not_empty` | problem [extraction-incomplete](https://docs.pnbr.io/troubleshooting/extraction-incomplete.md) |
| The start is refused with `extraction_level_unavailable` | card rule [extraction.14](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-14) |
| A key is refused for a whole collection | card rule [collection.7](https://docs.pnbr.io/concepts/collection.md#rule-collection-7) |
| The run finished with Files not fully read | problem [extraction-incomplete](https://docs.pnbr.io/troubleshooting/extraction-incomplete.md) |
| After the owner approved, you cannot follow the run | problem [cannot-follow-approved-extraction](https://docs.pnbr.io/troubleshooting/cannot-follow-approved-extraction.md) |
| An item says `updates existing`, or the workspace holds a repeat | problem [extraction-duplicates-existing-entity](https://docs.pnbr.io/troubleshooting/extraction-duplicates-existing-entity.md) |
| A File has a new version, and the Graph shows the old text | problem [new-version-not-extracted](https://docs.pnbr.io/troubleshooting/new-version-not-extracted.md) |
| Try again is refused with `retry_not_offered` or `retry_stale` | card rule [extraction.13](https://docs.pnbr.io/concepts/extraction.md#rule-extraction-13) |
| The reply to the start was lost | recipe [recover-after-a-lost-reply](https://docs.pnbr.io/guides/recover-after-a-lost-reply.md) |

## Worth offering

* Offer to run the plan again on a smaller set first, so the person sees what
  one File costs before the whole collection.
* Offer to search the Graph for the items that stayed new, and to relate the
  new entities to existing ones.
* If the run was partial, offer to try again for the rest and to say what it
  adds to the cost.

## Report

Say which Files and versions the run read, the Shape and the extraction level. Give the
cost range you showed and the cost that was charged. Say how many items wait
in which workspace, and which Files were not read to the end. Say which items
update existing entities, and which items may duplicate entities that exist. Say that nothing is in the Graph until the
person approves the submit, and who must approve it.

## Keep going

[Extraction](https://docs.pnbr.io/concepts/extraction.md) · [Workspace](https://docs.pnbr.io/concepts/workspace.md) · [Approval](https://docs.pnbr.io/concepts/approval.md) · [Collection](https://docs.pnbr.io/concepts/collection.md) · [Shape](https://docs.pnbr.io/concepts/shape.md) · [File](https://docs.pnbr.io/concepts/file.md) · [Check that a File's text arrived intact](https://docs.pnbr.io/guides/check-file-text.md) · [Review a workspace and submit it](https://docs.pnbr.io/guides/review-and-submit.md) · [An extraction did not read every File](https://docs.pnbr.io/troubleshooting/extraction-incomplete.md) · [The owner approved an extraction and you cannot follow it](https://docs.pnbr.io/troubleshooting/cannot-follow-approved-extraction.md) · [An extraction found an entity the Graph already has](https://docs.pnbr.io/troubleshooting/extraction-duplicates-existing-entity.md) · [A change returned pending](https://docs.pnbr.io/troubleshooting/change-waits-for-approval.md)

Penumbra Grammar 0.6.26


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.