Extract entities from Files

​

Outcome

A workspace holds the items that the run found, as entities and relationships that nobody has approved yet. You know which Files and versions it read, how much it cost, and which Files it did not read to the end. Nothing is in the Graph until the workspace is submitted.

​

Before you start

• Orient first (recipe orient). A key or a connected agent waits for approval to start a run (card rule extraction.5). A whole collection needs more for an API key, a connected agent or a Kit. An owner must turn on unattended collection extraction (card rule collection.7). Check it with penumbra usage unattended.
• Check the text of each File first (recipe check-file-text). A run reads the extracted text. A File whose text is partial gives partial entities, and you pay for the run.
• Choose one Shape. A run takes one Shape or one typeset, never several (card rule extraction.2). List them with penumbra shapes.
• Choose the Files by ID, or one collection, never both. A run takes 1 to 256 Files (card rule extraction.3).
• Do not open a workspace for the run. The run opens its own and names it (card rule extraction.15). Leave that workspace alone until the run ends (card rule extraction.8).

​

Steps

1. Plan the run. This holds nothing and charges nothing (card rule extraction.1).
   • CLI: penumbra extractions plan --file-id <id> --file-id <id> --shape "<Shape>", or penumbra extractions plan --collection <collection-id> --shape "<Shape>". Add --level standard or --level fast to choose how thorough it is.
   • Expect these lines, in order. A Plan: read … line counts the Files and their text. One line follows for each File, with its version. Then come Shape: <name>, Types by route:, Expected cost: and Most a run can be charged:. A plan: digest and the command to start the run end the answer.
   • MCP: none. No MCP tool plans a run. Use the CLI.
2. Read the plan. Check that the list holds the Files and versions you meant. A File you replace after the start is not read by that run (card rule extraction.4). More than 256 Files is refused with collection_too_large (card rule collection.5).
3. Give the person the credit range and the Shape. Wait for a clear yes. The high number is an estimate, not a cap.
4. Start the run once, with a request id that you keep (card rule extraction.5).
   • CLI: penumbra graph extract --file-id <id> --shape "<Shape>" --level <level> --request-id <key>, or with --collection <collection-id>.
   • Expect Extraction started or Extraction is queued, the number of Files, the name of the new workspace that it opened, Expected cost:, and a line extraction: <run-id>. Use the run id and the workspace name exactly as they are printed.
   • A key or a connected agent gets exit code 4 and outcome: pending. Give the person the approval address and wait (problem change-waits-for-approval). Do not start it again. The run and its workspace do not exist until the owner approves (problem cannot-follow-approved-extraction).
   • MCP: extraction_start in the full catalog or in a Kit that selects it. The default catalog has none. It takes no workspace; the answer names the one it opened.
5. Follow the run.
   • CLI: penumbra graph extractions "<workspace>" --run <run-id> --watch. Expect Extracting… N items so far and a count for each type. The count can fall as well as rise. Ctrl-C stops the watch and not the run (card rule extraction.10).
   • MCP: extraction_progress and extraction_findings in the same catalog.
6. Stop only if the person asks. penumbra graph extractions stop "<workspace>" --run <run-id>. The items so far stay, the rest is not read, and you are charged for the work done. A run that is still queued costs nothing (card rule extraction.6).
7. Read how it ended.
   • Extraction finished (<level>) — ready to review. N items in the review "<name>". is a whole run. The level is Standard or Fast. The CLI’s <name> is the workspace name.
   • Extraction partly finished (<level>) — ready to review., then Partly finished: N Files weren't fully read, is a partial run. What was read is in the workspace, and you pay only for that. Decide on Try again now, before you read or edit the workspace, because the first read can change it (card rule extraction.13). Run penumbra extractions try-again <run-id> to read the rest (card rule extraction.12; problem extraction-incomplete). It opens a new workspace, so follow the name and the run id that it prints.
   • Stopped early after a stop means the rest was not read. Try again is not offered. Start a new extraction for the rest (card rule extraction.11).
   • Extraction didn't finish with a reason is a failed run. Nothing was added and nothing was charged.
   • These items weren't double-checked appears after a Fast run. Tell the person.
8. Compare what the run read with what it should have read (problem extraction-incomplete). The plan listed the Files and versions. penumbra graph extractions "<workspace>" --run <run-id> --json names the Files that were not fully read.
9. Check the items against the Graph. Extraction matches an item to the one entity with the same Shape type and name (card rule extraction.9).
   • CLI: penumbra graph workspace show "<workspace>" lists those items as updates existing <Type> "<name>". Check each one. Run penumbra graph workspace keep-new "<workspace>" <handle> for one that is a different thing. For each other item, search: penumbra graph search "<name>" and penumbra graph nodes --type <Type> (problem extraction-duplicates-existing-entity).
10. Review and submit (recipe review-and-submit).

​

Check

• penumbra graph extractions "<workspace>" shows the run as Ready to review with its item count, or as partly finished with the Files that were not read.
• penumbra graph workspace show "<workspace>" lists the items. Open some of them and confirm that each cites a File version that the plan named.
• You can state the Files and versions the run read, its cost, and what is left to read.
• Nothing is in the Graph yet. You said so to the person (card rule workspace.2).

​

If it goes wrong

​

Worth offering

• Offer to run the plan again on a smaller set first, so the person sees what one File costs before the whole collection.
• Offer to search the Graph for the items that stayed new, and to relate the new entities to existing ones.
• If the run was partial, offer to try again for the rest and to say what it adds to the cost.

​

Report

Say which Files and versions the run read, the Shape and the extraction level. Give the cost range you showed and the cost that was charged. Say how many items wait in which workspace, and which Files were not read to the end. Say which items update existing entities, and which items may duplicate entities that exist. Say that nothing is in the Graph until the person approves the submit, and who must approve it.

​

Keep going

Extraction · Workspace · Approval · Collection · Shape · File · Check that a File’s text arrived intact · Review a workspace and submit it · An extraction did not read every File · The owner approved an extraction and you cannot follow it · An extraction found an entity the Graph already has · A change returned pending