Skip to main content
Concept · extraction
An extraction reads pinned Files through one Shape or typeset, costs AI credits, and opens its own new workspace.

Use when

You turn the text of Files into entities and relationships, or you check on, stop or finish an extraction that already started.

Looking for something else?

Facts a person tells you or you write by hand (recipe Save facts a person tells you as entities). Changing entities that are already in the Graph (card Workspace). Reading a File (card File).

​
For

An extraction reads the text of Files and finds entities and relationships that fit a Shape. It opens its own new workspace and stages them there. Nobody has approved them yet. A submit of that workspace puts them in the Graph. An extraction costs AI credits and takes time, so you plan it, start it once, follow it, and then review what it staged.

​
Use when

  • A person asks for the knowledge in one File, several Files or a collection.
  • You must know the cost of a run before you start it.
  • A run is slow, has failed, or ended with some Files unread.
  • A run costs more than you want, and you must stop it.

​
Not this

  • Facts you already know: stage them yourself (recipe capture-by-hand).
  • An entity that is already in the Graph and needs a change: use card workspace.
  • Whether a File is ready to read: use card Collection and its readiness check.

​
Rules

  1. extraction.1 Plan before you start. Your job: run penumbra extractions plan with the Files or the collection, a Shape and an extraction level. It lists the Files and versions the run would read, the types, and the credit range. It holds nothing and charges nothing. Give the person the range before you start.
  2. extraction.2 A run takes one perspective: one Shape or one typeset. Enforced: penumbra graph extract refuses both or neither with invalid_arguments. Your job: to use several Shapes, start one extraction for each Shape. Each one opens its own workspace.
  3. extraction.3 A run reads Files by id or reads one collection, never both. Enforced: the CLI refuses both or neither with invalid_arguments. One run takes 1 to 256 Files, and a larger collection is refused (card rule collection.5).
  4. extraction.4 A run reads the versions that are current when it starts. Your job: a File that you replace, or add to the collection, after the start is not read by that run. Start another run for it (problem new-version-not-extracted).
  5. extraction.5 Starting a run costs AI credits and, with an API key or as a connected agent, waits for approval (card rules approval.2 and approval.3). Your job: pass --request-id <key> and keep it. Enforced when you ask: the same key returns the same extraction and does not charge again. Without the flag, the CLI makes a new key, and a second run charges a second time.
  6. extraction.6 Penumbra charges for work done. A failed run is not charged. A stopped run is charged for the work done, and a queued run that is stopped is not charged. Your job: tell the person what a stop costs before you stop a run that is already reading.
  7. extraction.7 What an extraction finds waits in the workspace that the extraction opened. Your job: review it and submit it (recipe review-and-submit). Nothing is in the Graph until the submit returns applied (card rule workspace.2).
  8. extraction.8 Leave that workspace alone until the extraction ends. Your job: do not modify it while the run is working. When the run ends in part, decide on Try again before you read or edit the workspace (rule extraction.13). Then review the items with penumbra graph workspace show.
  9. extraction.9 An extraction compares each item with the entities that the Graph already has. The comparison runs the first time anyone reads, validates, edits or submits the workspace. When it matches or flags an item, it saves that as a change to the workspace. An item with the same Shape type and the same name as exactly one live entity becomes an update to that entity. The workspace shows it as updates existing. Every other item stays a new entity. Your job: check each updates existing item before you submit. Run penumbra graph workspace keep-new for one that is a different thing (problem extraction-duplicates-existing-entity). Search for an item whose entity has another name, because only the exact name matches.
  10. extraction.10 Follow a run with its extraction id. Your job: run penumbra graph extractions "<workspace>" --run <id> --watch, with the workspace name that penumbra graph extract printed. The count of items so far can go down as well as up. Ctrl-C ends the watch and does not stop the run. Use the id exactly as penumbra graph extract printed it. Do not build one.
  11. extraction.11 Stop keeps what the run has found. Enforced: penumbra graph extractions stop does not wait for approval and is safe to repeat. A stopped run is not offered Try again: it is refused with retry_not_offered. Your job: after a stop, the rest of the Files are not read, and the stop message does not say which Files were read. To read the rest, start a new extraction for those Files (recipe extract-files). A new extraction charges for the Files it reads.
  12. extraction.12 Try again finishes a run that failed, or that ended in part because the engine could not finish some work (the CLI says Partly finished). Your job: run penumbra extractions try-again <run-id>. It reads the same Files at the same versions with the same Shape, reuses the work that finished, and charges only for the rest. It costs credits and waits for approval under the same rules as a start. It closes the old workspace and opens a new one under a new name. Follow the workspace name and the run id that it prints (rule extraction.10), and not the old ones.
  13. extraction.13 Try again is offered only on the newest run of a lineage, while its workspace is still open and has not been changed. Enforced: it is refused with retry_not_offered, retry_in_progress, retry_review_edited or retry_stale. Your job: decide on Try again before you read or edit a workspace that ended in part. The first read can change it (rule extraction.9). If the workspace changed, submit it or close it (rule workspace.11), then start a new extraction.
  14. extraction.14 The extraction level sets how thorough a run is, standard or fast. Enforced: a level that the project does not offer is refused with extraction_level_unavailable, and the reply names the levels it offers. Without --level, the run uses the project’s default.
  15. extraction.15 Every extraction opens its own new workspace, and you do not name one. Enforced: the start takes no workspace argument, and the answer names the workspace it opened. That workspace holds only the items of that run. Your job: use that name to follow, review and submit the run. Do not capture into it.

​
Failure signs

​
Doors

A Kit with extraction_start also has the four tools that follow and finish a run.

​
HTTP reference

​
Keep going

Workspace · Approval · Collection · Shape · File · Entity · An extraction did not read every File · The owner approved an extraction and you cannot follow it · An extraction found an entity the Graph already has