Skip to main content
Concept · files

Use when

You group Files for a job, check whether a group is ready, extract from a group, or choose what a View includes.

Looking for something else?

One document and its versions (card File). A folder kept current from a source (card Synced folder). The part of a project a person may see (card View).

​
For

A collection is a name for a set of Files. A File can belong to several collections, and joining one does not change the File, its path or its versions. The collection holds the File itself, so a new edition of a member is still a member. Collections are how a job works on a corpus. You bring Files in, put them in a collection, and check that the collection is ready. You then search inside it and extract from it. A View can include the collection, and a Brain includes that View, so the same collection decides what a client sees. Membership is read when a command runs, not when you set it up. Adding a File to a collection changes what every search, extraction scope and View on that collection reads next.

​
Use when

  • You bring in a set of documents and must treat them as one corpus.
  • You need to know whether every File in a group is searchable.
  • You extract from many Files at once.
  • You search inside one or more collections, with penumbra search --collection or with collections in the MCP tool omnisearch (card Search).
  • You choose which Files a View, and so a Brain, includes.

​
Not this

  • One document and its versions: use card File.
  • A folder on a machine or in a drive that Penumbra keeps current: use a source (card Synced folder).
  • What one person or agent may read: use a View (card View).

​
Rules

  1. collection.1 A collection name is unique in the project, and case does not matter. Enforced: the CLI refuses a second collection with the same name with collection_conflict. Files commands and penumbra search --collection take a collection by ID or by name, an exact match first, then one that differs only in case. penumbra graph extract and penumbra extractions plan take the ID only.
  2. collection.2 Deleting a collection keeps its Files. Enforced: penumbra files collections delete is refused with confirmation_required unless you pass --yes. It also waits for approval when you act with an API key or as a connected agent. The result is pending with an approval link. Your job: do not retry. A deleted collection cannot be restored. A new collection has a new ID, so a View that named the old one does not include it.
  3. collection.3 Check readiness before you search a collection or extract from it. Your job: run penumbra files collections readiness. It splits the Files into four groups that add up to the File count: searchable, still processing, unreadable and without text. Search reads only the searchable ones. An extraction is refused with source_unavailable when a member has no readable current version.
  4. collection.4 Some searchable Files are not yet searchable by meaning. Your job: the readiness report counts them apart. Search them by words, or wait, before you rely on a meaning search of the collection.
  5. collection.5 An extraction from a collection reads each member’s current version when it starts. Enforced: an empty collection is refused with collection_empty. A collection with more than 256 Files, or more text than one run reads, is refused with collection_too_large. No credit is held.
  6. collection.6 An admitted extraction keeps its Files and versions. Your job: a File you add or a version you replace after the start is not read by that run. Run penumbra extractions plan first to see the Files, versions and cost, because it changes nothing and charges nothing.
  7. collection.7 An API key, a connected agent or a Kit can start a collection extraction only when an owner or admin allows it. Enforced: while that setting is off, which is the default, the start is refused with unattended_collection_extraction_off. With it on, a run above the per-run cap is refused with collection_run_over_cap. Only a signed-in owner or admin can change the setting.
  8. collection.8 A pending approval of an extraction names the collection as it stood. Enforced: if a File joins or leaves, or a member gets a new version, before the owner approves, the approval is refused with target_changed. Start the extraction again.
  9. collection.9 A View that includes a collection shows its members as they are when someone reads. Your job: before you add a File to such a collection, decide that everyone limited to that View may read it. It appears at once, and removing the File hides it at once.
  10. collection.10 A collection that a source keeps current follows its folder. Your job: do not treat its membership as yours. Files that the person deletes from the folder leave the collection.
  11. collection.11 The collection list comes a page at a time. Your job: penumbra files collections list shows 50 by default and up to 200 with --limit, and prints more: rerun with --cursor <cursor> when more remain. On a managed connection, the MCP tool list_collections shows only the first 200 collections. Check a collection beyond that by its exact name, with penumbra files suggest collections.

​
Failure signs

​
Doors

The default catalog has no tool that starts an extraction. The MCP tool extraction_start does, in the full catalog and in a Kit that selects it. The MCP tool set_collection_membership names a File by its path. A File with no path cannot be added that way. Give it a path with penumbra files place, or use the CLI command penumbra files collections add. No MCP tool reports the readiness of a whole collection. The tool list_files with the collection and the current version expanded shows each File’s processing. Use the CLI, the SDK or HTTP for the grouped report.

​
The CLI word

The CLI calls a View a Context Slice.

​
HTTP reference

​
Keep going

File · Search · Extraction · View · Brain · Synced folder · Bring a set of documents into the project · An extraction did not read every File