Skip to main content
Troubleshooting · files

Use when

You added or replaced a File, and a search does not find its content yet, or finds it only by exact words.

Looking for something else?

A search that finds nothing for a File that is ready (problem Search returns nothing). A File whose text is wrong (recipe Check that a File’s text arrived intact).

​
Signs

  • penumbra files list shows the File with a status in brackets after its id: [processing] for a new File, [updating] for a File that Penumbra replaces.
  • penumbra files show <file> prints (none yet) on its version: line, or a line that ends with (lexical ready, semantic pending).
  • A search by exact words finds the File. A search by meaning does not.
  • penumbra files search prints semantic coverage: partial with two counts, and a warning semantic_coverage_partial.
  • The coverage reads full, and the new File is still missing from the results.

​
Why it happens

A File goes through steps. Penumbra stores the bytes, reads the text, splits the text into passages, and then builds the data for search by meaning. Each step ends at a different time. Search reads only versions that are ready. A version that Penumbra still reads is not in any search, and the coverage count does not include it. So the coverage can read full while your new File is missing. When the text step ends, the version becomes current and the File answers exact words. The version joins search by meaning only when every passage has its data. A PDF page can fail to read for a few minutes. Penumbra then reads that page again, up to about 10 minutes, before it finishes the version. The pages that Penumbra already read stay as they are. If the page reads in time, the File comes out whole, with no warning. If it does not, the File finishes without that page’s text, and penumbra files inspect shows a warning: line that names the page. A replace follows the same steps. The old version stays current until the new version has its text. Then the old passages leave search. If the new version has no data for meaning yet, the File drops out of search by meaning until that data is ready.

​
What Penumbra does

By itself:
  • Reads the text of a new version in the background, and makes the version current only when exact-word search is ready for it.
  • Leaves a version out of search until it is ready. A version that is still processing is not counted in the coverage.
  • Reads a failed PDF page again for up to about 10 minutes, so the first version of the File is whole.
  • Adds a version to search by meaning only when all its passages have their data.
  • Reports the coverage as partial or none, with a warning, when some versions wait for that data.
  • Keeps the old version of a replaced File current until the new version is ready.
  • Gives a File with no readable text no passages. Such a File never appears in search.
You must:
  • Wait until the File is ready, or search by exact words in the meantime.
  • Tell the person which kind of search covers the File now.
  • Treat a full coverage count as true only for the versions that were ready.
Not handled:
  • The coverage count does not include versions that are still processing, so a full count does not prove that your new File is searchable.

​
Do

  1. Read the state of the File: penumbra files show <file>. For a version that is not current yet, run penumbra files versions <file>, then penumbra files inspect <file> --version <version-id>. The second command prints processing: with lexical= and semantic=.
  2. To wait, add --wait to penumbra files upload or penumbra files replace. Or run penumbra files wait <operation-id>. The upload printed the operation id. Read the processing: line when the wait ends. A PDF can take up to about 10 minutes longer when Penumbra reads a page again. --wait waits up to 15 minutes, so it covers that. No other action is needed.
  3. If the status is [processing] or [updating], or the lexical state is pending, wait. Do not draw a conclusion from a search yet.
  4. If the lexical state is ready and the semantic state is pending, search by exact words now: penumbra files search "<words>" --mode lexical --file-id <file-id>. MCP: call omnisearch in lexical mode. Search by meaning after the semantic state is ready.
  5. If the status is [failed], run penumbra files wait <operation-id>. It prints error: with a code. Send the content again with penumbra files replace <file> <local-path>. If the text is empty, the document may be a scan with no readable text (recipe check-file-text).
  6. For a batch of Files, run penumbra files collections readiness <collection>. It counts the Files that are ready, the Files that Penumbra could not read, and the Files that are searchable but not by meaning.
  7. If a View limits your credential, the new File stays outside the View until it joins a collection that the View includes (problem hidden-reads-as-missing).

​
Don’t

  • Don’t tell the person that a document is missing because one search did not find it. The File may still be processing.
  • Don’t upload the File again to make it appear. The same bytes give the same File and change nothing (problem same-bytes-no-new-version).
  • Don’t use search by meaning only (--mode semantic) while the File waits for its data. That search skips the File and can return nothing.
  • Don’t start an extraction right after a replace. The extraction reads the version that is current when it starts, and that is still the old version until the new one is ready.

​
Check

  • penumbra files inspect <file> prints lexical=ready and semantic=ready.
  • penumbra files search "<phrase from the File>" --file-id <file-id> returns a passage of the File, and its first line shows mode: hybrid -> hybrid.
Card file. Card search, with rule search.2. Recipes bring-in-a-corpus and check-file-text. Problems search-returns-nothing and same-bytes-no-new-version.

​
Keep going

File · Search · Passage and citation · Search returns nothing