Skip to main content
Troubleshooting · graph

Use when

A write returned graph_write_busy, HTTP status 503, or retry_after_seconds.

Looking for something else?

A write with no reply or outcome: unknown (problem The reply to a change was lost). A submit that ended partial (problem A submit is partly applied or unresolved).

​
Signs

  • The refusal code is graph_write_busy, with HTTP status 503.
  • The refusal has retry_after_seconds: 1 and the header Retry-After: 1.
  • The message reads Graph is busy. Retry the same request with the same command key.
  • The penumbra command exits with code 5.
  • In execute on the default MCP catalog, the effect has the outcome busy, the recovery is resume_with_same_program, and the result gives a retry_after time.

​
Why it happens

Graph writes in one project take turns. Another write held the project’s write lock for too long, so your attempt could not finish in time. Penumbra rolled your attempt back and then answered busy. A direct entity change can meet this. A workspace submit can also meet it when a Shape in the project has rules, because the whole submit is then one transaction.

​
What Penumbra does

By itself:
  • Answers graph_write_busy only after it confirmed the rollback. Nothing was applied.
  • Keeps the request id. The same request id with the same body is safe to send again.
  • Marks the effect as busy in an execute run, so you can resume it.
You must:
  • Wait retry_after_seconds.
  • Send the same request again, with the same request id and the same body.
Not handled:
  • The CLI does not wait or retry for you after a 503.
  • The SDK does not retry. Its submit call says that it never polls or retries.

​
Do

  1. Wait retry_after_seconds. The value is 1 second.
  2. Send the same command again with the same --request-id. For a submit, that is penumbra graph workspace submit "<name>" --revision <n> --request-id <key>.
  3. In execute, wait until retry_after. Then resume with the same program, as recovery says.
  4. If the Graph stays busy after several tries, tell the person. Keep the same request id.

​
Don’t

  • Don’t send the request at once in a loop. The Graph needs time to free its lock.
  • Don’t change the body and keep the request id. Penumbra refuses it with idempotency_conflict.
  • Don’t treat busy as unknown. A busy answer means a confirmed rollback. An unknown outcome needs the recovery in problem reply-lost.

​
Check

  • The retry returns applied, or a clear refusal.
  • A read of the state shows the change once.
Card rules workspace.4 and workspace.5. Recipe review-and-submit. Problems reply-lost and submit-partly-applied.

​
Keep going

Workspace · Entity · The reply to a change was lost · A submit is partly applied or unresolved