#160 Cconcurrent Agent starts can desynchronize runtime and persisted credentials

[Anonymous]

Originally created by: Hemantabhusal
Originally owned by: kalibetre

Summary

Two concurrent start requests for the same Agent can both succeed.

Each request generates new runtime credentials. Because the lifecycle status is
checked before any cross-process serialization, both requests may proceed to
runtime provisioning and persist different credentials.

This can leave the Kubernetes Secret containing credentials from one request
while the database contains credentials from the other.

## Expected behavior

Only one lifecycle mutation should run for an Agent at a time. A competing
request should receive HTTP 409 Conflict.

## Actual behavior

Both concurrent start requests can return HTTP 200. The Agent is recorded as
running, but its persisted credentials may not match the credentials mounted
into the runtime.

This can cause ingest or communication authentication failures even though the
Agent appears healthy.

## Reproduction

1. Create a stopped Agent.
2. Send two start requests for that Agent concurrently.
3. Allow both requests to reach runtime provisioning before either commits.
4. Observe that both requests succeed.
5. Compare the decrypted database ingest key with the key in the Kubernetes
   Secret.

In a deterministic integration reproduction, both responses were HTTP 200 and
the two keys differed.

## Root cause

Agent lifecycle state was checked before Kubernetes operations, while the
database row was locked only during the final write. That lock did not prevent
both requests from independently provisioning the runtime.

## Impact

Medium-to-high reliability impact:

• The Agent may appear running but reject authenticated runtime communication.
• It can be triggered through multiple tabs, users, direct API clients, or
  retries.
• It affects one Agent at a time and is recoverable.
• No authorization bypass or data loss has been identified..

Proposed fix

Serialize runtime-mutating operations per Agent across API processes. After
acquiring the lock, reload the Agent and recheck authorization and lifecycle
state. Return HTTP 409 Conflict when another lifecycle operation is already in
progress.

A regression test should verify that concurrent start requests cannot leave
the runtime Secret and persisted credentials out of sync.

[Anonymous]

• assigned_to: kalibetre

[Anonymous]

Originally posted by: kalibetre

This was generated by AI during triage.

Agent Brief

Category: bug
Summary: Concurrent agent-start requests can leave the persisted (database) credentials out of sync with the credentials actually mounted into the runtime (Kubernetes Secret)

Current behavior:
The agent-start lifecycle mutation checks whether the agent is already running before doing any cross-process serialization. Because that check happens before any lock is held, two concurrent start requests for the same agent can both pass the check, each generate its own fresh set of runtime credentials, and each independently provision the runtime (including writing a runtime Secret) and persist its own credentials to the database. The only row-level lock in the current flow is acquired at the very end, immediately before the final database write — by that point both requests have already generated distinct credentials and already performed their runtime-provisioning side effects, so the lock only decides which request's data "wins" in the database, not which request's data is actually running. Both requests can return a 200 success response, and the database's persisted credentials can end up not matching the credentials actually present in the runtime, causing later authenticated communication with that agent to fail even though it appears healthy.

Desired behavior:
Only one lifecycle-mutating operation (start being the case described here, but the same category of operation) should be allowed to run for a given agent at a time, enforced across API processes — not just within a single process's memory. A request that arrives while another lifecycle mutation is already in progress for that agent should be rejected with HTTP 409 Conflict rather than being allowed to proceed. The lock must be held (or an equivalent per-agent serialization guarantee must be in effect) across the entire mutating operation — from the initial eligibility/status check through runtime provisioning through the final persistence step — not just at the final write. After acquiring the lock, the agent's current lifecycle state should be reloaded and rechecked before proceeding, since it may have changed since the initial check.

Key interfaces:

• Whatever function currently performs the "start agent" (and ideally other lifecycle-mutating: stop/restart) operation needs its status-check-then-mutate sequence to become atomic per-agent across processes. This codebase already has a working example of a cross-process, per-key serialization primitive (a Postgres advisory lock, keyed by hashing an identifier) used to solve an analogous "two processes racing to allocate/mutate the same row" problem elsewhere in the domain layer — look for that existing pattern before reaching for a new dependency (e.g. Redis-based locking).
• The lifecycle mutation's response contract gains a new error case: HTTP 409 Conflict when a competing lifecycle operation for the same agent is already in flight (distinct from the existing 409 returned when the agent is already in the target state).

Acceptance criteria:

• [ ] Two concurrent start requests for the same stopped agent cannot both succeed — exactly one returns success; the other returns HTTP 409 Conflict.
• [ ] After the operation settles, the credentials persisted in the database match the credentials actually present in the runtime (e.g. the Kubernetes Secret) for that agent.
• [ ] The per-agent serialization does not block concurrent lifecycle operations on different agents (no global lock).
• [ ] A regression test exercises two genuinely concurrent start requests (real concurrency — threads/async tasks/processes racing, not two sequential calls) against the same agent and asserts both the single-success/one-409 outcome and the credential-consistency outcome.
• [ ] Existing single-request lifecycle behavior (e.g. the current "already running" 409, normal successful start) is unchanged.

Out of scope:

• Reconciling or auto-repairing agents that are already in a desynchronized state from before this fix — this brief only covers preventing new desynchronization going forward.
• Changing the credential generation scheme itself.
• Adding a new locking dependency/infrastructure component if an existing cross-process serialization primitive in this codebase already solves the problem.

Verification performed during triage: Traced the actual start-agent code path and confirmed the race is real, not hypothetical: the status check happens with no lock held, credential generation and runtime provisioning happen unsynchronized, and the only lock in the flow is acquired at the final database write — by which point it can only decide which request's data is persisted, not prevent both from having already mutated the runtime with different credentials. No existing test exercises true concurrency on this path.