Agent Substrate Files

What's Changed

⚠️ Breaking Changes

• SandboxConfig now holds multiple named versions with a default by @zoez7 in https://github.com/agent-substrate/substrate/pull/2333
• Replace SnapshotContentScope with the SnapshotFidelity ladder and preferred_fidelity by @dberkov in https://github.com/agent-substrate/substrate/pull/2364
• Add sandboxClasses with an optional SandboxConfig ref to WorkerPool by @zoez7 in https://github.com/agent-substrate/substrate/pull/2332
• gvisor: move the sandbox asset to release 20260907.0 by @ygao-g in https://github.com/agent-substrate/substrate/pull/2413
• Health probes move to their own server on :9091, and :9090 serves only /metrics by @JeffLuoo in https://github.com/agent-substrate/substrate/pull/2424
• Rename ContainerWakeupProbe to WakeupProbe by @laoj2 in https://github.com/agent-substrate/substrate/pull/2575
• Remove the substrate version label from the install by @HavenXia in https://github.com/agent-substrate/substrate/pull/1812
• Block link local egress in atunnel by @keithmattix in https://github.com/agent-substrate/substrate/pull/2319
• stop naming every cached sandbox asset runsc-<sha> by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2627
• Changed SandboxClass from enum to string to allow extensibility by @laoj2 in https://github.com/agent-substrate/substrate/pull/2586

Bug Fixes

• objectstore: finish snapshot cleanup when the bucket is missing by @cloudonly in https://github.com/agent-substrate/substrate/pull/1973
• Components honor OTEL_TRACES_EXPORTER=none and treat OTEL_METRICS_EXPORTER=prometheus as none where they serve /metrics by @QuentinBisson in https://github.com/agent-substrate/substrate/pull/2190
• Return the caller's context error when an atelet call fails by @eren23 in https://github.com/agent-substrate/substrate/pull/2236
• ateom-gvisor: attribute runsc output for the pause container by @cloudonly in https://github.com/agent-substrate/substrate/pull/2330
• Benchmark automation runners can reach ate-api-server again under ate-system's default-deny ingress by @haiyanmeng in https://github.com/agent-substrate/substrate/pull/2637
• ateapi: fail /readyz while the store is unreachable by @ericcurtin in https://github.com/agent-substrate/substrate/pull/1989
• run ateom under minimal init by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2626
• atunnel: bound the tunnel open by a deadline by @ygao-g in https://github.com/agent-substrate/substrate/pull/2654

Features: Networking and Egress

• atenet: keep the router pod unready until its Envoy is serving by @LiorLieberman in https://github.com/agent-substrate/substrate/pull/2325
• Egress gateway can mint and cache actor JWTs by @thompsonmax in https://github.com/agent-substrate/substrate/pull/2135
• Egress policies can inject actor JWTs by @thompsonmax in https://github.com/agent-substrate/substrate/pull/2141
• More robust Actor DNS Relay by @bowei in https://github.com/agent-substrate/substrate/pull/2370
• Add E2E HTTPS egress tests and reject WebSocket upgrades by @swiftdiaries in https://github.com/agent-substrate/substrate/pull/1634
• Add actor egress benchmarking tool by @yanavlasov in https://github.com/agent-substrate/substrate/pull/2505
• Reduce number of egress atunnels by multiplexing CONNECTs over H/2 by @yanavlasov in https://github.com/agent-substrate/substrate/pull/2425
• atenet-router: reach atunnel through mTLS CONNECT tunnels by default by @LiorLieberman in https://github.com/agent-substrate/substrate/pull/2618
• Router status private by @LiorLieberman in https://github.com/agent-substrate/substrate/pull/2573
• Block DNS if there is no egress policy configured by @bowei in https://github.com/agent-substrate/substrate/pull/2524

Features: Security and Identity

• atespace isolation tests by @siyanshen in https://github.com/agent-substrate/substrate/pull/1966
• security: prevent path traversal in snapshot storage locations by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2372
• atelet: introduce declarative validation for the AteomHerder RPCs by @shrutiyam-glitch in https://github.com/agent-substrate/substrate/pull/1997
• ate-system now denies ingress by default; in-cluster API clients must label their namespace ate.dev/api-access=true by @destijl in https://github.com/agent-substrate/substrate/pull/2522
• add support for connect from atenet (envoy) to atunnel by @LiorLieberman in https://github.com/agent-substrate/substrate/pull/2531

Features: Observability

• ateom-gvisor: emit actor usage records by @baizhenyu in https://github.com/agent-substrate/substrate/pull/2194
• ateom-microvm: emit actor usage records by @baizhenyu in https://github.com/agent-substrate/substrate/pull/2195
• atelet: drop its actor usage records and count each activation's CPU once by @baizhenyu in https://github.com/agent-substrate/substrate/pull/1984
• benchmarking/analysis: aggregate suspend/resume phase breakdown log r… by @Oneimu in https://github.com/agent-substrate/substrate/pull/2009
• Drop per-pod server.address label from rpc.client.call.duration metrics by @JeffLuoo in https://github.com/agent-substrate/substrate/pull/2576

Features: Workers and Actors

• fake-dataplane: let the fake data plane host actors by @haiyanmeng in https://github.com/agent-substrate/substrate/pull/2300
• Build the OCI spec in ateom instead of atelet by @HavenXia in https://github.com/agent-substrate/substrate/pull/2293
• Allow ActorTemplates to disable golden snapshot creation by @cloudonly in https://github.com/agent-substrate/substrate/pull/2328
• Benchmark external volumes with the DurDir load by @anishgang in https://github.com/agent-substrate/substrate/pull/2057
• ateom-microvm: boot the guest image over virtio-pmem with DAX by @jiayimeow in https://github.com/agent-substrate/substrate/pull/2218
• microvm: evict snapshot page cache on restore to prevent cgroup memory pressure by @chw120 in https://github.com/agent-substrate/substrate/pull/2030
• Move static runsc flags to configuration file by @fvoznika in https://github.com/agent-substrate/substrate/pull/2281
• drop wall-clock TestWaitForSocketReturnsPromptly, it flakes on loaded ci runners by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2625
• fix kind cluster delete hanging when nfs volumes are mounted by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2621

Features: Install and Operations

• benchmarking: pace boomer to a target rate and add the resumepaused user class by @haiyanmeng in https://github.com/agent-substrate/substrate/pull/2350
• ate-setup: have EnsureCRDs create only the missing CRDs by @HavenXia in https://github.com/agent-substrate/substrate/pull/2585
• Add --fake-data-plane to install a fake data plane for control-plane benchmarks by @haiyanmeng in https://github.com/agent-substrate/substrate/pull/2469

Dependencies

• Envoy, Alpine and dev-tool dependency bumps clear pre-GA CVEs by @annapendleton in https://github.com/agent-substrate/substrate/pull/2568

Documentation

• Lifecycle metrics omit empty template labels, and a crash during a delete reports the delete operation by @JeffLuoo in https://github.com/agent-substrate/substrate/pull/2356
• Upgrade Runbook GA by @HavenXia in https://github.com/agent-substrate/substrate/pull/1844

Other Changes

• Build the JWK set that publishes actor JWT verification keys by @thompsonmax in https://github.com/agent-substrate/substrate/pull/2132
• ate-setup: do not record demo and benchmark deploys by @HavenXia in https://github.com/agent-substrate/substrate/pull/2320
• ate-setup should record the Substrate version an install used by @HavenXia in https://github.com/agent-substrate/substrate/pull/2321
• Volume E2E tests wait for actor restoration before checking mounts by @cloudonly in https://github.com/agent-substrate/substrate/pull/2294
• docs(router): fix the request parking guide link by @ktsoator in https://github.com/agent-substrate/substrate/pull/2327
• Authenticate bundled PostgreSQL logins with separate pod certificates by @iplay88keys in https://github.com/agent-substrate/substrate/pull/2266
• ateapi: record one scheduler assignment for each resume by @JeffLuoo in https://github.com/agent-substrate/substrate/pull/2244
• Delete unused actor_template_ref from ActorAssigment sub-resource by @juli4n in https://github.com/agent-substrate/substrate/pull/2322
• benchmark(ateom-gvisor): log per-actor checkpoint and restore phase b… by @Oneimu in https://github.com/agent-substrate/substrate/pull/2302
• Don't default name in store layer. by @juli4n in https://github.com/agent-substrate/substrate/pull/2359
• kubectl ate admin can rotate the actor JWT signing key by @thompsonmax in https://github.com/agent-substrate/substrate/pull/2140
• cdiinject: confine device paths and host mounts to allowed roots by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2315
• Add sections about sub-resources and singletons to the API style guide. by @juli4n in https://github.com/agent-substrate/substrate/pull/2354
• ate-setup: choose the StorageClass of the bundled PostgreSQL by @ericcurtin in https://github.com/agent-substrate/substrate/pull/2314
• Move snapshot transfers behind the object-store snapshot plugin by @ILLUM1N0X in https://github.com/agent-substrate/substrate/pull/2224
• Fail snapshot plugin calls with Unavailable after 30s when the plugin is down by @aramase in https://github.com/agent-substrate/substrate/pull/2380
• docs(api): fix systeminfo volume examples by @ktsoator in https://github.com/agent-substrate/substrate/pull/2346
• ateom-gvisor logs a warning when a running actor's sandbox dies by @git286 in https://github.com/agent-substrate/substrate/pull/2258
• Update Snapshots API to support Actor Lifecycle v2 by @msau42 in https://github.com/agent-substrate/substrate/pull/2289
• Reject requests with duplicate ate-target-actor headers across HTTP and CONNECT by @destijl in https://github.com/agent-substrate/substrate/pull/2308
• docs: add substrate security guide and principles by @destijl in https://github.com/agent-substrate/substrate/pull/2326
• Rename some ExternalVolume fields by @juli4n in https://github.com/agent-substrate/substrate/pull/2404
• kubectl-ate: add --source to select container or lifecycle log lines by @erain in https://github.com/agent-substrate/substrate/pull/2337
• ate-setup sets the actor JWT issuer (ACTOR_JWT_ISSUER), and ate-api-server requires it by @thompsonmax in https://github.com/agent-substrate/substrate/pull/2257
• Rename target_ateom_uid to worker_pod_uid by @laoj2 in https://github.com/agent-substrate/substrate/pull/2423
• Rename MintActorJWTRequest.audience to audiences. by @juli4n in https://github.com/agent-substrate/substrate/pull/2421
• Remove makefile by @ahmedtd in https://github.com/agent-substrate/substrate/pull/2374
• Rename the object-store plugin proto package to objectstore.v1 by @aramase in https://github.com/agent-substrate/substrate/pull/2432
• podcertcontroller: do not rate-limit PCRs assigned to other replicas by @chw120 in https://github.com/agent-substrate/substrate/pull/2353
• Add an eviction section to the API guide by @HavenXia in https://github.com/agent-substrate/substrate/pull/2417
• Worker Self-Reported Sandbox & Hardware Compatibility by @shrutiyam-glitch in https://github.com/agent-substrate/substrate/pull/2381
• Update COLLABORATING.md to reflect our stronger processes by @ahmedtd in https://github.com/agent-substrate/substrate/pull/2437
• atenet-router no longer exposes its plaintext xDS server on the pod IP by @destijl in https://github.com/agent-substrate/substrate/pull/2443
• cmd/atenet: Remove the router's "all" mode by @ahmedtd in https://github.com/agent-substrate/substrate/pull/2441
• test: Fix macos unit test failures by @timflannagan in https://github.com/agent-substrate/substrate/pull/2429
• Update x/net to v0.60.0 to fix govulncheck failures by @cloudonly in https://github.com/agent-substrate/substrate/pull/2401
• ROOTFS snapshot fidelity: micro-VM capture and restore, API opened for micro-VM templates by @dberkov in https://github.com/agent-substrate/substrate/pull/2448
• fix(kubectl-ate): validate atespace flags before connecting by @ktsoator in https://github.com/agent-substrate/substrate/pull/2464
• atunnel: strip X-Ate-Assignment-Stale from upstream actor responses by @panchoruy in https://github.com/agent-substrate/substrate/pull/2410
• WorkerPools resolve their SandboxConfig and protect it from deletion by @zoez7 in https://github.com/agent-substrate/substrate/pull/2393
• Simplify Snapshot API for Actor Lifecycle v2 by @msau42 in https://github.com/agent-substrate/substrate/pull/2449
• Singleton subresource Get and Delete requests take a name by @ericcurtin in https://github.com/agent-substrate/substrate/pull/2452
• Fix a couple issues with plural commands in kubectl-ate by @thockin in https://github.com/agent-substrate/substrate/pull/2509
• e2e: skip the egress credential injection test by @yufan-su in https://github.com/agent-substrate/substrate/pull/2518
• atecontroller: add --worker-sync-concurrency flag by @chw120 in https://github.com/agent-substrate/substrate/pull/2334
• ate-api-server no longer links the GCS and S3 SDKs by @ericcurtin in https://github.com/agent-substrate/substrate/pull/2398
• Make DrainWorker return DrainWorkerResponse by @juli4n in https://github.com/agent-substrate/substrate/pull/2501
• ate-setup delete ate-system now deletes the ClusterTrustBundles Substrate published by @aramase in https://github.com/agent-substrate/substrate/pull/2418
• Update gVisor to 2026-10-10 nightly and use fs/ split checkpoint for durable dirs by @nybidari in https://github.com/agent-substrate/substrate/pull/2455
• ate-setup: roll ate-api-server once when its environment changes by @HavenXia in https://github.com/agent-substrate/substrate/pull/2371
• e2e: update credential injection e2e to call an in-cluster echo server by @yufan-su in https://github.com/agent-substrate/substrate/pull/2535
• Fetch private sandbox assets through the object-store plugin by @aramase in https://github.com/agent-substrate/substrate/pull/2439
• router authn/z part 1: Add simple mTLS client authentication to the ingress router by @ahmedtd in https://github.com/agent-substrate/substrate/pull/2387
• ateapi: [Part 3] authorize actor lifecycle, egress policy, and worker RPCs with OpenFGA by @SHRUTI6991 in https://github.com/agent-substrate/substrate/pull/2185
• Move the OIDC JWT verifier to internal/ so binaries other than ate-api-server can use it by @ILLUM1N0X in https://github.com/agent-substrate/substrate/pull/2519
• Rename the snapshot-plugin sidecar, image and flag to objectstore-plugin by @aramase in https://github.com/agent-substrate/substrate/pull/2540
• Report missing Kind and control-plane prerequisites before deployment by @cloudonly in https://github.com/agent-substrate/substrate/pull/2403
• fix(objectstorage): handle download reader close errors by @ktsoator in https://github.com/agent-substrate/substrate/pull/2446
• atecontroller: the :8080 metrics listener follows OTEL_METRICS_EXPORTER by @ericcurtin in https://github.com/agent-substrate/substrate/pull/1976
• Use TitleCase for initialisms in API proto names. by @juli4n in https://github.com/agent-substrate/substrate/pull/2520
• hack: warn when the host's neighbor table is at the kernel default by @dims in https://github.com/agent-substrate/substrate/pull/2497
• Credential provider tests no longer fail with text file busy by @ericcurtin in https://github.com/agent-substrate/substrate/pull/2549
• benchmarking: append the suspend/resume phase percentiles to stats.jsonl from the runner by @Oneimu in https://github.com/agent-substrate/substrate/pull/2148
• Align timestamp fields with AIP-142 naming conventions by @ganeshkumarashok in https://github.com/agent-substrate/substrate/pull/2438
• e2e: fetch in-cluster echo origins in the egress MITM suite by @dims in https://github.com/agent-substrate/substrate/pull/2561
• cmd/atelet: Join the pause-image pull in the prewarm download test by @dims in https://github.com/agent-substrate/substrate/pull/2564
• Fix ETXTBSY test flake in credentialprovider by @BenTheElder in https://github.com/agent-substrate/substrate/pull/2365
• ate-setup: refuse to point an external-database install at the bundled one by @HavenXia in https://github.com/agent-substrate/substrate/pull/2588
• ateapi: include actor identity in ateom certificates by @christian-posta in https://github.com/agent-substrate/substrate/pull/2582
• Add and use a CLI verbosity flag by @thockin in https://github.com/agent-substrate/substrate/pull/2475
• docs: document demo & benchmark install settings with their demos by @stshrive in https://github.com/agent-substrate/substrate/pull/2572
• Update atelet Snapshot API for to support asynchronous uploads by @msau42 in https://github.com/agent-substrate/substrate/pull/2592
• atelet: preserve tar entry mtimes when unpacking image layers by @jiayimeow in https://github.com/agent-substrate/substrate/pull/2216
• e2e: Update egress e2e to use in-cluster test server and remove every external dependencies by @yufan-su in https://github.com/agent-substrate/substrate/pull/2590
• Add WorkerService.ReportActorSnapshotState API for atelet snapshot reports by @igooch in https://github.com/agent-substrate/substrate/pull/2612
• ateapi: grant system component roles to Substrate's own components by @SHRUTI6991 in https://github.com/agent-substrate/substrate/pull/2220
• Wrap page size and next page token in a ListOptions message. by @juli4n in https://github.com/agent-substrate/substrate/pull/2593
• atelet prepares each local worker's sandbox assets and versions file by @zoez7 in https://github.com/agent-substrate/substrate/pull/2521
• Define a paused actor by its snapshots and delete the PAUSED and PAUSING states by @zoez7 in https://github.com/agent-substrate/substrate/pull/2620
• benchmarking: add the personal-assistant-light agent-session script by @aditya-shantanu in https://github.com/agent-substrate/substrate/pull/2583
• Mark workers unavailable if it's pod status changes by @sfunkenhauser in https://github.com/agent-substrate/substrate/pull/2055
• Stamp snapshots with SandboxRuntime and schedule restores on runtime-compatible workers by @shrutiyam-glitch in https://github.com/agent-substrate/substrate/pull/2536
• csi: require serverName when TLS is enabled. by @lekhoavu2909 in https://github.com/agent-substrate/substrate/pull/2603
• WorkerService.MintAteomActorJwt mints ateom-for-actor JWTs for atelet; actor JWT claims move from ate.dev to actor.ate.dev by @ILLUM1N0X in https://github.com/agent-substrate/substrate/pull/2523
• Move nighthawk benchmark server out of ate-system namespace by @yanavlasov in https://github.com/agent-substrate/substrate/pull/2656
• Add proto message options to attach resource metadata. by @juli4n in https://github.com/agent-substrate/substrate/pull/2454
• benchmarking: full request history, histogram restore/checkpoint, active_actors by @Nishanth29 in https://github.com/agent-substrate/substrate/pull/2221
• fix(atelet): fall back to anonymous GCS read with objectstore-plugin Internal by @annapendleton in https://github.com/agent-substrate/substrate/pull/2614
• atelet: Prefer local snapshot on Restore by @msau42 in https://github.com/agent-substrate/substrate/pull/2658
• Fix macOS DNS relay tests by @swiftdiaries in https://github.com/agent-substrate/substrate/pull/2667
• ateapi: [Part 4] deny RPCs without an authz rule and authorize tags by @SHRUTI6991 in https://github.com/agent-substrate/substrate/pull/2562
• Consolidate API validation for sandbox class across services by @laoj2 in https://github.com/agent-substrate/substrate/pull/2587

New Contributors

• @eren23 made their first contribution in https://github.com/agent-substrate/substrate/pull/2236
• @siyanshen made their first contribution in https://github.com/agent-substrate/substrate/pull/1966
• @erain made their first contribution in https://github.com/agent-substrate/substrate/pull/2337
• @panchoruy made their first contribution in https://github.com/agent-substrate/substrate/pull/2410
• @ganeshkumarashok made their first contribution in https://github.com/agent-substrate/substrate/pull/2438
• @christian-posta made their first contribution in https://github.com/agent-substrate/substrate/pull/2582
• @jiayimeow made their first contribution in https://github.com/agent-substrate/substrate/pull/2218
• @fvoznika made their first contribution in https://github.com/agent-substrate/substrate/pull/2281
• @lekhoavu2909 made their first contribution in https://github.com/agent-substrate/substrate/pull/2603

Full Changelog: https://github.com/agent-substrate/substrate/compare/v0.4.0...v0.5.0

Committers in this release

• @aditya-shantanu
• @ahmedtd
• @aiqubits
• @anishgang
• @annapendleton
• @aramase
• @baizhenyu
• @BenTheElder
• @bowei
• @christian-posta
• @chw120
• @cloudonly
• @dberkov
• @destijl
• @dims
• @erain
• @eren23
• @ericcurtin
• @fvoznika
• @ganeshkumarashok
• @git286
• @haiyanmeng
• @hajiler
• @HavenXia
• @igooch
• @ILLUM1N0X
• @iplay88keys
• @JeffLuoo
• @jiayimeow
• @juli4n
• @keithmattix
• @ktsoator
• @laoj2
• @lekhoavu2909
• @LiorLieberman
• @msau42
• @Nishanth29
• @nybidari
• @Oneimu
• @panchoruy
• @QuentinBisson
• @sfunkenhauser
• @SHRUTI6991
• @shrutiyam-glitch
• @siyanshen
• @stshrive
• @swiftdiaries
• @thockin
• @thompsonmax
• @timflannagan
• @yanavlasov
• @ygao-g
• @yufan-su
• @zoez7