v1.5.4 (2026-08-19)

v1.5.4 is the final release of the 1.5 line. It is a stability release: it backports fixes for a webhook certificate-renewal outage that could block all pod creation, a KubeVirt VMI validation failure, several release-blocking crashloops (velero, cert-manager, SeaweedFS, flux-shard-operator), a silent PostgreSQL restore data-integrity bug, and a raft of smaller reliability and CI fixes accumulated on the branch. It also closes the SeaweedFS 4.31 rename fallout on the 1.5.x line and pins the CAPI kubeadm bootstrap objects that a later upgrade to 1.6 would otherwise prune — both of which need an operator to act, and both of which are covered in the section below.

These notes are measured against v1.5.2, not v1.5.3. v1.5.3 was tagged but its GitHub Release was left a draft and never published, so no user ever received it and every operator upgrading arrives from v1.5.2. Comparing against v1.5.3 would silently drop four commits — two user-facing fixes — that nobody has seen in a release. The range is v1.5.2..v1.5.4, 82 commits across 27 pull requests.

⚠️ Breaking Changes and Required Actions

There are no breaking API or values changes in v1.5.4. There are two things that need an operator, and both of them can cost data or wedge an upgrade if they are skipped. Read this section in full before applying the v1.5.4 Platform Package.

Pre-upgrade checks

Run these against the management cluster before upgrading.

1. SeaweedFS 4.31 rename — classify every instance, and re-run the audit even if you have run it before.

Cozystack v1.5.0 bumped the vendored SeaweedFS chart from 4.0.405 to 4.31.0. Before 4.31 the chart named its workloads after the chart (seaweedfs-master, seaweedfs-filer, seaweedfs-volume), ignoring the Helm release name. 4.31 names them after the release, and the data-plane release is <name>-system, so every StatefulSet wanted to become seaweedfs-system-*. StatefulSet names are immutable, so Helm could not rename in place — it stood up a second, duplicate set beside the running one.

What that duplicate does depends on the cluster. With as many nodes as master replicas the new masters cannot schedule (hard pod anti-affinity against the old ones), so the duplicate sits Pending/CrashLoopBackOff and the original keeps serving. With more nodes than masters the new, empty set comes up — and because both sets carry identical pod labels, the seaweedfs-s3 Service load-balances across them while both filers write to the same seaweedfs-db Postgres metadata store pointing at different volume servers. That is a data-integrity incident, not a cosmetic duplicate: reads of existing objects through the new endpoint miss, new writes land on empty volumes, and two master sets hand out volume IDs from independent sequences into one shared metadata table.

Separately, and landing on the same upgrade, the v1.5.0 database split moved the CNPG Cluster/seaweedfs-db — the filer metadata store, i.e. the index for every object in the tenant's S3 — out of the <name>-system release into its own <name>-db release. Migration 43 shipped comparing the owning release against the literal string seaweedfs-system, so it only ever fired for an instance named seaweedfs; an instance named anything else was skipped and had its Cluster pruned as a removed resource, with CNPG taking the PVC along with it. That prune is not a one-shot: Helm computes deletions by diffing the last deployed revision against the new manifest, so a tenant whose <name>-system last succeeded on a pre-split revision recomputes the same deletion on every upgrade attempt, including attempts that fail for unrelated reasons.

v1.5.4 pins fullnameOverride: seaweedfs in system/seaweedfs, so workloads are named after the chart exactly as they were before 4.31 and upgrading adopts the running set and its volumes in place. Migration 43 is fixed to match the -system suffix, and migration 45 re-runs the hand-over for clusters that already ran the hardcoded version. Two states cannot be adopted that way, and the chart fails the render rather than guess — the enforcing guard is packages/system/seaweedfs/templates/naming-guard.yaml, with a sibling copy in extra/seaweedfs so the refusal is visible on the SeaweedFS application itself.

Step 0 — seaweedfs-db ownership (read-only, do this first). This one destroys data rather than duplicating it, so clear it before anything else.

:::sh
kubectl get cluster.postgresql.cnpg.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.annotations.meta\.helm\.sh/release-name,KEEP:.metadata.annotations.helm\.sh/resource-policy'

Read the rows where NAME is seaweedfs-db:

OWNER	KEEP	Meaning
<name>-db	keep	Handed over. Nothing to do.
<name>-db	(none)	Installed fresh on ≥ v1.5.0. Safe — <name>-system never rendered the Cluster, so it is not in that release's prune baseline.
<name>-system	(none)	At risk. Migration 45 hands it over on the next platform upgrade. Do not reconcile <name>-system before the migration runs.
(no row at all)		Already lost. The metadata index is gone: that tenant's S3 returns 500 and its objects are unreachable even though the volume PVCs still hold the bytes. No migration can rebuild it — restore the seaweedfs-db Postgres from a backup, or treat that tenant's object storage as lost. Note that <name>-db may still report Ready while this is true; trust the kubectl get cluster output, not the HelmRelease status.

Step 1 — classify every SeaweedFS instance (read-only, mutates nothing).

:::sh
hack/seaweedfs-naming-audit.sh                 # whole cluster
hack/seaweedfs-naming-audit.sh tenant-foo      # or named namespaces

CLASS	State	Action
L	Only the chart-named generation is present.	None. The upgrade adopts it in place.
S	Only the release-named generation is present — the instance was installed fresh on 1.5.x, and its data lives on data1-seaweedfs-system-volume-* PVCs.	Re-bind those volumes onto the chart-named PVC names before upgrading. Pinning the chart name without that renames the workloads away from the data, and Helm cannot move data between PVCs. Runbook Step 2.
MIXED	Both generations are present.	One is an empty duplicate and one holds the data, and nothing durable in the object graph says which — so the chart refuses. Classify the tenant and delete the empty generation so exactly one remains; the render then adopts the survivor with no further action. Runbook Step 1, then 2a or 3.

Read the exit code, not just the table. The audit fails closed: a kubectl call that fails, or a Helm release payload it cannot decode, prints FATAL and exits non-zero, and the partial table must not be trusted. Only exit 0 means the table is the whole answer — and an empty table with exit 0 is a genuinely clean fleet.

If you have already run this audit, run it again on v1.5.4. The version of the script that shipped in v1.6.0 silenced every kubectl failure with 2>/dev/null, so a timeout or an RBAC denial produced an empty, "all clean" table byte-identical to an honestly clean fleet — a false clean, on the script whose output gates a runbook step that deletes PVCs. v1.5.4 is the first release on the 1.5 line to carry the audit at all, and it carries the fail-closed version (v1.6.1 and later carry it too). A clean result from a v1.6.0 checkout, or from main between 2026-07-20 and 2026-07-28, is not evidence of anything.

One unrelated filer change lands on the same upgrade and is worth knowing about while you are looking at this: the filer's postgres2 connection pool to that same seaweedfs-db metadata store was unconfigured, so every metadata lookup opened a fresh PostgreSQL connection and added seconds of latency to every S3 request. That is fixed here too (see the postgres2 connection pool entry below), and it needs no operator action — but if you have been treating slow S3 as a symptom of the rename, it may well have been this instead.

Recovery for S and MIXED is docs/operations/seaweedfs-431-rename-recovery.md. Do not guess which generation holds the data — the runbook exists because a duplicate that briefly served writes and later crashed is indistinguishable, on every durable signal, from one that never scheduled. A tenant that went 1.4.x straight to 1.6 never renamed and is unaffected by any of this; duplicates exist only on tenants that passed through 1.5.x.

2. The platform migration targetVersion moves from 45 to 46 — which changes what a later upgrade to 1.6 runs.

v1.5.4 is the first 1.5.x release stamped targetVersion: 46; v1.5.0 through v1.5.3 were all stamped 45. run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster that reaches 46 and later upgrades to v1.6 (targetVersion: 54) runs slots 46 through 53 — and never executes 1.6's own slot 45. This is a skip, not an ordering problem.

Slot 45 does not hold the same thing on both branches. On release-1.5 it is the SeaweedFS seaweedfs-db hand-over repair described above. On main and release-1.6 it is the pin that stamps helm.sh/resource-policy: keep onto the CAPI KubeadmConfigTemplate objects. 1.6 drops KubeadmConfigTemplate from the tenant kubernetes chart entirely — workers move to TalosConfigTemplate — so on that upgrade Helm sees the object in the previous release manifest, absent from the new one, and deletes it while the kubeadm-backed MachineSet is still mid-rollover with its bootstrap.configRef pointing at it. controller-manager then floods with reconcile errors, and where the Talos image fetch is slow or a MachineHealthCheck remediates, workers can hang pending with nothing to bootstrap from. Tenant Kubernetes only, and a noisy broken rollover rather than data loss — but it is not self-healing.

v1.5.4 closes this in two places, covering two disjoint populations, and dropping either would leave a real hole:

• Existing clusters (stamped 45 or lower) pick the pin up from the migration: release-1.5's slot 45 now runs the keep-pin after the SeaweedFS repair, so the annotation is already in place by the time a later 1.6 upgrade skips 1.6's slot 45. The SeaweedFS half runs first deliberately — a missed hand-over loses a tenant's filer metadata, a missed pin is recoverable by hand — and both halves fail closed ahead of the version stamp, so a half-completed attempt is safe and the Job retries the whole slot. The pin selects on the app.kubernetes.io/managed-by=Helm label that Helm injects, so the KubeadmConfig children CAPI spawns from the template (owned by their Machine, never pruned by Helm) are correctly left alone. It is idempotent: an object already carrying keep is skipped without a write.

• Fresh v1.5.4 installs cannot be reached by any migration, so the chart covers them instead. The platform chart renders the cozystack-version ConfigMap directly at targetVersion when it does not already exist, and the migration hook emits its Job only when that ConfigMap is already present — so a fresh install is stamped 46 having never run slot 45, or any other slot. The kubernetes chart therefore stamps helm.sh/resource-policy: keep on the KubeadmConfigTemplate at render time, so the object is born pinned no matter which path a cluster took to get there.

Verify after upgrading:

:::sh
# Should print 46.
kubectl get configmap cozystack-version -n cozy-system -o jsonpath='{.data.version}{"\n"}'

# Every row should show keep.
kubectl get kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io -A \
  -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,KEEP:.metadata.annotations.helm\.sh/resource-policy'

A KubeadmConfigTemplate row showing <none> means the pin did not land on that object, and a later 1.6 upgrade will prune it out from under a live MachineSet. Recovering from that means recreating the template by hand with the right Helm ownership metadata, so fix it before upgrading to 1.6.

Manual actions required

• BEFORE upgrading — recover SeaweedFS tenants classified S or MIXED. The chart refuses to render for them, which blocks the tenant's upgrade until an operator resolves it. Run hack/seaweedfs-naming-audit.sh and follow docs/operations/seaweedfs-431-rename-recovery.md.

• BEFORE trusting an earlier audit result — re-run the audit from v1.5.4. The v1.6.0 copy of the script could report a false clean (see pre-upgrade check 1).

• AFTER upgrading — confirm the migration stamp is 46 and every KubeadmConfigTemplate carries helm.sh/resource-policy: keep (see pre-upgrade check 2). This is what makes the later 1.6 upgrade safe.

• No action, but expect one rollout each: the cert-manager webhook (it moves off port 10250) and cainjector (raised memory limit), the kube-ovn webhook, velero (new startupProbe), and the CNPG instances of every Postgres release with backup.enabled or bootstrap.enabled, whose rendered Cluster gains one spec.env entry for the S3 checksum fix.

Fixes

• fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal: kube-ovn-webhook only loaded its TLS certificate once at startup. Once cert-manager rotated the backing Secret and the webhook pod outlived the old certificate's expiry (roughly a year after install), the pod kept serving the expired certificate, kube-apiserver rejected the TLS handshake, and — because the MutatingWebhookConfiguration uses failurePolicy: Fail — every pod creation in tenant namespaces was blocked, including virt-launcher, so no VM could start. The webhook now reloads its certificate from disk on renewal through a GetCertificate closure, with renewBefore widened to 720h. The reload path is hardened against the cases that would have made it unreliable in practice: the certificate file's mtime is observed before the key pair is read, so a Secret swap racing the read is retried on the next handshake rather than cached under a newer mtime and missed; change is detected by mtime inequality rather than strictly-newer, catching equal-mtime replacements and backward clock steps; a load attempt is recorded whatever the outcome, so a persistently unreadable file is retried once its mtime advances instead of on every handshake; and stat failures are logged rather than swallowed, so a broken mount is visible before expiry. The same change also bounds the webhook's http.Server with ReadHeaderTimeout, ReadTimeout, WriteTimeout and IdleTimeout — previously it set none, so a client that opened a connection and sent headers slowly could hold a goroutine and a file descriptor indefinitely (@IvanHunters in [#3557], backport [#3879]).

• fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy: The cloned per-shard helm-controller inherited HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the flux-aio container and lacked a startupProbe, so on clusters behind a corporate HTTP proxy the shard could not reach the (proxy-unreachable) in-cluster API server and crashlooped before ever becoming ready. Proxy environment variables are now stripped from the cloned container and a startupProbe is added so slow starts are tolerated instead of killed. The cloned probe inherits its timeoutSeconds from the liveness handler rather than being forced to 1s, so a future flux-aio shipping a larger liveness timeout cannot end up with a stricter startup probe and recreate the very crashloop this fixes (@IvanHunters in [#3546], backport [#3883]).

• fix(velero): add startupProbe so slow startup does not crashloop the install gate: Velero binds its health endpoint only after loading plugins and connecting to the API server, which can take minutes under a heavy parallel platform install — long enough to trip the liveness probe and crashloop the pod before the install-readiness gate ever passes. A startupProbe now defers liveness checks until the server has actually finished starting (@lexfrei in [#3138], backport [#3533]).

• fix(cert-manager): move the webhook off the kubelet's port: The cert-manager webhook listened on port 10250, the kubelet's own port. When the webhook Service resolved to a node IP instead of the pod IP, the connection landed on the kubelet, which answered with its own certificate, and the API server rejected every cert-manager admission call cluster-wide. The webhook now listens on a port the kubelet does not own (@lexfrei in [#3359], backport [#3366]).

• fix(cert-manager): raise cainjector memory limit to unblock caBundle injection: cainjector loads every webhook configuration, APIService, and CRD into its informer caches at startup; on a full Cozystack install that working set exceeded the 128Mi memory limit, so the single leader-elected cainjector pod was OOMKilled during cache population and crashlooped before it could inject any CA bundle, breaking every webhook that depends on cert-manager-issued CAs. The memory limit is raised so cache population can complete (@IvanHunters in [#3199], backport [#3202]).

• fix(mariadb): widen startup probe budget so bootstrap cannot be killed: The MariaDB operator built its startup probe without a failureThreshold, so it fell back to the Kubernetes default of 3 — giving a fresh bootstrap only ~40 seconds before being killed by the liveness probe the startup probe exists to defer. The budget is widened so a legitimately slow first bootstrap is no longer mistaken for a hung container (@lexfrei in [#3344], backport [#3364]).

• fix(seaweedfs): make naming audit fail closed on kubectl and payload errors: hack/seaweedfs-naming-audit.sh, used as the pre-upgrade gate before PVC deletion, silenced every kubectl failure with 2>/dev/null, so a transient API error produced an empty, "all clean" table indistinguishable from an honestly clean fleet. Every enumeration now routes through one fail-closed helper that names the failed query on stderr and exits non-zero, and a Helm release Secret whose payload cannot be decoded is fatal rather than a silent skip. The same change fixes a false clean reachable with no corruption at all: the chart-name extraction was a greedy (last-match) sed, and since Helm marshals the release's config after chart, a values subtree spelling chart.metadata.name shadowed the real chart name — the release then read as non-SeaweedFS and the tenant vanished from the report. It now takes the first match, folds newlines so a pretty-printed payload parses at all, and says which path and shape it expected when it cannot read one (@myasnikovdaniil in [#3436], backport [#3878]).

• fix(seaweedfs): close the 4.31 rename fallout on the 1.5.x→1.6 upgrade path: Cozystack v1.5.0 bumped the vendored SeaweedFS chart to 4.31.0, which renames workloads after the Helm release name; because StatefulSet names are immutable, upgrading through the 1.5.x line stood up a second, empty StatefulSet beside the running one instead of adopting it. This closes the remaining gaps in the adoption path: system/seaweedfs pins fullnameOverride: seaweedfs so the upgrade adopts the running workloads and volumes in place, the naming guard moves into the chart a platform upgrade actually re-renders and refuses when both naming generations exist, cluster-scoped COSI RBAC is named per namespace again (the 4.31 release-based names collided across tenants), and the seaweedfs-db hand-over now runs for every instance name — previously an instance not named seaweedfs had its filer metadata database pruned on upgrade. Migration 45 repairs clusters that already ran the old hand-over, and hack/seaweedfs-naming-audit.sh plus docs/operations/seaweedfs-431-rename-recovery.md guide classification and recovery. The refusal is expected for tenants that passed through 1.5.x — see the required-actions section above before upgrading (@myasnikovdaniil in [#3339], backport [#3370], building on the adopt-in-place pin from [#3282]).

• fix(seaweedfs): configure postgres2 connection pool for the filer: The vendored SeaweedFS chart ships connection-pool settings only for the mysql store, which Cozystack does not use. The postgres2 store, which is enabled, had none — so the Go SQL pool kept zero idle connections and every filer metadata lookup opened a fresh PostgreSQL connection (TCP + TLS + SCRAM, roughly 300ms each). At around five path lookups per S3 operation that added about two seconds of latency to every S3 request, regardless of cluster load. Measured on a 12-node cluster with two filers: PUT of a 4KiB object went from 10s to 0.28s (p50) and HEAD from 2.6s to 0.16s. max_open is capped at 40 so two filer replicas stay under CNPG's default max_connections of 100. This fix was tagged in v1.5.3, which was never published, so v1.5.4 is the first release to carry it (@mattia-eleuteri in [#2906], backport [#3194]).

• fix(cozystack-basics): gate the hostname VAP policies on the VAP API: The route/gateway/ingress hostname ValidatingAdmissionPolicy templates rendered unconditionally. On a cluster missing the VAP API, an operator-generated HelmRelease with drift detection off would render the policies out on first install and never add them back. The templates are now gated on .Capabilities.APIVersions.Has, so installs on VAP-less clusters no longer silently drop hostname validation (@lexfrei in [#3409], backport [#3876]).

• fix(keycloak-configure): patch HelmRelease in release namespace on teardown: The keycloak-configure pre-delete teardown Job cleared the Flux HelmRelease finalizers with a kubectl patch aimed at a hardcoded namespace that didn't match the release's actual cozy-keycloak namespace, so the teardown Job's own RBAC could never reach the object it needed to patch and uninstalling the release could get stuck. The patch now targets the correct release namespace (@lexfrei in [#3372], backport [#3877]).

• fix(platform): forward backupStorage overrides to the backupstrategy-controller Package: The documented admin override for cozy-default backup S3 coordinates (spec.components.backupstrategy-controller.values on the platform Package) had no effect, because the cozystack-platform PackageSource exposes only a single platform component and silently ignored the override block, so admin-configured backup storage settings never reached the running backup controller. The platform chart now forwards the override through correctly (@androndo in [#3333], backport [#3402]).

• fix(backups): request S3 checksum only when required for barman-cloud (non-AWS S3 / Ceph RGW): Since botocore ~1.36 (early 2025) the default RequestChecksumCalculation is when_supported, so barman-cloud's boto3 attaches a flexible checksum to every PutObject. AWS S3 accepts it, but several S3-compatible backends — Ceph RADOS Gateway, the platform's own SeaweedFS system bucket, some MinIO and Cloudflare R2 builds — reject it with InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, failing every backup and WAL-archive upload. AWS_REQUEST_CHECKSUM_CALCULATION=when_required is now set through the CNPG Cluster's spec.env, which reaches the instance pods and therefore the barman-cloud subprocess the instance manager execs. On this branch that one setting covers all three S3 paths at once: the chart-rendered legacy spec.backup.barmanObjectStore, the same field SSA-patched by the CNPG backup driver in the useSystemBucket flow, and externalClusters recovery. Note this is a release-1.5-native equivalent of the fix on main, not a cherry-pick — main sets spec.instanceSidecarConfiguration.env on barmancloud.cnpg.io ObjectStore objects, and release-1.5 has no barman-cloud plugin and no ObjectStore CRD, so those objects would template cleanly and then apply to nothing. It is gated on backup.enabled or bootstrap.enabled, so a Postgres release with no S3 configured is untouched (@androndo in [#3417], backport [#3882]).

• fix(objectstorage-controller): converge BucketClaim readiness and speed up COSI provisioner failover: The COSI control plane (built from upstream container-object-storage-interface v0.2.2) never watched Bucket objects and dropped no-op resync deltas, so a BucketClaim whose backend Bucket became ready after the first reconcile stayed bucketReady=false forever, and the consuming application never got its BucketAccess. The controller now requeues until the backend Bucket is ready, emits a Warning event while it waits, and shortens the COSI leader-election lease so failover is faster (@lexfrei in [#3034], backport [#3532]).

• fix(tenant): inherit full ancestor label chain from parent namespace: Tenant namespace labelling only derived one ancestor level from .Release.Namespace, so a direct child of tenant-root never received the labels needed for its grandparent's <ancestor>-egress CiliumClusterwideNetworkPolicy to allow traffic, silently breaking cross-tenant egress for nested tenants more than one level deep. Namespaces now inherit the full ancestor label chain from their parent (@IvanHunters in [#2912], backport [#3191]).

• fix(backups/postgres): purge stale recovery cluster on repeat in-place restore: A repeat in-place PostgreSQL RestoreJob into a target that had already been restored once silently reported Succeeded while doing nothing — the PVC was untouched, no recovery pods appeared, and the database kept its pre-restore contents, a silent data-integrity failure. The restore path now purges the stale recovery cluster before restoring again, so a repeat restore actually recovers data instead of quietly no-opping. The freshness discriminator that decides whether a leftover Cluster belongs to this restore or a previous one is deliberately strict (creationTimestamp > StartedAt), so an exact timestamp tie resolves to "not fresh" and the caller purges — the conservative direction, since a genuinely fresh Cluster is always created well after StartedAt (@IvanHunters in [#3318], backport [#3321]).

• fix(fluxcd): omit empty distribution.artifact in FluxInstance for guest clusters: Guest Kubernetes clusters with the fluxcd addon enabled never became Ready, because the flux-instance template rendered spec.distribution.artifact unconditionally (unlike its sibling optional fields), and the empty string broke the intended air-gapped default of using the operator's embedded manifests. The field is now guarded like its siblings, so guest clusters with fluxcd enabled converge to Ready again (@IvanHunters in [#3284], backport [#3292]).

• fix(dashboard): unbreak CORS on expired session for k8s API calls: The dashboard SPA broke with CORS errors once the kc-access cookie expired: the Keycloak client's webOrigins was never set (so it stuck on a stale hostname after any rootUrl change), and oauth2-proxy mishandled the expired-session case for Kubernetes API calls. Both are fixed so an expired session no longer surfaces a broken, CORS-blocked SPA (@IvanHunters in [#2788], backport [#3291]).

• fix(apps/vpn): remove invalid foo field from urls Secret: A leftover debug foo field in the VPN app's <release>-urls Secret template is not part of the core/v1 Secret schema. Client-side apply tolerated it, but server-side apply's stricter validation rejected the object outright, so any VPN application upgrade failed and the release was left in a broken state. The stray field is removed (@IvanHunters in [#3281], backport [#3290]).

• fix(kubevirt): update KubeVirt to v1.8.4: Updates the vendored kubevirt-operator from v1.8.2 to v1.8.4, backporting the VMI checksum status-field validation fix (uint32 range) that Kubernetes 1.36 requires — without it, strict CRD numeric-format validation could leave VMIs stuck in Scheduled and unable to start. The update also carries assorted upstream bug fixes and CVE remediations; the KubeVirt custom resource itself is unchanged (@lexfrei in [#2940], backport [#3285]).

• fix(kubevirt-instancetypes): restore persistent EFI/TPM state: v1.5.1 stripped persistent: true from both preferredEfi and preferredTPM on the six shipped Windows preferences — windows.11, windows.2k22, windows.2k25 and their .virtio variants — to unblock live migration. Secure Boot and the vTPM stayed present, so the VMs still booted, but their state was no longer persisted: EFI NVRAM and vTPM contents were discarded on every VM restart, so anything the guest sealed to the vTPM (BitLocker being the common case) or wrote to NVRAM — Secure Boot key enrollment, boot entries — did not survive a power cycle. This affects anyone running Windows 11, Server 2022 or Server 2025 guests created from the shipped preferences on v1.5.1 or v1.5.2; v1.5.0 had persistence, and it is restored here. The strip was motivated by a real-looking concern that turned out to rest on an outdated premise — that the persistent-state-for-<vm> backend-storage PVC, ReadWriteOnce on the default replicated StorageClass, pins the VM to its node and blocks live migration and node drains, stalling cluster upgrades under evictionStrategy: LiveMigrate. KubeVirt has in fact migrated RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): it creates a fresh target state PVC and copies the small state blob during migration. Verified on the default replicated storage that a persistent-firmware VM reports LiveMigratable=True and gets its copy-on-target PVC, so persistence and live migration now hold at the same time. Existing VMs pick the change up on their next restart; already-running VMs are unaffected until then. This fix was tagged in v1.5.3, which was never published, so v1.5.4 is the first release to carry it (@kvaps in [#3154], backport [#3212], reverting [#3006]).

• fix(migrations): pin KubeadmConfigTemplate on the 1.5 slot 45 migration and in the chart: Migration slot 45 holds different things on different branches — release-1.5 uses it for the SeaweedFS seaweedfs-db hand-over repair, while main and release-1.6 use it for the helm.sh/resource-policy: keep pin on the CAPI KubeadmConfigTemplate objects. Cutting v1.5.4 is what first ships targetVersion: 46, and run-migrations.sh loops seq CURRENT (TARGET - 1), so a cluster that reaches 46 later runs seq 46 53 on the way to 1.6 and never executes 1.6's slot 45 at all — the pin is skipped, and Helm then prunes the template while the kubeadm-backed MachineSet is still mid-rollover. 1.5's own slot 45 now applies the pin (after the SeaweedFS repair, and fail-closed) so an existing cluster picks it up on the way to 46, and the kubernetes chart stamps the annotation at render time so the object is born pinned — which is the only thing that covers a fresh v1.5.4 install, since such a cluster has its cozystack-version ConfigMap rendered straight at 46 and the migration hook never emits a Job at all. See the required-actions section above for the verification commands (@myasnikovdaniil in [#3892]).

• fix(postgres-operator): align CNPG CRDs with 1.27.3: The vendored CloudNativePG CRDs were still pinned to 1.27.1 metadata while the operator image was already on 1.27.3, which prunes the new Backup.status.instanceID.sessionID field the newer instance-manager writes — causing false instance-manager restart errors and failed backups. The CRDs are aligned with the pinned 1.27.3 image so backups no longer fail on this mismatch (@myasnikovdaniil in [#3526]).

Development, Testing, and CI/CD

• ci(release): move the v1.5.4 release path off the decommissioned runner pool: Every job on the release-1.5 release path was pinned to a [self-hosted] runner pool that no longer exists, so the release would silently sit Queued forever with nothing red to point at instead of failing loudly. The release, tag, and cache-warmer jobs are moved to the same runner shape release-1.6 already uses, unblocking future patch releases on this branch (@myasnikovdaniil in [#3906]).

• ci: warm the build cache on release-1.5 pushes: release-1.5's build-cache warmer was dead code — it triggered on pushes to main (which never happens on this branch) and targeted the same decommissioned [self-hosted] runner — so every PR build on the branch ran cold and could exceed its 30-minute timeout. The warmer now triggers on release-1.5 pushes and targets a working runner, and the Build job is pointed at the branch-scoped cache namespace the warmer actually writes; without that the warmer would run and nothing would read it, because the cache-from refs still resolved to the shared namespace main writes, whose layers carry main's sources and miss on most of this branch's 28 packages. Builds only read the cache (WRITE_CACHE defaults to 0), so concurrent PR builds cannot race on a cache manifest and a miss degrades to a cold build rather than an error. The timeout is raised to 75 minutes at the same time: a warm build is around 15 minutes, but a cold one is around 45, and 30 minutes only ever fit the warm case — when it did not fit, the job reported a timeout instead of whatever actually went wrong (@myasnikovdaniil in [#3469]).

• fix(ci, kubernetes): build container disks concurrently to unblock Build: Every release-1.5 PR Build job was dying at its timeout, and effectively all of the budget went to one target: image-ubuntu-container-disk builds one disk per supported Kubernetes minor — six of them — and each spends four to six minutes inside a libguestfs appliance installing that minor's kubelet and kubeadm into its own copy of the cloud image, roughly 28 minutes serially. None of it was cached and no warmer could fix that, because only build-main.yaml sets WRITE_CACHE=1 and main dropped this image from make image shortly before release-1.5 was cut, so the buildcache refs it reads had never been written — leaving release-1.5 as the only branch still building the image at all. The six builds share no per-version work, so they now run concurrently in a sub-make carrying its own -j (keeping the root make build serial), with buildkit deduplicating the shared guestfish and cloud-image stages across the concurrent solves and --output-sync=target keeping each version's log readable. The job also moves off the 4cpu/16gb runner shape, since six concurrent libguestfs appliances want roughly 9GB of RAM and a core each (@myasnikovdaniil in [#3469]).

• ci: install flux in the release-1.5 cache warmer: The warmer's make build step shells out to flux push artifact, which the previous self-hosted runner had baked in but the new ephemeral runner shape does not; the cache warmer is updated to install the flux CLI so it can complete (@myasnikovdaniil in [#3480]).

• test(seaweedfs): make the guard-parity and fake-kubectl assertions able to fail: The four assertions holding the invariant that neither SeaweedFS chart classifies on mutable claim timestamps or pod liveness were written as ! grep -qF X "$f", which cannot fail — POSIX and bash both exempt a !-negated pipeline from errexit, so the command ran, returned 1, and the test carried on reporting success. Those four were the entire body of the test, so it asserted nothing at all; and converting them naively fails, because all four names legitimately appear in the comment prose that records why each was rejected as a discriminator. They now assert against the template logic with comment blocks stripped, forbidding a live reference, and are proven in both directions. The fake kubectl used by the migration tests likewise now fails on unmodelled calls instead of returning success for anything it does not recognise (@myasnikovdaniil in [#3436], backport [#3878]; [#3892]).

• Regression coverage shipped alongside the fixes above: Each of the backported fixes carried its own test, and they are listed here rather than left silent because they are what stops these bugs coming back. A reconcile-level regression drives reconcileCNPGRestore end to end against a fake client, covering the actual call site of the repeat-restore bug rather than only its helpers (@IvanHunters in [#3318], backport [#3321]); the e2e suite now asserts BucketAccess reaches accessGranted=true, asserts a BucketClaim converges to bucketReady promptly for both the bucket and Harbor paths, and dumps controller pods on a bucket failure (@lexfrei in [#3034], backport [#3532]); keycloak-configure gains a run of the chart's helm-unittest suite (@lexfrei in [#3372], backport [#3877]); the flux-shard-operator asserts the full startupProbe contract and rejects an alias regression, with the proxy-env drop and probe recorded in the sanitisation list (@IvanHunters in [#3546], backport [#3883]); the kubeadm keep-pin folded into slot 45 has migration coverage (@myasnikovdaniil in [#3892]); and the PR labeler now maps the objectstorage-controller scope to area/storage (@lexfrei in [#3034], backport [#3532]).

Documentation

• [website] fix(backup): simplify UX with default backupclass and correct the admin override path: Rewrites the backup guides to match the corrected backupStorage override path shipped in [#3402]/#3333 above, so the documented admin override actually works and the default backup-class UX is simpler to follow (@androndo in cozystack/website#622).

• [website] docs(backups): document PostgreSQL point-in-time recovery (PITR): Adds a guide for restoring a PostgreSQL cluster to a specific point in time, covering the required backup configuration and the restore procedure (@androndo in cozystack/website#629).

• [website] docs(monitoring): add OIDC authentication guide for Grafana: New guide walking operators through enabling OIDC-based single sign-on for Grafana, including the admin-group and mode-toggle caveats (@IvanHunters in cozystack/website#597).

• [website] docs(operations): document the gateway.http2 platform value: Documents the gateway.http2 platform value so operators know how to enable/disable HTTP/2 on the ingress gateway (@lllamnyp in cozystack/website#625).

• [website] docs(talm): describe the preset value knobs for network and registries: Documents the new talm preset value knobs for tuning network sysctls and registry mirror configuration (@lexfrei in cozystack/website#633).

• [website] docs: fix mermaid edge label and document wildcardSecretName for v1.5: Fixes an unrenderable mermaid diagram in the certificate documentation and documents the wildcardSecretName option for v1.5 (@lexfrei in cozystack/website#615).

• [website] docs: correct the publishing reference and document the existingSecret cert mode: Corrects the certificate publishing reference and documents the third, existingSecret, certificate mode (@lexfrei in cozystack/website#619).

• [website] docs(gpu): drop the manual KubeVirt patch step for GPU passthrough: The platform wires permittedHostDevices into the KubeVirt CR itself from packages/core/platform/files/gpu-passthrough-defaults.yaml, so the manual patch documented since v1.4 is obsolete; the guide now covers the pre-upgrade migration steps for hand-edited entries instead. Merged just after the v1.5.2 tag, so v1.5.4 is the first release these notes cover it in (@lexfrei in cozystack/website#556).

• [website] docs: import the operator guides that lived in the cozystack repo: Moves the operator-facing guides that previously lived in the cozystack repository into the website's documentation, consolidating operator docs in one place (@myasnikovdaniil in cozystack/website#648).

• [website] docs: regenerate the managed apps reference for v1.4.5 and v1.5.2: Refreshes the auto-generated managed-application reference pages so the published values tables match what those releases actually ship (@app/cozystack-ci in cozystack/website#600, cozystack/website#601).

Other repositories

talm (v0.32.0 → v0.34.0)

• [talm] feat(charts): add preset value knobs: Adds configurable preset value knobs for network and registry settings, letting operators tune Talos machine config presets without hand-editing templates (@lexfrei in cozystack/talm#232).

• [talm] chore(deps): migrate to Helm 4 and drop the cozystack/talos fork: Migrates talm's Helm library from v3 to v4 and drops the previously vendored cozystack/talos fork in favor of upstream Talos v1.13.7 (@lexfrei in cozystack/talm#231).

• [talm] security: disable, then restore, KVM nested virtualization in Talos presets (CVE-2026-53359): talm first disabled KVM nested virtualization in its Talos presets to mitigate CVE-2026-53359, then reverted the blanket disable once the upstream fix landed, so presets keep nested virtualization available instead of being permanently disabled (@kvaps in cozystack/talm#224, @lexfrei in cozystack/talm#225).

• [talm] feat(version): warn when project charts drift from the talm binary: talm now warns when a project's vendored charts have drifted from the version embedded in the talm binary, plus chart-drift-detection support in init --update, catching stale vendored charts before they cause a surprising render diff (@lexfrei in cozystack/talm#216).

ansible-cozystack (v1.5.2 → v1.6.1)

• [ansible-cozystack] fix(prepare): keep KVM nested virtualization enabled: Mirrors talm's CVE-2026-53359 response in the Ansible prepare role — nested virtualization is disabled to mitigate the CVE and then restored once the fix landed, matching the talm preset behavior (@kvaps in cozystack/ansible-cozystack#60, @lexfrei in cozystack/ansible-cozystack#61).

• [ansible-cozystack] fix(examples): make prepare and k3s.orchestration runnable as separate invocations: The example playbooks can now be run as independent steps rather than requiring a single combined invocation (@lexfrei in cozystack/ansible-cozystack#52).

• [ansible-cozystack] chore(deps): track the cozy-installer version: The installer image is bumped in step with the Cozystack releases cut during this period — v1.5.3, v1.6.0 and v1.6.1 (@app/renovate in cozystack/ansible-cozystack#59, cozystack/ansible-cozystack#63, cozystack/ansible-cozystack#65).

No tags were released in boot-to-talos (latest v0.7.1, 2026-03-19), cozyhr (latest v1.6.1, 2026-01-27), cozy-proxy (latest v0.3.0, 2026-04-28) or external-apps-example (no tags) during this release period.

Contributors

Thanks to everyone who contributed to this patch release:

• @androndo
• @IvanHunters
• @kvaps
• @lexfrei
• @lllamnyp
• @mattia-eleuteri
• @myasnikovdaniil

Full Changelog: https://github.com/cozystack/cozystack/compare/v1.5.2...v1.5.4