cozystack Files

Cozystack v0.31.0 is a significant release that brings new features, key fixes, and updates to underlying components. This version enhances GPU support, improves many components of Cozystack, and introduces a more robust release process to improve stability. Below, we'll go over the highlights in each area for current users, developers, and our community.

Major Features and Improvements

GPU support for tenant Kubernetes clusters

Cozystack now integrates NVIDIA GPU Operator support for tenant Kubernetes clusters. This enables platform users to run GPU-powered AI/ML applications in their own clusters. To enable GPU Operator, set addons.gpuOperator.enabled: true in the cluster configuration. (@kvaps in https://github.com/cozystack/cozystack/pull/834)

Check out Andrei Kvapil's CNCF webinar showcasing the GPU support by running Stable Diffusion in Cozystack.

Cilium Improvements

Cozystack's Cilium integration received two significant enhancements. First, Gateway API support in Cilium is now enabled, allowing advanced L4/L7 routing features via Kubernetes Gateway API. We thank Zdenek Janda @zdenekjanda for contributing this feature in https://github.com/cozystack/cozystack/pull/924.

Second, Cozystack now permits custom user-provided parameters in the tenant cluster's Cilium configuration. (@lllamnyp in https://github.com/cozystack/cozystack/pull/917)

Cross-Architecture Builds (ARM Support Beta)

Cozystack's build system was refactored to support multi-architecture binaries and container images. This paves the road to running Cozystack on ARM64 servers. Changes include Makefile improvements (https://github.com/cozystack/cozystack/pull/907) and multi-arch Docker image builds (https://github.com/cozystack/cozystack/pull/932 and https://github.com/cozystack/cozystack/pull/970).

We thank Nikita Bykov @nbykov0 for his ongoing work on ARM support!

VerticalPodAutoscaler (VPA) Expansion

The VerticalPodAutoscaler is now enabled for more Cozystack components to automate resource tuning. Specifically, VPA was added for tenant Kubernetes control planes (@klinch0 in https://github.com/cozystack/cozystack/pull/806), the Cozystack Dashboard (https://github.com/cozystack/cozystack/pull/828), and the Cozystack etcd-operator (https://github.com/cozystack/cozystack/pull/850). All Cozystack components that have VPA enabled can automatically adjust their CPU and memory requests based on usage, improving platform and application stability.

Tenant HelmRelease Reconcile Controller

A new controller was introduced to monitor and synchronize HelmRelease resources across tenants. This controller propagates configuration changes to tenant workloads and ensures that any HelmRelease defined in a tenant stays in sync with platform updates. It improves the reliability of deploying managed applications in Cozystack. (@klinch0 in https://github.com/cozystack/cozystack/pull/870)

Virtual Machine Improvements

Configurable KubeVirt CPU Overcommit: The CPU allocation ratio in KubeVirt (how virtual CPUs are overcommitted relative to physical) is now configurable via the cpu-allocation-ratio value in the Cozystack configmap. This means Cozystack administrators can now tune CPU overcommitment for VMs to balance performance vs. density. (@lllamnyp in https://github.com/cozystack/cozystack/pull/905)

KubeVirt VM Export: Cozystack now allows exporting KubeVirt virtual machines. This feature, enabled via KubeVirt's VirtualMachineExport capability, lets users snapshot or back up VM images. (@kvaps in https://github.com/cozystack/cozystack/pull/808)

Support for various storage classes in Virtual Machines: The virtual-machine application (since version 0.9.2) lets you pick any StorageClass for a VM's system disk instead of relying on a hard-coded PVC. Refer to values systemDisk.storage and systemDisk.storageClass in the application's configs. (@kvaps in https://github.com/cozystack/cozystack/pull/974)

Other Features and Improvements

• [platform] Introduce options expose-services, expose-ingress, and expose-external-ips to the ingress service. (@kvaps in https://github.com/cozystack/cozystack/pull/929)
• [cozystack-controller] Record the IP address pool and storage class in Workload objects. (@lllamnyp in https://github.com/cozystack/cozystack/pull/831)
• [apps] Remove user-facing config of limits and requests. (@lllamnyp in https://github.com/cozystack/cozystack/pull/935)

New Release Lifecycle

Cozystack release lifecycle is changing to provide a more stable and predictable lifecycle to customers running Cozystack in mission-critical environments.

• Gradual Release with Alpha, Beta, and Release Candidates: Cozystack will now publish pre-release versions (alpha, beta, release candidates) before a stable release. Starting with v0.31.0, the team made three release candidates before releasing version v0.31.0. This allows more testing and feedback before marking a release as stable.

• Prolonged Release Support with Patch Versions: After the initial vX.Y.0 release, a long-lived branch release-X.Y will be created to backport fixes. For example, with 0.31.0's release, a release-0.31 branch will track patch fixes (0.31.x). This strategy lets Cozystack users receive timely patch releases and updates with minimal risks.

To implement these new changes, we have rebuilt our CI/CD workflows and introduced automation, enabling automatic backports. You can read more about how it's implemented in the Development section below.

For more information, read the Cozystack Release Workflow documentation.

Fixes

• [virtual-machine] Add GPU names to the virtual machine specifications. (@kvaps in https://github.com/cozystack/cozystack/pull/862)
• [virtual-machine] Count Workload resources for pods by requests, not limits. Other improvements to VM resource tracking. (@lllamnyp in https://github.com/cozystack/cozystack/pull/904)
• [virtual-machine] Set PortList method by default. (@kvaps in https://github.com/cozystack/cozystack/pull/996)
• [virtual-machine] Specify ports even for wholeIP mode. (@kvaps in https://github.com/cozystack/cozystack/pull/1000)
• [platform] Fix installing HelmReleases on initial setup. (@kvaps in https://github.com/cozystack/cozystack/pull/833)
• [platform] Migration scripts update Kubernetes ConfigMap with the current stack version for improved version tracking. (@klinch0 in https://github.com/cozystack/cozystack/pull/840)
• [platform] Reduce requested CPU and RAM for the kamaji provider. (@klinch0 in https://github.com/cozystack/cozystack/pull/825)
• [platform] Improve the reconciliation loop for the Cozystack system HelmReleases logic. (@klinch0 in https://github.com/cozystack/cozystack/pull/809 and https://github.com/cozystack/cozystack/pull/810, @kvaps in https://github.com/cozystack/cozystack/pull/811)
• [platform] Remove extra dependencies for the Piraeus operator. (@klinch0 in https://github.com/cozystack/cozystack/pull/856)
• [platform] Refactor dashboard values. (@kvaps in https://github.com/cozystack/cozystack/pull/928, patched by @llamnyp in https://github.com/cozystack/cozystack/pull/952)
• [platform] Make FluxCD artifact disabled by default. (@klinch0 in https://github.com/cozystack/cozystack/pull/964)
• [kubernetes] Update garbage collection of HelmReleases in tenant Kubernetes clusters. (@kvaps in https://github.com/cozystack/cozystack/pull/835)
• [kubernetes] Fix merging valuesOverride for tenant clusters. (@kvaps in https://github.com/cozystack/cozystack/pull/879)
• [kubernetes] Fix ubuntu-container-disk tag. (@kvaps in https://github.com/cozystack/cozystack/pull/887)
• [kubernetes] Refactor Helm manifests for tenant Kubernetes clusters. (@kvaps in https://github.com/cozystack/cozystack/pull/866)
• [kubernetes] Fix Ingress-NGINX depends on Cert-Manager. (@kvaps in https://github.com/cozystack/cozystack/pull/976)
• [kubernetes, apps] Enable topologySpreadConstraints for tenant Kubernetes clusters and fix it for managed PostgreSQL. (@klinch0 in https://github.com/cozystack/cozystack/pull/995)
• [tenant] Fix an issue with accessing external IPs of a cluster from the cluster itself. (@kvaps in https://github.com/cozystack/cozystack/pull/854)
• [cluster-api] Remove the no longer necessary workaround for Kamaji. (@kvaps in https://github.com/cozystack/cozystack/pull/867, patched in https://github.com/cozystack/cozystack/pull/956)
• [monitoring] Remove legacy label "POD" from the exclude filter in metrics. (@xy2 in https://github.com/cozystack/cozystack/pull/826)
• [monitoring] Refactor management etcd monitoring config. Introduce a migration script for updating monitoring resources (kube-rbac-proxy daemonset). (@lllamnyp in https://github.com/cozystack/cozystack/pull/799 and https://github.com/cozystack/cozystack/pull/830)
• [monitoring] Fix VerticalPodAutoscaler resource allocation for VMagent. (@klinch0 in https://github.com/cozystack/cozystack/pull/820)
• [postgres] Remove duplicated template entry from backup manifest. (@etoshutka in https://github.com/cozystack/cozystack/pull/872)
• [kube-ovn] Fix versions mapping in Makefile. (@kvaps in https://github.com/cozystack/cozystack/pull/883)
• [dx] Automatically detect version for migrations in the installer.sh. (@kvaps in https://github.com/cozystack/cozystack/pull/837)
• [dx] remove version_map and building for library charts. (@kvaps in https://github.com/cozystack/cozystack/pull/998)
• [docs] Review the tenant Kubernetes cluster docs. (@NickVolynkin in https://github.com/cozystack/cozystack/pull/969)
• [docs] Explain that tenants cannot have dashes in their names. (@NickVolynkin in https://github.com/cozystack/cozystack/pull/980)

Dependencies

• MetalLB images are now built in-tree based on version 0.14.9 with additional critical patches. (@lllamnyp in https://github.com/cozystack/cozystack/pull/945)
• Update Kubernetes to v1.32.4. (@kvaps in https://github.com/cozystack/cozystack/pull/949)
• Update Talos Linux to v1.10.1. (@kvaps in https://github.com/cozystack/cozystack/pull/931)
• Update Cilium to v1.17.3. (@kvaps in https://github.com/cozystack/cozystack/pull/848)
• Update LINSTOR to v1.31.0. (@kvaps in https://github.com/cozystack/cozystack/pull/846)
• Update Kube-OVN to v1.13.11. (@kvaps in https://github.com/cozystack/cozystack/pull/847, @lllamnyp in https://github.com/cozystack/cozystack/pull/922)
• Update tenant Kubernetes to v1.32. (@kvaps in https://github.com/cozystack/cozystack/pull/871)
• Update flux-operator to 0.20.0. (@kingdonb in https://github.com/cozystack/cozystack/pull/880 and https://github.com/cozystack/cozystack/pull/934)
• Update multiple Cluster API components. (@kvaps in https://github.com/cozystack/cozystack/pull/867 and https://github.com/cozystack/cozystack/pull/947)
• Update KamajiControlPlane to edge-25.4.1. (@kvaps in https://github.com/cozystack/cozystack/pull/953, fixed by @nbykov0 in https://github.com/cozystack/cozystack/pull/983)
• Update cert-manager to v1.17.2. (@kvaps in https://github.com/cozystack/cozystack/pull/975)

Documentation

• Installing Talos in Air-Gapped Environment: new guide for configuring and bootstrapping Talos Linux clusters in air-gapped environments. (@klinch0 in https://github.com/cozystack/website/pull/203)

• Cozystack Bundles: new page in the learning section explaining how Cozystack bundles work and how to choose a bundle. (@NickVolynkin in https://github.com/cozystack/website/pull/188, https://github.com/cozystack/website/pull/189, and others; updated by @kvaps in https://github.com/cozystack/website/pull/192 and https://github.com/cozystack/website/pull/193)

• Managed Application Reference: A set of new pages in the docs, mirroring application docs from the Cozystack dashboard. (@NickVolynkin in https://github.com/cozystack/website/pull/198, https://github.com/cozystack/website/pull/202, and https://github.com/cozystack/website/pull/204)

• LINSTOR Networking: Guides on configuring dedicated network for LINSTOR and configuring network for distributed storage in multi-datacenter setup. (@xy2, edited by @NickVolynkin in https://github.com/cozystack/website/pull/171, https://github.com/cozystack/website/pull/182, and https://github.com/cozystack/website/pull/184)

Fixes

• Correct error in the doc for the command to edit the configmap. (@lb0o in https://github.com/cozystack/website/pull/207)
• Fix group name in OIDC docs (@kingdonb in https://github.com/cozystack/website/pull/179)
• A bit more explanation of Docker buildx builders. (@nbykov0 in https://github.com/cozystack/website/pull/187)

Development, Testing, and CI/CD

Testing

Improvements:

• Introduce cozytest --- a new BATS-based testing framework. (@kvaps in https://github.com/cozystack/cozystack/pull/982)

Fixes:

• Fix device_ownership_from_security_context CRI. (@dtrdnk in https://github.com/cozystack/cozystack/pull/896)
• Increase timeout durations for capi and keycloak to improve reliability during e2e-tests. (@kvaps in https://github.com/cozystack/cozystack/pull/858)
• Return genisoimage to the e2e-test Dockerfile (@gwynbleidd2106 in https://github.com/cozystack/cozystack/pull/962)

CI/CD Changes

Improvements:

• Use release branches release-X.Y for gathering and releasing fixes after initial vX.Y.0 release. (@kvaps in https://github.com/cozystack/cozystack/pull/816)
• Automatically create release branches after initial vX.Y.0 release is published. (@kvaps in https://github.com/cozystack/cozystack/pull/886)
• Introduce Release Candidate versions. Automate patch backporting by applying patches from pull requests labeled [backport] to the current release branch. (@kvaps in https://github.com/cozystack/cozystack/pull/841 and https://github.com/cozystack/cozystack/pull/901, @nickvolynkin in https://github.com/cozystack/cozystack/pull/890)
• Support alpha and beta pre-releases. (@kvaps in https://github.com/cozystack/cozystack/pull/978)
• Commit changes in release pipelines under github-actions <github-actions@github.com>. (@kvaps in https://github.com/cozystack/cozystack/pull/823)
• Describe the Cozystack release workflow. (@NickVolynkin in https://github.com/cozystack/cozystack/pull/817 and https://github.com/cozystack/cozystack/pull/897)

Fixes:

• Improve the check for versions_map running on pull requests. (@kvaps and @klinch0 in https://github.com/cozystack/cozystack/pull/836, https://github.com/cozystack/cozystack/pull/842, and https://github.com/cozystack/cozystack/pull/845)
• If the release step was skipped on a tag, skip tests as well. (@kvaps in https://github.com/cozystack/cozystack/pull/822)
• Allow CI to cancel the previous job if a new one is scheduled. (@kvaps in https://github.com/cozystack/cozystack/pull/873)
• Use the correct version name when uploading build assets to the release page. (@kvaps in https://github.com/cozystack/cozystack/pull/876)
• Stop using ok-to-test label to trigger CI in pull requests. (@kvaps in https://github.com/cozystack/cozystack/pull/875)
• Do not run tests in the release building pipeline. (@kvaps in https://github.com/cozystack/cozystack/pull/882)
• Fix release branch creation. (@kvaps in https://github.com/cozystack/cozystack/pull/884)
• Reduce noise in the test logs by suppressing the wget progress bar. (@lllamnyp in https://github.com/cozystack/cozystack/pull/865)
• Revert "automatically trigger tests in releasing PR". (@kvaps in https://github.com/cozystack/cozystack/pull/900)
• Force-update release branch on tagged main commits. (@kvaps in https://github.com/cozystack/cozystack/pull/977)
• Show detailed errors in the pull-request-release workflow. (@lllamnyp in https://github.com/cozystack/cozystack/pull/992)

Community and Maintenance

Repository Maintenance

Added @klinch0 to CODEOWNERS. (@kvaps in https://github.com/cozystack/cozystack/pull/838)

New Contributors

• @etoshutka made their first contribution in https://github.com/cozystack/cozystack/pull/872
• @dtrdnk made their first contribution in https://github.com/cozystack/cozystack/pull/896
• @zdenekjanda made their first contribution in https://github.com/cozystack/cozystack/pull/924
• @gwynbleidd2106 made their first contribution in https://github.com/cozystack/cozystack/pull/962

Full Changelog

See https://github.com/cozystack/cozystack/compare/v0.30.0...v0.31.0