Cozystack v1.2.0

⚠️ WARNING: Do not use this release. This version includes CloudNativePG operator, which updates the default PostgreSQL image to version 18. CNPG is unable to perform the migration from the previous major version automatically, which will cause PostgreSQL clusters to fail to start after the upgrade. Please use v1.2.1 instead.

Cozystack v1.2.0 delivers significant platform enhancements: a fully managed OpenSearch service joining the application catalog, VPC peering for secure inter-tenant networking, tenant workload placement control via the new SchedulingClass system, a highly-available VictoriaLogs cluster replacing the single-node setup, and Linstor volume relocation for optimized clone and snapshot restore placement. Additional highlights include external-dns as a standalone extra package, multi-node RWX volume fixes, and a wave of dashboard and monitoring improvements.

Feature Highlights

OpenSearch: Managed Search and Analytics Service

Cozystack now ships OpenSearch as a fully managed PaaS application — supporting OpenSearch v1, v2, and v3 in a multi-role topology with dedicated master, data, ingest, coordinating, and ML nodes. TLS is enabled by default, HTTP Basic auth is provided out of the box, and custom user definitions allow per-application credentials. The optional OpenSearch Dashboards UI can be enabled alongside the engine. External access, topology spread policies, and a comprehensive JSON schema are all included.

A companion opensearch-operator system package wraps the upstream Opster OpenSearch Operator v2.8.0 and adds a sysctl DaemonSet to configure the required vm.max_map_count kernel parameter on every node automatically. An ApplicationDefinition package ties everything into the Cozystack platform dashboard with schema validation and resource management.

SchedulingClass: Tenant Workload Placement

Cozystack now supports a SchedulingClass CRD that allows platform operators to define cluster-wide scheduling constraints — pinning tenant workloads to specific data centers, hardware generations, or node groups without requiring tenants to manage scheduler configuration themselves. Tenants declare a schedulingClass in their Tenant spec; the platform injects the appropriate schedulerName into all workloads in that namespace.

The lineage-controller-webhook has been extended to verify the referenced SchedulingClass CR before injection, and child tenants inherit their parent's scheduling constraints (children cannot override). A SchedulingClass dropdown in the Tenant creation form in the dashboard makes the feature fully self-service. The underlying cozystack-scheduler — a custom kube-scheduler extension with SchedulingClass-aware affinity plugins — is now installed and enabled by default as part of the platform.

VPC Peering for Multi-Tenant Environments

The vpc application gains bilateral VPC peering using Kube-OVN's native vpcPeerings mechanism, allowing tenants to securely interconnect their private networks without routing traffic through public endpoints. Peering link-local IPs (169.254.0.0/16) are allocated deterministically from a hash of the sorted VPC pair names, ensuring stable addresses across reconciliations. Static route support (staticRoutes) enables fine-grained inter-VPC routing policies. A cozy-lib helper (hexToInt) performs the deterministic IP allocation, and a JSON Schema validation enforces the ^tenant- namespace pattern for peered VPCs.

VictoriaLogs: Clustered Mode for High Availability

The platform's log storage has been upgraded from the deprecated single-node VLogs CR to a VLCluster deployment with separate vlinsert, vlselect, and vlstorage components, each running with 2 replicas by default — consistent with the existing VMCluster setup. This brings horizontal scalability and resilience to the logging tier. VPA autoscaling is enabled for all VLCluster components, and the victoria-metrics-operator has been upgraded from v0.55.0 to v0.68.1 to add VLCluster CRD support.

Linstor CSI: Volume Relocation After Clone and Restore

The Linstor CSI driver now carries upstream patches enabling automatic replica relocation after PVC clone and snapshot restore operations. Two new parameters control the behavior: linstor.csi.linbit.com/relocateAfterClone on StorageClasses moves replicas to optimal nodes after a clone, and snap.linstor.csi.linbit.com/relocate-after-restore on VolumeSnapshotClasses does the same after a restore. VolumeSnapshotClasses for Velero and Kasten use cases are pre-configured. This enables full PVC-level backup and restore workflows with automatic data rebalancing, a key prerequisite for production Velero/Kasten integrations.

Major Features and Improvements

• [apps] Add managed OpenSearch service: Deployed as a PaaS application supporting OpenSearch v1/v2/v3 with multi-role node topology, TLS, HTTP Basic auth, custom users, optional OpenSearch Dashboards UI, external access, and topology spread policies; backed by the opster OpenSearch Operator v2.8.0 and a sysctl DaemonSet for vm.max_map_count (@matthieu-robin in [#1953]).

• [vpc] Add VPC peering support for multi-tenant environments: Bilateral VPC peering via Kube-OVN's vpcPeerings, deterministic link-local IP allocation from sorted VPC pair hash, static routes support, ConfigMap peer discovery enrichment, and JSON Schema validation enforcing ^tenant- namespace pattern (@mattia-eleuteri in [#2152]).

• [monitoring] Migrate VictoriaLogs from VLogs to VLCluster: Replaced deprecated single-node VLogs CR with clustered VLCluster (vlinsert/vlselect/vlstorage, 2 replicas each), added VPA for all components, upgraded victoria-metrics-operator to v0.68.1 (@sircthulhu in [#2153]).

• [scheduler] Integrate SchedulingClass support for tenant workloads: Added schedulingClass Tenant parameter with inheritance enforcement, scheduling.cozystack.io/class namespace label, lineage-webhook extension to verify and inject schedulerName, SchedulingClass dropdown in Tenant dashboard form (@sircthulhu in [#2223]).

• [cozystack-scheduler] Add custom scheduler as an optional system package: Vendored cozystack-scheduler from github.com/cozystack/cozystack-scheduler — a kube-scheduler extension with SchedulingClass-aware affinity plugins, including Helm chart with RBAC, ConfigMap, Deployment, and CRD (@lllamnyp in [#2205]).

• [platform] Enable cozystack-scheduler by default: The cozystack-scheduler and SchedulingClass CRD are now installed as default system packages; the backup tool has been moved to optional packages (@lllamnyp in [#2253]).

• [extra] Add external-dns as a standalone extra package: Packaged external-dns as an installable extra (tenant-level) component for automatic DNS record management from Kubernetes Service and Ingress resources (@mattia-eleuteri in [#1988]).

• [linstor] Add linstor-csi patches for clone/snapshot relocation: New patch enabling relocateAfterClone StorageClass parameter and relocate-after-restore VolumeSnapshotClass parameter; pre-configured VolumeSnapshotClasses for Velero and relocation workflows; CDI switched to csi-clone strategy (@kvaps in [#2133]).

• [monitoring] Add inlineScrapeConfig support to tenant vmagent: Tenants can now define inline scrape configurations directly in their VMAgent spec, enabling custom metrics collection from services that are not discoverable via standard Kubernetes service discovery (@mattia-eleuteri in [#2200]).

• [monitoring] Add Slack dashboard URL, vmagent environment label, and dynamictext Grafana plugin: Added SLACK_DASHBOARD_URL and SLACK_SUMMARY_FMT environment variables for richer alert notifications, per-vmagent environment label for metric source identification, and the dynamictext-panel plugin for Grafana dashboards (@vnyakas in [#2210]).

• [monitoring] Scope infrastructure dashboards to tenant-root only: Infrastructure-level Grafana dashboards are now scoped to the tenant-root namespace only, preventing them from appearing in tenant sub-namespaces and reducing dashboard noise (@mattia-eleuteri in [#2197]).

• [tenant] Allow egress to virt-handler for VM metrics scraping: Extended tenant NetworkPolicy to permit egress to virt-handler pods, enabling Prometheus to scrape VM-level metrics from KubeVirt without additional policy exceptions (@mattia-eleuteri in [#2199]).

• [dashboard] Add keycloakInternalUrl for backend-to-backend OIDC requests: Added a keycloakInternalUrl platform value for the dashboard backend to perform OIDC token introspection via an internal cluster URL, avoiding external round-trips and improving reliability in air-gapped environments (@sircthulhu in [#2224]).

• [dashboard] Add secret-hash annotation to KeycloakClient for secret sync: Added a secret-hash annotation to the KeycloakClient resource so that changes to the client secret trigger automatic reconciliation and propagation to dependent components (@sircthulhu in [#2231]).

• [docs] Add OpenAPI and Go types code generation for apps: Added tooling to generate OpenAPI schemas and Go types from Helm chart values, enabling type-safe programmatic access to managed application configurations and automatic API reference generation (@myasnikovdaniil in [#2214]).

Improvements (minor)

• [cozystack-scheduler] Update to v0.2.0: Updated the cozystack-scheduler to v0.2.0 with improved SchedulingClass affinity handling (@lllamnyp in [#2244]).

• [platform] Ensure cozystack-packages OCIRepository updates reliably: Added configuration to ensure the cozystack-packages OCIRepository resource is consistently reconciled and reflects the latest package versions on upgrade (@sircthulhu in [#2246]).

• [etcd] Add protective limits to defrag CronJob: Added CPU and memory resource limits to the etcd defragmentation CronJob to prevent it from starving other workloads during scheduled defragmentation runs (@sircthulhu in [#2233]).

• [platform] Add missing apps to tenant admin RBAC: Extended the tenant admin ClusterRole to include RBAC permissions for recently added applications that were missing from the role binding (@sircthulhu in [#2268]).

Bug Fixes

• [keycloak] Fix health probe configuration for Keycloak v26.x+: Replaced deprecated KC_PROXY=edge with KC_PROXY_HEADERS=xforwarded/KC_HTTP_ENABLED=true; replaced liveness/readiness probes with management port endpoints (/health/live, /health/ready) and added a startupProbe to handle slow Keycloak startup without triggering premature restarts (@mattia-eleuteri in [#2162]).

• [migrations] Handle missing RabbitMQ CRD in migration 34: Fixed a crash in migration script 34 that occurred when the RabbitMQ CRD was not yet installed, allowing upgrades from environments where RabbitMQ was never deployed (@IvanHunters in [#2168]).

• [platform] Fix VM MAC address not preserved during migration: Fixed the virtual-machine to vm-instance migration script to correctly carry over the MAC address, preventing network identity changes after upgrading existing VM resources (@sircthulhu in [#2169]).

• [dashboard] Fix External IPs factory EnrichedTable rendering: Corrected the External IPs factory component to use the EnrichedTable renderer, resolving blank/broken rendering of the external IP address list in the dashboard (@IvanHunters in [#2175]).

• [dashboard] Preserve disabled/hidden state on MarketplacePanel reconciliation: Fixed a regression where MarketplacePanel reconciliation would reset the disabled and hidden flags set by operators, causing hidden applications to reappear in the catalog (@IvanHunters in [#2176]).

• [dashboard] Exclude hidden MarketplacePanel resources from sidebar menu: Fixed the sidebar to omit applications that have been hidden via MarketplacePanel flags, preventing inaccessible menu entries from being displayed to users (@IvanHunters in [#2177]).

• [etcd-operator] Replace deprecated kube-rbac-proxy image: Replaced the unmaintained gcr.io/kubebuilder/kube-rbac-proxy sidecar with the actively maintained quay.io/brancz/kube-rbac-proxy image to eliminate deprecation warnings and ensure continued security updates (@kvaps in [#2181]).

• [backups] Fix RBAC and backupstrategy-controller location: Corrected role bindings and the deployment location for the backup strategy controller to restore full backup and restore functionality (@myasnikovdaniil in [#2149]).

• [api] Skip OpenAPI post-processor for non-apps group versions: Fixed the API server to bypass OpenAPI schema post-processing for non-apps group versions, preventing schema corruption in unrelated API groups (@kvaps in [#2212]).

• [bucket] Fix s3manager endpoint mismatch with COSI credentials: Corrected the S3 Manager UI to use the actual S3 endpoint from the BucketInfo COSI resource rather than a hardcoded value, resolving connection failures when the S3 endpoint differs from the default (@IvanHunters in [#2211]).

• [kubernetes] Fix tenant Kubernetes cluster creation for versions < 1.32: Resolved a template rendering error that prevented creation of tenant Kubernetes clusters with versions older than 1.32 (@myasnikovdaniil in [#2209]).

• [kube-ovn] Fix MASTER_NODES detection for multi-master Kubernetes clusters: Updated kube-ovn configuration to discover control-plane nodes via the standard node-role.kubernetes.io/control-plane label rather than relying on static node lists, fixing OVN connectivity issues in multi-master generic Kubernetes deployments (@lexfrei in [#2245]).

• [kubernetes] Fix CiliumNetworkPolicy endpointSelector for multi-node RWX volumes: Corrected the CiliumNetworkPolicy endpoint selector for NFS-based ReadWriteMany volumes to properly allow NFS traffic when data is spread across multiple Linstor storage nodes (@mattia-eleuteri in [#2227]).

• [csi] Hide disk.img and lost+found from RWX NFS mounts: Fixed the Linstor CSI NFS server to exclude internal files (disk.img, lost+found) from being visible inside NFS-mounted volumes, preventing application errors caused by unexpected files in volume root directories (@mattia-eleuteri in [#2243]).

• [dashboard] Fix broken backup menu links missing cluster context: Restored cluster context in backup-related sidebar navigation links, fixing 404 errors when navigating to BackupJob and Plan pages from the cluster-level dashboard view (@kvaps in [#2232]).

• [dashboard] Fix StorageClass dropdown "Error" state by granting RBAC read access: Added a ClusterRole/ClusterRoleBinding to grant authenticated users read access to StorageClass resources, resolving the "Error" state displayed in StorageClass dropdowns on application forms (@sircthulhu in [#2267]).

• [postgres] Fix database deletion lifecycle management: Added cleanup stages to delete databases and orphaned roles when removed from values.databases, enabling declarative database lifecycle management and preventing stale data retention (@sircthulhu in [#2247]).

• [dashboard] Fix JSONPath crash on Tenant details with resourceQuotas: Restored fallback protection for unresolved flatMap placeholders in the ResourceQuota "Used" column, preventing JSONPath parser crashes on the Tenant details page (@sircthulhu in [#2249]).

• [system] Fix tenant RBAC for endpointslices read access: Added discovery.k8s.io/endpointslices read permissions to tenant ClusterRoles, resolving 403 errors on the Service details page when displaying the "Pod serving" section (@sircthulhu in [#2257]).

• [linstor] Fix swapped VMPodScrape job labels: Corrected the job label relabeling in LINSTOR VictoriaMetrics PodScrape templates, fixing linstorControllerOffline alerts that incorrectly reported satellite endpoints as controller failures (@sasha-sup in [#2264]).

• [piraeus-operator] Fix LINSTOR satellite alert labels and scrape flapping false positives: Fixed non-existent name label in linstorSatelliteErrorRate alert annotations (changed to hostname) and prevented false positives caused by scrape flapping and stale metric counters (@sasha-sup in [#2265]).

• [dashboard] Fix EndpointSlice column definitions for Pod serving table: Added missing CustomColumnsOverride for the EndpointSlice table on service details page, replacing "Raw:" prefixes and "Invalid Date" values with proper Pod, Addresses, Ready, and Node columns (@sircthulhu in [#2266]).

Dependencies & Version Updates

• [cilium] Update Cilium to v1.19.1: Upgraded the Cilium CNI to v1.19.1 with latest bug fixes and performance improvements (@BROngineer in [#2173]).

• [keycloak-operator] Update to v1.32.0: Updated the Keycloak Operator to v1.32.0 (based on epam/edp-keycloak-operator with upstream patches), bumping Keycloak to 26.5.2 (@lllamnyp in [#2206]).

• [postgres-operator] Update to v1.27.3: Upgraded the Postgres Operator (Patroni-based) to v1.27.3 with latest upstream fixes (@dmpopoff in [#2226]).

• [objectstorage-controller] Update to v0.2.2, drop upstreamed patches: Updated the object storage controller to v0.2.2 and removed patches that were accepted upstream, reducing the maintenance delta (@lexfrei in [#2261]).

• [kilo] Switch from fork to upstream squat/kilo: Replaced the Cozystack-maintained Kilo fork with the upstream squat/kilo image now that required patches (--internal-cidr, allowed-location-ips fix, preferred source for WireGuard routes, Cilium IPIP overlay support) have been merged upstream (@lexfrei in [#2259]).

• [talos] Bump Talos to v1.12.6: Updated the pinned Talos version to v1.12.6 (@kvaps in [#2254]).

• [talm] Release v0.22.4 (github.com/cozystack/talm): Fixed --file/--template flag requirement to prevent ambiguous invocations (@kvaps in cozystack/talm#112).

• [boot-to-talos] Release v0.7.0 (github.com/cozystack/boot-to-talos): Added support for ISO, RAW, and HTTP image sources (@lexfrei in cozystack/boot-to-talos#13); permanent MAC addresses for predictable interface names (@IvanHunters in cozystack/boot-to-talos#14); detection and workaround for 5-level paging (LA57) incompatibility with kexec (@IvanHunters in cozystack/boot-to-talos#15).

• [boot-to-talos] Release v0.7.1 (github.com/cozystack/boot-to-talos): Fixed EFI boot entry creation to use the target disk rather than relying on the installer disk, preventing boot failures on bare-metal systems (@kvaps in cozystack/boot-to-talos#16).

Development, Testing, and CI/CD

• [tests] Stabilize E2E Kubernetes tests: Comprehensive improvements to E2E test stability: pre-cleanup of leftover resources, fixes for port-forward race conditions and leaks, improved NFS PVC timeout and debug output, proper EXIT trap handling, and increased CAPI deployment timeouts (@lexfrei in [#2262]).

• [ci] Fix E2E check blocking docs-only PRs: Moved path filtering to the job level so that documentation-only pull requests are not blocked by pending E2E CI checks (@IvanHunters in [#2170]).

• [ci] Add timeout-minutes to Build and E2E jobs: Added explicit timeout-minutes constraints to Build and E2E workflow jobs to prevent stuck runners from consuming CI resources indefinitely.

Documentation

• [website] Complete CA rotation documentation: Added comprehensive CA certificate rotation procedures for all Cozystack system components (@kvaps in cozystack/website#406).

• [website] Add Ansible automated installation guide: Added a step-by-step guide for automated Cozystack installation using Ansible playbooks (@lexfrei in cozystack/website#442).

• [website] Add self-signed certificates configuration guide for OIDC: Added documentation for configuring Cozystack to use self-signed TLS certificates with OIDC providers, covering certificate authority setup and kubeconfig integration (@IvanHunters in cozystack/website#443).

• [website] Add custom metrics collection guide: Added a guide explaining how to configure custom Prometheus scrape targets using the new inlineScrapeConfig feature of tenant VMAgent (@IvanHunters in cozystack/website#444).

• [website] Add PackageSource/Package architecture to Key Concepts: Documented the PackageSource and Package CRD architecture, explaining how operators extend the platform with custom application catalogs (@IvanHunters in cozystack/website#445).

• [website] Add SchedulingClass operations guide: Added a guide covering SchedulingClass CRD creation, tenant assignment, and workload placement verification (@lllamnyp in cozystack/website#455).

• [website] Add VMInstance and VMDisk backups documentation: Added user-facing documentation for backing up and restoring virtual machine instances and VM disk images using Velero (@myasnikovdaniil in cozystack/website#456).

• [website] Update developer guide: Updated the developer guide with current build, test, and contribution workflows including OCIRepository and migration tooling (@myasnikovdaniil in cozystack/website#458).

• [website] Document keycloakInternalUrl platform value: Added documentation explaining how to configure keycloakInternalUrl for backend-to-backend OIDC token introspection in cluster-internal environments (@sircthulhu in cozystack/website#452).

• [website] Add DependenciesNotReady troubleshooting guide: Added a troubleshooting article explaining how to diagnose and resolve the DependenciesNotReady package status condition (@kvaps in cozystack/website#450).

• [website] Reorder installation steps for operator-before-platform: Updated the installation guide to install the cozystack-operator before applying the platform package, reflecting the correct dependency order (@sircthulhu in cozystack/website#449).

• [website] Update managed apps reference: Updated the automatically generated managed applications reference documentation to reflect new apps and schema changes in this release (@app/github-actions in cozystack/website#448).

• [website] Update screenshots for Cozystack v1.1: Refreshed dashboard screenshots to reflect the updated UI in Cozystack v1.1 (@kvaps in cozystack/website#447).

• [website] Enhance operator backups guide: Improved the backup and recovery guide for operators with additional recovery scenarios and procedures (@androndo in cozystack/website#440).

Contributors

We'd like to thank all contributors who made this release possible:

• @androndo
• @BROngineer
• @dmpopoff
• @IvanHunters
• @kvaps
• @lexfrei
• @lllamnyp
• @mattia-eleuteri
• @matthieu-robin
• @myasnikovdaniil
• @sasha-sup
• @sircthulhu
• @tym83
• @vnyakas

---

Full Changelog: https://github.com/cozystack/cozystack/compare/v1.1.0...v1.2.0