Conduit v0.20.2
v0.20.2 is a patch release. It fixes a bug where avro.encode could silently write wrong data when
two pipelines in one Conduit process use schemas that share type names. It also fixes four security
issues, and adds a guard so that v0.20.2 refuses to run pipelines stored by v0.21. Embedded libraries:
conduit-commons v0.7.0 → v0.7.1, conduit-schema-registry v0.2.6 → v0.2.7. The built-in connectors are
unchanged (postgres v0.14.4, s3 v0.9.5, kafka v0.12.6).
Upgrading from v0.20.0? Read the v0.20.1 notes first. Their upgrade and downgrade warnings still apply.
Upgrade if you encode Avro
avro.encode could write records with another schema's encoder (conduit-commons#305,
conduit-commons#308, GHSA-hx9g-59x7-9h4x). Affected: v0.7.0 through v0.20.1: avro.encode since v0.9.0,
and the encodewithschemakey / encodewithschemapayload processors in v0.7 and v0.8, which use the
same code. There is no error or warning, and the records are acked.
It happens when one Conduit process encodes with two schemas that both define a named type (a
record, enum or fixed, including a decimal fixed) with different definitions, and refer to it
by name inside an array, map or union. Typical cases are two pipelines whose subjects share type
names, or two versions of one subject. The second schema is encoded with the first schema's encoder:
- record fields are written in the other schema's order;
- enum values point at the wrong symbol;
- decimals are written at the wrong scale (1.5 becomes 15000).
Which pipeline gets the wrong encoder depends on which one encodes first, so a restart can move the corruption rather than clear it. Built-in connectors that supply their own schemas go through the same encoder and are exposed the same way. Decoding is not affected. A single self-contained schema, and the schemas the connector SDK extracts on its own, never trigger it.
What to do:
- Upgrade to v0.20.2. Each schema now encodes with its own encoder cache, and the bytes for schemas that never collided are identical.
- Records written before the upgrade are not repaired. If your schemas match the conditions above, decode a sample of the output with each record's own schema and look for swapped fields, wrong enum symbols, or decimals off by a power of ten.
- Standalone connectors and processors carry their own copy of conduit-commons. They are exposed only if they encode colliding schemas in their own process. Rebuild them against conduit-commons v0.7.1 or v0.8.1.
- If you cannot upgrade yet: give every named type a full name that is unique to its schema (for example a namespace per subject and version), or run pipelines with colliding schemas in separate Conduit processes.
A Conduit regression test with two pipelines using avro.encode with shared type names is in
[#3106].
Security fixes
Details are in each advisory.
- Schema identity (GHSA-94xc-rjcp-83cr, GHSA-pfr7-p89q-w2jp, GHSA-95wx-33xh-gpvw). conduit-commons, the built-in schema registry (conduit-schema-registry) and Conduit's schema registry client could treat two different schemas as the same schema. Someone who can register schemas in the same registry, for example by creating a pipeline, could cause their schema to be used in place of another one. Fixed in conduit-commons v0.7.1, conduit-schema-registry v0.2.7 and Conduit (#3106). All three fixes are needed together; v0.20.2 ships all three.
- Standalone plugins inherit Conduit's environment (GHSA-gwvc-9h8p-x5r7). A standalone connector plugin receives every environment variable of the Conduit process, including credentials meant only for Conduit. v0.20.2 does not change what plugins receive. It names the credential-like variables at startup so you can act on them (see below). A future release restricts the plugin environment to an allowlist.
Before you upgrade
- New startup warning for standalone plugins (#3045, [#3054], [#3065]; backport [#3084]). When
connectors.pathholds at least one standalone connector, Conduit logs one warning per process with the codeplugin.standalone_env_passed. It lists the names, never the values, of host variables that look like credentials and that the plugins will receive.conduit doctorreports the same condition asplugins.standalone_env, including in--json; the MCPdoctortool reports only a count. Start Conduit with a minimal environment. Once you have reviewed the list, silence the warning withconnectors.standalone.env-warning: false,--connectors.standalone.env-warning=falseorCONDUIT_CONNECTORS_STANDALONE_ENV_WARNING=false. That only silences the warning. Runbook:docs/operations/standalone-plugin-environment.md; site: Standalone plugin environment. The allowlist becomes the default in the minor release after this one, and its notes will say so. - Downgrade guard for v0.21 databases (#3062). v0.21 stores
${VAR}references in a newSettingRefsfield and takes them out of the settings. v0.20.2 refuses to start a pipeline whose stored connector, processor or DLQ settings carrySettingRefs. The pipeline is markedDegradedwithpipeline.settings_references_unsupported. Nothing is read or acked, and the stored records are left as they are. Without the guard, such a pipeline would run with those settings missing, and an optional credential could silently fall back (S3 to the default AWS credential chain, Kafka to no SASL). Deleting such a pipeline or connector is refused too, so a connector's cleanup never runs without its settings. Heal it by running v0.21 or later, or by re-provisioning it from its pipeline file. Runbook:docs/operations/settings-references-unsupported.md. v0.20.0 and v0.20.1 do not have this check: do not downgrade from v0.21 to them unless the pipelines directory is mounted and the reconcile succeeds. - Avro: a schema that uses a type it does not define now fails to parse (conduit-commons#308).
Before, it borrowed the definition from whichever schema in the process had defined that name
first. The same schema already failed in a fresh process. It now fails with
could not parse avro schemaevery time. - Avro encoders are built once per schema, not once per process. Schemas are cached, so each one compiles once on the normal path. This was not benchmarked, and we make no performance claim.
- Downgrading to v0.20.1 or earlier brings [#305] back. Nothing stored changes between v0.20.1 and v0.20.2, so downgrading is otherwise as safe as the v0.20.1 notes describe.
Behavior changes
| Change | v0.20.1 | v0.20.2 | Code | PR |
|---|---|---|---|---|
Start a pipeline whose stored settings carry SettingRefs |
started without those settings | refused, Degraded |
pipeline.settings_references_unsupported, FailedPrecondition |
[#3062] |
| Delete such a pipeline or connector | cleanup ran without the settings | refused, retried on next boot | pipeline.settings_references_unsupported |
[#3062] |
| Standalone plugins with credential-like host variables | nothing | one startup WARN and a doctor finding | plugin.standalone_env_passed |
[#3084] |
| Avro schema using a type it does not define | resolved from another schema | parse fails | — | [#3106] |
| Avro encode with a colliding named type | another schema's encoder | own encoder | — | [#3106] |
New flag: connectors.standalone.env-warning (default true).
Known issues
- Everything under Known issues in the v0.20.1 notes still applies.
- A self-referencing Avro schema hangs the pipeline (conduit-commons#314). A schema that refers
to itself, directly or through another type (a linked list
L{next: ["null","L"]}, a treeT{kids: array<"T">}), makes schema parsing recurse without bound: the pipeline hangs and memory grows until the process dies. One source emitting such a schema is enough. Affects v0.20.0, v0.20.1 and v0.20.2. Workaround: do not use self-referencing Avro schemas. Fix in v0.20.3. - Some nested Avro unions can silently lose map keys on decode (conduit-commons#312). This
affects
avro.decodeand the connector SDK's schema middleware, so destinations that decode Avro are affected too. It happens when a union has arecord,enum,fixed,maporarraybranch that itself contains another union, for example["null", map<["null","int"]>]. Map keys can be dropped with no error, and some values can stay in their union wrapper; one shape fails withexpected single value encoded as a mapinstead. Workaround: avoid such schemas. Fix in v0.20.3, which follows shortly. - Open conduit-commons Avro issues: a decoded nullable record or map does not re-encode with its own
schema (conduit-commons#306);
local-timestamp-*values go throughtime.Local(conduit-commons#307).
Verify this release
Requires cosign v3 or later.
Download checksums.txt, checksums.txt.sigstore.json and your files into one directory:
:::sh
cosign verify-blob \
--certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
--bundle checksums.txt.sigstore.json \
checksums.txt
# only once the signature verifies (on macOS: shasum -a 256 --check --ignore-missing)
sha256sum --check --ignore-missing checksums.txt
The Docker image (pin the digest it prints):
:::sh
cosign verify \
--certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
ghcr.io/conduitio/conduit:v0.20.2
The SBOM is attested on each platform digest, not the tag or index:
:::sh
DIGEST=$(docker buildx imagetools inspect --raw ghcr.io/conduitio/conduit:v0.20.2 \
| jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "arm64") | .digest')
cosign verify-attestation \
--type spdxjson \
--certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.2' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
"ghcr.io/conduitio/conduit@$DIGEST" \
| jq -r '.payload | @base64d | fromjson | .predicate' > conduit-linux-arm64.spdx.json
Use amd64 for the other platform.
