Security advisories: GHSA-fhm8-v27h-q65j (conduit-commons#342), GHSA-2cr9-8q6v-733r (conduit-commons#312)

Conduit v0.20.5

v0.20.5 is a patch release. It fixes three ways records could be changed without an error: copies of a record sharing nested data, Avro decoding dropping map keys, and Avro encoding rewriting the values it was given. It also fixes a self-referencing Avro schema hanging the pipeline. Embedded library: conduit-commons v0.7.1 → v0.7.2 (#3216). The built-in Postgres connector is v0.14.4 → v0.14.5, which fixes a silent value conversion after a column type change.

These fixes were planned for v0.20.3. v0.20.3 and v0.20.4 were cut on conduit-commons v0.7.1 and do not have them.

Upgrading from v0.20.1 or earlier? Read the v0.20.2 notes first.

Upgrade if you fan out, clone or split records

Copies of a record shared nested data (conduit-commons#336). Affected: v0.8.0 through v0.20.4.

Conduit copies a record when a pipeline sends it to two or more destinations (with one destination there is no copy), and the clone and split processors copy records too. Those copies shared their nested arrays and nested objects with each other and with the original. A processor that changed one of them in place changed every copy. The most likely case is a JavaScript processor (custom.javascript) that edits an array element on one destination's branch: the other destinations then received the edited value too. There was no error.

You are affected only if a pipeline has two or more destinations, or a clone or split processor, and a processor after that point changes a nested array or nested object in place. Standalone (WASM) processors are not affected unless the processor itself calls Record.Clone; rebuild those against a processor SDK built on conduit-commons v0.8.3 or later (SDK releases on fixed commons are in progress).

How to check: compare what two destinations of the same pipeline received for one source record, or compare a destination record with the source. A nested value that changed on a branch whose processors never touch it is the sign.

• Upgrade to v0.20.5. Each copy is now independent. Inferred Avro schemas and field types do not change.
• Records written before the upgrade are not repaired.

Upgrade if you decode Avro

Some nested Avro unions lost map keys on decode (conduit-commons#312). Affected: v0.20.0 through v0.20.4, reproduced. Earlier releases back to v0.9.0, the first with avro.decode, are probably affected but were not checked.

It happens when a union has a record, enum, fixed, map or array branch that itself contains another union, for example ["null", map<["null","int"]>]. Map keys could be dropped with no error, a nested map could be flattened one level, and some values stayed in their union wrapper. One shape failed with expected single value encoded as a map instead.

It affects avro.decode and the connector SDK's schema middleware, which decodes for every built-in destination when a record carries a schema.

How to check: see whether any schema you decode has the shape above. If one does, compare a sample of destination records with the source and look for missing map entries and values one level higher than expected.

• Upgrade to v0.20.5. Union values are now unwrapped by walking the schema together with the value.
• Records decoded before the upgrade are not repaired. Records that hit the error case failed and were not written.
• Standalone connectors and processors carry their own conduit-commons. Rebuild them once connector-sdk and processor-sdk releases built on conduit-commons v0.7.2/v0.8.3 or later are out (in progress).

Upgrade if you encode Avro

Avro encoding rewrote the values it was given (conduit-commons#342). Affected: v0.11.0 through v0.20.4.

To encode a union, the encoder wrapped each value in place, inside the record's own maps and arrays: "bar" became {"string": "bar"}. Anything else holding the same map or array saw the wrapped value afterwards. If that value was encoded again, the second encode either failed, or, when the wrapper happened to fit another branch of the union (for example a map of strings), succeeded and wrote the wrapper instead of the value. There was no error in that case.

This is the encoder behind avro.encode and behind the connector SDK's and processor SDK's schema middleware, which encode records by default (sdk.schema.extract.payload.enabled and .key.enabled are true).

You are affected only if a schema has a union with a map or array branch, or a union inside a map or array, and the same nested map or array is encoded twice. That happens when a record's Before and After share a nested value, when copies of a record shared nested data (#336 above), or when a connector reuses its own data between records.

How to check: decode destination records with their schema and look for values that arrived as a one-key map such as {"string": "…"} where a plain value was expected. Encode errors on records that share data with an earlier record are the loud form of the same bug.

• Upgrade to v0.20.5. Encoding no longer changes its input, and two encodes of one value give the same bytes.
• Standalone connectors and processors encode in their own process with their own copy of conduit-commons, so upgrading Conduit does not fix them. Rebuild them once connector-sdk and processor-sdk releases built on conduit-commons v0.7.2/v0.8.3 or later are out (in progress).
• Records written before the upgrade are not repaired.

Fixes

• A self-referencing Avro schema no longer hangs the pipeline (conduit-commons#314). A schema that refers to itself, directly or through another type (a linked list L{next: ["null","L"]}, a tree T{kids: array<"T">}), made schema parsing recurse until the process ran out of memory. Parsing now stops at a reference to a type already on the current path. Such schemas now parse, encode and decode.
• The built-in Postgres destination no longer converts values after a column type change (conduit-connector-postgres#367, fixed in built-in postgres v0.14.5, #3218). Affected: built-in postgres v0.14.4 and earlier. A running destination kept the statements it had prepared, with their parameter types, so after ALTER TABLE ... ALTER COLUMN ... TYPE on a table it writes, Postgres converted new values to the new type without an error: 5/2 written to a column changed from numeric to integer was stored as 3. The destination now describes each statement before it runs it (pgx describe_exec), so the next write after the change is typed for the new column and a value the new type cannot hold fails the write (the record is nacked) instead of being converted. Rows written before the upgrade are not repaired. An explicit default_query_exec_mode in the destination's url, keyword/value connection string or service file is kept as set, and an explicit cache mode logs a warning at start. Opt-out: add default_query_exec_mode=cache_statement to the url to reuse prepared statements as before. Use it on a direct connection only, and only if the tables the pipeline writes never change type while it runs; with it the bug above returns, and behind PgBouncer or another pooler that shares prepared statements a restart does not clear it.

Before you upgrade

• Some Avro values that encoded before now fail to encode. A value that fits more than one branch of a union is now an error (ErrSchemaValueMismatch, "ambiguous") instead of being written as one of the branches by guess. Examples: a map that is also a valid record of another branch, a string that is also an enum symbol in a union with an array or map branch, a time.Time in a union with both a date and a timestamp. In a test of 20,000 random union-heavy schema shapes, 18 now fail this way. Before, those 18 were written wrong without an error. The record is nacked and goes to the DLQ or stops the pipeline, depending on its DLQ settings. To fix it, pass the value in its named form ({"<branch name>": value}) or make the union's branches distinguishable.
• Record copies no longer share nested data (conduit-commons#336). A processor that relied on an in-place change on one copy showing up in another, on purpose or by accident, no longer sees it. Copying a record now allocates its nested arrays and objects instead of sharing them.
• Encoding allocates more. For schemas with unions through maps or arrays, each encode copies the containers it wraps.
• Neither cost was benchmarked on the v0.20 line, and we make no performance claim.
• Postgres destination: a value that does not fit the column type now fails the write. After an ALTER COLUMN ... TYPE on a running pipeline, such a record is nacked and goes to the DLQ or stops the pipeline, depending on its DLQ settings. Before, it was converted without an error. Describing each statement adds a round trip. Measured on the postgres connector's main branch, throughput was 35% and 16% lower in the measured cases; it was not measured on v0.14.x or on the v0.20 line. Opt-out: default_query_exec_mode=cache_statement (see Fixes).
• Decoded Avro maps can change shape. For schemas with the conduit-commons#312 shape, decoded records now carry every map key, and values are no longer left in their union wrapper. Consumers that worked around the old shape need to change back.
• Self-referencing Avro schemas now work. Before, they hung the pipeline, so nothing relies on the old behavior.
• Downgrading to v0.20.4 or earlier brings back these bugs. Nothing stored changes between v0.20.4 and v0.20.5.

Known issues

• Processors that address a nested field fail after fan-out, clone or split. field.set, field.exclude, field.rename and other processors that use a nested reference such as .Payload.After.a.b fail on a record that was copied by fan-out to two or more destinations, or by clone / split. The record is nacked and goes to the DLQ or stops the pipeline; nothing is changed silently. With one destination and no clone or split, they work. This is unchanged from earlier v0.20 releases. Workaround: put nested-path processors before the fan-out, as pipeline-level processors rather than destination-level ones. Fixed in v0.21.
• Everything under Known issues in the v0.20.1 and v0.20.2 notes still applies, except conduit-commons#312 and [#314], which are fixed here.

Verify this release

Requires cosign v3 or later. Download checksums.txt, checksums.txt.sigstore.json and your files into one directory:

:::sh
cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.5' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json \
  checksums.txt

# only once the signature verifies (on macOS: shasum -a 256 --check --ignore-missing)
sha256sum --check --ignore-missing checksums.txt

The Docker image (pin the digest it prints):

:::sh
cosign verify \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.5' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/conduitio/conduit:v0.20.5

The SBOM is attested on each platform digest, not the tag or index:

:::sh
DIGEST=$(docker buildx imagetools inspect --raw ghcr.io/conduitio/conduit:v0.20.5 \
  | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "arm64") | .digest')

cosign verify-attestation \
  --type spdxjson \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.5' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  "ghcr.io/conduitio/conduit@$DIGEST" \
  | jq -r '.payload | @base64d | fromjson | .predicate' > conduit-linux-arm64.spdx.json

Use amd64 for the other platform.

---

Changelog

• 54f2ba703c5ce39cb93771d8999fb6e2e09c6f87: fix(deps): bump conduit-commons to v0.7.2 (#3216) (@devarismeroxa)
• 37d8d8a48459852e85d743dd24f4e892c0d07871: fix(deps): bump conduit-connector-postgres to v0.14.5 (#3218) (@devarismeroxa)

Release Artifacts

OSX Darwin

• 📦 conduit_0.20.5_Darwin_arm64.tar.gz (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Darwin_x86_64.tar.gz (conduit.gateway.scarf.sh)

Linux

• 📦 conduit_0.20.5_Linux_arm64.deb (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_arm64.rpm (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_arm64.tar.gz (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_i386.deb (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_i386.rpm (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_i386.tar.gz (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_x86_64.deb (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_x86_64.rpm (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Linux_x86_64.tar.gz (conduit.gateway.scarf.sh)

Windows

• 📦 conduit_0.20.5_Windows_arm64.zip (conduit.gateway.scarf.sh)
• 📦 conduit_0.20.5_Windows_x86_64.zip (conduit.gateway.scarf.sh)

Verifying

checksums.txt is signed with Sigstore keyless signing (checksums.txt.sigstore.json), and each archive has an SPDX SBOM. The Docker image is signed by digest. See Verifying a release.

cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.5' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json checksums.txt
sha256sum --check --ignore-missing checksums.txt

Docker images

To pull the image, you can execute the following command:

docker pull conduit.docker.scarf.sh/conduitio/conduit:v0.20.5