Conduit v0.20.1

v0.20.1 is a patch release. It fixes data-loss bugs in the built-in Postgres, S3 and Kafka connectors, and lifecycle bugs where a failed status write could leave a run that could not be stopped, or let one pipeline run twice. Embedded connectors: postgres v0.14.2 → v0.14.4, s3 v0.9.3 → v0.9.5, kafka v0.12.5 → v0.12.6. Read Before you upgrade first: some Postgres pipelines stop on purpose, the first restart can re-deliver records, and downgrading is not safe for every pipeline.

Upgrading from v0.19.0? Read the v0.20.0 notes first. Their behavior changes still apply.

Upgrade if you run v0.20.0

• Postgres CDC could lose rows on restart (conduit-connector-postgres#331), since 2022: interleaved transactions and rows sharing one log position (COPY, multi-row inserts). Fixed in v0.14.3 (#2938), included in v0.14.4 (#3052).
• Kafka destination could ack a record that failed to write in a multi-record batch (conduit-connector-kafka#308). Fixed in v0.12.6 (#3049).
• Postgres destination acked a batch whose COMMIT failed (conduit-connector-postgres#342), since v0.6.0. Fixed (conduit-connector-postgres#351).
• Postgres DELETE stopped Avro pipelines (conduit-connector-postgres#379), since v0.11.0: the pipeline restarted on the same record forever while the slot kept WAL. Fixed (conduit-connector-postgres#385); upgrading alone unsticks it.
• Postgres wrote unchanged TOASTed columns as NULL (conduit-connector-postgres#317). Fixed (conduit-connector-postgres#387); one case still halts, see Known issues.
• Postgres NUMERIC (conduit-connector-postgres#364, conduit-connector-postgres#368, conduit-connector-postgres#375): stale scale at the destination; source panic on NaN/Infinity; source halt on an unconstrained numeric of 10000 or more. Fixed (conduit-connector-postgres#366, conduit-connector-postgres#372, conduit-connector-postgres#382).
• Postgres lifecycle (conduit-connector-postgres#346, conduit-connector-postgres#347, conduit-connector-postgres#354): wedged CDC, panic on stop during start, Teardown under a running Write, a goroutine leak. Fixed (conduit-connector-postgres#353, conduit-connector-postgres#357, conduit-connector-postgres#363).
• S3 CDC skipped objects for good (conduit-connector-s3#1029, conduit-connector-s3#1032). Fixed in v0.9.5 (conduit-connector-s3#1037).
• S3 destination panicked on a delete record, and on every restart after (conduit-connector-s3#1030). Fixed (conduit-connector-s3#1039).
• S3-compatible stores such as MinIO now work via aws.endpoint and aws.pathStyle, both off by default (#2940).
• Shutdown could hang if a source failed during stop (#2969). Fixed (#2975).
• A failed status write could orphan a live run, degrade a running pipeline, or start a second run on one source (#2898, [#2899]). Fixed (#2941, [#2942], [#2955], [#2962]).
• One failed connector state write failed every connector in its shared write (#2930), including other pipelines'. Fixed (#2950).

Before you upgrade

• Postgres: some pipelines stop on purpose (v0.14.4).
• REPLICA IDENTITY USING INDEX on an index whose key columns exclude the primary key is refused at start with postgres.replica_identity.excludes_key. Such tables emitted deletes with a null key, which the destination applied as a no-op, so deleted rows stayed. Use REPLICA IDENTITY FULL or a unique index that includes the primary key, and check destinations for rows that should be gone.
• Under Avro, infinity/-infinity in a date, timestamp or timestamptz halts with postgres.source.temporal_not_representable. It used to be written silently as 1970-01-02 / 1969-12-31, so those rows are wrong downstream. Restart once with logrepl.withAvroSchema: false from the same position, then switch back. NaN in a numeric(p,s) halts with postgres.source.numeric_not_representable (it used to panic).
• Postgres: record shape changes (v0.14.4).
• Under the default replica identity, deletes and key-changing updates no longer carry Payload.Before; the key is in Key. Unchanged TOASTed columns are omitted instead of NULL (with Avro on, the source halts instead, conduit-connector-postgres#383). Under REPLICA IDENTITY FULL they carry their value.
• An unconstrained numeric is now an Avro string. Consumers see a new schema version and must handle "NaN", "Infinity" and "-Infinity".
• The destination no longer checks that a column is numeric: a decimal string is stored in any column whose text input accepts it. That is a silent coercion if a schema maps a numeric field to the wrong column. Rows written by the old rounding (2.50 stored as 3) are not repaired.
• Postgres: duplicates on the first restart. v0.14.2 positions carry no transaction information, so the first restart after the upgrade uses a fallback. It loses nothing but can re-deliver acked records: up to everything committed while the checkpointed record's transaction was open, plus that transaction's delivered rows. That is bounded by how long the transaction was open, not by a small count. Non-idempotent sinks and processors see the duplicates once; upsert-by-key destinations absorb them. After the first new checkpoint, resumes are exact. Positions are now format version 2 (tx_commit_lsn, tx_seq).
• Postgres: downgrading is not safe. Downgrading to v0.20.0 can lose CDC rows on the first restart and may stop the subscription; the log then shows handler error: out of order commit while the pipeline still reports running. Do not downgrade a pipeline with interleaved transactions or COPY traffic in flight. If you must roll back, expect to re-snapshot.
• Postgres: SCRAM channel binding is off by default. v0.14.4 maps channel_binding=prefer (the default, or set in the URL or PGCHANNELBINDING) to disable in source and destination. Servers with an Ed25519 certificate, and TLS re-terminating proxies that pass SASL through, failed with SCRAM channel binding negotiation error or channel binding check failed; they now work. To keep channel binding, set channel_binding=require in the connection URL.
• S3: CDC re-delivers recent objects (v0.9.5). After a restart and at the snapshot handoff, objects modified within lookbackPeriod (default 5 minutes) of the newest one, plus everything written during the snapshot, are delivered again. Destinations must tolerate duplicates. A freshly bulk-loaded bucket delivers its last window twice. If tracking state would exceed about 256 MiB (roughly a million objects), the source shortens its window and logs s3.source.dedupe_window_shrunk; lower lookbackPeriod for such pipelines. It halts with s3.source.dedupe_window_overflow when too many versions share one second, or when future-stamped entries (a store clock ahead of the host) do not fit the budget. See the s3 v0.9.5 notes.
• S3: writes that got HTTP 200 with an <Error> body now fail and are retried. Before, they were acked. Only stores that answer that way are affected. To roll back to v0.20.0, first remove aws.endpoint and aws.pathStyle from every pipeline: S3 v0.9.3 refuses them as unrecognized parameters. Positions are unchanged between v0.9.3 and v0.9.4 (#2940).
• Settings updates with "***" (#2926). A redacted "***" keeps the stored secret only if nothing else in the update changes; otherwise the update is refused. Clients that change one field and send the rest back as "***" must send the real secret values.
• Avro decode limit. An array or map that declares more than schema.avro.max-elements elements fails to decode with schema.avro.limit_exceeded; nothing is truncated. Default 1,000,000; 0 means no limit; a negative value fails config validation. Set it with --schema.avro.max-elements, CONDUIT_SCHEMA_AVRO_MAX_ELEMENTS or schema.avro.max-elements in conduit.yaml. If you decode legitimate arrays or maps over 1,000,000 elements, raise it before you upgrade. It covers decodes inside the Conduit process, not standalone plugins. Sources still encode records over the limit, so a destination can reject its own source's record, failing the whole batch. With the default DLQ the pipeline stops Degraded or, with max-retries: -1, retries forever in Recovering (#2997). Runbook: docs/operations/avro-limit-exceeded.md. The codec is now iskorotkov/avro/v2, a maintained fork; the wire format is unchanged.
• Storage engines. badger v4.9.1 → v4.9.6, modernc.org/sqlite v1.38.0 → v1.46.1. New CI tests open databases across old and new versions in both directions and resume a stored position each way. badger lists no on-disk format change.
• New error codes: pipeline.stopping, pipeline.status_persist_failed, pipeline.config_changing, pipeline.archv2_destination_batching_unsupported, three orchestrator.redacted_setting_* codes, schema.avro.limit_exceeded and registry.processor_validation_timeout.

Behavior changes

Notes:

• "***": the three codes are HTTP 400, CLI exit 2. Create and apply calls refuse "***", and a literal *** can only be set through YAML. Keys left out of an update are still removed.
• arch-v2 batching: remove sdk.batch.size and sdk.batch.delay from the destination, or use the default engine. Runbook: docs/operations/archv2-destination-batching.md.
• Status writes: failures log pipeline.status_persist_failed and increment conduit_pipeline_status_persist_failures_total. It should stay at 0; when it rises, look at the database.

Fixes

• Source teardown waited 10s after a plugin stream ended (#2900, [#2947]).
• arch-v2 destination batching stalled (#2979); now refused (#2987).
• generate.provider was not read (#2908) and --max-retries showed 0 (#2933); fixed in [#2923].
• --help listed wrong exit codes (#2907, [#2920]).
• A cancelled call could wedge a standalone WASM processor (#2981, [#3039]).
• conduit run --dev restarted its debounce window twice; no apply was dropped (#2995).
• Also: arch-v2 fan-out record sharing (#2946), pinned release tooling (#2919), image SBOMs (see Verify this release), docs (#2982, [#2984]).

Known issues

• Secrets passed as ${VAR} in a pipeline config file are stored in plaintext in Conduit's database (#2988). The expanded value is stored in badger, SQLite or Postgres and in every backup; the API redacts it, the store does not. Restrict access to the database files and treat the database and its backups as secret. A fix is being designed for v0.21; existing databases will need a migration or a re-provision.
• Postgres v0.14.4:
• An UPDATE leaving a TOASTed column unchanged halts the source under the default replica identity with Avro on. Workaround: REPLICA IDENTITY FULL (conduit-connector-postgres#383).
• A NULL in a nullable column can fail Avro encoding (conduit-connector-postgres#326).
• A key-changing UPDATE leaves a duplicate at the destination (conduit-connector-postgres#386).
• Zero numeric reads 0 in snapshot, 0.000 in CDC (conduit-connector-postgres#391).
• NULL uuid is emitted as "" (conduit-connector-postgres#392).
• int8 max as a JSON number is stored as its minimum on amd64 (conduit-connector-postgres#402).
• A cancelled write can still be applied by the server. It is not acked and is redelivered, so a destination table without a key can get a duplicate row.
• S3 v0.9.5:
• CDC misses uploads longer than lookbackPeriod and reports no deletes in unversioned buckets.
• The snapshot's last object is delivered again (conduit-connector-s3#1040).
• A restart during the snapshot starts it over (conduit-connector-s3#1041).
• Static keys only, no AWS default credential chain (conduit-connector-s3#1021); see [#2988].
• A real AWS regional aws.endpoint logs a false warning (conduit-connector-s3#1022).
• arch-v2:
• The batching refusal at boot is only logged; the pipeline shows stopped with no error (#2989).
• A stopping pipeline can report the wrong teardown error; nothing is lost (#2966).
• Destinations buffering outside the SDK can still stall; SIGTERM does not stop them (#2990).
• Default engine: a source fetcher can spin after its node stops; nothing is acked or dropped (#2976).
• Status writes are not bounded on badger, the default store (#2964).
• No chaos test for a source failure racing a graceful stop (#2977).
• pipeline.shutting_down rarely reaches API clients; they see common.unavailable (#2918).
• Concurrent config writes on one pipeline can leave orphans. Avoid them (#2992).
• Advisories GO-2026-5046/5047/5048 (via the Postgres connector's hamba/avro/v2) and GO-2026-5932 (golang.org/x/crypto/openpgp) have no fix and are in the govulncheck baseline.

Deferred

• A single pipeline engine (v0.21, design [#2971]). Until then, the RAG template needs --preview.pipeline-arch-v2.
• Debezium-compatible Postgres CDC output (design [#3033]) and Protobuf decode → v0.21.
• The v0.22 items in the v0.20.0 notes are unchanged.

Verify this release

Requires cosign v3 or later. Download checksums.txt, checksums.txt.sigstore.json and your files into one directory:

:::sh
cosign verify-blob \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  --bundle checksums.txt.sigstore.json \
  checksums.txt

# only once the signature verifies (on macOS: shasum -a 256 --check --ignore-missing)
sha256sum --check --ignore-missing checksums.txt

The Docker image (pin the digest it prints):

:::sh
cosign verify \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/conduitio/conduit:v0.20.1

The SBOM is attested on each platform digest, not the tag or index:

:::sh
DIGEST=$(docker buildx imagetools inspect --raw ghcr.io/conduitio/conduit:v0.20.1 \
  | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "arm64") | .digest')
# or: DIGEST=$(crane digest --platform linux/arm64 ghcr.io/conduitio/conduit:v0.20.1)

cosign verify-attestation \
  --type spdxjson \
  --certificate-identity 'https://github.com/ConduitIO/conduit/.github/workflows/release.yml@refs/tags/v0.20.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  "ghcr.io/conduitio/conduit@$DIGEST" \
  | jq -r '.payload | @base64d | fromjson | .predicate' > conduit-linux-arm64.spdx.json

Use amd64 for the other platform. Against the tag or index, verify-attestation fails with none of the attestations matched the predicate type; that is expected.