Merged PRs

dolt

• 10773: go: cmd/dolt: sqlserver: metrics_listener.go: Only run the background goroutine for updating system metrics and cluster state if we actually have a metrics port configured. Related to feature request [#10563].
• 10772: Relax PK column mapping to match by name if tags don't match
• 10771: go: sqle: commit_hooks.go: Quiesce async_push_sync_replica thread when there is no work. Signal over a channel when new work is potentially pending. Related to feature request [#10563].
• 10770: go: sqle: auto_gc.go: Remove periodic call to checkForGC in autoGCCommitHook thread. Previously we wanted standby replicas to run auto gc, but standby replicas did not fire commit hooks when receiving writes over remotesapi. To get autogc working on standby replicas, we had the thread wake up every second and check the database sizes. The hooks behavior was changed in [#10722]. As a consequence, we no longer need to wake up every second and check database sizes. It is more efficient to not do so. One behavior change to note: this periodic check would also apply to long running transactions which generate lots of writes/garbage but which never commit and so never generate database writes. Now that pattern will have to wait for an actual database write to trigger an auto gc. A sequence of transactions which generate a lot of garbage but which always rollback will also fail to trigger auto gc. To fix this, auto gc should be triggering on a table file persister callback, not on a commit callback. Related to the user request in [#10563].
• 10761: go: store/nbs: Disable conjoin when the store is being garbage collected. A conjoin running during GC could violate the assumptions made by the garbage collector about which files it has created and were safe to reference in the future. A separate issue on a specialized write path which GC uses to interact with the store means that the store can be updated to reference files which no longer exist. The end result was that the GC would fail and the database would enter a state that would require expert manual invervention to recover. Trigger this path requires a dolt_gc('--full') to be called when the oldgen is already in a very particular state. This path does not effect auto_gc.
• 10752: Bump fast-xml-parser from 5.5.6 to 5.5.9 in /.github/actions/ses-email-action Bumps fast-xml-parser from 5.5.6 to 5.5.9.
  
  Commits

  • See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

• 10747: /go/go.{mod,sum} bump google.golang.org/grpc to 1.79.3
• 10744: go: sqle: txLocks: Move txLocks to be on the DatabaseProvider. This has no real impact on dolt itself. Removing global state is part of improving dolthub/driver support.
• 10740: Prevent panic when first branch control load fails
• 10739: Delay autogc during heavy CPU load autogc and archiving causes spikes during TPCC benchmarks, specifically the delivery transaction. benchmark. This PR implements a very naive scheduler that delays garbage collection when the CPU is busy. There is an environment variable and logic that allows us to select different gc schedulers (if more are added in the future), and it currently defaults to NAIVE.
• 10736: Fix getting gen sys tables Related to my last change, decided on this design.
• 10735: go: sqle: replication.go: Use unique thread names in BackgroundThreads for each async replication background thread. While this does not impact behavior today, because BackgroundThreads is happy to overwrite existing map entries and leave the old threads still running, if BackgroundThreads adds more requirements or instrospection in the future, having unique names will be nice.
• 10733: go: cmd/dolt: cli/stdio.go: Redirect Stdout and Stderr to DevNull for now. Yeeting it into random UUID files in $TMPDIR is not the right approach.
• 10731: Fix dolt push failing when remote has ignored tables in working set dolt push returned "target has uncommitted changes. --force required to overwrite" when the remote's working set contained tables listed in dolt_ignore, even though dolt status showed clean. Push now succeeds in this case, consistent with how other commands (rebase, pull, stash) already handle ignored tables. Fix [#10727]
• 10725: restore forward-compatibility testing We can only go back to around v1.50.0, because beyond that clients can't read the archive format created by a remote push (test setup involved pushing to a remote and then cloning from it). But this will at least alert us when we're about to make a change that can't accommodate older clients going forward.
• 10718: Get generated sys tables correctly Some current issues where when looking a specific scheme, we call getGeneratedSytemTables, which gets system tables from all schemas, resulting in trying to check for tables that don't exist in scope.
• 10638: go: dolt_backup.go: More efficiently restore backups. When dolt backup restore was migrated to SQL, it lost the ability to restore from table files. It always did the merkle dag pull instead. This restores the ability of dolt backup restore to do a table file pull when that is appropriate.
• 10616: Codeaucafe/10488 Companion [#10549]
• 10549: [#10488]: Add --overwrite-ignore and --no-overwrite-ignore This PR adds --overwrite-ignore and --no-overwrite-ignore to dolt checkout (for both CLI and SQL procedure). This attempts to mirror Git's behavior. By default, --overwrite-ignore is implied, and checkout silently overwrites ignored tables during a branch switch. When --no-overwrite-ignore is specified, checkout aborts if the target branch would overwrite ignored tables in the working set. Per Git docs, the check only triggers "when the new branch contains ignored files," so tables that only exist locally are not flagged.
• Register flag constants and argparser entries
• Add overwriteIgnore bool through all checkout paths: doDoltCheckout -> checkoutNewBranch / checkoutExistingBranch / checkoutRemoteBranch -> doGlobalCheckout -> MoveWorkingSetToBranch -> transferWorkingChanges -> RootsForBranch
• Add FindOverwrittenIgnoredTables() using dolt_ignore to detect ignored tables that differ between working set and target branch
• Add ErrCheckoutWouldOverwriteIgnoredTables error type with clear user-facing message Closes [#10488] Closes [#10616]

go-mysql-server

• 3490: bug fix: add check for unmatched column conjCollector maintains a map of column ordinals, but the code in addEq() and addIneq() wasn't normalizing the column name before doing a lookup in the column ordinal map. These functions also weren't checking if there wasn't a matching column in the ordinal map. This resulted in one of our query plan tests choosing the wrong index, because the zero value of the missed ordinal lookup incorrectly pointed to an entry in the ordinal map that contained the wrong value.
• 3488: Fix WHERE filter on view columns backed by string literals fails with index out of bounds When using CREATE VIEW v(c1, c2) AS SELECT ..., 'literal' from t, filtering the view with WHERE col2 = 'something' would panic with "unable to find field".
• Fix WHERE filters on views with explicit column name lists failing when a renamed column maps to a string literal in the inner SELECT Fix [#10741]
• 3487: /.github/workflows: bump checkout action
• 3485: Fix prefix index validation for non-utf8mb4 charsets Fixes incorrect rejection of prefix indexes on CHAR, VARCHAR, and TEXT columns when the column's charset is not utf8mb4. For example, code(12) on a VARCHAR(32) CHARACTER SET latin1 column now works as expected.
• Prefix length validation now uses the column's actual bytes-per-character instead of assuming 4 (utf8mb4)
• ErrInvalidIndexPrefix, ErrKeyTooLong, ErrInvalidBlobTextKey, and ErrKeyZero now map to their correct MySQL error codes (1089, 1071, 1170, 1391) Fix [#3459]
• 3483: Fix MATCH AGAINST fails when fulltext-indexed table is accessed via JOIN alias Fix MATCH AGAINST queries that use a JOIN returning no results or duplicate rows.
• Fix the query planner skipping the fulltext index when the table carrying the index was accessed through an alias.
• Fix the row evaluator reading key values from the wrong position in a joined row for LEFT JOIN queries.
• Fix incorrect column positions in join conditions when the query optimizer placed the fulltext-filtered table on the left side of an INNER JOIN. The IndexedTableAccess node for fulltext was created without column ID metadata, so execution index assignment resolved join condition references to the wrong positions in the joined row and returned duplicate rows. Fix [#10729]
• 3482: Allow infinity and NaN float64 values fixes [#10710] Inf and NaN values are allowed in Postgres. MySQL handles Inf values as an out-of-range +/-MaxFloat64; sometimes erroring out and sometimes not. To be more compatible with Doltgres and to avoid handling Inf values on a case-by-case basis, we've decided to just allow Inf and NaN values. The original guarding against Inf and NaN was done to fix [#3264], where we were panicking when converting Inf and NaN values to Decimals. This was because shopspring/decimal does not allow converting NaN and Inf values to decimal.Decimal. This PR will still guard against NaN and Inf being converted to decimal.Decimal. Doltgres bump: dolthub/doltgresql#2502
• 3481: Improve performance when displaying zero timestamps. Calling appendMicroseconds with a known value of 0 microseconds is slow because we end up looping for each digit we write. It's faster to just cache the zero string for each level of precision and write those directly.
• 3480: Improve performance when displaying timestamps with precision. This avoids writing extra characters that will get stripped afterward, which could be expensive if it causes the slice to be reallocated.
• 3479: sql: background_threads.go: Have Shutdown return nil error on success.
• 3476: Display correct precision for time types Fixes https://github.com/dolthub/dolt/issues/10089 Previously we were stripping trailing zeros from sub-second precision, but to match MySQL we should display the same number of decimal places as specified by the original type. (e.g. DATE(1) always displays one sub-second digit, etc.) Unfortunately this makes https://github.com/dolthub/dolt/issues/10661 more visible in places where users use a raw TIME type. TIME in MySQL is the same as TIME(0) (no sub-second precision), while TIME in Dolt is the same as TIME(6) (microsecond precision). Previously, if a user created a column of type TIME and only inserted timestamps without sub-second precision, those values would be displayed without sub-seconds, resulting in the same values as MySQL over the wire and creating the illusion that the types were the same. Now we always return microsecond precision for TIME, making the disparity more visible.
• 3474: Bump google.golang.org/grpc from 1.56.3 to 1.79.3 Bumps google.golang.org/grpc from 1.56.3 to 1.79.3.
  
  Release notes

  Sourced from google.golang.org/grpc's releases.

  Release 1.79.3

  Security

  • server: fix an authorization bypass where malformed :path headers (missing the leading slash) could bypass path-based restricted "deny" rules in interceptors like grpc/authz. Any request with a non-canonical path is now immediately rejected with an Unimplemented error. (#8981](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8981">/issues/8981))

  Release 1.79.2

  Bug Fixes

  • stats: Prevent redundant error logging in health/ORCA producers by skipping stats/tracing processing when no stats handler is configured. (grpc/grpc-go#8874](https://github.com/href="https://redirect.github.com/grpc/grpc-go/pull/8874">grpc/grpc-go/issues/8874))

  Release 1.79.1

  Bug Fixes

  • grpc: Remove the -dev suffix from the User-Agent header. (grpc/grpc-go#8902](https://github.com/href="https://redirect.github.com/grpc/grpc-go/pull/8902">grpc/grpc-go/issues/8902))

  Release 1.79.0

  API Changes

  • mem: Add experimental API SetDefaultBufferPool to change the default buffer pool. (#8806](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8806">/issues/8806))
    • Special Thanks: @​vanja-p
  • experimental/stats: Update MetricsRecorder to require embedding the new UnimplementedMetricsRecorder (a no-op struct) in all implementations for forward compatibility. (#8780](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8780">/issues/8780))

  Behavior Changes

  • balancer/weightedtarget: Remove handling of Addresses and only handle Endpoints in resolver updates. (#8841](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8841">/issues/8841))

  New Features

  • experimental/stats: Add support for asynchronous gauge metrics through the new AsyncMetricReporter and RegisterAsyncReporter APIs. (#8780](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8780">/issues/8780))
  • pickfirst: Add support for weighted random shuffling of endpoints, as described in gRFC A113.
    • This is enabled by default, and can be turned off using the environment variable GRPC_EXPERIMENTAL_PF_WEIGHTED_SHUFFLING. (#8864](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8864">/issues/8864))
  • xds: Implement :authority rewriting, as specified in gRFC A81. (#8779](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8779">/issues/8779))
  • balancer/randomsubsetting: Implement the random_subsetting LB policy, as specified in gRFC A68. (#8650](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8650">/issues/8650))
    • Special Thanks: @​marek-szews

  Bug Fixes

  • credentials/tls: Fix a bug where the port was not stripped from the authority override before validation. (#8726](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8726">/issues/8726))
    • Special Thanks: @​Atul1710
  • xds/priority: Fix a bug causing delayed failover to lower-priority clusters when a higher-priority cluster is stuck in CONNECTING state. (#8813](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8813">/issues/8813))
  • health: Fix a bug where health checks failed for clients using legacy compression options (WithDecompressor or RPCDecompressor). (#8765](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8765">/issues/8765))
    • Special Thanks: @​sanki92
  • transport: Fix an issue where the HTTP/2 server could skip header size checks when terminating a stream early. (#8769](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8769">/issues/8769))
    • Special Thanks: @​joybestourous
  • server: Propagate status detail headers, if available, when terminating a stream during request header processing. (#8754](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8754">/issues/8754))
    • Special Thanks: @​joybestourous

  Performance Improvements

  • credentials/alts: Optimize read buffer alignment to reduce copies. (#8791](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8791">/issues/8791))
  • mem: Optimize pooling and creation of buffer objects. (#8784](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8784">/issues/8784))
  • transport: Reduce slice re-allocations by reserving slice capacity. (#8797](https://github.com/href="https://redirect.github.com/grpc/grpc-go/issues/8797">/issues/8797))

  ... (truncated)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

• 3472: Match MySQL output when marshalling datetimes in JSON objects. MySQL always displays microsecond precision in datetimes. We should too. Fixes https://github.com/dolthub/dolt/issues/10706
• 3471: Changed some reflection checks to use type equality reflect.DeepEquals has issues with some types represented by integrators, so we're using type.Equals in such scenarios.

Closed Issues

• 10758: DELETE ... WHERE col < ? LIMIT N fails with 'column not found in scope'
• 10741: CREATE VIEW doesn't respect column naming
• 10710: Handle negative and positive infinity in comparisons
• 8572: Potential TUI tool like Lazygit
• 10131: [feature request] dolt stash apply
• 10729: MATCH AGAINST fails when FULLTEXT-indexed table is accessed via JOIN
• 10488: Add --overwrite-ignore and --no-overwrite-ignore to dolt checkout
• 10732: dolt leaks zero-byte /tmp/<uuid> files on every process start
• 10727: dolt push fails with 'uncommitted changes' when remote has ignored tables in working set
• 10706: JSON_OBJECT on datetime field doesn't represent fractional seconds
• 10698: Interactive rebase fails with 'changes in branch' when ignored tables exist in working set
• 10692: Server permanently unresponsive after client crash during CREATE/DROP DATABASE
• 3264: Panic When Comparing Decimal against NaN
• 3459: False ErrInvalidIndexPrefix on latin1 prefix index