Distributed Concourse (ALPHA)

• Added support for running Concourse as a distributed cluster. This ALPHA release allows Concourse to operate across multiple nodes while maintaining strong consistency guarantees.
• Distributed Concourse operates as a CP system with optimistic availability, meaning it prioritizes data consistency while maximizing availability when possible.
• Each node in the cluster operates as a peer - there is no single leader, allowing clients to connect to any node to perform operations.
• Distributed deployment can be enabled by configuring the new cluster settings in your configuration file:
• cluster.nodes: A list of host:port entries for each node in the cluster
• cluster.replication_factor: Specifies the minimum number of nodes that should receive a copy of each piece of data (defaults to n/2 + 1, where n is the number of nodes)
• IMPORTANT: This is an ALPHA release with several limitations:
• Cannot convert an existing single-node deployment with data into a distributed cluster
• Cannot have nodes with different configurations within the same cluster
• Cannot have nodes running different Concourse versions
• Cannot add or remove nodes from an existing cluster dynamically
• Not recommended for production systems or mission-critical data
• The focus of this initial implementation is on core correctness of the distributed system protocol, data routing, and consistency - consider it a proof of concept at this stage.

Batch Transporter for Improved Automatic Indexing

We've introduced a new mechanism to control how Concourse Server transports data from the Buffer to the Database, significantly improving system throughput and responsiveness:

• New Transporter Configuration: Added the transporter configuration option in concourse.yaml that allows you to specify how Concourse moves data from the write-optimized Buffer to the read-optimized Database, where it becomes fully indexed.

• Two Transport Strategies:

• Streaming Transporter (legacy): Processes writes incrementally in small batches, with transport operations competing with reads and writes for system resources. While this provides consistent throughput by amortizing indexing costs across operations, it can lead to "stop-the-world" situations during high load when transport operations block all reads (and implicitly writes, which perform verification reads).

• Batch Transporter (default/new): Performs indexing entirely in the background, allowing reads and writes to continue uninterrupted during the indexing phase. Only a brief critical section is required when merging the fully-indexed data into the Database. This approach dramatically improves overall system throughput by eliminating the resource contention between transport operations and normal read/write operations.

• Configuration Options:

  Simple configuration

  transporter: batch # or "streaming"

  Advanced configuration

  transporter: type: batch # or "streaming" num_threads: 2 # default: 1

• Performance Benefits: The Batch Transporter significantly improves system throughput by:

• Moving the time-consuming indexing work to background threads
• Minimizing the duration of critical sections where locks block concurrent operations
• Reducing "stop-the-world" situations during high load

• Making system performance more predictable and responsive

• Use Case Recommendation: The Batch Transporter is particularly beneficial for workloads with high concurrent activity or large data volumes that require extensive indexing.

This enhancement represents a fundamental improvement to Concourse's architecture, addressing a key bottleneck in the storage engine by separating the indexing process from the critical path of normal operations.

Search

We made several changes to improve search performance and accuracy:

• Fulltext Search in Queries: Fulltext search is now supported directly in query operations via the new operators CONTAINS and NOT_CONTAINS. This enhancement eliminates the need to separately invoke the search method and manually intersect query result sets, allowing you to filter results using search logic inline with your queries.
• Preservation of Stopwords in Indexing and Search: Stopwords are no longer removed. As a result, searches that contain stopwords may return different, yet more accurate and contextually relevant results.
• Previous Configuration: In earlier versions, Concourse Server could be configured using the conf/stopwords.txt file to exclude common stopwords from indexing and search operations. This approach was designed to reduce storage requirements and improve search performance by removing frequently occurring, but generally less significant words.
• Rationale for Change: Preserving stopwords is crucial for maintaining context, which can significantly enhance the accuracy of search results and the effectiveness of ranking algorithms. Since affordable storage and computational resources are more abundant, resource usage is no longer a concern and it makes more sense to prioritze better search accuracy and system robustness. Lastly, preserving stopwords eliminates corner case bugs that are inherent to the way Concourse's search algorithm interacts with the buffered storage system.
• Upgrade Implications: Upon upgrading to this version, an automatic reindexing task will be initiated to ensure that all previously indexed data conforms to the new no-stopword-removal policy. It's important to allocate downtime for this reindexing to occur. And, it is wise to anticipate more storage spaced being used due to stopwords being included in the search corpus.
• Changed the default value of the max_search_substring_length configuration option to 40. The previous default allowed unlimited substring lengths, which increased search index size and hurt performance. Existing explicit configurations for this option remain unchanged.
• Compiled Search Queries: Since query-based searches consult indexed data instead of the full corpus (as direct searches do), we added logic to compile search queries when they must be compared against multiple stored values (e.g. in the Buffer, or within atomic operations or Transactions). This compilation allows the search algorithm to optimize over simple substring matching—especially when the search term is a single token/word—by leveraging efficient algorithms like Boyer-Moore.

Locking Optimizations

We made several changes to improve the safety, scalability and operational efficiency of the Just-in-Time (JIT) locking protocol:

• Eliminated redundant logic and localized the determination of when an Atomic Operation or Transaction becomes preempted by another commit. Previously that determination was managed globally in the Engine and relied on the JVM garbage collector (GC) to remove terminated operations from listening for data conflicts. Under contention, If many terminated operations accumulated between GC cycles, write performance could become degraded for hot data topics. As a result of this change, JIT locking is generally more memory efficient.
• Reduced lock metadata by consolidating the provisioning for all locks to a single broker. Previously, range locks and granular locks were issued and managed independently by different services.
• Improved the CPU efficiency of range locks by scheduling range blocked operations to park instead of busy waiting.
• Eliminated a known race condition that made it possible for two different conflicting commits to violate ACID semantics by concurrently acquiring different locks for the same resource.
• Switched the basis for all storage engine locks from java.util.concurrent.locks.ReenteantReadWriteLock to either java.util.concurrent.locks.StampedLock or other synchronization primitives that are generally shown to have better throughput.

YAML Configuration

• Concourse now supports YAML configuration files. Going forward, YAML files are preferred over preferences files for configuration.
  • Concourse Server can be configured with concourse.yaml and concourse.yaml.dev files.
  • Concourse Shell and other Java Driver based clients can be configured with a concourse_client.yaml file.
  • Usage of concourse.prefs, concourse.prefs.dev and concourse_client.prefs is now deprecated.
• Existing configuration defined in .prefs files is still recognized and backwards compatability is fully preserved.
• Configuration that is defined in .yaml files take precedence over configuration defined in .prefs files, with the exception that concourse.prefs.dev takes precedence over concourse.yaml to honor the convention of prioritizing dev configuration.
• The stock concourse.prefs file will no longer be updated when new configuration options are available. All new configuration templates will be defined in the stock concourse.yaml file.
• Concourse Server will not automatically migrate custom configuration from .prefs files to the corresponding .yaml files. While .prefs files are still functional, users are encouraged to manually copy custom configuration to the new format in case support for .prefs files goes away at a future date.
• concourse.yaml supports an option to specify custom credentials for the root administrator account under the init.root object. If either init.root.username or init.root.password is provided, it takes precedence over any value provided for init_root_username or init_root_password, respectively.

Plugin Framework Improvements

We've enhanced the plugin framework to support plugins that are compiled against different Java versions than the server, enabling greater flexibility in plugin development and deployment:

• Cross-Version Plugin Compatibility: Plugins compiled with newer Java versions (e.g., Java 21) can now run on servers using older Java versions (e.g., Java 8). This is achieved through a new external bootstrapping mechanism that discovers plugin classes by scanning bytecode directly, rather than loading classes into the server's JVM, which would cause UnsupportedClassVersionError when version incompatibilities exist.
• Plugin Bootstrapper Configuration: Added the plugins.bootstrapper configuration option in concourse.yaml to control how plugins are bootstrapped during activation:
• external (default): Bootstraps plugins by scanning class file bytecode externally, which avoids loading plugin classes into the server's JVM. This enables cross-version compatibility and automatically falls back to internal bootstrapping if external processing fails.
• internal: Bootstraps plugins within the server's JVM using Reflections-based class loading. This is the legacy approach that requires compatible Java versions between the server and plugin.
• Custom Java Runtime for Plugins: Added the java_home and java_binary configuration options for plugins. Both options allow each plugin to respectively specify the path to a custom JDK or java binary that should be used to launch its JVM. If neither is configured, plugins use the same Java runtime as the server.
• Improved Java 9+ Compatibility: Fixed an issue where the server could not correctly determine the system classpath on Java 9 and later versions. The server now uses the java.class.path system property as a fallback when the system ClassLoader is not a URLClassLoader.
• Improved Plugin IPC Reliability: Refactored inter-process communication between the server and plugins to use CompletableFuture for response handling, replacing a custom spin-wait synchronization mechanism. Also hardened message framing to correctly handle partial socket reads and added write serialization to prevent message interleaving under concurrent load.

Concourse Automation Framework

• Added the concourse-automation framework to provide a central set of tools to programatically interact with the Concourse codebase and release artifacts in automated tests and devops workflows. For the most part, the concourse-automation framework is comprised of tools that were previously available in the concourse-ete-test-core framework.
  • ConcourseCodebase - Provides programmatic interaction with a local copy of the Concourse source code. Can be used to build installer artifacts.
  • ConcourseArtifacts - Provides factory methods to retrieve local copies of Concourse artifacts for any version. Can be used to download the installer for a released version.
  • ManagedConcourseServer - Provdes the ability to control an external Concourse Server process within another application.

New Functionality and Enhancements

• Reduced the amount of heap space required for essential storage metadata.
• Efficient Metadata: Added the enable_efficient_metadata configuration option to further reduce the amount of heap space required for essential storage metadata. When this option is set to true, metadata will occupy approximately one-third less heap space and likely improve overall system performance due to a decrease in garbage collection pauses (although per-operation performance may be slightly affected by additional overhead).
• Asynchronous Data Reads: Added the enable_async_data_reads configuration option to allow Concourse Server to potentially use multiple threads to read data from disk. When data records are either no longer cached or not eligible to ever be cached (due to space limitations), Concourse Server streams the relevant information from disk on-demand. By default, this is a synchronous process and the performance is linear based on the number of Segment files in the database. With this new configuration option, Concourse Server can now stream the data using multiple threads. Even under high contention, the read performance should be no worse than the default synchronous performance, but there may be additional overhead that reduces peak performance on a per-operation basis.
• Improved write performance of the set method in large transactions by creating normalized views of existing data, which are consulted during the method's implicit select read operation.
• Improved the performance of the verifyOrSet method by removing redundant internal verification that occurred while finalizing the write.

Bug Fixes

• GH-454: Fixed an issue that caused JVM startup options overriden in a ".dev" configuration file to be ignored (e.g., heap_size).
• GH-535: Fixed an issue that caused JVM startup options overriden in an environment variable to be ignored (e.g., CONCOURSE_HEAP_SIZE).
• GH-491 Fixed a race condition that made it possible for a range bloked operation to spurriously be allowed to proceed if it was waiting to acquire a range lock whose intended scope of protection intersected the scope of a range lock that was concurrently released.
• Fixed a bug that caused range locks to protect an inadequate scope of data once acquired.
• GH-490: Fixed a bug that made it possible for a write to a key within a record (e.g., key A in record 1) to erroneously block a concurrent write to a different key in the same record (e.g., key B in record 1). The practial consquence of this bug was that more Atomic Operations and Transactions failed than actually necessary.
• Fixed an issue that occurred when using a navigation key in a Criteria/Condition that was passed as a parameter to a Concourse Server command. Previously, the individual stops of navigation keys were not individually registered as condition keys. As a result, the Strategy framework didn't have all the relevant information to accurately determine all the ideal lookup sources when traversing the document graph to retrieve the values along the path. Now, in addition to the entire navigation key, each individual stop is registered as a condition key, which means that the Strategy framework will have enough information to determine if more efficient to use any index data (as opposed to table data) for lookups.

API Breaks and Deprecations

• Concourse CLIs have been updated to leverage the lib-cli framework. There are no changes in functionality, however, in the concourse-cli framework, the following classes have been deprecated:
  • CommandLineInterface in favor of ConcourseCommandLineInterface
  • CommandLineInterfaceRunner in favor of com.cinchapi.lib.cli.CommandLineInterfaceRunner from the lib-cli framework.
  • NoOptions in favor of creating a new Options object.
  • Options in favor of com.cinchapi.lib.cli.Options from the lib-cli framework.
• As a result of Concourse's new support for YAML configuration:
  • Usage of concourse.prefs, concourse.prefs.dev and concourse_client.prefs is deprecated in favor of concourse.yaml, concourse.yaml.dev and concourse_client.yaml respectively.
  • The ConcourseServerPreferences handler is deprecated in favor of using ConcourseServerConfiguration, which provides the same functionality.
  • The ConcourseClientPreferences handler is deprecated in favor of using ConcourseClientConfiguration, which provides the same functionality.
  • ManagedConcourseServer#prefs() is deprecated in favor of ManagedConcourseServer#config().
  • The Concourse#connectWithPrefs methods have been deprecated in favor of Concourse#connect methods that take one or more configuration file Paths or a ConcourseClientConfiguration handler, respectively.
• With the introduction of the concourse-automation framework, duplicate classes in the concourse-ete-test-core framework have been deprecated:
  • com.cinchapi.concourse.util.ConcourseCodebase has been deprecated in favor of using com.cinchapi.concourse.automation.developer.ConcourseCodebase which provides the same functionality, but some methods have been renamed for clarity.
  • ConcourseServerDownloader has been deprecated in favor of using ConcourseArtifacts which provides the same functionality, but some methods have been renamed for clarity.
  • com.cinchapi.concourse.server.ManagedConcourseServer has been deprecated in favor of using com.cinchapi.concourse.automation.server.ManagedConcourseServer which provides the same functionality, but some methods have been renamed for clarity.
• The com.cinchapi.concourse.util.Processes utility class has been removed in favor of using com.cinchapi.common.process from accent4j.
  • This was removed without deprecation because the utility provided by the accent4j version is nearly identical to the one that was provided in Concourse and accent4j is naturally available to users of Concourse frameworks by virtue of being a transitive dependency.
  • The waitFor and waitForSuccessfulCompletion methods of accent4j's Processes utility return a ProcessResult, which provides access to the process's exit code, output stream and error stream (in the Concourse version, these methods had a void return type). This means that an Exception will be thrown if an attempt is made to use the getStdErr or getStdOut method on a process that was submitted to waitFor or waitForSuccessfulCompletion.