Pathway Files

Added

• pw.io.gdrive.read now supports the "only_metadata" format. When this format is used, the table will contain only metadata updates for the tracked directory, without reading object contents.
• Detailed metrics can now be exported to SQLite. Enable this feature using the environment variable PATHWAY_DETAILED_METRICS_DIR or via pw.set_monitoring_config().
• pw.io.kinesis.read and pw.io.kinesis.write methods for reading from and writing to AWS Kinesis.

Fixed

• A bug leading to potentially unbounded memory consumption that could occur in Table.forget and Table.sort operators during multi-worker runs has been fixed.
• Improved memory efficiency during cold starts by compacting intermediary structures and reducing retained memory after backfilling.

Changed

• The frequency of background operator snapshot compression in data persistence is limited to the greater of the user-defined snapshot_interval or 30 minutes when S3 or Azure is used as the backend, in order to avoid frequent calls to potentially expensive operations.
• The Google Drive input connector performance has been improved, especially when handling directories with many nested subdirectories.
• The MCP server tool method now allows to pass the optional data title, output_schema, annotations and meta to inform the LLM client.
• Relaxed boto3 dependency to <2.0.0.