Dataherald Files

Release Notes for Version 1.0.0

What's New

1. New Resources, Attributes, and Endpoints

• Finetuning: One of our new exciting features is automatic finetuning GPT family models on your golden questions/SQLs pairs.
• POST /api/v1/finetuning: By calling this endpoint you can create a fientuning job on your golden question/SQL pairs. The only required parameter is the db_connection_id and you have the option to specify which golden question/SQL pairs you want to use for finetuning process.
• GET /api/v1/finetuning/{finetuning_di}: With this endpoint you can retrieve the status of the finetuning process and once the status is SUCCEEDED you can use the model for SQL generation.
• POST /api/v1/finetuning/{finetuning_id}/cancel: If you want to cancel the finetuning for whatever reason you can call this endpoint.
• GET /api/v1/finetuning: List all of the finetuned models for a given db_connection_id
• DELETE /api/v1/finetuning/{finetuning_di}: Delete a given finetuned model from the finetunings collection.
• Metadata: All resources now include a metadata attribute, allowing you to store additional information for internal purposes. Soon, GET list endpoints will support filtering based on metadata fields.

2. Resource and Endpoint Changes

• Renaming questions to prompts: The entity has been renamed to Prompt, and the collection is now called prompts. You can use the following endpoints to interact with this resource:
• GET /api/v1/prompts: List all existing prompts.
• POST /api/v1/prompts: Create a new prompt.
• GET /api/v1/prompts/{prompt_id}: Retrieve a specific prompt.

• PUT /api/v1/prompts/{prompt_id}: Update the metadata for a prompt.

• Splitting responses into sql_generation and nl_generation: The previous responses resource has been divided into sql_generations and nl_generations. You can work with them as follows:

• POST /api/v1/prompts/{prompt_id}/sql-generations: Create a sql-generation from an existing prompt.
• POST /api/v1/prompts/sql-generations: Create a new prompt and a sql-generation.
• GET /api/v1/prompts/sql-generations: List sql-generations.
• GET /api/v1/sql-generations/{sql_generation_id}: Retrieve a specific sql-generation.
• PUT /api/v1/sql-generations/{sql_generation_id}: Update the metadata for a sql-generation.
• GET /api/v1/sql-generations/{sql_generation_id}/execute: Execute the created SQL and retrieve the result.

• GET /api/v1/sql-generations/{sql_generation_id}/csv-file: Execute the created SQL and generate a CSV file using the result.

• POST /api/v1/sql-generations/{sql_generation_id}/nl-generations: Create an nl-generation from an existing sql-generation.

• POST /api/v1/prompts/{prompt_id}/sql-generations/nl-generations: Create a sql-generation and an nl-generation from an existing prompt.
• POST /api/v1/prompts/sql-generations/nl-generations: Create a prompt, sql-generation, and nl-generation.
• GET /api/v1/nl-generations: List all nl-generations.
• GET /api/v1/nl-generations/{nl_generation_id}: Retrieve a specific nl-generation.
• GET /api/v1/sql-generations/{sql_generation_id}: Retrieve a specific sql-generation.

• PUT /api/v1/nl-generations/{nl_generation_id}: Update the metadata for an nl-generation.

• Renaming golden_records to golden_sqls: We've updated the name for all endpoints, entities, and collections.

3. Migration Script

• Purpose: To facilitate a smooth transition from version 0.0.5 to version 1.0.0, we've introduced a migration script.
• Data Modifications: The script performs the following actions:
• Renames the golden_records collection to golden_sqls.
• Replaces all related data types from ObjectId to strings.
• Updates table descriptions by changing "SYNCHRONIZED" status to "SCANNED" and "NOT_SYNCHRONIZED" to "NOT_SCANNED."
• Utilizes the existing questions collections to create the prompts collection.
• Converts responses collections into sql_generations and nl_generations collections.

To run the migration script, use the following command:

docker-compose exec app python3 -m dataherald.scripts.migrate_v006_to_v100

We hope that these changes enhance your experience with our platform. If you have any questions or encounter any issues, please don't hesitate to reach out to our support team.