Future

• groovy: change version to 2.5.7
• getl.lang.Getl: added support for easy DSL language work with data sources and file systems
• getl.jdbc.TableDataset: added where and order properties
• getl.proc.Flow: added support class of parameters for methods
• getl.Config.ConfigSlurper: added a new Groovy Slurper configuration management class with support convert old format configuration files
• getl.utils.MapUtils: added a method FindKeys to search for items by mask
• getl.jdbc.TableDataset: added "offs", "limit", "directive" and "forUpdate" properties
• getl.h2.H2Table: added table with H2 specified options
• getl.hive.HiveTable: added table with Hive specified options
• getl.driver.Driver: added options INSERT, UPDATE, DELETE and READ_METADATA to operations enum
• getl.hive.HiveConnection: added version property
• getl.jdbc.JDBCDriver: improved dropping of tables, generated the "if exists" option when possible
• getl.vertica.VerticaTable: added support label and direct hint
• getl.files.SFTPManager: added "hostKey" property (RSA key)
• getl.mysql.MySQLDriver: added support JDBC driver version 6 and later, improved write speed to tables
• getl.mssql.MSSQLDriver: added support for transactional model for working with DDL operators and table truncate
• getl.proc.Executor: added the exec statement to run the thread list
• getl.files.HDFSManager: Added support for hdfs client libraries for version 3.2.0
• getl.jdbc.JDBCDataset: added queryParams property

Fixed bugs

• getl.FileUtils.ParseArguments: fixed a bug where the argument without spaces is enclosed in double quotes
• getl.MapUtils.ProcessArguments: fixed bug with extra space between arguments
• getl.h2.H2Connection: enabled by default option CASE_INSENSITIVE_IDENTIFIERS and ALIAS_COLUMN_NAME
• getl.hive.HiveDriver: fixed bug with working with non-standard of object names, disable support the insert, update and delete operations, changed the method read information of table
• getl.oracle.OracleDriver: fixed bug by working for blob and clob fields as use dynamic load of file driver
• getl.vertica.VerticaDriver: fixed bug for write clob field
• getl.mysql.MySQLDriver: fixed reading of table metadata when its schema is not specified
• getl.posgresql.PostgreSQLDriver: fixed table lock bug after run the truncate statement
• getl.mssql.MSSQLDriver: fixed table lock bug after run the ddl and truncate statements
• getl.hive.HiveDriver: fixed a bug when you immediately specified a vendor when creating a connection
• getl.files.Manager: fixed a bug when the current directory was lost after downloading files

GETL DSL

The implemented language extension makes it easier to work with data sources, file systems and simplifies the development of data processing processes.

Object-oriented coding is not required to work with data objects or file systems. All work is supported at the level of a simple scripting language. The language has built-in support for all types of data sources and file systems of the GETL classes.

To simplify the development of complex logic, support has been added for the local repository of the described data sources and file systems.

The language has added support for nested script calls for developing patterns of logic for working with data sources.

To extend the functionality of intermediate calculations, there is support for tables in the in-memory database and temporary text files.

To create parallelization of data processing, support for multi-threaded user code has been implemented. Connections and data sources that are registered in the repository are automatically cloned to work in threads.

The following data sources are supported

• csv files
• excel files
• H2 database
• Hadoop Hive
• json files
• MS SQLServer
• MySql
• Oracle
• PostgreSql
• Vertica
• xml files

The following file systems are supported

• local and sharing files
• ftp
• ssh file (sftp)
• hadoop hdfs

How use DSL in code classes

getl.lang.Getl.Dsl {
 ... user code ...
}

Example: "src/test/groovy/getl/lang/GetlTest.groovy"

How use DSL in groovy script

import groovy.transform.BaseScript
@BaseScript getl.lang.Getl getl
 ... user code ...
}

Examples: "src/main/groovy/getl/examples/*"

Example of copying records from excel to a temporary text file

@BaseScript getl.lang.Getl getl
import groovy.transform.BaseScript
import getl.data.Field
import getl.utils.FileUtils

excel { dataset -> // define excel dataset
    connection = excelConnection { // define excel connection
        path = '/excel'
        fileName = 'demo.xlsx'
        listName = 'workbook1'
        header = true
    }

    // copy rows from excel to csv file
    copyRows(dataset, csvTempWithDataset('demo', dataset))
}

// Read csv file rows and print to log
rowProcess(csvTemp('demo')) {
    process { logInfo(it) }
}

Documentation

Full Russian language documentation will be posted later on www.easydata.ru site. In the future, the basic part of the documentation will be translated into English and posted on Githab site.