apache-logs-to-mysql Files

Log Generator created data with many values outside of "real-life" logs. All prior testing was done with "real" logs. Some values outside of "real-life" values causing unseen issues that required changing some table columns from TINYINT to INT. The data also caused error routines to be truly tested. Ran 5 million Combined Access records and 1 Million Error records through MySQL and MariaDB. Used environment variables ERROR_SERVER, ERROR_SERVERPORT, COMBINED_SERVER, COMBINED_SERVERPORT to assign domains. This should be the last of the database structure changes. I did think I was done last version but the Log Generator shook things up!

MariaDB 11.6 is consistently twice as fast as MySQL 9.1 in every benchmark on Windows 11. I have not run benchmarks on Ubuntu or MacOS.

• [3.2.5] modify process_access_import to correct remoteLogName and remoteUser column processing. Values were switched in tables stored. Running log generator stress tests flushed this out.
• [3.2.5] modify logs2mysql.py to add timing variables for all child processes to display in logs and store to import_load TABLE.
• [3.2.5] modify TABLE import_load added six columns for process execution durations in seconds.
• [3.2.5] modify logs2mysql.py reworked all process message logging verbiage to provide child process summary information at each phase. Running log generator stress tests flushed this out.
• [3.2.5] modify logs2mysql.py reduced number of cursor objects created by reusing only two cursor objects. Reduced all variables for import_file TABLE processing using same variables for all.
• [3.2.5] modify TABLES access_log_useragent and log_client added indexes for use in logs2mysql.py processing.