apache-logs-to-mysql download

Apache Log Parser and Data Normalization Application

Python handles File Processing & MySQL handles Data Processing

ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.

Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0.

4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution.

Database system designed to accommodate unlimited domains. Step-by-step guide for easy installation.

Web interface with Drill Down Capability and apache/echarts Log Visualization integration in development.

MySQL apache_logs schema currently has 49 Tables, 853 Columns, 168 Indexes, 66 Views, 7 Stored Procedures and 43 Functions to process Apache Access log in 4 formats & Apache Error log in 2 formats. Database normalization at work!

Features

Creates MySQL schema - apache_logs
Includes 2 Python modules
Imports Apache Access Logs - common, combined and vhost_combined
Imports Apache Error Logs
Complete Apache Log Data Normalization
MySQL does all Data Manipulation & Processing
Python handles polling of log file folders
Python executes MySQL Database LOAD DATA statements
Python executes Stored Procedures & Functions and SQL Statements
Requires installation of 4 Python modules - requirements.txt included

Project Samples

MySQL View - apache_logs.access_ua_browser_list - data from LogFormat: combined & extended

MySQL View - apache_logs.access_requri_list - data from LogFormat: combined & extended

MySQL View - apache_logs.error_apachemessage_list

Stored Procedures & Functions in apache_logs Schema

Project Activity

See All Activity >

License

Apache License V2.0

Follow apache-logs-to-mysql

apache-logs-to-mysql Web Site

Other Useful Business Software

MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.

Start Free

Rate This Project

User Reviews

Be the first to post a review of apache-logs-to-mysql!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Intended Audience

Developers, Information Technology, Management, System Administrators

User Interface

Command-line, Console/Terminal

Programming Language

Python

Database Environment

Python Database API, SQL-based

Related Categories

Python Log Analysis Software, Python Log Management Tool, Python Log Parsers

Registered

2024-11-19

Similar Business Software

Tungsten Clustering

Tungsten Clustering is the only complete, fully-integrated, fully-tested MySQL HA, DR and geo-clustering solution running on-premises and in the cloud combined with industry-best and fastest, 24/7 support for business-critical MySQL, MariaDB, & Percona Server applications. It allows...

See Software
Windocks

Windocks is a leader in cloud native database DevOps, recognized by Gartner as a Cool Vendor, and as an innovator by Bloor research in Test Data Management. Novartis, DriveTime, American Family Insurance, and other enterprises rely on Windocks for on-demand database environments for development,...

See Software
Jet Profiler

Jet Profiler for MySQL 4 is a real-time query performance and diagnostics tool for the MySQL database server. Jet Profiler focuses on queries, tables and users. This gives you the information you need in order to quickly fix performance problems in your code, such as most frequent queries, most...

See Software
Releem

Releem is an AI-powered MySQL performance monitoring tool that delivers consistent performance through continuous database profiling, configuration tuning, and SQL query optimization. Releem automates analysis, performance issues detection, configuration tuning, query optimization and schema...

See Software
DiskInternals MySQL Recovery

DiskInternals MySQL Recovery is the best and most advanced MySQL database recovery tool. Be sure, this awesome tool is as easy to use as MySQL database recovery can be. The recovery wizard, which will appear instantly, will give you a choice of several options. Decide on one, click on the...

See Software
TencentDB

TencentDB for MySQL makes it easy for you to deploy and use MySQL database, one of the world's most popular open source relational database, in the cloud. TencentDB for MySQL is cost-efficient and flexible; it only takes minutes to deploy a scalable MySQL database instance and you can resize...

See Software