apache-logs-to-mysql download

Apache Log Parser and Data Normalization Application

Python handles File Processing & MySQL handles Data Processing

ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.

Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0.

4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution.

Database system designed to accommodate unlimited domains. Step-by-step guide for easy installation.

Web interface with Drill Down Capability and apache/echarts Log Visualization integration in development.

MySQL apache_logs schema currently has 49 Tables, 853 Columns, 168 Indexes, 66 Views, 7 Stored Procedures and 43 Functions to process Apache Access log in 4 formats & Apache Error log in 2 formats. Database normalization at work!

Features

Creates MySQL schema - apache_logs
Includes 2 Python modules
Imports Apache Access Logs - common, combined and vhost_combined
Imports Apache Error Logs
Complete Apache Log Data Normalization
MySQL does all Data Manipulation & Processing
Python handles polling of log file folders
Python executes MySQL Database LOAD DATA statements
Python executes Stored Procedures & Functions and SQL Statements
Requires installation of 4 Python modules - requirements.txt included

Project Samples

MySQL View - apache_logs.access_ua_browser_list - data from LogFormat: combined & extended

MySQL View - apache_logs.access_requri_list - data from LogFormat: combined & extended

MySQL View - apache_logs.error_apachemessage_list

Stored Procedures & Functions in apache_logs Schema

Project Activity

See All Activity >

License

Apache License V2.0

Follow apache-logs-to-mysql

apache-logs-to-mysql Web Site

Other Useful Business Software

Gen AI apps are built with MongoDB Atlas

The database for AI-powered applications.

MongoDB Atlas is the developer-friendly database used to build, scale, and run gen AI and LLM-powered apps—without needing a separate vector database. Atlas offers built-in vector search, global availability across 115+ regions, and flexible document modeling. Start building AI apps faster, all in one place.

Start Free

Rate This Project

User Reviews

Be the first to post a review of apache-logs-to-mysql!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Intended Audience

Information Technology, System Administrators, Developers, Management

User Interface

Console/Terminal, Command-line

Programming Language

Python

Database Environment

Python Database API, SQL-based

Related Categories

Python Log Analysis Software, Python Log Management Tool, Python Log Parsers

Registered

2024-11-19

Similar Business Software

Tungsten Clustering

Tungsten Clustering is the only complete, fully-integrated, fully-tested MySQL HA, DR and geo-clustering solution running on-premises and in the cloud combined with industry-best and fastest, 24/7 support for business-critical MySQL, MariaDB, & Percona Server applications. It allows...

See Software
Windocks

Windocks is a leader in cloud native database DevOps, recognized by Gartner as a Cool Vendor, and as an innovator by Bloor research in Test Data Management. Novartis, DriveTime, American Family Insurance, and other enterprises rely on Windocks for on-demand database environments for development,...

See Software
Jet Profiler

Jet Profiler for MySQL 4 is a real-time query performance and diagnostics tool for the MySQL database server. Jet Profiler focuses on queries, tables and users. This gives you the information you need in order to quickly fix performance problems in your code, such as most frequent queries, most...

See Software