apache-logs-to-mysql download

Apache Log Parser and Data Normalization Application

Python handles File Processing & MySQL handles Data Processing

ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.

Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0.

4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution.

Database system designed to accommodate unlimited domains. Step-by-step guide for easy installation.

Web interface with Drill Down Capability and apache/echarts Log Visualization integration in development.

MySQL apache_logs schema currently has 49 Tables, 853 Columns, 168 Indexes, 66 Views, 7 Stored Procedures and 43 Functions to process Apache Access log in 4 formats & Apache Error log in 2 formats. Database normalization at work!

Features

Creates MySQL schema - apache_logs
Includes 2 Python modules
Imports Apache Access Logs - common, combined and vhost_combined
Imports Apache Error Logs
Complete Apache Log Data Normalization
MySQL does all Data Manipulation & Processing
Python handles polling of log file folders
Python executes MySQL Database LOAD DATA statements
Python executes Stored Procedures & Functions and SQL Statements
Requires installation of 4 Python modules - requirements.txt included

Project Samples

MySQL View - apache_logs.access_ua_browser_list - data from LogFormat: combined & extended

MySQL View - apache_logs.access_requri_list - data from LogFormat: combined & extended

MySQL View - apache_logs.error_apachemessage_list

Stored Procedures & Functions in apache_logs Schema

Project Activity

See All Activity >

License

Apache License V2.0

Follow apache-logs-to-mysql

apache-logs-to-mysql Web Site

Other Useful Business Software

Orchestrate Your AI Agents with Zenflow

The multi-agent workflow engine for modern teams. Zenflow executes coding, testing, and verification with deep repo awareness

Zenflow orchestrates AI agents like a real engineering system. With parallel execution, spec-driven workflows, and deep multi-repo understanding, agents plan, implement, test, and verify end-to-end. Upgrade to AI workflows that work the way your team does.

Try free now

Rate This Project

User Reviews

Be the first to post a review of apache-logs-to-mysql!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Intended Audience

Developers, Information Technology, Management, System Administrators

User Interface

Command-line, Console/Terminal

Programming Language

Python

Database Environment

Python Database API, SQL-based

Related Categories

Python Log Analysis Software, Python Log Management Tool, Python Log Parsers

Registered

2024-11-19

Similar Business Software

Windocks

Windocks is a leader in cloud native database DevOps, recognized by Gartner as a Cool Vendor, and as an innovator by Bloor research in Test Data Management. Novartis, DriveTime, American Family Insurance, and other enterprises rely on Windocks for on-demand database environments for development,...

See Software
Google Cloud SQL

Fully managed relational database service for MySQL, PostgreSQL, and SQL Server with rich extension collections, configuration flags, and developer ecosystems. New customers get $300 in free credits to spend on Cloud SQL. You won’t be charged until you upgrade. Reduce maintenance costs with...

See Software
Cerberus FTP Server

Cerberus FTP Server provides a secure and reliable file transfer solution for the demanding IT professional in any industry. Supporting SFTP and SCP, FTP/S, and HTTP/S, Cerberus is able to authenticate against Active Directory and LDAP, run as a Windows service, has native x64 support, includes...

See Software
ManageEngine Endpoint Central

ManageEngine Endpoint Central is built to secure the digital workplace while also giving IT teams complete control over their enterprise endpoints. It delivers a security-first approach by combining advanced endpoint protection with comprehensive management, allowing IT teams to manage the...

See Software
CMW Platform

Low-code BPMS, helps mid-size and large companies automate, and improve business processes while staying aligned with enterprise architecture and IT policies. Business and IT teams can quickly build and adapt workflows without deep coding skills. BPM suite supports common use cases like CapEx...

See Software
New Relic

There are an estimated 25 million engineers in the world across dozens of distinct functions. As every company becomes a software company, engineers are using New Relic to gather real-time insights and trending data about the performance of their software so they can be more resilient and...

See Software