Search Results for "text processing"

Trafilatura is a Python package and command-line tool designed to gather text on the Web. It includes discovery, extraction and text-processing components. Its main applications are web crawling, downloads, scraping, and extraction of main texts, metadata and comments. It aims at staying handy and modular: no database is required, the output can be converted to various commonly used formats. Going from raw HTML to essential parts can alleviate many problems related to text quality, first by avoiding the noise caused by recurring elements (headers, footers, links/blogroll etc.) and second by including information such as author and date in order to make sense of the data. ...

...Newspaper4k also includes natural language processing capabilities that can generate summaries and identify keywords from extracted article text. Newspaper4k supports both single-article extraction and full news site processing, allowing users to build sources representing entire publications and iterate through their articles. It maintains compatibility with the original project so that existing code written for newspaper3k can continue working with minimal changes.

...It provides an integrated pipeline that crawls news sites, retrieves article pages, and extracts structured information such as headlines, authors, publication dates, and article text. news-please can recursively follow internal links and read RSS feeds to gather both recent and archived articles from a news outlet when given only the root URL of a site. It combines several established technologies and libraries to perform web crawling and content extraction, enabling reliable processing across a wide range of news sources. ...

...It focuses on automating the collection of large sets of images by programmatically parsing page content and iterating through gallery entries. mzitu also includes a simple analysis script that processes downloaded folder names to generate statistics and visualizations. Using text segmentation and frequency analysis, the project can create a word cloud representing common keywords found in the dataset. This makes the repository both a scraping example and a small data analysis experiment built around the collected content. Overall, mzitu serves as a learning-oriented implementation of Python web scraping, data processing, and visualization techniques.

uncaptcha is an open-source proof-of-concept system designed to demonstrate vulnerabilities in Google’s audio reCAPTCHA challenges by automatically solving them using speech recognition techniques. The project uses browser automation to navigate to CAPTCHA challenges, extract audio files, and process them through multiple speech-to-text services. By combining outputs from several transcription engines, the system increases the likelihood of correctly identifying the spoken digits or phrases required to solve the challenge. It employs signal processing techniques such as segmenting audio clips into individual components before transcription, which improves accuracy in noisy or complex audio conditions. ...

More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation...

Teng is a general purpose templating engine written in C++ (i.e. library). It is also available as Python module or PHP extension. The main idea of teng is to strictly separate application logic from presentation layer. Widely used on dynamic web sites.

arCHMage is a reader and decompiler for files in the CHM format. This is the format used by Microsoft HTML Help, and is also known as Compiled HTML.

SilverCity is a lexing package, based on Scintilla, that can provide lexical analysis for over 20 programming and markup langauges. Scripting language bindings currently exist for Python.

This tool converts html to mediawiki markup language.

A Python-based template and view-controller framework derived from HTML::Mason. Supports the full featureset of Mason, allowing component-based web development with Python-embedded HTML, and includes many new concepts and features not found in Mason.

This is simple and tiny template framework module. It processing is speedy. And provides extract variables, dictionary reference and sequencial variable loop. Import a tinpy module and call the build function, so it became generate document with templat

Concurrence is a networked file editing program that enables multiple people to modify a document simultaneously. It is written entirely in Python, and uses the wxPython library for the GUI and the Twisted library for networking.

pdfplayground is a collection of tools written in python to read/write PDF files.

Pyana is a extension module that allows Python programs to interface with the Apache Software Foundation's Xalan XSLT transformation engine.

A multiplatform visual implementation of the Unix utility grep

Process OpenOffice.org Writer Files and transform them to PDF without installing OpenOffice.org What is PyOpenOffice? * It is a class library, written in the Python Language. * It is a platform-independent command-line utility (many abilitie

NovaX is a set of programs that is being devloped for small company's and beginners in the fields of HTML and programming. Coded in Python and C++, this is also a good replacement for MS Office. NOTE: This requires Python to be on your PC. ( Python.org )

Mamba is a extensible xml templates preprocessor wrote in Python. Using it, you can rapidly develop powerful applications ready to integrate with the internet. It can be used to work as a generic CGI program or for generate content.

SchemaDoc is a XML-based markup language for documenting XML schemas. The work products include both the vocabulary and a set of tools for combining it with the schema source (e.g. a DTD) to produce documentation in HTML, XML DocBook, LaTeX, etc.