Search Results for "python web crawler"

An advanced command-line tool designed to brute force directories and files in webservers, AKA web path scanner. Wordlist is a text file, each line is a path. About extensions, unlike other tools, dirsearch only replaces the %EXT% keyword with extensions from -e flag. For wordlists without %EXT% (like SecLists), -f | --force-extensions switch is required to append extensions to every word in wordlist, as well as the /. To use multiple wordlists, you can separate your wordlists with commas....

AnySearch Skill is a real-time search engine skill for AI agents. It gives agents a structured way to search the web, run vertical searches, perform parallel batch searches, and extract full-page content. The project is packaged as a skill rather than a standalone search application, so it is meant to be installed into compatible AI-agent environments. It supports multiple domain-specific search categories, making it useful when general web search is too broad. The skill can also fetch page...

rag-search is a lightweight Retrieval-Augmented Generation API service designed to provide structured semantic search and answer generation through a simple FastAPI backend. The project integrates web search, vector embeddings, and reranking logic to retrieve relevant context before passing it to a language model for response generation. It is built to be easily deployable, requiring only environment configuration and dependency installation to run a functional RAG service. The system...

FreeAskInternet is a self-hosted search and answer application that combines web results with large language models. A local SearXNG instance sends queries to multiple search engines and returns a broader collection of sources. The backend crawls selected pages, supplies their content to a configured model, and streams a synthesized response to the chat interface. It supports several hosted model services as well as custom local backends such as Ollama and llama.cpp. The web interface is...

Search Online (ver.Chrome Selinium) [Improved.Simplified.Alternative] A desktop application developed using python 3.6.8 and other add-on libaries. Uses chrome driver and python 'Selinium' library to automate web browsing. Compatible only for windows OS.

Meresco is both an OAI Data Provider and a Service Provider. SourceForge is only used to host the source control (subversion). Sources: http://sources.meresco.org/ Binaries: http://repository.cq2.org/ Mail: http://groups.google.com/group/meresco

A tool for autonomous and virtual topical data integration using the focused web-harvesting method.

Document summarization system. By adding document content to system, user queries will generate a summary document containing the available information to the system.

Nucular Archiving System for creating full text indices for fielded data. Python API, web, and command line interfaces. Fast. Very light weight. Concurrent read/writes with no possible locking issues. No server process. Proximity. Facets. Funny name.

DiskAt is disk/media catalogue app supporting multiple categories per item, good search and features which allow to use it as Movie/DVD/etc database. Written with PHP/Python/SQLite.

Universal information crawler is a fast precise and reliable Internet crawler. Uicrawler is a program/automated script which browses the World Wide Web in a methodical, automated manner and creates the index of documents that it accesses.

A configurable knowledge management framework. It works out of the box, but it's meant mainly as a framework to build complex information retrieval and analysis systems. The 3 major components: Crawler, Analyzer and Indexer can also be used separately.

Wilma is a program for quickly finding text lurking in the files on your computer. It does this by creating an index of what words are in which files, which allows it to later find files containing a given word or set of words almost instantaneously.

Xapian is a Search Engine Library, written in C++ with bindings for Perl, Python, PHP, Java, Tcl, C# and Ruby. Xapian allows you to easily add advanced indexing and search facilities to your applications. See www.xapian.org for more information.

Djucene is intended to provide access to a lucene index via web services powered by the django framework