Search Results for "web crawler bot" - Page 2
Sort By:
Showing 53 open source projects for "web crawler bot"
View related business solutions-
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
1
bors-ng
A merge bot for GitHub Pull Requests
bors-ng is a GitHub merge bot that enforces an “evergreen” main branch by queueing approved pull requests, testing them together on a staging branch, and only advancing main with the exact, passing commit set. Instead of maintainers repeatedly rebasing and merging one PR at a time, bors-ng batches r+-ed PRs, triggers your existing CI (e.g., GitHub Actions), and merges automatically when the batch is green. If a batch fails, it bisects the batch into smaller runs to isolate the culprit,... -
2
TorBot
Dark Web OSINT Tool
Contributions to this project are always welcome. To add a new feature fork the dev branch and give a pull request when your new feature is tested and complete. If its a new module, it should be put inside the modules directory. The branch name should be your new feature name in the format <Feature_featurename_version(optional)>. On Linux platforms, you can make an executable for TorBot by using the install.sh script. You will need to give the script the correct permissions using chmod +x... -
3
Gerapy
Distributed Crawler Management Framework Based on Scrapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Someone who has worked as a crawler with Python may use Scrapy. Scrapy is indeed a very powerful crawler framework. It has high crawling efficiency and good scalability. It is basically a necessary tool for developing crawlers using Python. If you use Scrapy as a crawler, then of course we can use our own host to crawl when crawling, but when the crawl is very large, we can’t... -
4
Goutte
Goutte, a simple PHP Web Scraper
...The method returns a Crawler object (Symfony\Component\DomCrawler\Crawler). To use your own HTTP settings, you may create and pass an HttpClient instance to Goutte. For example, to add a 60 second request timeout. Read the documentation of the BrowserKit, DomCrawler, and HttpClient Symfony Components for more information about what you can do with Goutte. Goutte is a thin wrapper around the following Symfony Components: BrowserKit, CssSelector, DomCrawler, and HttpClient. -
Secure File Transfer for Windows with Cerberus by RedwoodCerberus supports unlimited users and connections on a single IP, with built-in encryption, 2FA, and a browser-based web client — all deployable in under 15 minutes with a 25-day free trial.
-
5
ReconSpider
Most Advanced Open Source Intelligence (OSINT) Framework
...Reconnaissance is a mission to obtain information by various detection methods, about the activities and resources of an enemy or potential enemy, or geographic characteristics of a particular area. A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing (web spidering). -
6
CEF Python
Python bindings for the Chromium Embedded Framework (CEF)
Python bindings for the Chromium Embedded Framework (CEF). CEF Python is an open source project founded by Czarek Tomczak in 2012 to provide Python bindings for the Chromium Embedded Framework (CEF). The Chromium project focuses mainly on Google Chrome application development while CEF focuses on facilitating embedded browser use cases in third-party applications. Lots of applications use CEF control, there are more than 100 million CEF instances installed around the world. There are... -
7
chat-bubble
Simple chatbot UI for the Web with JSON scripting
Simple chatbot UI for the Web with JSON scripting. -
8
taskt
Free and open-source robotic process automation (rpa)
taskt (formerly sharpRPA) is the first truly free, easy-to-use, and open-source process automation client built on the .NET Framework in C#. taskt allows you to build and design process automation without needing to write application code. taskt allows you to automate the boring stuff and create efficienies by giving you the power to craft a digital workforce that executes and performs rule-based automation. No API? No Problem! Included is a "what you see is what you get" bot designer with... -
9
koa-isbot
Fast Middleware detect bot crawler for Koa
Koa detects robots. Fast Middleware detects bot crawler for Koa. -
Stop vibe-debugging.AppSignal's MCP server hands Claude, Cursor, or Zed your real errors, traces, and the deploy that shipped them. AI writes the fix; you review the diff.
-
10
ShadowSocksShare
Python ShadowSocks framework
This project obtains the shared ss(r) account from the ss(r) shared website crawler, redistributes the account and generates a subscription link by parsing and verifying the account connectivity. Since Google plus will be closed on April 2, 2019, almost all the available accounts crawled before come from Google plus. So if you are building your own website, please keep an eye on the updates of this project and redeploy using the latest source code. -
11
Mojo-Webqq
Perl-based smart QQ client framework
Mojo-Webqq is an unofficial Perl client and bot framework for the QQ messaging platform’s web endpoints, built on the Mojolicious real-time web stack. It manages login flows, session refresh, and long-polling or websocket-style message reception to deliver events into an asynchronous event loop. Plugins and hooks let developers implement auto-replies, keyword filters, logging, forwarding, and integrations with external services or databases. -
12
Claudia Bot Builder
Create chat bots for Facebook Messenger, Slack, Amazon Alexa, etc.
Claudia Bot Builder helps developers create and deploy chat-bots for various platforms in minutes to AWS Lambda. It simplifies the messaging workflows, automatically sets up the correct web hooks, and guides you through configuration steps, so that you can focus on important business problems and not have to worry about infrastructure code. This code is enough to operate bots for all supported platforms. -
13
Catberry
Catberry is an isomorphic framework
Catberry is an isomorphic framework for building universal front-end apps using components, Flux architecture and progressive rendering. Catberry builds a bundle for running the application in a browser as a Single Page Application. Cat-Components – similar to web-components but organized as directories, can be rendered on the server and published/installed as NPM packages. The entire architecture of the framework is built using the Service Locator pattern, which helps to manage module dependencies and create plugins, and Flux, for the data layer. Search crawler receives a full page from the server. ... -
14
A Java Web crawler: multi-threaded, scalable, with high performance, extensible and polite. It can be used to crawl and index any web or enterprise domain and is configurable through a XML configuration file.
-
15
wechat4u.js
JavaScript implementation of WeChat wechat web interface
JavaScript implementation of WeChat WeChat web interface, compatible with Node and browsers, WeChat robot. Export and import necessary data to keep WeChat login bot.botData. Modify contact remarks bot.updateRemarkName(UserName, RemarkName). Modify group name bot.updateChatRoomName(ChatRoomUserName, NewName). Forward message bot.forwardMsg(msg, toUserName). Revoke message bot.revokeMsg(MsgID, toUserName). A class of methods that send a message that returns the full response data on success.... -
16
Un-WordPress
A WordPress Theme to hide WordPress signatures
DEVELOPMENT IS INDEFINITELY ON HOLD All WordPress sites are under constant crack attempts - often successful - the Un-WordPress Theme provides a way to hide all WordPress signatures - paths, file names, comment form data, etc. This is version 1.0 and will probably break many a plugin... But it does work. Changes may be required to the WordPress install location - but no changes to WordPress itself are made. With the Un-WordPress theme no bot can locate any WordPress files (just... -
17
DotNetWikiBot Framework
Make robots for MediaWiki-powered sites!
The DotNetWikiBot Framework was developed so that it can offer a helping hand with many complicated and routine tasks of wiki site development and maintenance. DotNetWikiBot Framework is a cross-platform full-featured client API, that allows you to build programs and web robots easily to manage information on MediaWiki-powered sites. DotNetWikiBot Framework can also be used for learning C# and .NET. -
18
sitecheck
Modular web site spider for web developers.
More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation... -
19
Zoozle Search & Download Suchmaschine
Zoozle 2008 - 2010 Webpage, Tools and SQL Files
Download search engine and directory with Rapidshare and Torrent - zoozle Download Suchmaschine All The files that run the World Leading German Download Search Engine in 2010 with 500 000 unique visitors a day - all the tools you need to set up a clone. Code Contains: - PHP Files for zoozle - Perl Crawler for gathering new content to database and all other cool tools i have... -
20
** Guys I have built a much more powerful Fully Featured CMS system at: https://github.com/MacdonaldRobinson/FlexDotnetCMS Macs CMS is a Flat File ( XML and SQLite ) based AJAX Content Management System. It focuses mainly on the Edit In Place editing concept. It comes with a built in blog with moderation support, user manager section, roles manager section, SEO / SEF URL
-
21
LogCrawler is an ANT task for automatic testing of web applications. Using a HTTP crawler it visits all pages of a website and checks the server logfiles for errors. Use it as a "smoketest" with your CI system like CruiseControl.
-
22
Globsy is a PHP bot framework for the Google Talk network. Globsy can be executed from a regular browser on a PHP enabled server, connect to the Google Talk network and continue running in the background, without the browser needing to remain open.
-
23
yuik is an abstract widget library for creating GUIs. The actual frontend can be chosen at runtime which allows the program to run under Tk, as a shell program, and even as a web-application. More frontends like an irc-bot, wxWidgets, or XUL are planned.
-
24
-
25
JIB - Java IM Bot, aimed to built a pluggable, customizable, expandable and dynamic bot framework for instant messager. With JIB user can control remote IM Bots over web based interface, configure bots to perform various tasks or custom behavior.