With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.
You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
Try free now
Cloud tools for web scraping and data extraction
Deploy pre-built tools that crawl websites, extract structured data, and feed your applications. Reliable web data without maintaining scrapers.
Automate web data collection with cloud tools that handle anti-bot measures, browser rendering, and data transformation out of the box. Extract content from any website, push to vector databases for RAG workflows, or pipe directly into your apps via API. Schedule runs, set up webhooks, and connect to your existing stack. Free tier available, then scale as you need to.
LittleLink Custom is an open source self-hosted alternative to services like Linktree and many.link. LittleLink Custom allows you to link all your social media platforms easily accessible on one page. No coding or commandline knowledge required. Just easy plug and play.
More than just a link checker, sitecheck is a website spider (also known as a crawler) which can assist with SEO by testing an entire site plus both inbound links from search engines and outbound links to other sites for the following issues: looping redirects (HTTP 301/302), broken links (HTTP 404), server errors (HTTP 500), spelling mistakes, low readability scores (using the Flesch Reading Ease test), missing/empty/duplicate meta tags, duplicate content, slow page speed, W3C validation...
AWBot is an easy to use tool to test a web site.:
AWbot connects to your web site and make URL requests like any other visitors.
AWBot is not a web indexing robot but a web client tool that emulate some visitors browsing on your site to test its reliab
New Homepage: http://wummel.github.io/linkchecker/
Linkchecker features:
- recursive and multithreaded checking and site crawling
- output in colored or normal text, HTML, SQL, CSV, XML or a sitemap graph in different formats
- HTTP/1.1, HTTPS, FTP, mailto:, news:, nntp:, Telnet and local file links support
- restrict link checking with regular expression filters for URLs
- proxy support
-...
Dun and Bradstreet Connect simplifies the complex burden of data management
Our self-service data management platform enables your organization to gain a complete and accurate view of your accounts and contacts.
The amount, speed, and types of data created in today’s world can be overwhelming. With D&B Connect, you can instantly benchmark, enrich, and monitor your data against the Dun & Bradstreet Data Cloud to help ensure your systems of record have trusted data to fuel growth.
Protection by an optimized hosts file. Over 29300 DNS entires against Tracking, Adware, Spyware, Viruses, Popups and Redirecting. For more security and privacy. Points the insecure domains to the localhost that they don't get resolved by DNS Servers
=DOES NOT WORK ANYMORE AS DSA HAS PUT CAPTCHA= DSA Practical Driving Test Monitor helps you find any available practical driving test slot within specified date range. Runs on Linux/Mac/Windows and automates your manual task of finding the test slot.
WebInspector is a websites watcher, a python console program that periodically checks a list of websites for changes and updates. If such an update is detected, selected by the user program (or programs) is executed. By default it's a web browser.
This project is designed to optimize search engine results by managing your web server sitemaps. The software combines both commandline processes and a web user interface with a highly configurable architecture.
Auto Proxy Filter Test (APFT) automates the testing of safe and unsafe URLs against a content filtering proxy (such as Dansguardian) and helps prevent regressions. APFT is useful to people who are designing filter rules.
rss2imap is a tool which gets RSS (RDF Site Summary) from web and delivers to the IMAP server as an e-mail message. it enables you to use IMAP supported MUA as a RSS reader, and to unify mail check and site update check with ONE client.
Sperowider Website Archiving Suite is a set of Java applications, the primary purpose of which is to spider dynamic websites, and to create static distributable archives with a full text search index usable by an associated Java applet.
Robust featureful multi-threaded CLI web spider using apache commons httpclient v3.0 written in java. ASpider downloads any files matching your given mime-types from a website. Tries to reg.exp. match emails by default, logging all results using log4j.
JoBo is a web site mirroring tool. It has a graphical UI but there is a also commandline version. Supports robot exclusion protocol (but this can be disabled)