Search Results for "crawl site links"
Sort By:
Showing 297 open source projects for "crawl site links"
View related business solutions-
Calmerry is beneficial for businesses of all sizes, particularly those in high-stress industries, organizations with remote teams, and HR departments seeking to improve employee well-being and productivity
-
Google Cloud is a cloud-based service that allows you to create anything from simple websites to complex applications for businesses of all sizes.
-
1
Heimdall
An Application dashboard and launcher
As the name suggests Heimdall Application Dashboard is a dashboard for all your web applications. It doesn't need to be limited to applications though, you can add links to anything you like. Heimdall is an elegant solution to organize all your web applications. It’s dedicated to this purpose so you won’t lose your links in a sea of bookmarks. Why not use it as your browser start page? It even has the ability to include a search bar using either Google, Bing or DuckDuckGo. You can use the app... -
2
TorBot
Dark Web OSINT Tool
... install.sh Now you can run ./install.sh to create the torBot binary. Run ./torBot to execute the program. Crawl custom domains.(Completed). Check if the link is live.(Completed). Built-in Updater.(Completed). TorBot GUI (In progress). Social Media integration.(not Started). -
3
Web-Check
All-in-one OSINT tool for analysing any website
Comprehensive, on-demand open source intelligence for any website. Get an insight into the inner-workings of a given website: uncover potential attack vectors, analyse server architecture, view security configurations, and learn what technologies a site is using. Currently the dashboard will show: IP info, SSL chain, DNS records, cookies, headers, domain info, search crawl rules, page map, server location, redirect ledger, open ports, traceroute, DNS security extensions, site performance... -
4
Laravel Sitemap
Create and generate sitemaps with ease
This package can generate a sitemap without you having to add urls to it manually. This works by crawling your entire site. The generator has the ability to execute JavaScript on each page so links injected into the dom by JavaScript will be crawled as well. The easiest way is to crawl the given domain and generate a sitemap with all found links. The destination of the sitemap should be specified by $path. If you don't want a crawled link to appear in the sitemap, just don't return... -
Deliver a personalized customer experience with every interaction, across every channel, with uContact, net2phone’s cloud contact center solution.
-
5
Sphinx
Main repository for the Sphinx documentation builder
Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. It was originally created for the Python documentation, and it has excellent facilities for the documentation of software projects in a range of languages. Of course, this site is also created from reStructuredText sources using Sphinx! HTML (including Windows HTML Help), LaTeX (for printable PDF versions), ePub, Texinfo, manual pages, plain text... -
6
Image Downloader
Download images from Google, Bing, Baidu
Crawl and download images using Selenium Using python3 and PyQt5. Supported Search Engine: Google, Bing, Baidu. Keywords input from the keyboard or input from line separated keywords list file for batch process. Download image using a customizable number of threads. Fully supported conditional search (eg. filetype:, site:). Switch for Google safe mode. Proxy configuration (socks, HTTP). CMD and GUI ways of using are provided. -
7
SQL Explorer
Easily share data across your company via SQL queries
SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can add to an existing Django site, or use as a standalone business intelligence tool. Quickly write and share SQL queries in a simple, usable SQL editor, preview the results in the browser, share links, download CSV, JSON, or Excel files (and even expose queries as API endpoints, if desired), and keep the information flowing! Comes with support for multiple... -
8
fullPage.js
Create beautiful fullscreen scrolling websites fast and easy
fullPage.js is an easy-to-use library for creating beautiful, fullscreen scrolling websites/ onepage sites/ single page websites complete with all the features you need. With fullPage.js you can add landscape sliders and links to sections of your site, create smaller or bigger sections, use extensions and more! fullPage.js is compatible with all modern browsers and even some old ones like IE9 and Opera 12. It also provides touch support designed for mobile devices and touch screen computers... -
9
Django jazzmin
Jazzy theme for Django
Welcome to Jazzmin, intended as a drop-in app to jazz up your django admin site, with plenty of things you can easily customize, including a built-in UI customizer. 4 different Change form templates (horizontal tabs, vertical tabs, carousel, collapsible). Bootstrap 4 modal (instead of the old popup window, optional). Search bar for any given model admin. Customizable UI (via Live UI changes, or custom CSS/JS). Select2 drop-downs. Bootstrap 4 & AdminLTE UI components. You can add links... -
It protects your internal resources such as behind-the-firewall applications, teams, and devices.
-
10
Whoogle Search
A self-hosted, ad-free, privacy-respecting metasearch engine
Get Google search results, but without any ads, javascript, AMP links, cookies, or IP address tracking. Easily deployable in one click as a Docker app, and customizable with a single config file. Quick and simple to implement as a primary search engine replacement on both desktop and mobile. Autocomplete/search suggestions. POST request search and suggestion queries (when possible). View images at full res without site redirect (currently mobile only). Light/Dark/System theme modes... -
11
SiteOne Crawler
SiteOne Crawler is a website analyzer and exporter
SiteOne Crawler is a very useful and easy-to-use tool you'll ♥ as a Dev/DevOps, website owner or consultant. Works on all popular platforms - Windows, macOS, and Linux (x64 and arm64 too). It will crawl your entire website in depth, analyze and report problems, show useful statistics and reports, generate an offline version of the website, generate sitemaps, or send reports via email. Watch a detailed video with a sample report for Astro. build website. This crawler can be used as a command... -
12
notfoundbot
fix & archive outgoing links on your website
notfoundbot is a GitHub Action that helps you automatically maintain the correctness of your website's outgoing links. It finds links that need fixing and opens pull requests that fix them. This action is intended for websites and blogs powered by static site generators. By using post dates derived from filenames, notfoundbot searches for Wayback Machine archives of linked resources that are contemporary to the post itself: broken links in a 2011 blog post will be linked to archives from around... -
13
crwlr
Library for Rapid (Web) Crawler and Scraper Development
This library provides kind of a framework and a lot of ready-to-use, so-called steps, that you can use as building blocks, to build your own crawlers and scrapers with. Before diving into the library, let's have a look at the terms crawling and scraping. For most real-world use cases, those two things go hand in hand, which is why this library helps with and combines both. A (web) crawler is a program that (down)loads documents and follows the links in it to load them as well. A crawler could... -
14
jekyll-theme-8bit
A Jekyll theme inspired by classic 8bit games
A Jekyll theme inspired by classic 8-bit games. Jekyll-theme-8bit has been scaffolded by the Jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. Snippets of code within the _includes directory that can be inserted in multiple layouts (and another include-file as well) within the same theme-gem. -
15
Goutte
Goutte, a simple PHP Web Scraper
Goutte is a screen scraping and web crawling library for PHP. Goutte provides a nice API to crawl websites and extract data from the HTML/XML responses. Goutte depends on PHP 7.1+. Add fabpot/goutte as a require dependency in your composer.json file. Create a Goutte Client instance (which extends Symfony\Component\BrowserKit\HttpBrowser). Make requests with the request() method. The method returns a Crawler object (Symfony\Component\DomCrawler\Crawler). To use your own HTTP settings, you may... -
16
ACHE Focused Crawler
ACHE is a web crawler for domain-specific search
.... ACHE also automatically learns how to prioritize links in order to efficiently locate relevant content while avoiding the retrieval of irrelevant pages. While ACHE was originally designed to perform focused crawls, it also supports other crawling tasks, including crawling all pages in a given web site and crawling Dark Web sites (using the TOR protocol). -
17
YYeTsBot
Renren Film and Television bot, fully connected to Renren resources
Renren Film and Television bot, fully connected to all Renren Film and television resources without deletion of resources. The database of this site is permanently open source and free. You can directly send the name of the episode you want to watch, and you can choose to share the webpage or link (ed2k and magnet links). When searching for resources, it will be searched according to my predetermined priority (everyone video offline, subtitle man), of course, you can also use commands to force... -
18
Nextra
Simple, powerful and flexible site generation framework
Simple, powerful, and flexible site generation framework with everything you love from Next.js. Nextra automatically converts Markdown links and images to use Next.js Link and Next.js Image when possible. No slow navigation or layout shift. -
19
PHPScraper
A universal web-util for PHP
..., including interesting attributes. You can filter and combine these to your needs. In some cases there is an option to get a simple or detailed version. PHPScraper can assist in collecting feeds such as RSS feeds, sitemap.xml-entries and static search indexes. This can be useful when deciding on the next page to crawl or building up a list of pages on a website. -
20
LinkChecker
Check links in web documents or full websites
LinkChecker is a free, GPL licensed website validator. LinkChecker checks links in web documents or full websites. It runs on Python 3 systems, requiring Python 3.8 or later. The version in the pip repository may be old, to find out how to get the latest code, plus platform-specific information and other advice see doc/install.txt in the source code archive. If you do not want to install any additional libraries/dependencies you can use the Docker image which is published on GitHub Packages. -
21
Minimal Mistakes Jekyll theme
Jekyll theme for building a personal site, blog, project documentation
A flexible two-column Jekyll theme. Perfect for building personal sites, blogs, and portfolios. Everything from the menus, sidebars, comments, and more can be configured or set with YAML Front Matter. Built with HTML5 + CSS3. All layouts are fully responsive with helpers to augment your content. Free to use however you want under the MIT License. Clone it, fork it, customize it, etc. Settings that affect your entire site can be changed in Jekyll’s configuration file: _config.yml, found... -
22
Minima
Minima is a one-size-fits-all Jekyll theme for writers
It's Jekyll's default (and first) theme. It's what you get when you run jekyll new. Minima has been scaffolded by the jekyll new-theme command and therefore has all the necessary files and directories to have a new Jekyll site up and running with zero-configuration. From Minima v3 onwards, the base layout is named base.html instead of default.html to avoid confusing new users into assuming that name holds a special status. Users migrating from older versions with customized _layouts... -
23
Polaris React
Shopify’s admin product component library
Our design system helps us work together to build a great experience for all of Shopify’s merchants. Downloads, links, and third-party tools to help you and your teams learn, draft, design, and build products for Shopify. Use the API to build Figma plugins, text editor extensions, or even a completely new site. While we do offer a CSS-only version, we strongly recommend using the React versions of our components. It’s the version that we use at Shopify. It allows for rich, complex components... -
24
WFDownloader App
Free batch downloader for image, wallpaper, video, audio, document,
Use as an image gallery, wallpaper, audio/music, video, document, and other media bulk downloader from supported websites. Also use to download sequential website urls that have a certain pattern (e.g. image01.png to image100.png). Also use app's built-in site crawler for advanced link search or extraction. There is also special support for forum media and open directory downloading. It's a programmable downloader and also works with password protected sites. Say goodbye to downloading one... -
25
OnionLinksV3
2023!!Onion Links V3!!(Forum&Chat&Onion Site)TR-EN
2023!!Onion Links V3!!(Forum&Chat&Onion Site)TR-EN