Search Results for "python web crawler"
Sort By:
664 projects for "python web crawler" with 2 filters applied:
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Simply solve complex auth. Easy for devs to set up. Easy for non-devs to use.Custom auth drains 25% of dev time and risks 62% more breaches, stalling enterprise deals. Frontegg platform delivers a simple login box, seamless authentication (SSO, MFA, passwordless), robust multi-tenancy, and a customizable Admin Portal. Integrate fast with the React SDK, meet compliance needs, and focus on innovation.
-
1
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
2
img2dataset
Easily turn large sets of image urls to an image dataset
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine. Also supports saving captions for url+caption datasets. Opt-out directives: Websites can pass the http headers X-Robots-Tag: noai, X-Robots-Tag: noindex , X-Robots-Tag: noimageai and X-Robots-Tag: noimageindex By default img2dataset will ignore images with such headers. -
3
dude uncomplicated data extraction
dude uncomplicated data extraction: A simple framework
Dude is a very simple framework for writing web scrapers using Python decorators. The design, inspired by Flask, was to easily build a web scraper in just a few lines of code. Dude has an easy-to-learn syntax. Dude is currently in Pre-Alpha. Please expect breaking changes. You can run your scraper from terminal/shell/command-line by supplying URLs, the output filename of your choice and the paths to your python scripts to dude scrape command. -
4
Gunicorn
WSGI HTTP Server for UNIX, fast clients and sleepy applications
Gunicorn 'Green Unicorn' is a Python WSGI HTTP Server for UNIX. It's a pre-fork worker model. The Gunicorn server is broadly compatible with various web frameworks, simply implemented, light on server resources, and fairly speedy. You can run Gunicorn by using commands or integrate with popular frameworks like Django, Pyramid, or TurboGears. For deploying Gunicorn in production see Deploying Gunicorn. After installing Gunicorn you will have access to the command line script gunicorn. Gunicorn... -
Sales CRM and Pipeline Management Software | PipedrivePipedrive’s simple interface empowers salespeople to streamline workflows and unite sales tasks in one workspace. Unlock instant sales insights with Pipedrive’s visual sales pipeline and fine-tune your strategy with robust reporting features and a personalized AI Sales Assistant.
-
5
Requests for PHP
Requests for PHP is a humble HTTP request library
Requests is a HTTP library written in PHP, for human beings. It is roughly based on the API from the excellent Requests Python library. Requests is ISC Licensed (similar to the new BSD license) and has no dependencies, except for PHP 5.6+. Despite PHP’s use as a language for the web, its tools for sending HTTP requests are severely lacking. cURL has an interesting API, to say the least, and you can’t always rely on it being available. Sockets provide only low-level access and require you... -
6
Easyspider - Distributed Web Crawler
Easy Spider is a distributed Perl Web Crawler Project from 2006
Easy Spider is a distributed Perl Web Crawler Project from 2006. It features code from crawling webpages, distributing it to a server and generating xml files from it. The client site can be any computer (Windows or Linux) and the Server stores all data. Websites that use EasySpider Crawling for Article Writing Software: https://www.artikelschreiber.com/en/ https://www.unaique.net/en/ https://www.unaique.com/ https://www.artikelschreiben.com/ https://www.buzzerstar.com/ https... -
7
Eric Integrated Development Environment
Python Development Environment with all batteries included
Eric is a Python IDE written using PyQt and QScintilla. It provides various features such as any number of open editors, an integrated (remote) debugger, project management facilities, unit test, refactoring and much more. -
8
MOFO Linux
A live Linux environment for computing without censorship barriers.
... or on the most modest of home desktop machines. It is a collection of office, multimedia, web browsing, file sharing, and internet messaging applications assisted by a collection of anonymity and anti-censorship tools. MOFO Linux contains Encrypted DNS over HTTPS, OpenVPN, Tor, Freenet, I2P, and other software tools which provide anonymous and / or secure access to the internet and circumvention of state censorship restrictions. -
9
ZK is an open-source Java framework for building modern web and mobile applications. It enables developers to create rich, interactive UIs using only Java — no JavaScript required. With 200+ Ajax-powered components, event-driven architecture, and support for popular technologies like Spring, Java EE, and JSP/JSF, ZK makes it simple to deliver powerful and user-friendly web applications.
-
Test your software product anywhere in the worldGlobal App Testing is a managed pool of freelancers used by Google, Meta, Microsoft, and other world-beating software companies.
-
10
ciwiki
Personnal or familly wiki with low ressource requirement.
Personal lightweight wiki based on DidiWiki. Upgraded to accept text and highlight color, image resize and video (youtube, dailymotion...) embedded. Written in C, doesn't require a lot of RAM. Works fine on Raspbian (Raspberry Pi). Example of Ciwiki running on Raspberry Pi B+ (700MHz, 512MB): http://inphilly.dyn.dhs.org -
11
http-proxy-tunnel
Create nested tunnels through HTTP proxies
Http-proxy-tunnel creates TCP tunnels through http proxies that permit the CONNECT method. It differs from other proxy tunnelling programs in that it can tunnel through multiple proxies, and can use SSL tunnels. These abilities mean that in combination with a web server that can proxy (such as Apache) you can serve normal web pages from ports 80 and 443 and connect to the server (using ssh say) via those ports at the same time. All available documentation can be read online at http... -
12
CerberusCMS5
Cerberus Content Management System
Cerberus Content Management System is a dynamic, secure and infinitely expandable CMS designed after a Unix-Like model. It is a custom written Web Application Framework ( W.A.F. ) with a consistent and custom written Pre-Hyper-Text-Post-Processor Programming Code Framework ( P.C.F. ). This Web Application Software Project' aim is to be the fastest and most secure Web Application Framework, Web Application Programming Code Framework, Text, Voice and Video Communications Platform and Content... -
13
A general purpose source code indexer and cross-referencer that provides web-based browsing of source code with links to the definition and usage of any identifier. Supports multiple languages. Up-to-date information in http://lxr.sourceforge.net
-
14
Cinemagoer
Python package to retrieve and manage data of the IMDb
Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb movie database about movies, people, characters and companies. Platform-independent, it can retrieve data from both the IMDb's web server and a local copy of the whole db. -
15
Webware for Python
The Classic Webware for Python
Webware for Python is a suite of components for dynamic, server-side web development. -
16
Zenoss Community Edition
Zenoss - Intelligent IT Operations Management
Zenoss provides software-defined IT operations for the world’s largest organizations. We deliver the ultimate level of IT service health with simplicity by providing the most granular and intelligent IT service modeling possible, at any scale, and sharing these unique insights with other IT operations management (ITOM) tools to make them more efficient. Zenoss Community Edition is not a “demo” or trial version of Zenoss Enterprise or Zenoss Cloud! Before You install Zenoss Community... -
17
Liferay Portal
The world's leading open source portal
Liferay Portal is the world's leading enterprise open source portal framework, offering integrated Web publishing and content management, an enterprise service bus and service-oriented architecture, and compatibility with all major IT infrastructure. Check GitHub for our latest releases: https://github.com/liferay/liferay-portal/releases https://github.com/liferay/liferay-ide/releases -
18
-
19
Zero Install is a decentralised cross-distribution software installation system. Create one package that works everywhere! With dependency handling and automatic updates, full support for shared libraries, and integration with native package managers
-
20
TimothyDocs
Timothy is a cloud base storage system designed to document your work
Timothy is a cloud based documentation system. Timothy will document any endeavor because it will store not only the documents created during the project but also store information about those files. Like most storge schemes timothy creates a hierarchy of categories through which one may browse. Timothy displays information about the document or category as well as its name. This use of meta data explains the structure and content of the project to the user as he browses. Users... -
21
PHP mini vulnerability suite
Multiple server/webapp vulnerability scanner
github: https://github.com/samedog/phpmvs -
22
SFM2Web reads text and database files encoded with SFMs (Standard Format Markers) and then generates a web site according to flags specified in control files. This is useful for web publication of MDF lexicons, USFM Bible books, texts, phrasebooks, etc.
-
23
OpenSearchServer Search Engine
An open source search engine with RESTFul API and crawlers
OpenSearchServer is a powerful, enterprise-class, search engine program. Using the web user interface, the crawlers (web, file, database, etc.) and the client libraries (REST/API , Ruby, Rails, Node.js, PHP, Perl) you will be able to integrate quickly and easily advanced full-text search capabilities in your application: Full-text with basic semantic, join queries, boolean queries, facet and filter, document (PDF, Office, etc.) indexation, web scrapping,etc. OpenSearchServer runs on Windows... -
24
icemac.addressbook
Multi user address book application accessable through the web.
Multi user address book application accessable through the web. Store, edit, search and export addresses, phone numbers, … using a web browser. Code moved to https://bitbucket.org/icemac/icemac.addressbook Documentation see https://icemacaddressbook.readthedocs.io/en/latest/ New releases (after 6.0.2) see https://pypi.org/project/icemac.addressbook/#history -
25
SpiderFoot
Open Source Intelligence Automation.
SpiderFoot is an open source intelligence automation tool. Its goal is to automate the process of gathering intelligence about a given target, which may be an IP address, domain name, hostname or network subnet. SpiderFoot can be used offensively, i.e. as part of a black-box penetration test to gather information about the target or defensively to identify what information your organisation is freely providing for attackers to use against you.
