Showing 749 open source projects for "python web crawler"

View related business solutions
  • Gen AI apps are built with MongoDB Atlas Icon
    Gen AI apps are built with MongoDB Atlas

    The database for AI-powered applications.

    MongoDB Atlas is the developer-friendly database used to build, scale, and run gen AI and LLM-powered apps—without needing a separate vector database. Atlas offers built-in vector search, global availability across 115+ regions, and flexible document modeling. Start building AI apps faster, all in one place.
    Start Free
  • Simple, Secure Domain Registration Icon
    Simple, Secure Domain Registration

    Get your domain at wholesale price. Cloudflare offers simple, secure registration with no markups, plus free DNS, CDN, and SSL integration.

    Register or renew your domain and pay only what we pay. No markups, hidden fees, or surprise add-ons. Choose from over 400 TLDs (.com, .ai, .dev). Every domain is integrated with Cloudflare's industry-leading DNS, CDN, and free SSL to make your site faster and more secure. Simple, secure, at-cost domain registration.
    Sign up for free
  • 1
    Python Web

    Python Web

    Course to learn frontend web development

    This repository is a beginner-friendly template for creating Python web applications using Flask. Designed by @mouredev for learning and practice, it provides a simple, minimalistic structure for serving HTML pages and static content. Ideal for educational purposes and small-scale web projects, it also includes preconfigured files to simplify deployment and local development.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 2
    Spatie Crawler

    Spatie Crawler

    An easy to use, powerful crawler implemented in PHP

    Spatie Crawler is a PHP library that allows developers to crawl websites and extract information efficiently. It can be used for web scraping, link checking, or automated testing of web pages. The library is simple to use and supports customizable crawling strategies, including controlling crawl depth and handling redirects. It’s suitable for building crawlers that navigate large or dynamically generated websites.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 3
    WebMagic

    WebMagic

    A scalable web crawler framework for Java

    WebMagic is a scalable crawler framework. It covers the whole lifecycle of crawler, downloading, url management, content extraction and persistent. It can simplify the development of a specific crawler. WebMagic is a simple but scalable crawler framework. You can develop a crawler easily based on it. WebMagic has a simple core with high flexibility, a simple API for html extracting. It also provides annotation with POJO to customize a crawler, and no configuration is needed. Some other...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 4
    Best-of Web Development with Python

    Best-of Web Development with Python

    A ranked list of awesome python libraries for web development

    ...If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! A ranked list of awesome python libraries for web development. Updated weekly.
    Downloads: 0 This Week
    Last Update:
    See Project
  • The All-in-One Commerce Platform for Businesses - Shopify Icon
    The All-in-One Commerce Platform for Businesses - Shopify

    Shopify offers plans for anyone that wants to sell products online and build an ecommerce store, small to mid-sized businesses as well as enterprise

    Shopify is a leading all-in-one commerce platform that enables businesses to start, build, and grow their online and physical stores. It offers tools to create customized websites, manage inventory, process payments, and sell across multiple channels including online, in-person, wholesale, and global markets. The platform includes integrated marketing tools, analytics, and customer engagement features to help merchants reach and retain customers. Shopify supports thousands of third-party apps and offers developer-friendly APIs for custom solutions. With world-class checkout technology, Shopify powers over 150 million high-intent shoppers worldwide. Its reliable, scalable infrastructure ensures fast performance and seamless operations at any business size.
    Learn More
  • 5
    Playwright for Python

    Playwright for Python

    Python version of the Playwright testing and automation library

    Playwright enables reliable end-to-end testing for modern web apps. Single API to automate Chromium, Firefox and WebKit. Capable automation for single page apps that rely on the modern web platform. Use the Playwright API in JavaScript & TypeScript, Python, .NET and, Java. With Playwright, test how your app behaves in Apple Safari with WebKit builds for Windows, Linux and macOS. Test locally and on CI.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 6
    Best-of Python

    Best-of Python

    A ranked list of awesome Python open-source libraries

    ...If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! Ranked list of awesome python libraries for web development. Correctly generate plurals, ordinals, indefinite articles; convert numbers. Libraries for loading, collecting, and extracting data from a variety of data sources and formats. Libraries for data batch- and stream-processing, workflow automation, job scheduling, and other data pipeline tasks.
    Downloads: 4 This Week
    Last Update:
    See Project
  • 7
    Heritrix

    Heritrix

    Internet Archive's open-source, web-scale, web crawler project

    Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.
    Downloads: 3 This Week
    Last Update:
    See Project
  • 8
    FastAPI Python

    FastAPI Python

    FastAPI framework, high performance, easy to learn, fast to code

    FastAPI framework, high performance, easy to learn, fast to code, ready for production. FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints.
    Downloads: 4 This Week
    Last Update:
    See Project
  • 9
    AUTOMATIC1111 Stable Diffusion web UI
    AUTOMATIC1111's stable-diffusion-webui is a powerful, user-friendly web interface built on the Gradio library that allows users to easily interact with Stable Diffusion models for AI-powered image generation. Supporting both text-to-image (txt2img) and image-to-image (img2img) generation, this open-source UI offers a rich feature set including inpainting, outpainting, attention control, and multiple advanced upscaling options. With a flexible installation process across Windows, Linux, and...
    Downloads: 53 This Week
    Last Update:
    See Project
  • Enterprise and Small Business CRM Solution | Clear C2 C2CRM Icon
    Enterprise and Small Business CRM Solution | Clear C2 C2CRM

    Voted Best CRM System with Top Ranked Customer Support. CRM Management includes Sales, Marketing, Relationship Management, and Help Desk.

    C2CRM consists of four modules that integrate to provide a comprehensive CRM solution: Relationship Management, Sales Automation, Marketing Automation, and Customer Service. Only buy what each user needs.
    Learn More
  • 10
    web.py

    web.py

    Web framework for python that is as simple as it is powerful

    web.py is a web framework for Python that is as simple as it is powerful. web.py is in the public domain, you can use it for whatever purpose with absolutely no restrictions. web.py was originally published while Aaron Swartz worked at reddit.com, where the site used it as it grew to become one of the top 1000 sites according to Alexa and served millions of daily page views.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 11
    The Falcon Web Framework

    The Falcon Web Framework

    The no-nonsense REST API and microservices framework

    Falcon is a minimalist WSGI library for building speedy web APIs and app backends. We like to think of Falcon as the Dieter Rams of web frameworks. When it comes to building HTTP APIs, other frameworks weigh you down with tons of dependencies and unnecessary abstractions. Falcon cuts to the chase with a clean design that embraces HTTP and the REST architectural style. Highly optimized, extensible code base. Easy access to headers and bodies through request and response objects. DRY request...
    Downloads: 2 This Week
    Last Update:
    See Project
  • 12
    Selenium-python Helium

    Selenium-python Helium

    Selenium-python but lighter: Helium is the best Python library

    Under the hood, Helium forwards each call to Selenium. The difference is that Helium's API is much more high-level. In Selenium, you need to use HTML IDs, XPaths and CSS selectors to identify web page elements. Helium on the other hand lets you refer to elements by user-visible labels. As a result, Helium scripts are typically 30-50% shorter than similar Selenium scripts. What's more, they are easier to read and more stable with respect to changes in the underlying web page. Selenium-python is great for web automation. Helium makes it easier to use. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 13
    Zappa - Serverless Python

    Zappa - Serverless Python

    Serverless Python

    Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as "serverless" web hosting for your Python apps. That means infinite scaling, zero downtime, zero maintenance - and at a fraction of the cost of your current deployments! With a traditional HTTP server, the server is online 24/7, processing requests one by one as they come in. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 14
    REST APIs with Flask and Python

    REST APIs with Flask and Python

    Projects and e-book for our course, REST APIs with Flask and Python

    A full course to teach you how to use Flask and Python to make REST APIs using multiple Flask extensions and PostgreSQL. Learn Flask, Docker, PostgreSQL, and more. Build professional-grade REST APIs with Python. No more outdated tutorials. Use Python 3.10+ and the latest versions of every Flask extension and library. Run your apps in Docker, host your code with Git, write documentation with Swagger, and test your APIs while developing. Learn how to perform user authentication using JWTs and...
    Downloads: 3 This Week
    Last Update:
    See Project
  • 15
    Flask

    Flask

    The Python micro framework for building web applications

    Flask is a lightweight WSGI web application framework designed to help developers get started with their web applications quickly and easily with the ability to scale up to complex applications. Being a “micro” framework does not mean that your whole web application must fit into a single Python file (although it can) or that it be limited; rather it means that Flask aims to keep the core simple but extensible.
    Downloads: 201 This Week
    Last Update:
    See Project
  • 16
    X-Crawl

    X-Crawl

    Flexible Node.js AI-assisted crawler library

    A high-performance web crawling and scraping framework for Node.js, designed for large-scale data extraction.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 17
    FastAPI

    FastAPI

    FastAPI framework, high performance, easy to learn, fast to code

    FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints. Great editor support. Completion everywhere. Less time debugging. Designed to be easy to use and learn. Less time reading docs. Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs. Get production-ready code. With automatic interactive documentation.
    Downloads: 48 This Week
    Last Update:
    See Project
  • 18
    AWS Lambda Python Runtime Interf Client

    AWS Lambda Python Runtime Interf Client

    Seamlessly extend your preferred base images to be Lambda compatible

    ...The Lambda Runtime Interface Client is a lightweight interface that allows your runtime to receive requests from and send requests to the Lambda service. The Lambda Python Runtime Interface Client is vended through pip. You can include this package in your preferred base image to make that base image Lambda compatible. To make it easy to locally test Lambda functions packaged as container images we open-sourced a lightweight web-server, Lambda Runtime Interface Emulator (RIE), which allows your function packaged as a container image to accept HTTP requests. ...
    Downloads: 1 This Week
    Last Update:
    See Project
  • 19
    AWS X-Ray SDK for Python

    AWS X-Ray SDK for Python

    AWS X-Ray SDK for the Python programming language

    AWS X-Ray SDK for the Python programming language. The AWS X-Ray SDK for Python is compatible with Python 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, and 3.9. X-Ray Python SDK will by default generate no-op trace and entity id for unsampled requests and secure random trace and entity id for sampled requests. If customer wants to enable generating secure random trace and entity id for all the (sampled/unsampled) requests (this is applicable for trace id injection into logs use case) then they should set the AWS_XRAY_NOOP_ID environment variable as False. ...
    Downloads: 1 This Week
    Last Update:
    See Project
  • 20
    NiceGUI

    NiceGUI

    Create web-based user interfaces with Python

    NiceGUI is a Python-based UI framework that enables developers to create interactive web applications using only Python code. It abstracts away the complexities of HTML, CSS, and JavaScript, allowing for rapid development of web interfaces directly from Python scripts. NiceGUI is suitable for building dashboards, control panels, and other web-based tools, especially in contexts like robotics and data visualization.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 21
    Streamlit

    Streamlit

    The fastest way to build data apps in Python

    ...Effortlessly share, manage and deploy your apps, directly from Streamlit. Streamlit lets you turn data scripts into sharable web apps in minutes, not weeks. It’s all Python, open-source, and free! And once you’ve created an app you can use our cloud platform to deploy, manage, and share your app! Streamlit can also be installed in a virtual environment on Windows, Mac, and Linux.
    Downloads: 48 This Week
    Last Update:
    See Project
  • 22
    CadQuery

    CadQuery

    A python parametric CAD scripting framework based on OCCT

    ...Provide a non-proprietary, plain text model format that can be edited and executed with only a web browser. The scripts use a standard programming language, Python, and thus can benefit from the associated infrastructure. This includes many standard libraries and IDEs. CadQuery's CAD kernel Open CASCADE Technology (OCCT) is much more powerful than the CGAL used by OpenSCAD. Features supported natively by OCCT include NURBS, splines, surface sewing, STL repair, STEP import/export, and other complex operations.
    Downloads: 111 This Week
    Last Update:
    See Project
  • 23
    H2O Wave

    H2O Wave

    Realtime Web Apps and Dashboards for Python and R

    No HTML, CSS, Javascript skills are required. Build rich, interactive web apps using pure Python. Broadcast live information, visualizations, and graphics using Wave's low-latency real-time server. Instant control over every connected web browser using a simple and intuitive programming model. Preview your app live as you code. Dramatically reduce the time and effort to build web apps. Easily share your apps with end-users, get feedback, improve and iterate. ~10MB static executables for Linux, Windows, OSX, BSD, Solaris on AMD64, 386, ARM, PPC. ...
    Downloads: 6 This Week
    Last Update:
    See Project
  • 24
    liquidpy

    liquidpy

    A port of liquid template engine for python

    A port of liquid template engine for Python, on the shoulder of jinja2.
    Downloads: 1 This Week
    Last Update:
    See Project
  • 25
    pywebview

    pywebview

    Build GUI for your Python program with JavaScript, HTML, and CSS

    ...If you choose to freeze your application, pywebview does not bundle a heavy GUI toolkit or web renderer with it keeping the executable size small. pywebview is compatible with Python 3.
    Downloads: 2 This Week
    Last Update:
    See Project
  • Previous
  • You're on page 1
  • 2
  • 3
  • 4
  • 5
  • Next