Search Results for "web crawler source code" - Page 10
Sort By:
Showing 374 open source projects for "web crawler source code"
View related business solutions-
Gemini 3 and 200+ AI Models on One PlatformBuild generative AI apps with Vertex AI. Switch between models without switching platforms.
-
Try Google Cloud Risk-Free With $300 in CreditUse your credit across every product. Compute, storage, AI, analytics. When it runs out, 20+ products stay free. You only pay when you choose to.
-
1
Jupyter Server Proxy
Jupyter notebook server extension to proxy web services.
Jupyter Server Proxy lets you run arbitrary external processes (such as RStudio, Shiny Server, Syncthing, PostgreSQL, Code Server, etc) alongside your notebook server and provide authenticated web access to them using a path like /rstudio next to others like /lab. Alongside the Python package that provides the main functionality, the JupyterLab extension (@jupyterhub/jupyter-server-proxy) provides buttons in the JupyterLab launcher window to get to RStudio for example. -
2
WeChatSogou
Python library to crawl and retrieve data from WeChat accounts
WechatSogou is an open source Python library designed to retrieve data from WeChat official accounts by using the Sogou WeChat search service as its data source. It provides developers with a programmatic way to search for public accounts and collect article information without manually browsing the search interface. It functions as a crawler interface that sends requests to the search engine, retrieves results, and converts the returned pages into structured data that can be used in... -
3
pyspider
A powerful Spider(Web Crawler) system in Python
pyspider is a powerful Spider(Web Crawler) system in Python. Components are connected by message queue. Every component, including message queue, is running in their own process/thread, and replaceable. That means, when process is slow, you can have many instances of processor and make full use of multiple CPUs, or deploy to multiple machines. This architecture makes pyspider really fast. benchmarking. Since pyspider has various components, you can just run pyspider to start a standalone and... -
4
haipproxy
Distributed proxy IP pool for web crawlers using Scrapy and Redis
HAipproxy is a distributed proxy IP pool system designed to collect, manage, and provide large numbers of proxy addresses for web crawling tasks. It automatically crawls proxy resources from the internet and aggregates them into a centralized pool that can be accessed by distributed spiders and scraping systems. It is built using Python and relies on Scrapy for high-performance crawling while Redis is used for data storage, communication, and task coordination between components. It includes... -
Add Two Lines of Code. Get Full APM.Works out of the box for Rails, Django, Express, Phoenix, and more. Monitoring exceptions and performance in no time.
-
5
gain
Asyncio-based Python framework for building fast web crawling spiders
Gain is a Python web crawling framework designed to simplify the process of building efficient and scalable web scrapers. It is built on top of asynchronous technologies such as asyncio, aiohttp, and uvloop to support high-performance crawling with concurrent network requests. It provides a structured framework for creating spiders that can navigate websites, extract structured data, and process the collected results. Developers define crawlers using components such as spiders, parsers, and... -
6
Toapi
Convert websites into structured APIs automatically with Python tool
Toapi is a Python library designed to transform ordinary websites into usable API services. Instead of building a traditional web crawler that collects and stores data before exposing it through an API, Toapi simplifies the process by allowing developers to define data structures that automatically generate an API layer from existing web pages. It works by parsing HTML content from a source site and mapping selected elements into structured data that can be returned as JSON through API endpoints. ... -
7
Holarse
website software for holarse
HolaCMS 3 Source Code which will power the new Holarse website. -
8
OTPspot
Wireless hotspot with OTP captive portal
Run a captive portal on your raspberry (or any linux box) to allow your guests to register before accessing your Wifi at home. Users will be requested for an OTP code that you can generate on your phone through the Google Authenticator/Authy app. Get rid of captive portal's static username and password, without the need for a complex radius server. OTPspot (since version 2.0) is fully compatible with nodogsplash and can run as a FAS service. In this configuration, nodogsplash will... -
9
JavaScript Lint is a development tool that validates JavaScript code for best coding practices, finding common mistakes without actually running the script or opening the web page.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
10
Assorted projects. General-purpose libraries for Python, C++, Scala, bash, and others. Meta-programming tools. System utilities. UI components. Web APIs. Configuration files. Benchmarks. Programming competition entries. And much more.
-
11
owfs and owhttpd
Project moved to https://github.com/owfs/owfs/
Please note that OWFS source code, and all Issue/Tickets/merge requests have now been moved to https://github.com/owfs/owfs/. Developer mailing lists will still be kept at Sourceforge. ---- OWFS -- 1-Wire file system. Use the Dallas 1-Wire and iButton chips with standard filesystem commands. Create temperature loggers. Monitor everything. OWHTTPD -- same system, only used as a light weight web server. -
12
jd-autobuy
Python tool that automates JD.com login and product purchase tasks
jd-autobuy is an open source Python-based automation tool designed to simulate the purchasing process on the JD e-commerce platform. It uses web scraping and HTTP request techniques to log into an account, check product availability, and attempt to purchase specified items automatically. It supports login through methods such as QR code authentication, allowing users to sign in through the platform’s mobile application. -
13
icemac.addressbook
Multi user address book application accessable through the web.
Multi user address book application accessable through the web. Store, edit, search and export addresses, phone numbers, … using a web browser. Code moved to https://bitbucket.org/icemac/icemac.addressbook Documentation see https://icemacaddressbook.readthedocs.io/en/latest/ New releases (after 6.0.2) see https://pypi.org/project/icemac.addressbook/#history -
14
A Complete Beginner's Guide to Django
A Complete Beginner's Guide to Django - Code Samples
Code samples from the Django tutorial series. I’m starting a new tutorial series about Django fundamentals. It’s a complete beginner’s guide to start learning Django. The material is divided into seven parts. We’re going to explore all the basic concepts in great detail, from installation, and preparation of the development environment, models, views, templates, URLs to more advanced topics such as migrations, testing, and deployment. I wanted to do something different. A tutorial that would... -
15
diskover
File system crawler and disk space usage software
diskover is a file system crawler and disk space usage software that uses Elasticsearch to index your file metadata. diskover crawls and indexes your files on a local computer or remote storage server over network mounts. diskover helps manage your storage by identifying old and unused files and give better insights into data change "hotfiles", file duplication "dupes" and wasted space. It is designed to help deal with managing large amounts of data growth and provide detailed storage... -
16
Flasky
Companion code to my O'Reilly book "Flask Web Development"
Flasky is a comprehensive example web application built with the Flask microframework that demonstrates best practices for developing real-world Python web applications, covering everything from project structure and configuration to database models, authentication, and deployment. It serves as both a tutorial and sample codebase that walks developers through building a full-featured web application, including user registration and login, role-based permissions, user profiles, and content... -
17
SaberNet DCS is a labor data collection system, designed to allow organizations to rapidly capture their labor data in real-time. Optimized for bar code input, DCS is the perfect way to automate and improve the accuracy of your time tracking.
-
18
PyXB (“pixbee”) is a pure Python package that generates Python source code for classes that correspond to data structures defined by XMLSchema. In concept it is similar to JAXB for Java and CodeSynthesis XSD for C++.
-
19
A Python script to automate creation of supporting EtherCAT files including SSI EEPROM, EtherCATInfo XML (ESI file), and certain related source and header files for the Beckhoff slave stack code. mesicat accepts a .mesi file (Meta ESI) containing a description of the EtherCAT CoE dictionary and other stack configuration information. Make commands in this file then trigger the creation of sane defaults, SII EEPROM, XML files, and so forth. The .mesi file is inspired by the various IDL files used in web space (e.g. protobuf) and uses a C-like syntax to reduce cognitive overhead when developing slave code. ...
-
20
Flask JSONDash
Build complex dashboards without any front-end code
Build complex dashboards without any front-end code. Use your own endpoints. JSON config only. Ready to go. Easily configurable, chart dashboards from any arbitrary API endpoint. JSON config only. Ready to go. This project is a flask blueprint that allows you to create sleek dashboards without writing any front-end (or backend) code. Everything is powered through simple JSON configurations for declaring arbitrary charts. Leveraging popular libraries like C3.js and D3.js, and much more. Also... -
21
wxpy
Probably the most elegant Wechat personal number API
On the basis of itchat, wxpy improves the ease of use of the module through a large number of interface optimizations and provides rich function expansion. All in all, it can be used to automate the operation of various WeChat personal accounts. Judging from the recent feedback (late June 2017), there is a certain probability that the use of robots may be restricted from logging in. It is mainly manifested in the inability to log in to Web WeChat (but does not affect other platforms such as... -
22
TensorFlow World
Simple and ready-to-use tutorials for TensorFlow
This repository aims to provide simple and ready-to-use tutorials for TensorFlow. The explanations are present in the wiki associated with this repository. There are different motivations for this open source project. TensorFlow (as we write this document) is one of / the best deep learning frameworks available. The question that should be asked is why has this repository been created when there are so many other tutorials about TensorFlow available on the web? Deep Learning is in very high... -
23
NTK RTMP SERVER
Naam Tamilar Web TV Live Streamer
Naam Tamilar RTMP Server This project updated as open source for future use of Naam Tamilar Political Party. To contribute to the party and in some case if there is any possibilities if i cannot support them for long term. I thought of sharing this source code so in future it may be helpful for the community and party in which other software developers can help them to upgrade. This source is forked from - https://github.com/arut/nginx-rtmp-module and modified with multiple broadcast... -
24
Icon Font to PNG
Python script (and library) for exporting icons from icon fonts
Python script (and library) for easy and simple export of icons from web icon fonts (e.g. Font Awesome, Octicons) as PNG images. The best part is the provided shell script, but you can also use it’s functionality directly in your (probably awesome) Python project. There’s also font-awesome-to-png script for backward compatibility with the first iteration of the concept. You can use IconFont (and IconFontDownloader for that matter) directly inside your Python project. There's no proper... -
25
pyadselfservice is a software created using Python 3.5 and Django 1.10. This project aims to provide web based password change interface to the end users, for their Active Directory account. While changing the password, users won't not need to enter their current password. Which means users can change their password even if they have forgotten their current password. Moreover, while changing the password, this software will automatically unlock the user account if it is locked. The...
