Scrapy

Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD.

Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring and automated testing.

Features

Built-in support for the selection and extraction of data from HTML/XML sources through extended XPath expressions and CSS selectors, with helper methods for extracting using regular expressions
Interactive shell console for using CSS and XPath expressions to scrape data
Built-in support for generating feed exports in several different formats (JSON, XML, CSV) and their storage in various backends (S3, FTP, local filesystem)
Support for encoding and auto-detection
Strong extensibility support
Plenty of built-in extensions and middlewares for cookies and session handling, robots.txt, crawl depth restriction and more
Telnet console to hook into a Python console running inside your Scrapy process
Reusable spiders, media pipeline, caching DNS resolver and more

Project Samples

Project Activity

See All Activity >

License

BSD License

Follow Scrapy

Scrapy Web Site

Other Useful Business Software

Earn up to 16% annual interest with Nexo.

More flexibility. More control.

Generate interest, access liquidity without selling, and execute trades seamlessly. All in one platform. Geographic restrictions, eligibility, and terms apply.

Get started with Nexo.

Rate This Project

User Reviews

Be the first to post a review of Scrapy!

Additional Project Details

Operating Systems

BSD, Linux, Mac, Windows

Programming Language

Python

Related Categories

Python Internet Software, Python Automated Testing Tool, Python Web Scrapers

Registered

2020-04-28

Similar Business Software

Oxylabs

Oxylabs is a market leader in web intelligence with enterprise-grade, ethical, and compliant solutions. Its proxy infrastructure spans one of the largest global networks, offering residential, ISP, mobile, datacenter, & dedicated datacenter proxies, along with Web Unblocker – an AI-driven...

See Software
Apify

Apify is a full-stack web scraping and automation platform helping anyone get value from the web. At its core is Apify Store, a marketplace with over 10,000 Actors where developers build, publish, and monetize automation tools. Actors are serverless cloud programs that extract data, automate...

See Software
pCloud Business

pCloud is a cloud-based digital asset management platform and cloud storage that provides access to all your digital content including images, video, audio, docs, and more- anytime, anywhere, on any device. Keep all of your important files safe and centralized in one place. You can share with...

See Software
Google Cloud Platform

Google Cloud is a cloud-based service that allows you to create anything from simple websites to complex applications for businesses of all sizes. New customers get $300 in free credits to run, test, and deploy workloads. All customers can use 25+ products for free, up to monthly usage...

See Software
JAMS

JAMS is an automation orchestration and job scheduling solution that works across applications, APIs, and scripting languages. Run, monitor, and manage critical IT processes—from simple batch jobs to cross-platform workflows—from a single pane of glass. JAMS can automate jobs on any...

See Software
Square 9

Square 9 removes the frustration of extracting data from documents, forms, and all external sources, so you can harness the full power of your information. Release your team from repetitive tasks while your work flows freely in areas like Accounts Payable, Order Processing, Customer and...

See Software