Scrapy
Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. Built-in support for selecting and extracting data from HTML/XML sources using extended CSS selectors and XPath expressions, with helper methods to extract using regular expressions. Built-in support for generating feed exports in multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, S3, local filesystem). Robust encoding support and auto-detection, for dealing with foreign, non-standard and broken encoding declarations.
Learn more
Nightwatch.js
Nightwatch.js is an integrated, easy to use End-to-End testing solution for web applications and websites, written in Node.js. It uses the W3C WebDriver API to drive browsers and perform commands and assertions on DOM elements. Simple but powerful syntax which enables you to write tests very quickly, using Javascript (Node.js) and CSS or Xpath selectors. Typescript is supported as well. Built-in command-line test runner which runs the tests either sequentially or in parallel, with retries and implicit waits. Also supports grouping of test suites and tags. Manages automatically Selenium or WebDriver services (ChromeDriver, GeckoDriver, Edge, Safari) in a separate child process. Fluent and easy to work with Page Object Model support to better organise elements and sections, with support for both CSS or Xpath selectors.
Learn more
jsoup
jsoup is a Java library that simplifies working with real-world HTML and XML. It offers an easy-to-use API for URL fetching, data parsing, extraction, and manipulation using DOM API methods, CSS, and XPath selectors. jsoup implements the WHATWG HTML5 specification and parses HTML to the same DOM as modern browsers. With jsoup, you can scrape and parse HTML from a URL, file, or string; find and extract data using DOM traversal or CSS selectors; manipulate HTML elements, attributes, and text; clean user-submitted content against a safelist to prevent XSS attacks; and output tidy HTML. jsoup is designed to deal with all varieties of HTML found in the wild, from pristine and validating to invalid tag-soup, creating a sensible parse tree. For example, you can fetch the Wikipedia homepage, parse it to a DOM, and select the headlines from the "In the news" section into a list of elements.
Learn more
UI-licious
Stop writing brittle tests with hard-coded CSS, XPATH selectors and waits. Write tests that are meaningful to people, effortless to maintain, and reusable. The reason for this is because writing your tests using hard-coded CSS and XPATH selectors is like pouring concrete onto a specific UI implementation. This creates very high-maintenance tests that break at the slightest changes to the UI and are very hard for humans to read. UI-licious uses dynamic code analysis to understand the structure of your website based on the use of semantic HTML and ARIA accessibility attributes, as well as context of the previous commands, to identify which is the intended element to target for every command. This means that even if the HTML code for the UI changes underneath the code, the test is valid as long as the user journey remains the same. That said, your website doesn't to be written perfectly for UI-licious to work.
Learn more
