Search Results for "scraping"

...Clean API. Fast (>1k request/sec on a single core) Manages request delays and maximum concurrency per domain. Automatic cookie and session handling. Sync/async/parallel scraping. Distributed scraping. Caching, automatic encoding of non-unicode responses. Robots.txt support. Google App Engine support.

...It fills a monitoring niche by being the glue between applications that do not export their own internal state (other than via logs) and existing monitoring systems, such that system operators do not need to patch those applications to instrument them or writing custom extraction code for every such application. The extraction is controlled by mtail programs which define patterns and actions. Metrics are exported for scraping by a collector as JSON or Prometheus format over HTTP, or can be periodically sent to a collectd, StatsD, or Graphite collector socket. Precompiled binaries for released versions are available in the Releases page on Github. Using the latest production release binary is the recommended way of installing mtail.

CLI tool which enables you to log in and retrieve AWS temporary credentials using ADFS or PingFederate Identity Providers. Aside from Okta, most of the providers in this project are using screen scraping to log users into SAML, this isn't ideal and hopefully, vendors make this easier in the future.

...The exporter fetches information from an Elasticsearch cluster on every scrape, therefore having a too short scrape interval can impose load on ES master nodes, particularly if you run with --es.all and --es.indices. We suggest you measure how long fetching /_nodes/stats and /_all/_stats takes for your ES cluster to determine whether your scraping interval is too short. As a last resort, you can scrape this exporter using a dedicated job with its own scraping interval. Commandline parameters start with a single - for versions less than 1.1.0rc1. Username and password can be passed either directly in the URI or through the ES_USERNAME and ES_PASSWORD environment variables. ...

Golang Web Scraper library for extracting data from web pages. Save results as CSV, JSON, XML