Search Results for "tasks"

...The library allows developers to control Chrome or Chromium programmatically from Go applications without relying on external browser drivers or additional runtime dependencies. Using chromedp, developers can automate tasks such as navigating websites, filling out forms, clicking elements, capturing screenshots, and extracting structured data from pages. The library typically operates in headless mode but can also run with a visible browser window when needed for debugging or development. Because it communicates directly with Chrome’s debugging interface, chromedp offers high performance and reliable automation compared with tools that rely on intermediary drivers. ...

Geziyor is a high-performance web crawling and web scraping framework built for the Go programming language. It is designed to help developers crawl websites and extract structured information from web pages efficiently. It focuses on speed and scalability, allowing large numbers of requests to be processed concurrently. Geziyor supports use cases such as data mining, monitoring web content, and automated testing workflows. It provides a flexible architecture where developers define parsing...

go-dork is an open source command-line tool designed to automate search engine dorking and reconnaissance tasks. Written in the Go programming language, it focuses on speed and efficiency when executing advanced search queries across multiple search engines. It allows users to run specialized queries, often referred to as “dorks,” to discover publicly exposed data, misconfigurations, or potentially vulnerable resources. It supports several major search engines and enables users to switch between them depending on the target or query requirements. go-dork can retrieve results from multiple pages of search results and process them sequentially for broader coverage during scans. go-dork also supports custom HTTP headers and proxy configuration, which can help users work around restrictions such as captchas or filtering mechanisms. ...

...The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. Master node and worker nodes communicate with each other via gRPC (a RPC framework). Tasks are scheduled by the task scheduler module in the master node, and received by the task handler module in worker nodes, which executes these tasks in task runners. Task runners are actually processes running spider or crawler programs, and can also send data through gRPC (integrated in SDK) to other data sources, e.g. MongoDB.

...Asynchronous read and write. SO_REUSEPORT port reuse support. Automatically clean up idle connections. Support WebSocket/Protobuf, custom protocols. Support for scheduled tasks and delayed tasks. High-performance web socket server.

gocrawl is a lightweight web crawling library written in the Go programming language that enables developers to build custom web crawlers and data extraction tools. gocrawl focuses on providing a minimal yet powerful crawling engine that can be easily extended and adapted for different web scraping or indexing tasks. It is designed to be polite when accessing websites by respecting crawling rules such as robots.txt policies and applying crawl delays for each host. It executes requests concurrently using Go’s goroutines, allowing efficient and scalable page retrieval across multiple URLs. Developers have full control over the crawling workflow, including which URLs are visited, inspected, and processed during execution. gocrawl integrates with HTML parsing tools so responses can be inspected and queried in a structured way while crawling. ...