OpenSERP Files

Maintenance release. The main fix: Google results point to real pages again - encrypted /goto links are resolved to destination URLs. Also new: a switch to turn engines off, correct links in extracted content, and configurable CLI logging.

Highlights

Google /goto links resolve to destination URLs. Google now gives automated clients encrypted /goto?url=... redirects instead of result links, so url pointed at google.com and URL-based rank tracking and /mega dedup broke (#53). OpenSERP now resolves each one with a single redirect request: no cookies, the destination page is never fetched, and ads are never touched. It's on by default for /google/* and /mega/*, in browser and raw mode. Use resolve_urls=false to turn it off, or resolve_urls=N to resolve only the top N organic results. The same setting is google.resolve_urls in config and --resolve-urls in the CLI. If Google pushes back (403/429, captcha or consent page), resolution through that IP or proxy pauses for 10 minutes, and results keep the wrapper URL until then.

Wrapped links keep their attribution. When a link stays wrapped (resolution off, paused or failed), OpenSERP recovers the domain and favicon URL from the visible breadcrumb or site name and generates a best-effort stable result ID. Plaintext /url?q= redirects are unwrapped locally without an extra request. Contributed by @ramgalin in [#51].

Turn engines off. Set <engine>.disabled: true in config, or OPENSERP_<ENGINE>_DISABLED=true, e.g. OPENSERP_BING_DISABLED=true. A disabled engine gets no endpoints and is left out of /mega, in browser and raw mode. serve refuses to start if every engine is disabled. Requested in [#45], contributed by @hasandenizuk in [#62].

Relative links in extracted content now resolve correctly. Extraction honors HTML <base href> and the final page URL after redirects, fixing links, images, and canonical URLs in raw and rendered modes, including full-page output. Fixes [#42], builds on the base URL fix contributed by @Yoomitech in [#43].

Configurable CLI log levels. Set --log_level error|warn|info|debug|trace, OPENSERP_APP_LOG_LEVEL, or app.log_level in config. CLI commands default to warn; serve defaults to info. Logs stay on stderr and results on stdout. Initial support contributed by @Spidy092 in [#52].

Other changes

• google.resolve_concurrency, google.resolve_timeout and google.resolve_cooldown tune redirect resolution (defaults: 4, 3s, 10m)
• Cache keys include the resolve_urls setting
• Google raw mode parses and classifies each page once, keeps original ranks with start, and dedups after resolving links
• Missing optional config files no longer produce warnings
• Explicit verbosity flags correctly override invalid log-level values from env/config
• Extraction metadata and content share the same charset-aware HTML document
• Long result display URLs are truncated without breaking UTF-8
• JS and Python quickstart examples print the request ID
• resolve_urls documented in README and the OpenAPI spec
• Added regression coverage for attribution, redirect resolution, extraction URL resolution, engine toggles, and logging

Upgrade notes

• resolve_urls is on by default. Each Google search now makes extra requests to Google (one per wrapped organic link, 4 at a time, 3s budget) through the same exit as the search, which adds some latency. To keep the old behavior, pass resolve_urls=false, or set google.resolve_urls: false / OPENSERP_GOOGLE_RESOLVE_URLS=false.
• Malformed or unreadable config files now stop startup instead of continuing with defaults. An absent optional config file is still allowed.
• --log_level takes precedence over -q, -v, and -d. Explicit -q / -v / -d flags take precedence over env/config log levels.

New Contributors

• @ramgalin made their first contribution in https://github.com/karust/openserp/pull/51
• @Yoomitech made their first contribution in https://github.com/karust/openserp/pull/43
• @Spidy092 made their first contribution in https://github.com/karust/openserp/pull/52
• @hasandenizuk made their first contribution in https://github.com/karust/openserp/pull/62

Full Changelog: https://github.com/karust/openserp/compare/v0.8.12...v0.8.14