remotepower Files

v3.0.2 — multi-webhook, self-monitoring, IaC generator, mitigation, ACME, AI

Covers v2.6.1 → v3.0.2 (everything since v2.6.0 was tagged).

═══════════════════════════════════════════════════════════════════ NEW FEATURES ═══════════════════════════════════════════════════════════════════

Multi-webhook destinations (v3.0.2)

• Up to 20 destinations, each with its own format adapter: Discord (severity-coloured embed), Slack (markdown), Pushover (form-encoded with token+user, internal priority mapped, critical stays at high not emergency), Microsoft Teams (MessageCard), ntfy.sh (plain body + Title/Priority/Tags headers), generic JSON.
• Per-destination filters: event allowlist, min-priority threshold.
• Pushover credentials write-once-and-redacted in UI + backup export.
• Per-destination test button using a synthetic config (no persistence).
• Legacy webhook_url still honoured, deduplicated against the array.

Server self-monitoring (v3.0.2)

• New "Server status" sidebar entry + GET /api/self/status endpoint.
• Reports: process RSS, devices freshness, webhook success rate (24h + 7d), DATA_DIR disk usage + top 20 largest files, audit log size, scheduled backup state, fleet events archive size.

Scheduled backup (v3.0.2)

• Daily gzipped tarball of /var/lib/remotepower triggered via heartbeat hook with 24h sentinel + stale-lock recovery.
• Retention configurable (default 14 days). Excludes the backups directory itself, in-flight .tmp.* writes, existing .gz archives.
• Owner/group stripped so restoring on a different host works.
• Manual "Run backup now" button (POST /api/self/backup-now).

Bulk operations (v3.0.2)

• Modal on the Devices page next to "Enroll device".
• Filter by all monitored / by group / by tag.
• Operations: package upgrade, reboot, shutdown, force package scan, force ACME rescan. Destructive ops require typing RUN.

Command palette + keyboard shortcuts (v3.0.2)

• / and Ctrl-K open the global palette. Indexes pages, devices (cache primed on open), quick actions.
• g-prefix navigation: g h/d/l/s/c/m/a/v.
• ? shows the cheat sheet.

Settings search bar (v3.0.2)

• Live filter across all setting sections. Per-tab match badges, dim tabs with zero matches, auto-switch when current tab has no hits. Re-indexes on every keystroke (handles dynamic sections).

Force ACME rescan (v3.0.2)

• One-shot flag per device. Agent re-walks ~/.acme.sh on next heartbeat instead of waiting for the hourly cadence. Same flag-on-heartbeat-lock pattern as force-upgrade.

Exponential lockout ladder (v3.0.2)

• Failed-login lockout escalates 10s → 1m → 5m → 30m → 2h per consecutive episode. Resets on successful login.

Audit log retention (v3.0.2)

• Age-based eviction (default 90 days, configurable). Old entries moved to audit_log_archive.jsonl.gz instead of being dropped.

────────────────────────────────────────────────────────────────────

ACME / acme.sh integration (v3.0.1)

• Agent walks ~/.acme.sh inventorying certs across the fleet.
• Per-cert: domain, challenge type, provider, dates, status.
• Issue/renew/remove via the Security → TLS / DNS expiry section.
• acme.sh-not-installed devices hidden from the table; count surfaced as a discreet hint above (v3.0.2).

Mitigation runners with AI (v3.0.1)

• 🩺 button on every Needs Attention card whose alert has a deterministic remediation. AI-generated playbook with operator sign-off; logged to mitigate_logs/<run_id>.json.

File-path log watching (v3.0.1)

• Log rules can reference an arbitrary file path, not just a systemd unit. Useful for Tomcat, custom daemons, etc.
• Dormant-file handling: fires once after 3 consecutive missing heartbeats then goes quiet — avoids alert fatigue.

IaC Generator (v3.0.0)

• New top-level page generates Terraform/Ansible/Pulumi/Salt/ cloud-init from live host inventory.
• 18 categories: OS & identity, packages, systemd, users, groups, SSH keys, networking, mounts, firewall, kernel modules, cron, timezone, etc. 5 output formats.
• Server-side secret masking on env vars matching SECRET|TOKEN| PASSWORD|KEY patterns.

Security audits (v2.9.0)

• Listening port audit — new ports fire new_port_detected.
• SSH key audit — new authorized_keys fire ssh_key_added.
• Brute-force detection — SSH + RemotePower-login fail tracking.
• Disk-space items in Needs Attention.

Expanded log sources (v2.8.0)

• Kernel log (dmesg), APT history, Proxmox snapshot ages.
• snapshot_old webhook event (edge-triggered per VM).

TLS/DANE certificate monitoring (v2.6.1)

• Cert expiry tracked across the fleet. tls_expiry webhook event fires once per cert when within the configurable threshold.
• reboot_required webhook event fires once per host on first detection of /var/run/reboot-required (Debian/Ubuntu).
• Stale-agent-version + Pending-reboot items in Needs Attention.

AI assistant (v2.7.0+)

• Provider-agnostic chat: Anthropic, OpenAI, OpenAI-compatible, Ollama. Streaming responses. Custom system prompts per use case (free-form chat, explain command output, find problem in logs, mitigation playbook).
• Privacy controls: send_hostnames, send_ips, send_journal, send_cmd_output. Project + fleet context injection.

Progressive Web App (v2.6.0 was first; ongoing iterations)

• manifest.json + sw.js. Installable; cached shell.
• sw.js CACHE_NAME versioned (current: remotepower-shell-v3.0.2) so activation cleans stale caches.

Host Configuration Management (v2.6.0)

• Agent collects current host config snapshot.
• Drift detection against baseline; config_drift webhook event.

═══════════════════════════════════════════════════════════════════ INFRASTRUCTURE / PERFORMANCE ═══════════════════════════════════════════════════════════════════

Per-request load() cache (v3.0.2)

• CGI = fresh interpreter per request, but within one handler CONFIG_FILE was being parsed up to 4× per heartbeat. Cache deduplicates. Invalidated on save() and _save_held() (and on aborted _LockedUpdate exits).
• Returns deep copies so caller mutations don't corrupt the cache.

Webhook delivery hardening (v3.0.2)

• Per-device monitored=false suppresses webhooks for ALL events carrying a device_id, not just device_offline. Previously metric_warning/critical, service_, log_alert, cve_found, drift_detected, custom_script_, container_*, patch_alert, brute_force, ssh_key_added, tls_expiry, reboot_required, new_port_detected, snapshot_old, backup_stale, config_drift all bypassed the device's monitored flag.

═══════════════════════════════════════════════════════════════════ BUG FIXES ═══════════════════════════════════════════════════════════════════

Static-audit catches (v3.0.2):

• _read_body() called in mitigation handlers but never defined — NameError, mitigation modal hit 500 on every click. Fixed to get_json_body().
• chat_anthropic / chat_openai_compatible called bare from _call_ai_with_prompts but defined in ai_provider module. Prefixed.
• current_username() referenced in three audit-log call sites but never defined; ACME Ignore/Cancel buttons appeared dead.
• _get_disk_thresholds called at line 9193 but never defined, guarded by always-false callable(globals().get(...)) — per- mount disk threshold overrides silently ignored. Replaced with canonical _resolve_metric_thresholds.
• server_url typo in agent force-upgrade path.
• pacman --disable-sandbox probe was checking pacman --help instead of pacman -S --help. CachyOS silently failed.
• Four undefined CSS variables (--bg-card, --bg-hover, --ok, --bg2, --bg3, --border2, --font-body) rendered with default browser colours in dark mode. Caught by a CSS-var sweep test; locked in via TestCssVarsDefined.

UI/UX fixes:

• URL bar stuck at #settings/<tab> when navigating other pages (showPage didn't write history.replaceState; switchSettingsTab used location.hash directly).
• Webhook test route mismatch (frontend /webhook-test, backend /webhook/test). Both aligned on the slash form.
• Bulk modal rendered transparent (var(--bg-card) undefined).
• Logs Search button silently froze on bad-regex 400 responses (TypeError reading data.results.length on {error: ...}).
• Settings search bar cached the section index once on first call, missing dynamic sections like the webhook destinations list.
• Command palette didn't have devices until you visited the Devices page first.
• Pending updates dashboard tile counted unmonitored devices.
• Drift + CVE dashboard tiles aggregated unmonitored devices.
• Dashboard empty-state message clarified ("no critical CVE findings" instead of "no CVE findings").
• ACME table no longer renders one row per device without acme.sh installed; suppressed count surfaced above.

Earlier fixes (v2.6.1 → v3.0.0):

• CVE tile always showed 0 (dashboard read non-existent devices[].findings field).
• Reboot/shutdown icon flash on mobile refresh (stale sw cache).
• HTML structure errors from injected panes landing outside their parent containers.
• JSON.stringify embedded in HTML attributes without escaping.
• Three duplicate /devices GETs on the Monitor page (one per loader). Request coalescing added.

Security fixes (v2.3.2+):

• PBKDF2-HMAC-SHA256 at 600k iterations replaced bare unsalted SHA-256 for password fallback.
• must_change_password flag on default admin with persistent banner.
• Backup export redacts secrets: Proxmox token, SMTP password, LDAP bind password, Pushover token + user.

═══════════════════════════════════════════════════════════════════ INTERNALS ═══════════════════════════════════════════════════════════════════

• 1422 tests passing (up from ~1000 at v2.6.0).
• AST audits: zero called-but-undefined names in any scope, zero respond() inside broad-Exception try blocks.
• CSS-var coverage: zero undefined var(--xxx) across .js/.html/.css.
• HTML balance: 1396
  /
  .
• Make targets: test, lint, typecheck, check (test + lint), dist (runs full suite against staged tarball, fails on missing files).
• .gitignore hardened: dist/, *.swp, .DS_Store, IDE dirs, build caches.
• Demo seeder (packaging/seed-demo-data.py) updated to v3.0.2 schema: multi-webhook example with Pushover destination, audit retention, session TTLs, scheduled backup config, one unmonitored device to demonstrate the suppression behaviour.

Lint scope (make check): cmdb_vault.py, openapi_spec.py, test_v190.py, test_v1100.py. Expanding to api.py deliberately deferred — single reformat would produce an unreviewable diff.

═══════════════════════════════════════════════════════════════════ SECURITY AUDIT (completed in v3.0.2) ═══════════════════════════════════════════════════════════════════

End-to-end audit covering api.py + nginx config + agent + extended modules (WebTerm handshake, cmdb_vault, ldap_auth, TOTP, API keys, AI provider, Proxmox client). Findings + fixes:

Server:

• CSRF defense-in-depth: _enforce_same_origin() runs before route dispatch. State-changing methods (POST/PUT/PATCH/DELETE) must carry no Origin/Referer (CLI/agents) or one matching Host. X-Token header was already CSRF-safe by CORS preflight; this is belt-and-suspenders against future regressions.
• XSS in toast(): rendered msg via innerHTML without escape. Fixed with escHtml(msg). AI markdown renderer was already escape-first.
• Optional SSRF guard: webhook_block_local config flag refuses POST to loopback/link-local/unspecified IPs (AWS metadata at 169.254.169.254 etc). RFC1918 deliberately allowed (homelab Gotify legitimate). Off by default.

Agent (client/remotepower-agent.py):

• MEDIUM /tmp/ symlink attack on three marker files (last-cmd, poll-interval, pending-cmd.json). Moved to STATE_DIR = /var/lib/remotepower (mode 0700) with _safe_state_write/read/ unlink helpers using O_NOFOLLOW. /tmp/ fallback for non-root deploys uses O_NOFOLLOW + pre-unlink.
• MEDIUM credentials write race in save_credentials: write_text then chmod left file world-readable in the window. Replaced with os.open(O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0o600) + pre-unlink + CONF_DIR forced to 0700.
• MEDIUM server-controlled log_watch paths: could exfiltrate /etc/shadow etc via heartbeat-back. Added _file_log_path_allowed() with deny list (/etc/shadow, /etc/gshadow, /etc/sudoers, /root/.ssh/, /home//.ssh/, /proc/, /sys/, /dev/) using os.path.realpath() to defeat symlink bypass.
• LOW URL scheme strip bug: lstrip('http://').lstrip('https://') strips characters not prefix. 'httpserver.com' → 'server.com'. Fixed with _strip_url_scheme() helper.
• LOW AI provider dead code: insecure_ssl flag documented but never honored. Now implements documented behaviour matching proxmox_client.py pattern.

Confirmed safe (no changes needed):

• shell=True on exec: commands — BY DESIGN, the product feature.
• TLS: CERT_REQUIRED + check_hostname=True; http_post() rejects non-HTTPS.
• Self-update: SHA-256 + hmac.compare_digest, atomic via mkstemp + shutil.move.
• All other subprocess: argv-list, no shell injection.
• WebTerm handshake: admin re-auth required, 32-byte tokens, one-shot use, source IP captured, fully audit-logged.
• cmdb_vault: AES-GCM + PBKDF2 600k iter (OWASP 2023 min), key never persisted server-side, no log leakage.
• LDAP: TLS verification on by default (CERT_REQUIRED).
• TOTP: 160-bit secret via secrets.token_bytes(20), RFC 4226.
• API keys: secrets.token_urlsafe(40) = 320 bits, never returned after creation, hmac.compare_digest verification, 50-key cap.
• Audit log + debug log: no secrets leaked.

Audit scope NOT covered (out of scope or low risk, deferred):

• remotepower-webterm daemon binary (separate repo).
• PowerShell client (install-client.ps1) — separate platform.
• prometheus_export, smtp_notifier, cve_scanner, containers, tls_monitor, openapi_spec, ai_context — data processing, no external attack surface.

Tests added: 19 for the audit (10 agent + 4 NA coverage + 2 multi- webhook suppression parity + 2 uptime seeder + 1 sentinel). Full posture, threat model, operator hardening checklist: docs/security.md

═══════════════════════════════════════════════════════════════════ KNOWN ISSUES (deferred to v3.1.x) ═══════════════════════════════════════════════════════════════════

• Mobile layout collapse on some viewports — blocked on a Chrome Console error trace.