EverOS Files

Background maintenance that fails loudly instead of quietly. A soak run on 1.2.2 found a table that had stopped reclaiming disk for 100 minutes while /health stayed green — nothing had failed, a call had simply never returned, and every signal was built from failure counters. Auditing for that shape turned up six more places it could happen: reads with no deadline (which stop the whole md to LanceDB projection, not just one table), background loops that die permanently on one exception with no log at all, an alert counter reset by the remediation it triggers. All of them are now bounded, and a stall that does happen names the table it happened to. Alongside that, agent-skill extraction is rescued from a retry-then-dead-letter loop, keyword search no longer returns 500 during an index rebuild, and the maintenance cadences moved into settings.

Fixed

• Agent skill extraction is no longer stuck in a retry-then-dead-letter loop. Target case data now travels on SkillClusterUpdated and existing skills for the cluster are read from markdown (strong-consistency), so the strategy never races cascade indexing. Prior to this fix, running a fresh agent trajectory produced zero SKILL.md files — .skills/ did not exist. The related stale-index clobber is fully closed only for clusters at or below MAX_SKILLS_IN_PROMPT (10). Above it, markdown still supplies the candidate set but LanceDB orders it, and the skill a lagging index omits is by definition the one written most recently — the one most likely to need update — so it can be ranked out of the prompt and re-added instead. The window is narrow (it needs a cluster over 10 skills and an index that has not caught up) and the consequence is the pre-existing full-replace, not a new failure mode.
• POST /api/v2/ome/trigger no longer masks strategy state. The status field now distinguishes not_dispatched (all dispatch gates rejected the strategy — usually a missing "force": true) from ok (dispatched and settled). The new runs field surfaces dead-lettered strategy runs that were previously invisible to the caller. If your client matches status exhaustively (Python Literal, TypeScript union), add a not_dispatched branch.
• Agentic search on agent memory now uses the skill-shaped rerank passage. The cross-encoder previously saw only the raw description field instead of the name + description + skill instruction triple that the HYBRID lane uses. A skill with empty description (a legal everalgo output — see everalgo/agent_memory/skill_ops.py:294) no longer causes HTTP 500 during the LLM sufficiency check.
• OME strategy retries now back off between attempts. A retry-class error (e.g. waiting on eventually-consistent state) previously exhausted its max_retries budget in milliseconds; the loop now sleeps min(base * 2**(attempt-1), cap) plus up to jitter seconds (defaults: 1s base / 10s cap / 0.5s jitter — code-only defaults, not currently exposed via everos.toml or ome.toml). engine_sem is now held per attempt rather than across the whole retry chain, so the backoff sleep does not occupy a concurrency slot. The cap bounds concurrent strategy work — LLM calls, embeddings, storage IO — and a coroutine waiting to retry consumes none of it; holding the slot would have turned a partial outage into a total stall, since enough simultaneously-failing runs park every one of the max_concurrent_runs slots in asyncio.sleep and starve strategies that would have succeeded. Backpressure on failing work is intended; backpressure on everything else is not.
• Path-traversal hardening for LLM-generated agent-skill names (CWE-22). AgentSkillFrontmatter.name comes straight from LLM output (extract_agent_skill) and was concatenated unsanitized into the skills/skill_<name>/ directory segment on both the write and read paths; given a sufficiently long ../ prefix, the write target could escape the memory root. This is the same class of defect previously fixed for knowledge-upload titles/categories (see knowledge_writer.py in an earlier 1.2.x). The sanitizer is now a single shared helper (everos.core.persistence.markdown.sanitize_dirname) used by both KnowledgeWriter and the new SkillPathMixin.skill_dir_name() / sanitize_skill_name(), instead of two independently maintained copies. extract_agent_skill now sanitizes the LLM-emitted name before constructing AgentSkillFrontmatter, so AgentSkillFrontmatter.name and the LanceDB agent_skill primary key now hold the sanitized name (spaces become _, characters outside [\w\-.] are dropped, capped at 50 chars), not the raw LLM output — a user-visible change for anything that reads a skill's name field expecting the verbatim LLM string. AgentSkillFrontmatter.name also gained a validator rejecting a name containing a path separator, or being exactly .., so a hand-edited SKILL.md that bypasses the writer's sanitization is caught on read rather than silently relocated (the substring form, e.g. a name that merely contains .., is deliberately allowed — sanitized output can legitimately contain runs of literal dots). sanitize_dirname itself falls back (not just on an empty result, but also on . or ..) so a short input that is itself a sanitizer fixpoint — e.g. "../" sanitizes to ".." verbatim without this fallback — cannot resolve to the same directory or its parent; this closes both the agent-skill case and an equivalent one-level escape on the knowledge-upload path, which has no skill_-style prefix protecting its sanitized segment. No data migration is needed for agent skills: extraction has never successfully produced a SKILL.md before this release (see the cascade-lag fix above), so there is no legacy skill corpus whose directory names would change. Knowledge documents do have a pre-existing corpus, and two inputs resolve to a different directory than before: a decomposed (NFD) topic or category now keeps its combining marks ("Résumé" no longer degrades to "Resume") because the shared helper NFC-normalizes first, and a topic or category of exactly . or .. now falls back instead of resolving onto the parent directory. Precomposed input — including CJK — is unaffected; the character class is unchanged from the previous private copy. Sanitizing is lossy: skills whose raw names differ only in characters the sanitizer drops or replaces (e.g. "fix django" vs. "fix_django") now share one SKILL.md, and so do names differing only in a combining mark regardless of script (e.g. Devanagari "किताब" vs. "कताब" — a combining mark alone is not \w and is stripped either way; same for Thai tone marks, Hebrew niqqud, Arabic harakat). The later write wins — the earlier skill's source_case_ids, maturity_score, and body are silently lost, not merged. Case is not folded, so "Fix Django" and "fix django" stay two distinct sanitized names — two LanceDB rows, but one directory on a case-insensitive filesystem (macOS APFS and Windows NTFS defaults), where the index then advertises a name whose content was overwritten. This is accepted for now rather than mitigated: detecting a collision and raising would reintroduce the dead-letter DoS the sanitizer was built to avoid, and a disambiguating suffix — the workable option — needs a collision probe plus a case-folding rule, so it is deferred to a deliberate pass rather than added here.
• A renamed skill no longer leaves an orphan directory that pollutes the next extraction. everalgo treats a name change as a first-class update (skill_ops._apply_update preserves prior.id while swapping the name), so the emitted skill was written to a new skill_<new_name>/ while the old directory survived carrying the same cluster_id. Because existing skills are now read from markdown rather than LanceDB, that orphan did not merely sit on disk — it came back in the next run's existing_relevant_skills as a duplicate of a skill the LLM had already renamed, feeding exactly the add-instead-of-update full-replace clobber this release set out to close, once more per rename. The old directory is now reaped after the new one is written, keyed on the skill's id (the only thing that survives a rename; a fresh add mints a uuid and can never match). A prior name that another skill in the same batch just claimed is never deleted.
• extract_agent_skill retire ops are documented as unimplemented rather than silently mispersisted. AgentSkillExtractor.aextract returns a flat list with no op discriminator, so a retirement arrives as an ordinary skill with confidence < retire_confidence and was written back like any other — staying in markdown, in the next prompt, and in search. The behaviour is unchanged; the module docstring no longer claims retire is handled. Honouring it is a design decision (delete the directory, or add a retired flag that the enumeration, cascade, and search all filter on) deferred to its own change.
• reference_name and script_filename are sanitized. Both are appended after the skill_<name> segment, so skill_dir_name never covered them; they now go through the same sanitize_dirname primitive on both the reader and the writer. No caller in src/ reaches them today, so nothing was exploitable — this closes the gap before progressive disclosure wires them up.
• A single unparseable SKILL.md no longer disables skill extraction for its whole cluster. AgentSkillReader.list_by_cluster propagated any frontmatter ValidationError, which aborted the enumeration that feeds extract_agent_skill its existing skills — so one hand-edited file (or, after a future schema revision adds a required field, every existing file at once) dead-lettered that cluster's extraction on every run. Offending files are now logged and skipped. read_main still raises, since a caller naming one specific skill needs to hear about corruption rather than receive the None that already means "not created yet". merged. This is accepted, not mitigated, on two grounds: a disambiguating suffix would break the name ≡ directory-suffix identity the reader/writer relies on, and detecting a collision and raising would reintroduce the dead-letter DoS the sanitizer was built to avoid. "fix django" vs. "fix_django") now share one SKILL.md, and the later write wins — the earlier skill's source_case_ids, maturity_score, and body are silently lost, not merged. This is accepted, not mitigated: the LLM's add/update decision is keyed on the name it sees, so a collision usually reads as an intended update anyway. under the new sanitizer. KnowledgeWriter and the new SkillPathMixin.skill_dir_name(), instead of two independently maintained copies. AgentSkillFrontmatter.name also gained a validator rejecting path separators / .. so a hand-edited SKILL.md is caught on read rather than silently relocated. No data migration: agent-skill extraction has never successfully produced a SKILL.md before this release (see the cascade-lag fix above), so there is no legacy skill corpus whose directory names would change under the new sanitizer.
• Reads now carry a deadline (count / get_by_id / find_where / find_where_paginated / search). The write-side deadline work skipped them on the reasoning that a read takes no lock and so blocks no writer — true, but the cascade drain loop reads on every batch and advances strictly one batch at a time, so a read that never returns stops the whole md → LanceDB projection: claimed rows stay processing forever, nothing new is indexed, and /health still reports healthy because a hang raises nothing. Budget 60s (~1000x the measured 62ms flat scan over 117k rows); expiry raises the retryable VectorStoreBusyError.
• Background loops are supervised. The drain / heartbeat / rebuild loops were plain create_task coroutines: one uncaught exception ended that loop permanently, and because the worker holds a strong reference to the task the interpreter never printed "Task exception was never retrieved" either — the loop's job simply stopped happening with zero output. Each now runs under a supervisor that logs and restarts with escalating backoff (5s / 15s / 45s), then asks the process to exit via SIGTERM so a restarting supervisor (systemd, Docker, k8s) can recover it. The restart budget counts consecutive quick crashes, not crashes over the process lifetime — a body that ran 60s+ before raising starts a fresh incident, so independent transients days apart cannot pool into a process exit (same windowed counting as systemd's StartLimitIntervalSec). A done-callback covers the case the supervisor itself ends unexpectedly.
• The optimize-failure alert is reachable again. The fallback rebuild reset the same counter the health verdict reads, so a table failing 100% of the time cycled 1..5 → 0 → 1.. and the threshold value existed only during the sub-second rebuild — roughly 1% observable against a 30s scrape, so cascade.healthy stayed green while the table never reclaimed a version. The rate limiter now lives in its own counter (failures_since_fallback); only a successful optimize clears the alert streak. Same shape as the cross-kind max() masking bug: a remediation path refreshing the signal meant to report it.
• A rebuild no longer leaves the column without an FTS index. rebuild_indexes dropped every index and recreated it, on the assumption that LanceDB falls back to a brute-force scan meanwhile. That holds for vector search and not for FTS: with no inverted index a BM25 query raises Cannot perform full text search unless an INVERTED index has been created, and since the recall legs are gathered without return_exceptions, one failing leg 500s the whole search request. Now uses create_index(replace=True), which swaps atomically — measured 0 failures across 49 queries spanning 3 replaces, versus 55 failures for the same test against drop-then-create — and collapses the live index fragment set exactly as before (7 index files back to 4).
• The empty-index-dir sweep is bounded by lance's own threshold. lance's cleanup.rs unlinks a superseded index's files but never its directory — it contains no rmdir at all, which is structural rather than an oversight: it targets object stores, where paths are flat keys and an empty directory does not exist. Only a local filesystem materialises them, and a soak run reached 13061 dirs, 98% empty. everos sweeps them, now with three independent guarantees instead of a self-chosen age: rmdir cannot delete a non-empty directory (the kernel refuses it, so no file can be lost and there is no check-then-act window), live index UUIDs are excluded via list_indices(), and anything else must outlive UNVERIFIED_THRESHOLD_DAYS = 7 — lance's own bound for deciding an unreferenced index UUID is dead rather than mid-build. The previous 300s was our invention, which is what made it indefensible. Two consequences worth knowing: the age gate reads the dir's mtime, which POSIX bumps when lance's cleanup empties it, so the effective reclaim horizon is up to ~14 days (file wait + age gate) and the ceiling-load steady state is ~1.8M dirs / ~7GB; and a sweep that blows its 60s deadline is swallowed inside prune() — the cleanup commit already succeeded, so escaping would bill a prune "failure" (feeding the fallback-rebuild threshold) and stall the prune-staleness clock for a cleanup stall that did not happen.
• The optimize runner's wait on an in-flight rebuild is bounded too. The two maintenance jobs park on each other — whichever arrives second waits — so an unbounded wait on this side is the same hazard as the one already fixed on the rebuild side, just seen from the other end: the kind's task slot stays occupied, _schedule_optimize keeps short-circuiting on it, and that table quietly stops being pruned. It was left open on the argument that rebuild_indexes carries its own 300s deadline, which covers its critical section but not the task's dispatch and teardown, so the transitive bound was never real. Now bounded at 180s, logging cascade_lancedb_optimize_skipped_rebuild_unfinished and skipping the beat rather than compacting under a live rebuild — the two commit on the same manifest, which is what the wait exists to prevent.

• A rebuild that loses a commit race is retried instead of waiting out the full 12h cadence. Lance labels the conflict Retryable and it is: a concurrent writer in another process won the manifest, nothing is wrong with the table. Retries are scheduled on the kind (10min / 30min / 3h) rather than slept through, so the other kinds in the sweep are not parked behind the backoff. A soak run at a 600s cadence hit 3 conflicts in 119 attempts, all while a concurrent CLI storm was running.

• The index-rebuild sweep can no longer park forever waiting on the optimize runner. That wait had no deadline, and the runner's loop condition is "keep going while there is unindexed data" — which under sustained writes is never, since the drain loop re-raises the flag every second against a 10s cooldown. Now bounded at 180s, logging cascade_lancedb_rebuild_skipped_optimize_unfinished and skipping the sweep rather than dropping indices under a live optimize. This makes the stall visible, not absent: under sustained ingest every sweep still times out, so active index-UUID / FTS part_N growth stays unbounded there. The functional fix requires the optimize runner to yield when a rebuild is pending, which changes the optimize/rebuild mutual-exclusion contract and needs its own validation — the rebuild cadence is 12h, longer than any soak run so far, so the periodic sweep has never been exercised under load.

• The memory-root lock wait is bounded and visible. Acquisition polls with LOCK_NB instead of blocking inside a worker thread: a blocking flock could not be bounded or cancelled — cancelling the awaiting coroutine left the thread to acquire the lock later with nobody to release it. The wait itself is by design (the second process is supposed to wait, then find the migration already done), but it now logs memory_root_lock_waiting and gives up after timeout_seconds (default 30min) instead of leaving a server startup looking like a hang whose last message is lifespan_provider_startup name=lancedb. The default sits an order of magnitude above the worst legitimate hold (a large migration is minutes) on purpose: the wait is already visible from the first poll, and against the one case the bound exists for — a holder that is alive but wedged — giving up at 5 minutes buys nothing over 30, while a bound near the legitimate hold turns a slow migration into startup crashes for every waiting process.

Added

• OfflineEngine.trigger_manual now returns tuple[BaseEvent, list[tuple[StrategyMeta, str]]] instead of None, enabling the dispatched/runs fields below.
• TriggerResponse gains dispatched: int and runs: list[RunSummary].
• OMEConfig gains retry_backoff_base_seconds, retry_backoff_cap_seconds, and retry_jitter_seconds for the retry-loop sleep.
• AgentSkillReader.list_by_cluster() enumerates the cluster's SKILL.md files from markdown (strong-consistency existence check).
• [cascade] settings section — the four maintenance cadences (optimize_heartbeat_seconds, optimize_prune_interval_seconds, optimize_prune_retention_seconds, optimize_rebuild_interval_seconds) are now configurable. They were already constructor arguments on CascadeWorker, but CascadeConfig did not carry them and no production path passed one, so the defaults were unreachable — which is why the 12h rebuild sweep could not be exercised by any soak run shorter than half a day. The deadlines that bound a hung call are deliberately not exposed: they are hang-catchers sized from measured durations, where too low manufactures failures on a healthy table and too high leaves a wedged one invisible for longer. Note optimize_prune_retention_seconds has a second effect worth reading before tuning — it also decides how long index files keep a manifest naming them, and below LanceDB's 7-day unverified window they then wait out the full 7 days.

Changed

• extract_foresight now ships disabled (enabled=False). Not because it is broken — the crash below is fixed — but because it is one LLM call per sender per memcell whose output nothing in EverOS reads today: no search route surfaces foresights and no prompt slot consumes them. Until something does, running it by default spends tokens on write-only data. Re-enable per install in ome.toml (hot-reloaded, no restart):

toml [strategies.extract_foresight] enabled = true

Editing default_ome.toml alone would not have reached existing installs — everos init does not overwrite an existing ~/.everos/ome.toml — so the code default is what changed.

• extract_foresight no longer crashes on a memcell containing tool calls. The sender scan read m.role off every item, but only ChatMessage carries it (ToolCallRequest has sender_id without it, ToolCallResult has neither), so the first tool call raised AttributeError — before any sender was resolved. The strategy was correct on plain user chat and dead-lettered every time on agent trajectories. everalgo explicitly contracts for the mixed case (user_memory/_render.chat_messages: the caller need not pre-filter), and every other user-memory extractor gets that for free by delegating; this was the one place the filter was hand-rolled. The scan now tests isinstance(m, ChatMessage), so a pure agent trajectory yields no senders and returns without an LLM call. Matters even with the strategy off by default: it is what makes the opt-in above actually usable.
• SkillClusterUpdated carries the case's 1024-dim embedding, growing the OME run_record table. The event payload is persisted verbatim in run_record.event_payload (and in the APScheduler jobstore while a job is queued), so a skill_cluster_updated record goes from roughly 0.8 KB to 14 KB. At the default max_records_per_strategy = 1000 ring buffer that is ~14 MB for this one strategy instead of ~0.8 MB. Operators sizing ~/.everos/.index/sqlite/ome.db should expect this. The vector is only read when a cluster holds more skills than MAX_SKILLS_IN_PROMPT, so it usually rides along unused; trimming it from the persisted copy is not a local change, because crash recovery replays event_payload to rebuild the event and a trimmed payload would silently take the recovered run down a different branch than the original. Tracked as a follow-up.
• cascade_lancedb_optimize_conflict now records pruned — which maintenance beat lost the commit race. Lance labels both beats' commit the same way (This Rewrite transaction was preempted by concurrent transaction …), so the message alone cannot separate a free loss from a costly one: a lost light beat retries ~10s later, while a lost heavy beat means that table skipped a whole prune cadence and its superseded files stay on disk. Attributing index-dir growth previously meant back-inferring which beats were heavy from the 300s cadence. Log level (debug) and the benign-conflict semantics are unchanged — this adds one field.

Upgrade

:::bash
pip install --upgrade everos   # or: uv sync

Two behaviour changes to know about before upgrading.

extract_foresight now ships disabled — a deployment relying on foresight entries must set enabled = true for it in ome.toml.

SkillClusterUpdated now carries the case's 1024-dim embedding, so a skill_cluster_updated row in the OME run_record table grows from ~0.8 KB to ~14 KB — about 14 MB for that strategy's default 1000-record ring buffer, against ~0.8 MB before. Anyone sizing ~/.everos/.index/sqlite/ome.db should account for it.

Nothing else needs action: the new [cascade] section is optional and a config written by an earlier version falls back to the same defaults (verified on a clean install).

One deployment note. When a supervised background loop crashes repeatedly and exhausts its restart budget, the worker now sends itself SIGTERM rather than serving on with a dead projection pipeline. That assumes something restarts the process — systemd Restart=always, Docker restart: unless-stopped, a k8s Deployment. Without one the process simply stops, which is still preferable to a server answering searches from a silently frozen index, but it is worth knowing before the first time it happens.

Full changelog: v1.2.2...v1.2.3