#477 Search engines get the pages of events with something of their own

[Anonymous]

Originally created by: athanclark

Closes [#467].

Every event of a public timeline used to go in the sitemap and to IndexNow, so the synthetic incident timeline's 2,299 log lines were 24% of timescale.info's sitemap. Each timeline now chooses which of its events' pages search engines get, under Sharing & settings → Search engines:

Setting	Lists
Automatically (default)	Events with at least 150 characters of notes and a title no other event of the timeline shares. None when the timeline's notes are templated.
All of them	Every event, as before
None	Only the timeline's own page

Templated notes (server/event-listing.mjs):

• Sample: up to 200 of a timeline's notes long enough to list, chosen by a hash of their IDs.
• Reduction: each note becomes its first 12 words, without numbers, addresses, paths, code and quoted text.
• Verdict: with at least 20 sampled notes, if half reduce to the same words, the timeline lists none automatically.
• Caching: the verdict is kept per timeline revision, so nothing extra is stored or rebuilt on save.

Checked against live timescale.info pages: 30 of 30 sampled incident notes reduce to the same words. Hand-written timelines top out at 2 of 30 (paleontology-history) and 1 of 30 (rome). So the incident timeline drops out automatically, without setting it to None.

What an unlisted event's page does:

• Kept: it keeps its page, and its timeline's page still links to it.
• Left out: of the sitemap and of IndexNow.
• Robots: its page says noindex, follow, so crawlers still follow its links to related events and its timeline.

Changes:

• Database: oc_timelines.index_events (auto, all or none).
• Setting it: indexEvents in PATCH /api/timelines/{id}/settings, documented in OpenAPI.
• Store: listedEntityKeys replaces entityKeys for the sitemap and IndexNow. listedOf filters a save's changed events for IndexNow, keeping removed ones so search engines hear they're gone. A timeline set to none is also left out by entitiesListed.
• Event pages: eventListed decides their robots tag.
• Settings page: a Search engines section with the three choices.
• Docs: docs/sharing.md#which-events-search-engines-list.

A model to judge this more finely is planned in [#476].

Tests:

• Unit:
• event-listing.test.mjs: note shapes and templated notes;
• entity-pages.test.mjs: noindex, follow for an unlisted event.
• PostgreSQL:
• sharing-postgres: in the sitemap and in each page's robots tag, it checks Automatically (a rich event listed; a thin one and two sharing a title not), a templated timeline, None, All of them, and a refused value. Its HTTP checks pass here; the browser steps run in CI.
• indexnow-postgres: on publishing and saving, only listed events are sent; with None, only the timeline's page.
• The existing sample timelines in both tests have short notes, so they're set to All of them.
• Run locally, passing: indexnow-postgres, labels-postgres, timeline-links-postgres, relationships-postgres, next-postgres, administration-postgres, summaries-postgres, and all unit tests except contents.test.mjs, which fails on main in this checkout as well.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fFY8eruEi1xAkkMBni13B

[Anonymous]

Ticket changed by: athanclark

• status: open --> closed