Add feed-backed threaded news imports

2 files changed, 155 insertions, 14 deletions

diff --git a/docs/feed-backed-news.md b/docs/feed-backed-news.md
new file mode 100644
index 0000000..8edcf79
--- /dev/null
+++ b/docs/feed-backed-news.md
@@ -0,0 +1,129 @@
+# Feed-backed threaded news
+
+Mobius can import a public RSS or Atom feed into an existing Hotline news
+category. This is useful for project announcements, Sparkle appcasts, and
+GitHub release feeds.
+
+When a user opens a mapped category, Mobius checks its source and adds entries
+it has not seen before as ordinary root articles. Imported articles live in
+`ThreadedNews.yaml`, alongside locally posted articles and replies. The category
+continues to work as ordinary Hotline news: authorized users can post roots,
+reply, and delete any article.
+
+## Configure a category
+
+First, create the target as an ordinary news category with a Hotline client.
+Every component in `CategoryPath`, including the final category, must already
+exist before the server starts.
+
+Then add a mapping to `config.yaml` and restart Mobius:
+
+```yaml
+NewsFeeds:
+  - CategoryPath: ["Software Updates", "Afterglow"]
+    URL: "https://morphing.cloud/afterglow/appcast.xml"
+
+  - CategoryPath: ["Software Updates", "Mobius"]
+    URL: "https://github.com/jhalter/mobius/releases.atom"
+```
+
+`CategoryPath` is the complete path from the root of threaded news to the
+existing category. `URL` must be an absolute public `http` or `https` URL and
+cannot contain embedded credentials. Only one feed may map to a category.
+
+Mobius detects RSS and Atom automatically. Sparkle appcasts are RSS; GitHub's
+`releases.atom` endpoints are Atom. JSON Feed and provider-specific APIs are not
+supported.
+
+## Import behavior
+
+Every article-list request for the mapped category performs a source check.
+Mobius sends the saved `ETag` and `Last-Modified` values when the source
+provides them, allowing an unchanged source to answer with a small `304 Not
+Modified` response. Concurrent requests for the same category share one
+in-flight check.
+
+On a successful response, Mobius:
+
+1. Finds entries not previously observed from that URL.
+2. Converts HTML descriptions to plain text and retains useful source, release
+   notes, and enclosure links.
+3. Imports new entries oldest-first as ordinary root articles.
+4. Saves the articles and import metadata together in one atomic update to
+   `ThreadedNews.yaml`.
+
+An entry needs a stable identity: a GUID or Atom ID, an HTTP(S) article link,
+or an HTTP(S) enclosure URL, in that order. Entries without any of these are
+skipped and logged. Mobius intentionally does not derive identities from titles
+or bodies because an edited entry could then be mistaken for a new article.
+
+Once an identity has been imported, later changes to that entry's title,
+author, date, or body are ignored. If an imported article is deleted locally,
+its identity remains in the seen set, so the feed does not recreate it. Items
+also remain in Hotline when they disappear from the source. This prevents a
+feed's rolling window from rolling articles out of the Hotline category.
+
+The initial import can only include entries returned by the source. Mobius does
+not paginate GitHub or reconstruct older releases that are already absent from
+its Atom feed.
+
+## Failures and limits
+
+A network error, timeout, non-success HTTP response, invalid feed, or failed
+disk write is logged. Mobius still returns the category's current local article
+list, including an empty list before the first successful import. A later
+category load tries the source again. Requests time out after 10 seconds and a
+response body may be at most 2 MiB.
+
+The Hotline protocol limits the complete encoded article-list payload for one
+category to 65,535 bytes. This listing contains article metadata—not article
+bodies—so the practical capacity depends mostly on the number and encoded
+length of titles and authors. Mobius checks the full list after each candidate
+article. It saves the oldest prefix that fits and stops before an addition
+would exceed the limit. It does not save new HTTP validators while unseen
+entries remain, so the next category load fetches and retries them. Mobius does
+not prune local or imported articles automatically; split a large source across
+categories when a category approaches this protocol limit.
+
+Individual imported titles and authors are limited to 255 encoded bytes, and
+article bodies to 65,535 encoded bytes, matching Hotline's field sizes.
+
+## Durable state and backups
+
+Feed history is part of each category in `ThreadedNews.yaml` under a YAML-only
+`FeedState` key. It records the current source URL, HTTP validators, and hashes
+of identities that have already been imported. This metadata is not sent to
+Hotline clients. Back up `ThreadedNews.yaml` as usual; there are no additional
+feed state or cache files.
+
+Changing a category's URL clears its HTTP validators but retains its seen
+history. Identity hashes include the source URL, so the new source can import
+its entries even if it uses the same GUID values. Switching back to a previous
+URL does not duplicate entries that URL imported earlier.
+
+The earlier experimental `FeedNewsState.yaml` and `FeedNewsCache.json` formats
+are not migrated or read. If they exist from a development build, Mobius
+ignores them; their contents do not participate in this feature.
+
+## Text encoding
+
+Feed parsers produce UTF-8 text. Mobius converts each entry once, when it is
+imported, according to the server-wide `Encoding` setting. The default
+`macintosh` setting converts to Mac Roman and replaces unsupported characters;
+`utf8` stores UTF-8 unchanged. Existing imported articles are ordinary raw-byte
+news data, so changing `Encoding` affects only future imports and does not
+rewrite history.
+
+## Troubleshooting
+
+- If startup fails, verify that every mapped path already exists and ends at a
+  news category rather than a bundle. Also check for duplicate paths and
+  invalid or credential-bearing URLs.
+- If new entries do not appear, open the category and inspect the server log.
+  Confirm that the URL is public RSS or Atom and that entries have stable IDs,
+  links, or enclosures.
+- If old releases are missing on the first import, inspect the source feed. Its
+  published window is the initial cutoff; Mobius does not use pagination or a
+  provider API.
+- If a source is unavailable, existing local news remains usable and the next
+  category load retries it.
diff --git a/docs/text-encoding.md b/docs/text-encoding.md
index 71bdada..f299b51 100644
--- a/docs/text-encoding.md
+++ b/docs/text-encoding.md
@@ -2,7 +2,7 @@
 
 ## Background
 
-The Hotline protocol was designed for classic Mac OS, which used an encoding called Mac Roman for text. Modern operating systems use UTF-8 instead. Mobius automatically translates between these two encodings so that classic Hotline clients and modern filesystems can work together.
+The Hotline protocol was designed for classic Mac OS, which used an encoding called Mac Roman for text. Modern operating systems use UTF-8 instead. Mobius translates filesystem names and newly imported feed news so that classic Hotline clients and modern UTF-8 sources can work together. Most other protocol text remains raw bytes.
 
 By default, Mobius assumes all clients use Mac Roman encoding. If your server exclusively serves modern UTF-8 clients, you can disable the Mac Roman conversion. This document explains how encoding works and how to configure it.
 
@@ -17,9 +17,20 @@ When a Hotline client uploads, downloads, browses, creates, or renames files and
 
 This means files on disk always use UTF-8 names, regardless of what encoding the client uses. You can place files with Unicode names in the server's file directory and clients will see them — as long as the characters have Mac Roman equivalents.
 
-### Chat, news, and usernames are NOT translated
+### Server-generated feed news is translated
 
-Text in chat messages, news articles, usernames, and private messages is passed through as raw bytes with no encoding conversion. This means:
+RSS and Atom sources are UTF-8 server data rather than client-authored Hotline
+text. Mobius encodes each imported article's title, author, and body when it is
+first added to `ThreadedNews.yaml`. With the default `macintosh` setting,
+characters that Mac Roman cannot represent are replaced. With `utf8`, feed text
+is stored unchanged.
+
+Imported entries become ordinary raw-byte news articles. Changing `Encoding`
+therefore affects future imports only; it does not rewrite existing articles.
+
+### Chat, user-authored news, and usernames are NOT translated
+
+Text in chat messages, existing news articles, usernames, and private messages is passed through as raw bytes with no encoding conversion. This means:
 
 - If all your users are on classic Mac clients, they'll see each other's text correctly (all Mac Roman).
 - If all your users are on modern UTF-8 clients, they'll also see each other's text correctly.
@@ -29,13 +40,13 @@ There is no way to configure this behavior — the Hotline protocol has no mecha
 
 ## Configuration
 
-The `Encoding` field in `config.yaml` controls how file and folder names are translated:
+The `Encoding` field in `config.yaml` controls how file and folder names and server-generated feed news are translated:
 
 ```yaml
 # Default — translates between Mac Roman and UTF-8 (compatible with classic Hotline clients)
 Encoding: macintosh
 
-# No-op — passes file names through without conversion (for servers with only modern UTF-8 clients)
+# No-op — passes file names and feed news through unchanged (modern UTF-8 clients only)
 Encoding: utf8
 ```
 
@@ -59,12 +70,13 @@ Modern Hotline clients that send UTF-8 for file operations may produce unexpecte
 
 ## Summary
 
-| What                      | Encoding translation? | Notes                                         |
-|---------------------------|----------------------|-----------------------------------------------|
-| File and folder names     | Yes                  | Mac Roman <-> UTF-8 at the filesystem boundary |
-| Chat messages             | No                   | Raw bytes, passed through as-is               |
-| News articles and titles  | No                   | Raw bytes, passed through as-is               |
-| Usernames                 | No                   | Raw bytes, passed through as-is               |
-| Private messages          | No                   | Raw bytes, passed through as-is               |
-| File comments             | No                   | Stored and retrieved as raw bytes             |
-| Login credentials         | No                   | Obfuscated, no charset conversion             |
+| What | Encoding translation? | Notes |
+| --- | --- | --- |
+| File and folder names | Yes | Mac Roman <-> UTF-8 at the filesystem boundary |
+| Newly imported feed articles | Yes | UTF-8 source text -> configured encoding at import time |
+| Chat messages | No | Raw bytes, passed through as-is |
+| Existing and user-authored news articles | No | Raw bytes, passed through as-is |
+| Usernames | No | Raw bytes, passed through as-is |
+| Private messages | No | Raw bytes, passed through as-is |
+| File comments | No | Stored and retrieved as raw bytes |
+| Login credentials | No | Obfuscated, no charset conversion |