Channel ingestion service #8

New issue

Closed

opened 2026-06-13 14:56:52 +00:00 by foresle · 1 comment

foresle commented

2026-06-13 14:56:52 +00:00

Owner

The run ingest service that captures posts.

Scope:

Keep a live connection to Telegram and receive new posts as they are published, rather than polling.
Peer filtering: store only posts from broadcast channels. Ignore private messages, basic groups, and supergroups.
Dynamic channel discovery: when a post arrives from a channel that is not yet known, record it automatically, with title and username taken from the post.
Store each post in the raw Telegram format. Editing a captured post updates the stored text. Deleting a post is ignored, so the archive is kept.
The same post is never stored twice.

Notes:

The first version captures only the posts that arrive while the ingestion service is connected. Backfill of missed posts is out of scope.
This service is the only writer to the database.

The `run ingest` service that captures posts. Scope: - Keep a live connection to Telegram and receive new posts as they are published, rather than polling. - Peer filtering: store only posts from broadcast channels. Ignore private messages, basic groups, and supergroups. - Dynamic channel discovery: when a post arrives from a channel that is not yet known, record it automatically, with title and username taken from the post. - Store each post in the raw Telegram format. Editing a captured post updates the stored text. Deleting a post is ignored, so the archive is kept. - The same post is never stored twice. Notes: - The first version captures only the posts that arrive while the ingestion service is connected. Backfill of missed posts is out of scope. - This service is the only writer to the database.

foresle added this to the 0.1.0 milestone

2026-06-13 14:56:52 +00:00

foresle added the

enhancement

label

2026-06-13 14:56:52 +00:00

foresle self-assigned this

2026-06-13 14:56:52 +00:00

foresle added this to the Developing project

2026-06-13 14:56:52 +00:00

foresle added reference dev

2026-06-22 13:27:57 +00:00

foresle started working

2026-06-24 10:00:04 +00:00

foresle commented

2026-06-24 10:02:23 +00:00

Author

Owner

The raw Telegram message bytes will not be stored. The raw payload is encoded against a specific TL schema layer that changes between Telegram API and grammers versions, so it is not a stable format to keep for future reprocessing. The extracted fields (text, text_entities, media reference) are the canonical storage.

The raw Telegram message bytes will not be stored. The raw payload is encoded against a specific TL schema layer that changes between Telegram API and grammers versions, so it is not a stable format to keep for future reprocessing. The extracted fields (`text`, `text_entities`, media reference) are the canonical storage.

foresle stopped working

2026-06-24 10:17:07 +00:00

17 minutes 3 seconds

foresle started working

2026-06-24 11:17:52 +00:00

foresle stopped working

2026-06-24 11:44:19 +00:00

26 minutes 27 seconds

foresle started working

2026-06-24 13:24:26 +00:00

foresle stopped working

2026-06-24 14:35:55 +00:00