Media download and object storage #9

New issue

Closed

opened 2026-06-13 14:57:40 +00:00 by foresle · 0 comments

foresle commented

2026-06-13 14:57:40 +00:00

Owner

Download attachments and store them in object storage. Part of ingestion, but a distinct module.

Scope:

For each attached file, download only when its size is at or below a single configurable threshold, applied uniformly to every type (photo, video, document, animation). Skip larger files. Proposed default threshold is 50 MB.
Identify each file by a hash of its contents and store it under a sharded path derived from the hexadecimal hash: /ab/cd/ef/<full-hex-hash>.<ext>, where the first three two-character segments are the leading characters of the hash and the file name is the full hash with the original extension.
Upload to a public-read bucket in the existing Garage instance, reached over a configurable S3-compatible endpoint.
Store only the object reference and metadata in SQLite, never the file contents. A configured public base URL lets feed items link media directly.

Download attachments and store them in object storage. Part of ingestion, but a distinct module. Scope: - For each attached file, download only when its size is at or below a single configurable threshold, applied uniformly to every type (photo, video, document, animation). Skip larger files. Proposed default threshold is 50 MB. - Identify each file by a hash of its contents and store it under a sharded path derived from the hexadecimal hash: `/ab/cd/ef/<full-hex-hash>.<ext>`, where the first three two-character segments are the leading characters of the hash and the file name is the full hash with the original extension. - Upload to a public-read bucket in the existing Garage instance, reached over a configurable S3-compatible endpoint. - Store only the object reference and metadata in SQLite, never the file contents. A configured public base URL lets feed items link media directly.