Durable queues, streams, pub/sub, and a cron scheduler – inside your SQLite file

· databases coding · Source ↗

TLDR

  • Honker adds Postgres-style NOTIFY/LISTEN, durable queues, pub/sub, and a cron scheduler as a SQLite loadable extension with bindings for 7 languages.

Key Takeaways

  • Enqueue atomically with business writes: queue.enqueue(..., tx=tx) inside a transaction means rollback drops the job and the row together.
  • Wake latency is ~0.7 ms p50 via polling PRAGMA data_version every millisecond – a ~3 µs read that signals any cross-process commit.
  • One poller thread per database fans wake signals to all subscribers; listener count scales without adding per-listener queries.
  • Targets apps already on SQLite (Bluesky PDS, Fly LiteFS, Turso) that want to avoid a second datastore like Redis + Celery.
  • Loadable extension model means any language with load_extension() support shares the same on-disk format across Python, Node, Rust, Go, Ruby, Bun, and Elixir.

Hacker News Comment Review

  • The 1 ms busy-poll on PRAGMA data_version drew skepticism: critics note it contradicts the claim of being lighter than kernel file watchers like inotify or kqueue.
  • Author confirmed polling is a known interim design and is actively prototyping inotify/kqueue/mmap shm alternatives after stat(2) proved unreliable.
  • Commenters questioned the core premise: SQLite’s single-writer constraint makes in-process ring buffers with futex/eventfd a simpler and faster IPC path than SQL rows.

Notable Comments

  • @russellthehippo: Author notes work-in-progress on inotify, kqueue, and mmap shm to replace polling once stat(2) direction proved unreliable.
  • @tengbretson: Raises that SQLite’s single-writer constraint makes application-layer queuing a simpler alternative.

Original | Discuss on HN