1 link tagged with all of: postgres + streaming + batching + performance
Click any tag below to further narrow down your results
Links
Postgres LISTEN/NOTIFY suffers a global exclusive lock on NOTIFY calls, which serializes commits and caps throughput around 2.9K writes/sec. By buffering notifications in memory and flushing them in batched transactions—plus a low-frequency polling fallback—the stream writer can leverage group commits and hit 60K writes/sec with 15–100ms latency.
- Postgres's NOTIFY takes a global exclusive lock at commit held until fsync completes, serializing all notifying transactions and capping throughput around 2.9K writes/sec regardless of spare CPU/IO.
- Batching notifications in memory and flushing them via a single background transaction lets individual writes use group commit instead of serializing on the lock.
- This approach pushed throughput to 60,000 inserts/sec on a single Postgres instance while keeping notify-to-read latency at 15–100ms, with CPU (not locking) as the bottleneck.
- Low-frequency polling as a fallback covers any notifications missed by the batching scheme, trading per-notify durability for table-level durability.