cascadeq

Fast disk-backed FIFO queue Link to heading

buffers in memory and overflows to disk transparently

For more information see cascadeq at github

cascadeq (cascade-queue) is a persistent, disk-backed FIFO queue for []byte items. It runs entirely in memory under normal load and spills to disk only when the configured memory limits are reached, making it fast when the queue is shallow and able to grow to any depth, limited only by disk space. Data in the queue is persisted to disk when the queue is closed and is available for reading when the queue is opened again.

Why I created this project Link to heading

I needed a high-performance buffer that could outgrow the host’s main memory, using secondary storage as needed, and retain almost all of the performance of a purely in-memory queue. Other disk-backed queues focused on ensuring the durability of writes, making sure each enqueued item was written to disk. My focus was on fast performance without limits of available main memory, with durability second to that.