What is VarMQ

VarMQ is a typed worker pool and job queue for Go. You give a worker a function and a concurrency limit, bind one or more queues to it, and add jobs to those queues. The worker runs each job through your function, never more at the same time than the limit allows.

worker := varmq.NewWorker(func(j varmq.Job[int]) {
	fmt.Printf("Processing %d\n", j.Data())
}, 2) // concurrency 2
defer worker.WaitUntilIdle()

queue := worker.BindQueue()
queue.Add(1)
queue.Add(2)

When to use it

Use VarMQ when a program has work to run in the background. It also lets you control how much of that work runs at the same time:

• Bounded concurrency with typed jobs. A Job[T] carries your data type. A job can also return an error (NewErrWorker) or a result and an error (NewResultWorker).
• Several sources, one pool. One worker can take jobs from plain queues, priority queues and topics at the same time.
• Control at runtime. Pause, resume, restart or resize a running worker.
• Jobs that outlive the process. Plug in a storage adapter, such as redisq or sqliteq. The adapter keeps jobs across restarts or shares a queue between instances.
• Fan-out. Publish to a topic once and every subscribed worker gets its own job. Topics are in-memory only.
• Inspection over HTTP. A Handler lists workers and pauses, resumes or resizes them.

If all you need is to run a fixed set of functions and wait for them, errgroup or a sync.WaitGroup is enough.

The model: a worker is a pool, a queue is a source

VarMQ splits the work between two things.

A worker is the pool. It owns your function and the concurrency limit. The limit is an upper bound, not a fixed number of goroutines. The worker starts goroutines while jobs are waiting. It lets them exit once the queue is empty.

A queue is a source of jobs. Each queue is bound to one worker. Add puts a job in the queue and returns immediately; the worker’s event loop then hands waiting jobs to free goroutines.

How it works shows this in a scene you can step through.

Next: install VarMQ, then run the quick start in your browser.