REST API

VarMQ ships an HTTP handler that lets you look at your workers and control them from outside the process. It serves JSON for a small set of routes, and it works on any worker that has a name.

Routes at a glance

The paths follow the prefix you give to Handler, for example /varmq.

Mounting it

Handler takes a path prefix and returns an http.Handler. It strips the prefix from each request before it looks for a route. Mount it on a mux with the prefix in the pattern, with a trailing slash:

mux := http.NewServeMux()
mux.Handle("/varmq/", varmq.Handler("/varmq"))
http.ListenAndServe("127.0.0.1:8080", mux)

You can also pass the handler to ListenAndServe itself, as examples/api-control does with varmq.Handler("/api"). Then the routes start at /api.

NewServerMux returns the same routes as an *http.ServeMux with no prefix, for mounting them yourself. Handler is NewServerMux behind that prefix, with one addition. It sets the instance header on every response. This includes the 404 and 405 replies that the mux generates for itself. With NewServerMux alone, those two replies do not carry the header.

The program below starts two named workers and an unnamed one, adds jobs to them, and serves the API under /varmq.

12 collapsed lines
package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/goptics/varmq"
)

func main() {
	// A string argument names the worker.
	emails := varmq.NewWorker(
		func(j varmq.Job[string]) {
			time.Sleep(2 * time.Second)
		},
		"email-sender",
		2,
	)
	emailQueue := emails.BindQueue()

	// WithName does the same.
	images := varmq.NewWorker(
		func(j varmq.Job[int]) {
			time.Sleep(500 * time.Millisecond)
		},
		varmq.WithName("image-processor"),
		varmq.WithConcurrency(4),
	)
	imageQueue := images.BindQueue()

	// This worker has no name, so the API does not list it.
	varmq.NewWorker(func(j varmq.Job[int]) {}).BindQueue()

	go func() {
		for i := range 20 {
			emailQueue.Add(fmt.Sprintf("mail-%d", i))
			imageQueue.Add(i)
		}
	}()

	mux := http.NewServeMux()
	mux.Handle("/varmq/", varmq.Handler("/varmq"))

	// The API has no authentication, so listen on loopback only.
	log.Println("listening on 127.0.0.1:8080")
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", mux))
}

Run locally

1. Make a module, save the program as main.go and add VarMQ:

   mkdir rest-demo && cd rest-demo
   go mod init rest-demo
   go get github.com/goptics/varmq

   Copy

2. Run it with go run ..

3. In another terminal, ask for the workers:

   curl localhost:8080/varmq/health
   curl localhost:8080/varmq/workers
   curl localhost:8080/varmq/workers/email-sender

   Copy

   The first prints {"status":"ok"}. The second prints a JSON array with two workers, email-sender and image-processor, and not the unnamed one. The third prints one worker with its counters, for example:

   {"name":"email-sender","status":"Running","concurrency":2,"num_processing":2,"num_pending":14,"num_idle_workers":0,"metrics":{"submitted":20,"completed":4,"successful":4,"failed":0}}

   Copy

   The numbers depend on when you ask.

4. Control a worker:

   curl -i -X PATCH localhost:8080/varmq/workers/email-sender/actions/pause
   curl localhost:8080/varmq/workers/email-sender
   curl -X PATCH localhost:8080/varmq/workers/email-sender/actions/resume
   curl -i -X PATCH localhost:8080/varmq/workers/email-sender/config/concurrency/4

   Copy

   The first prints a 200 OK with no body. While a job is running, the next call shows the worker as Pausing, and later Paused, as Pause does. Run resume once it reads Paused: on a worker that is still Pausing, it is refused (see the status codes). The last call changes the limit to 4.

The repository’s examples/api-control runs four named workers with a steady supply of jobs. vartop is a separate terminal dashboard for VarMQ workers that reads this API, and the example’s last comment suggests pointing it at http://localhost:8080/api.

Which workers it serves

The API serves the workers in WorkerRegistry, a package-level sync.Map from name to worker.

• Only a named worker is registered. WithName and a plain string argument to NewWorker both give a worker a name. A worker with no name is not in the registry and cannot be reached.
• A worker is registered when it starts. A worker created with WithAutoRun(false) is not in the registry until you call Start.
• A name holds one worker. When a second worker with the same name starts, it replaces the first in the registry. The API then reaches the second.
• A stopped worker stays for a while. After it reaches Stopped, the WithStoppedRetention time starts. The default is one minute. When that time has passed, the worker is removed from the registry, unless it was started again. Lifecycle has the details.
• The order of /workers is not defined. It follows the order the map yields entries in.

You can read the registry yourself. varmq.WorkerRegistry.Load("email-sender") returns the worker as an any. A type assertion to varmq.Worker gives you its methods. You can then build your own routes on it.

Routes

Routes at a glance has the table of routes.

The {action} is one of pause, resume, restart, start and stop. Each calls the worker method of the same name, which lifecycle describes. Two behaviours follow from that:

• pause and stop return immediately. A worker that is running jobs reads Pausing or Stopping until they end. Read the worker again to see when it has changed.
• restart waits. Restart stops the worker, waits until it has stopped, and starts it. The request is answered after that, so it can take as long as the running jobs do.

The concurrency route calls TunePool with the number. A number below 1 becomes the CPU count. Concurrency covers what a change does to jobs already running.

What a worker looks like

/workers returns a JSON array of these, and /workers/{name} returns one with a metrics object added:

Waiting covers what NumPending and NumProcessing count, and Metrics covers the four counters on in-memory queues. For a worker on a persistent or distributed queue, see the overview.

Responses

A response from one of these routes sets Content-Type: application/json, except a successful action, which has no body. An error is a JSON object with one field, error.

The last two rows follow the error the worker returns. start on a worker that is running, and a concurrency change to the number it already has, are 409. A worker that is Pausing, Paused, Stopping or Stopped answers some calls with ErrWorkerPausing, ErrWorkerPaused, ErrWorkerStopping or ErrWorkerStopped, and those are 500. So resume on a worker that is Pausing is a 500 with worker is pausing, and so is pause on a stopped one. Errors lists which call returns which error.

Two replies do not come from the routes. A path the mux does not know gets a plain-text 404 page not found. A known path with the wrong method gets a plain-text 405 Method Not Allowed with an Allow header.

Which instance answered

InstanceHeader is the name of a response header, X-Varmq-Instance. Its value is InstanceID. When several instances sit behind one URL, such as a load balancer, the header tells you which process answered.

InstanceID is a package variable. It starts as the host name, which on Kubernetes is the pod name, or as an empty string if the host name is not available. Set it before you start serving when several instances share a host:

varmq.InstanceID = "worker-a"

Every response from the routes carries the header, whether you use Handler or NewServerMux. As described above, only Handler also adds it to the mux’s own 404 and 405 replies.