JOB 03Reference

Configuration

VarMQ is configured with functions you pass when you create something: a worker, a job, a queue or a topic. This page lists all of them in one table, then explains how they combine. Each row links to the guide that covers the option in full.

Tick options in the scene below and it builds the call. A result list says what the worker gets.

Option explorer · simulation

An option explorer. Tick an option and the NewWorker call below it grows to include it. The result list says what the worker gets: its concurrency, whether it starts, how many idle goroutines it keeps, and what the error handler, the context and the other options do. An option that has no effect, such as a retention time on a worker with no name, is marked.

Worker function and options
  • s
  • min
What the worker gets
worker := varmq.NewWorker(process)
// switch an option on: the call builds itself here

Defaults come from defaultConfig in config.go. An option that has no effect, such as a retention time on a worker with no name, is marked.

Where each option goes

KindPassed to
Worker options, which return a ConfigFuncNewWorker, NewErrWorker or NewResultWorker, after the function
WithJobId, which returns a JobConfigFuncAdd, after the data (and after the priority, on a priority queue)
Queue options, which return a QueueConfigFunc: WithQueueCapacity and WithQueuePriorityBindQueue, BindPriorityQueue and the other binding methods, such as WithQueue
WithTopicCapacity, which returns a TopicConfigFuncNewTopic
WithTopicPriority, which also returns a TopicConfigFuncWithTopic, when you subscribe a worker
WithEndedJobs0: none kept
WithWorkflowIDgenerated
WithFailFastoff
WithUndoOnCanceltrue
WithResultLimit64 KiB
WithWorkflowRetention7 days

Every option

A Default* function sets the default of the matching worker option for workers created after the call. Its default is the same as the option’s. How the Default* functions work is below the table.

FunctionDefaultWhat it doesGuide
WithConcurrency1The most jobs the worker runs at once. A value below 1 means one per CPU.Concurrency
DefaultConcurrency1Sets the default concurrency. Below 1 means one per CPU.Concurrency
WithAutoRuntrueWhether the worker starts when it is created. With false it stays Initiated until you call Start.Lifecycle
WithNamenoneNames the worker. A named worker is added to WorkerRegistry when it starts, and the REST handler finds it there. A worker with no name is not registered.Worker types, REST API
WithStrategyPriorityWhich bound queue or topic gives the worker its next job: Priority, RoundRobin, MaxLen or MinLen.Strategies
DefaultStrategyPrioritySets the default strategy.Strategies
WithMinIdleWorkerRationot set: one idle goroutineThe share of the concurrency limit to keep idle, as a percentage from 1 to 100. A value outside the range is clamped: 0 becomes 1 and anything above 100 becomes 100.Concurrency
DefaultMinIdleWorkerRationot set: one idle goroutineSets the default idle ratio.Concurrency
WithIdleWorkerExpiryDurationnot set: no timerOnce per duration, removes the goroutines that have been idle longer than the duration, down to the idle minimum. A negative duration panics when the worker starts, which is at NewWorker unless auto-run is off.Concurrency
DefaultIdleWorkerExpiryDurationnot set: no timerSets the default expiry duration.Concurrency
WithJobIdGeneratornone: jobs have no IDA func() string. Add calls it, on the caller’s goroutine, for every job; its value is used when the job has no ID of its own. nil is ignored.Job IDs
DefaultJobIdGeneratornone: jobs have no IDSets the default generator. nil is ignored.Job IDs
WithErrHandlerdoes nothingA function that receives the Worker and the error, which the worker calls for every error it reports. nil is ignored.Errors
DefaultErrHandlerdoes nothingSets the default error handler. nil is ignored.Errors
WithContextnone: acts as context.Background()The worker shuts down when this context is done.Context
DefaultCtxnone: acts as context.Background()Sets the default context.Context
WithStoppedRetention1 minuteHow long a stopped, named worker stays in WorkerRegistry before it is removed. 0 keeps it there for good.Lifecycle
DefaultStoppedRetention1 minuteSets the default retention.Lifecycle
WithJobIdnone: the job has no ID, or the generator’sGives one job an ID. An empty string is ignored.Job IDs
WithQueueCapacityunlimitedThe most jobs the queue holds. Add returns false when it is full. A value below 1 means no limit.Queues
WithQueuePrioritymath.MaxInt, the largest rank, so it goes lastThe rank of a queue among the queues bound to one worker. A lower number goes first under the Priority strategy.Strategies
WithTopicCapacityunlimitedThe most unread payloads the slowest subscription may hold. Publish returns false once it does. A value below 1 means no limit.Topics
WithTopicPrioritymath.MaxInt, the largest rank, so it goes lastThe rank of a topic subscription among a worker’s sources, like WithQueuePriority.Topics

That is every exported With and Default function in the package. The methods that bind a source to a worker, such as WithQueue and WithTopic, are not options and are not in the table. A few details it leaves out:

  • Not setting the idle ratio is not the same as setting it to 1. Without it, the worker keeps one idle goroutine. WithMinIdleWorkerRatio(1) keeps 1% of the limit, rounded down, and never fewer than one, so with a limit of 200 it keeps two.
  • WithName and a string argument do the same thing. See below.
  • A queue option reaches only a queue that can use it. WithQueueCapacity takes effect only on a queue that implements CapacitySetter. The built-in queues implement it. See queues.

Options and positional arguments

NewWorker, NewErrWorker and NewResultWorker take config ...any after the worker function. They read each argument by its type:

ArgumentWhat it does
a ConfigFunc, such as the value WithAutoRun(false) returnsapplies that option
an intsets the concurrency, like WithConcurrency. A value below 1 means one per CPU.
a stringnames the worker, like WithName
any other typeis ignored, with no error

Only a value of type int sets the concurrency. A value of another integer type, such as int64 or uint, matches the last row and is ignored.

// 4 is the concurrency and "mailer" is the name.
worker := varmq.NewErrWorker(send, 4, "mailer", varmq.WithAutoRun(false))

The arguments are applied from left to right, over the defaults, so the last one that sets a value wins. varmq.NewWorker(f, 2, varmq.WithConcurrency(8)) runs 8 at once. A string and WithName set the same name, so with both, the later one decides. Worker types lists the same arguments next to the three constructors.

A value below 1 for the concurrency means one per CPU wherever it is given: as an int, in WithConcurrency, in DefaultConcurrency and in TunePool. The number is the logical CPU count of the machine, so it differs between machines.

The Default* functions

Each Default* function sets the default of one worker option. A worker copies the defaults when you create it. So a call changes only the workers created after it, not the ones that exist. An option or a positional argument for a worker still overrides the default.

func main() {
	// Defaults for the workers created below.
	varmq.DefaultConcurrency(8)
	varmq.DefaultErrHandler(func(w varmq.Worker, err error) {
		log.Printf("%s: %v", w.Name(), err)
	})

	emails := varmq.NewErrWorker(send, "emails")                                  // 8 at once, the handler above
	reports := varmq.NewResultWorker(build, 2, "reports", varmq.WithAutoRun(false)) // 2 at once, the handler above
}
  • They write a package-level value with no locking. Call them once at start-up, before any goroutine creates a worker. Calling one while another goroutine creates a worker is a data race.
  • DefaultJobIdGenerator(nil) and DefaultErrHandler(nil) do nothing, like their With twins. The other Default* functions have no such check. DefaultCtx(nil) goes back to no context.
  • Some options have no Default* function: WithAutoRun, WithName, and the job, queue and topic options.
Edit page

Last updated: