BoxLang 🚀 A New JVM Dynamic Language Learn More...

cbq

A protocol-based queueing system for ColdBox

Requirements

Adobe 2018+ or Lucee 5+ ColdBox 6+

Definitions

Queue Connection

A queue connection defines how to connect to a backend service like Redis, RabbitMQ, or even a database. Any given queue connection can have multiple "queues" which are named stacks of queued jobs or messages to be delivered.

Queue

A named stack of jobs or messages to be delivered. A queue connection must have at least one queue which is usually "default". A queue connection can have as many queues as desired. This is mostly used later when defining queue workers to scale different queues at different priorities.

Queue Provider

A queue provider is how a queue connection connects to a backend service like Redis, RabbitMQ, or a database. It implements the necessary interface to send the jobs and to work the queues. A queue provider can be used multiple times in a single application to define multiple queue connections with different configuration options.

A queue provider must extend the AbstractQueueProvider and implement the required abstract methods:

public any function push( required string queue, required string payload, numeric delay, numeric attempt )
public function function startWorker( required WorkerPool pool )

Additionally, the Queue Provider can use the following hooks to do additional processing or cleanup:

private void function beforeJobRun( required AbstractJob job )
private void function afterJobFailed( required any id, AbstractJob job, WorkerPool pool )

Job

A job is a CFC that follows the IDispatchableJob interface (easily done by extending the AbstractJob component). It defines how to serialize the job using a memento pattern and deserialize the job from the queue. It also holds the data needed to execute the job and a handle method that is called when working the job from the queue. Job components exist in the context of your application so you have access to all the models, services, and helpers you have already written. (Raw string messages can also be dispatched via cbq. The message will need to be handled directly by your queue worker.)

component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        sleep( 1000 ); // do some processing work
        log.info( "sending email - #this.getBody()#" );
    }

}

Providers

cbq provides the following providers out of the box:

SyncProvider
ColdBoxAsyncProvider
DatabaseProvider

Future planned providers include:

Mock (for testing)
RabbitMQ
Redis
Couchbase

(See the ROADMAP for other planned protocols.)

Each of the providers takes different configuration when creating a connection. Refer to the specific provider documentation for details.

Installation and Setup

To install cbq, install it from ForgeBox:

box install cbq

You can configure cbq in your moduleSettings inside config/ColdBox.cfc as follows:

moduleSettings = {
    "cbq" : {
        // The path the custom config file to register connections and worker pools
        "configPath" : "config.cbq",
        // Flag if workers should be registered.  If your application only pushes to the queues, you can set this to false.
        "registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
        // The interval to poll for changes to the worker pool scaling.  Defaults to 0 which turns off the scheduled scaling feature.
        "scaleInterval" : 0
    }
};

Most of the configuration for cbq happens inside the cbq config file, located at config/cbq.cfc by convention.

component {

    function configure() {
        newConnection( "default" )
            .setProvider( "SyncProvider@cbq" );

		newWorkerPool( "default", "default" );
    }

}

In configure you define one or more Connections. You must have at least one Connection called default.

New Connections are created using the newConnection function. It is a builder pattern object. Only a provider must be set. The other setters are optional.

newConnection( connectionName )
    .setProvider( providerMapping )
    .onQueue( name = "default" )
    .markAsDefault( /* true / false */ );    .

You can also define worker pools inside configure to work on queues for a given Connection that you defined previously.

New Worker Pools are created using the newWorkerPool function. It is a builder pattern object. All of the setters are optional.

newWorkerPool( name, connectionName )
    .quantity( numberOfWorkers )
    .onQueue( name = "default" )
    .backoff( backoffTimeInSeconds )
    .timeout( timeoutTimeInSeconds )
    .maxAttempts( maxNumberOfAttempts );

The config file follows ColdBox's environment overrides by calling a method matching the environment name if it is found. Inside that method you can use the withConnection and withWorkerPool methods to change the properties of connections defined in configure and worker pools defined in work:

component {

	function configure() {
		newConnection( "default" )
			.setProvider( "DBProvider@cbq" );

		newWorkerPool( "default", "default" )
			.setTimeout( 5 )
			.setMaxAttempts( 5 )
			.setQuantity( 3 );
	}

	function development() {
		withConnection( "default" )
			.setProvider( "SyncProvider@cbq" );

		withWorkerPool( "default" )
			.setMaxAttempts( 1 )
			.setQuantity( 1 );
	}

}

Usage

Job Components

Jobs are CFCs that extend cbq.models.Jobs.AbstractJob. You need to define a handle method that is ran when the Job is processed.

// GreetingJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {

    function handle() {
        // this is ran when the job is processed
        log.debug( "Hello world!" );
    }

}

To dispatch a job to a queue to be worked, call the dispatch method on a Job instance. Dispatching a job serializes it and sends it to the configured connection. It will later be picked up by a worker and processed.

getInstance( "GreetingJob" ).dispatch();

A Job is sent to the queue with any of its properties serialized. You can set the properties of a job by calling the setProperties method and passing a struct of properties.

getInstance( "GreetingJob" )
    .setProperties( { "greeting": "Hello" } )
    .dispatch();

Additional Job-level properties can be set before dispatching to override Worker Pool defaults on a per-Job basis.

getInstance( "GreetingJob" )
    .setProperties( { "greeting": "Hello" } )
    .setDelay( 10 ) // delay processing this job for 10 seconds
    .dispatch();

CBQ Model

A cbq model exists to make certain job- and dispatch-related actions easier to perform. You can access it by injecting cbq@cbq or simply @cbq. Here are the methods available to you:

`job`

Creates a Job instance.

Name	Type	Required	Default	Description
job	string or Job instance	true		A job instance or mapping string to a Job instance. Additionally, any string may be provided here, even if it doesn't exist as a CFC. If so, cbq will create a `NonExecutableJob` with the given mapping. This can only be used if the instance dispatching the jobs will never work the jobs.
properties	struct	false	`{}`	A struct of properties for the new Job.
chain	Job[]	false	`[]`	An array of Job instances to chain after this one.
queue	string	false	`null`	The queue to run this Job on. Overrides the Job queue and the default queue, if provided.
backoff	numeric	false	`null`	The backoff amount in seconds between Job attempts. Overrides the Job backoff and the default backoff, if provided.
timeout	numeric	false	`null`	The timeout amount in seconds before a Job run is considered timed out. Overrides the Job timeout and the default timeout, if provided.
maxAttempts	numeric	false	`null`	The maxAttempts amount before a Job run is considered failed. Overrides the Job maxAttempts and the default maxAttempts, if provided.

`dispatch`

Creates a Job instance and immediately dispatches it.

Name	Type	Required	Default	Description
job	string or Job instance OR array of Job instances	true		A job instance or mapping string to a Job instance. Additionally, any string may be provided here, even if it doesn't exist as a CFC. If so, cbq will create a `NonExecutableJob` with the given mapping. This can only be used if the instance dispatching the jobs will never work the jobs. If an array of Job instances are passed, this forwards it on to `chain` and dispatches the chain.
properties	struct	false	`{}`	A struct of properties for the new job.
chain	Job[]	false	`[]`	An array of Job instances to chain after this one.
queue	string	false	`null`	The queue to run this Job on. Overrides the Job queue and the default queue, if provided.
backoff	numeric	false	`null`	The backoff in seconds amount between Job attempts. Overrides the Job backoff and the default backoff, if provided.
timeout	numeric	false	`null`	The timeout amount in seconds before a Job run is considered timed out. Overrides the Job timeout and the default timeout, if provided.
maxAttempts	numeric	false	`null`	The maxAttempts amount before a Job run is considered failed. Overrides the Job maxAttempts and the default maxAttempts, if provided.

`chain`

Creates a Job Chain and returns the first Job in the chain. To dispatch the chain, you must call dispatch on the returned Job.

Name	Type	Required	Default	Description
chain	Job[]	false	`[]`	An array of Job instances to chain after this one.

v4.0.0

31 Jan 2025 — 17:41: 44 UTC

BREAKING

Batch: ACF currently errors when compiling PendingBatch.cfc (78c0c39)
FailedJobs: Use a unix timestamp as the failed job log failedDate column type (0f36ad8)
DBProvider: Better locking to avoid duplicate runs of the same job (eed4c61)
Config: Worker Pools can only define a single queue to work. (#15) (a417b09)
Config: Remove work method in favor of configure (eb94b08)

chore

DBProvider: Remove lockForUpdate flag and add debug logging (413760f)

feat

WorkerPool: Make shutdown timeout configurable (575651c)
DBProvider: Add back ability to work on multiple queues (364b9ca)
Interceptors: Add ability to restrict interceptor execution with jobPattern (552e8ae)
Job: Add support for before and after lifecycle methods (8cf8390)
Config: Add environment detection for config file (7004069)

fix

Batch: Don't override Batch job queues unless one is specifically requested (6926067)
SyncProvider: Fix stack overflow when releasing a job too many times (5997087)
SyncProvider: Add chained jobs to the Sync job memento. (b1af5aa)
SyncProvider: Un-nest chains to prevent stack overflows (b89c81a)
FailedJobs: Fix variable name (b6e5f40)
FailedJobs: Inject logbox into the interceptor (f759d4f)
FailedJobs: Use CF_SQL_VARCHAR as the SQL type for originalId (041f34b)
FailedJobs: Make originalId able to track all provider ids (ecbade0)
FailedJobs: Log errors logging failed jobs (fd4bffa)
DBProvider: Fix releasing job timeouts using the wrong value (276a473)
DBProvider: Allow picking up of jobs that were previously reserved but not released correctly (67c3659)
DBProvider: When claiming a job, the DBProvider should extend the availableDate by the job timeout, not backoff. (bb0b0e2)
WorkerPool: Correctly shutdown worker pools (fcaee85)
DBProvider: Fix for unwrapping an Optional in a log message (9025919)
DBProvider: Disable forceRun until we figure out why it's losing mappings (4418d4c)
SyncProvider: Add pool to releaseJob call (0430bcd)
ColdBoxAsyncProvider: Respect WorkerPools in ColdBoxAsyncProvider (a5011f3)
ColdBoxAsyncProvider: Fix unbound thread CPU usage in ColdBoxAsyncProvider (54ae0bf)
box.json: Upgrade to qb v9 (08b9e2e)
DBProvider: Fix duplicate job runs (5736613)
DBProvider: Missing Parameter to releaseJob (6b079d8)
SyncProvider: Pass the pool to getMaxAttemptsForJob (d5d6742)
Scheduler: Fix onAnyTaskError exception logging (43e56c2)

other

*: ci: Adds Coldbox 7 Tests and Experimental Matrix (#18) (1eb27a3)
*: fix: Replace typo excpetion with exception (56cdd07)
*: fix: Update syntax for Lucee 6.1 and ACF (a4f4502)
*: Fix missing shutdownTimeout variable (9bb553c)
*: feat: Add clean-up tasks for completed or failed jobs, failed job logs, and completed or cancelled batches. (80ee9e9)
*: v3.0.0-beta.1 (139302b)
*: fix: reload module mappings in an attempt to work around ColdBox Async losing them (152e282)
*: fix: Fix moduleSettings missing a queryOptions key for failed jobs (f47402d)
*: v1.0.0 (d4196ed)
*: fix: Adjust moduleSettings to be more internally consistent. (1898ebd)
*: chore: code cleanup (3e4baae)
*: fix: Update ColdBoxAsyncProvider for named worker pools (bb169c3)
*: feat: Add failed jobs table and interceptor (edc7952)
*: fix: Ensure batch jobs are recorded from SyncProvider (c3161f8)
*: feat: Allow setting connections on Batches (a612729)
*: feat: Allow for infinite attempts when setting maxAttempts to 0 (3467e26)
*: tests: Temporarily only test on Lucee 5 (84b200f)
*: tests: ACF-specific fixes (7328566)
*: tests: Fix for ACF and MockBox (d85e869)
*: tests: Install cfconfig in pipelines (e3b58dd)
*: tests: Add CFMigrations dependency (d49d906)
*: tests: Add database for DBProvider tests (0cee608)
*: docs: Add docblocks to PendingBatch (20ecb08)
*: feat: Manually release a job inside the handle method. (451b616)
*: feat: BatchableJob is no longer needed; it all goes through AbstractJob (751e138)
*: fix: Avoid calling getMemento() on null jobs in PendingBatch (01d8591)
*: chore: Update README newWorkerPool function (077772d)
*: fix: incorrect throw that causing raw HTML as the exception message (c8ae745)
*: feat: Provide cbq helper to jobs by default (dd0d5eb)
*: fix: Add missing queryOptions to qb calls (767c4f7)
*: feat: if a job defines an onFailure method, call it when the job fails (6b091f7)
*: fix: Allow for setting different datasources for Batches (4f54b79)
*: feat: Use module defaults for worker pool settings (81b3484)
*: chore: Clean up LogBox logs for better error grouping in StacheBox (ca4f738)
*: feat: Docblocks, listen block for Sync Provider, remove onQueue from QueueConnectionDefinition (750a166)
*: feat: Enable multiple worker pools per connection and queue priority order (56f0456)
*: fix: Fix issues where config properties were not correctly applied (ab78ce6)
*: fix: use cbq.job in case the job is not defined on this server (14a6dfd)
*: fix: Use job queue if present in chained jobs (a78dbbb)
*: fix: Allow for chains longer than 2 (aeb9229)
*: feat: Add connection and queue to mementos (a45ae5a)
*: feat: Add support for batched jobs (4f80090)
*: docs: fix typo (61d09a1)
*: docs: Fixed onQueue argument name in README (4b0d24c)
*: Merge pull request #1 from Daemach/main (cab3958)
*: Typo fix (c59afd8)
*: fix: Temporarily disable scale spec (1e40d8e)
*: 0.1.4 (af7bfe9)
*: fix: Don't do timeout or backoff in SyncProvider (a0a5e21)
*: 0.1.3 (ccd684b)
*: fix: Standardize on seconds for timeout and backoff (39d1107)
*: 0.1.2 (7037c29)
*: v0.1.1 (6cb9053)
*: feat: Add chained jobs and job helpers in cbq model (350047c)
*: v0.1.0 (65ebb72)
*: Initial commit (d601988)

perf

DBProvider: Increase throughput for DBProvider (6fb8fd3)

v3.0.13

20 Sep 2024 — 15:39: 44 UTC

other

*: fix: Replace typo excpetion with exception (56cdd07)

v3.0.12

03 Sep 2024 — 15:53: 21 UTC

other

*: fix: Update syntax for Lucee 6.1 and ACF (a4f4502)

v3.0.11

07 Aug 2024 — 18:40: 35 UTC

other

*: Fix missing shutdownTimeout variable (9bb553c)

v3.0.10

07 Aug 2024 — 16:37: 29 UTC

fix

Batch: Don't override Batch job queues unless one is specifically requested (6926067)

v3.0.9

25 Jun 2024 — 16:46: 56 UTC

fix

SyncProvider: Fix stack overflow when releasing a job too many times (5997087)

v3.0.8

12 Jun 2024 — 17:21: 18 UTC

fix

SyncProvider: Add chained jobs to the Sync job memento. (b1af5aa)

v3.0.7

23 May 2024 — 21:34: 49 UTC

fix

SyncProvider: Un-nest chains to prevent stack overflows (b89c81a)

v3.0.6

23 May 2024 — 17:07: 59 UTC

fix

FailedJobs: Fix variable name (b6e5f40)

v3.0.5

23 May 2024 — 17:03: 25 UTC

fix

FailedJobs: Inject logbox into the interceptor (f759d4f)

v3.0.4

23 May 2024 — 16:58: 00 UTC

fix

FailedJobs: Use CF_SQL_VARCHAR as the SQL type for originalId (041f34b)

v3.0.3

23 May 2024 — 16:40: 10 UTC

fix

FailedJobs: Make originalId able to track all provider ids (ecbade0)

v3.0.2

21 May 2024 — 14:01: 55 UTC

fix

FailedJobs: Log errors logging failed jobs (fd4bffa)

v3.0.1

16 May 2024 — 15:48: 35 UTC

fix

DBProvider: Fix releasing job timeouts using the wrong value (276a473)

v3.0.0

16 May 2024 — 14:58: 33 UTC

BREAKING

FailedJobs: Use a unix timestamp as the failed job log failedDate column type (0f36ad8)
DBProvider: Better locking to avoid duplicate runs of the same job (eed4c61)

chore

DBProvider: Remove lockForUpdate flag and add debug logging (413760f)

feat

WorkerPool: Make shutdown timeout configurable (575651c)

fix

DBProvider: Allow picking up of jobs that were previously reserved but not released correctly (67c3659)
DBProvider: When claiming a job, the DBProvider should extend the availableDate by the job timeout, not backoff. (bb0b0e2)
WorkerPool: Correctly shutdown worker pools (fcaee85)
DBProvider: Fix for unwrapping an Optional in a log message (9025919)

other

*: feat: Add clean-up tasks for completed or failed jobs, failed job logs, and completed or cancelled batches. (80ee9e9)
*: v3.0.0-beta.1 (139302b)

v2.1.0

13 Feb 2024 — 23:42: 12 UTC

feat

DBProvider: Add back ability to work on multiple queues (364b9ca)
Interceptors: Add ability to restrict interceptor execution with jobPattern (552e8ae)
Job: Add support for before and after lifecycle methods (8cf8390)

v2.0.5

09 Nov 2023 — 16:58: 45 UTC

fix

DBProvider: Disable forceRun until we figure out why it's losing mappings (4418d4c)

v2.0.4

06 Nov 2023 — 19:33: 46 UTC

other

*: fix: reload module mappings in an attempt to work around ColdBox Async losing them (152e282)

Eric Peterson
Published
4.0.0 is the latest of 66 release(s)
Published
Published on {{ getFullDate("2025-01-31T17:41:48Z") }}
coldbox-modules/cbq

$ box install cbq

How do I do this?

No collaborators yet.

cbq
Created On: {{ getFullDate("2022-09-06T02:44:07Z") }}
Last Publish: {{ getFullDate("2025-01-31T17:41:48Z") }}
Views: 7,067
Installs: 8,441

Report Abuse

cbq

A protocol-based queueing system for ColdBox

Requirements

Definitions

Queue Connection

Queue

Queue Provider

Job

Providers

Installation and Setup

Usage

Job Components

CBQ Model

job

dispatch

chain

v4.0.0

31 Jan 2025 — 17:41: 44 UTC

BREAKING

chore

feat

fix

other

perf

v3.0.13

20 Sep 2024 — 15:39: 44 UTC

other

v3.0.12

03 Sep 2024 — 15:53: 21 UTC

other

v3.0.11

07 Aug 2024 — 18:40: 35 UTC

other

v3.0.10

07 Aug 2024 — 16:37: 29 UTC

fix

v3.0.9

25 Jun 2024 — 16:46: 56 UTC

fix

v3.0.8

12 Jun 2024 — 17:21: 18 UTC

fix

v3.0.7

23 May 2024 — 21:34: 49 UTC

fix

v3.0.6

23 May 2024 — 17:07: 59 UTC

fix

v3.0.5

23 May 2024 — 17:03: 25 UTC

fix

v3.0.4

23 May 2024 — 16:58: 00 UTC

fix

v3.0.3

23 May 2024 — 16:40: 10 UTC

fix

v3.0.2

21 May 2024 — 14:01: 55 UTC

fix

v3.0.1

16 May 2024 — 15:48: 35 UTC

fix

v3.0.0

16 May 2024 — 14:58: 33 UTC

BREAKING

chore

feat

fix

other

v2.1.0

13 Feb 2024 — 23:42: 12 UTC

feat

v2.0.5

09 Nov 2023 — 16:58: 45 UTC

fix

v2.0.4

06 Nov 2023 — 19:33: 46 UTC

other

Use It

`job`

`dispatch`

`chain`