BoxLang 🚀 A New JVM Dynamic Language Learn More...
Adobe 2018+ or Lucee 5+ ColdBox 6+
Batch IDs use the Java UUID Generator bundled in
cbq/lib. Add that directory to your application's
Application.cfc Java load paths before creating batches
(adjust the path to your module installation):
this.javaSettings = {
loadPaths : [ expandPath( "/modules/cbq/lib" ) ],
reloadOnChange : false
};
If you already configure this.javaSettings, append this
directory to its existing loadPaths. Restart the
application after changing Java load paths. A
ClassNotFoundException for
com.fasterxml.uuid.Generators means this bundled library
is missing from the application's classpath; changing the Java version
alone does not add it. Use a Java version supported by your CFML engine.
A queue connection defines how to connect to a backend service like Redis, RabbitMQ, or even a database. Any given queue connection can have multiple "queues" which are named stacks of queued jobs or messages to be delivered.
A named stack of jobs or messages to be delivered. A queue connection must have at least one queue which is usually "default". A queue connection can have as many queues as desired. This is mostly used later when defining queue workers to scale different queues at different priorities.
A queue provider is how a queue connection connects to a backend service like Redis, RabbitMQ, or a database. It implements the necessary interface to send the jobs and to work the queues. A queue provider can be used multiple times in a single application to define multiple queue connections with different configuration options.
A queue provider must extend the
AbstractQueueProvider and implement the required abstract methods:
public any function push( required string queue, required
AbstractJob job, numeric delay, numeric attempt )
public function function startWorker( required WorkerPool pool )
Additionally, the Queue Provider can use the following hooks to do additional processing or cleanup:
private void function beforeJobRun( required AbstractJob job )
private void function afterJobFailed( required any id,
AbstractJob job, WorkerPool pool )
A job is a CFC that follows the IDispatchableJob
interface (easily done by extending the AbstractJob
component). It defines how to serialize the job using a memento
pattern and deserialize the job from the queue. It also holds the
data needed to execute the job and a handle method that
is called when working the job from the queue. Job components exist
in the context of your application so you have access to all the
models, services, and helpers you have already written. (Raw string
messages can also be dispatched via cbq. The message will need to be
handled directly by your queue worker.)
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
sleep( 1000 ); // do some processing work
log.info( "sending email - #this.getBody()#" );
}
}
cbq provides the following providers out of the box:
Future planned providers include:
(See the ROADMAP for other planned protocols.)
Each of the providers takes different configuration when creating a connection. Refer to the specific provider documentation for details.
For MySQL, DBProvider@cbq requires MySQL 8.0 or
later. Its reservation query uses FOR UPDATE SKIP
LOCKED, which MySQL 5.7 does not support. See MySQL
locking reads. cbq does not ship a MySQL 5.7 fallback grammar.
Upgrade MySQL before enabling database workers; removing the lock
clause changes concurrent worker behavior.
Use separate queues and explicitly configure each worker pool's queues when workers have different job components. An unknown mapping is logged with the job ID and worker pool. Database workers release that reservation without consuming an attempt or marking the job failed, then continue the polling pass. A worker with the required mapping can claim it. Queue routing prevents incompatible workers from repeatedly claiming the same job. Configure your LogBox appenders to send worker logs to a shared destination when central logging is needed.
Run migration
2000_01_01_000011_track_processed_batch_jobs.cfc before
deploying this version, including on custom batch tables. Batch
results are recorded once per job ID under a row lock; repeated
success or failure callbacks do not consume another pending job or
repeat lifecycle callbacks. The new nullable
processedJobIds column records results going forward.
Existing failed IDs remain protected, but successful job IDs from
before the migration cannot be reconstructed. Drain existing batches
before upgrading if they might receive duplicate callbacks. This
accounting does not make a job's external side effects execute exactly once.
DBProvider@cbq polls once every five seconds by default.
Each poll fetches only as many jobs as the worker pool currently has
available slots. For short jobs, polling can therefore determine
backlog drain time even when transport is fast.
Set pollIntervalMilliseconds in the connection
properties to configure the delay between completed polling passes. It
must be a positive integer; the default is 5000. This
configures each native database watcher on that connection and takes
effect when workers are registered. Worker quantity, job timeout,
retry backoff and ownership checks are unchanged.
newConnection( "mail" )
.setProvider( "DBProvider@cbq" )
.setProperties( { "pollIntervalMilliseconds": 250 } );
A shorter interval also increases polling while the queue is idle. Measure database load, worker capacity, transport limits and queue age before choosing it. This setting does not increase the worker pool's concurrency or guarantee a delivery time.
To install cbq, install it from ForgeBox:
box install cbq
You can configure cbq in your moduleSettings inside
config/ColdBox.cfc as follows:
moduleSettings = {
"cbq" : {
// The path the custom config file to register connections and worker pools
"configPath" : "config.cbq",
// Flag if workers should be registered. If your application only pushes to the queues, you can set this to false.
"registerWorkers" : getSystemSetting( "CBQ_REGISTER_WORKERS", true ),
// The interval to poll for changes to the worker pool scaling. Defaults to 0 which turns off the scheduled scaling feature.
"scaleInterval" : 0
}
};
Most of the configuration for cbq happens inside the cbq config file,
located at config/cbq.cfc by convention.
component {
function configure() {
newConnection( "default" )
.setProvider( "SyncProvider@cbq" );
newWorkerPool( "default", "default" );
}
}
In configure you define one or more
Connections. You must have at least one
Connection called default.
New Connections are created using the
newConnection function. It is a builder pattern object.
Only a provider must be set. The other setters are optional.
newConnection( connectionName )
.setProvider( providerMapping )
.onQueue( name = "default" )
.markAsDefault( /* true / false */ ); .
You can also define worker pools inside configure to
work on queues for a given Connection that
you defined previously.
New Worker Pools are created using the
newWorkerPool function. It is a builder pattern object.
All of the setters are optional.
newWorkerPool( name, connectionName )
.quantity( numberOfWorkers )
.onQueue( name = "default" )
.backoff( backoffTimeInSeconds )
.timeout( timeoutTimeInSeconds )
.maxAttempts( maxNumberOfAttempts );
The config file follows ColdBox's environment overrides by calling a
method matching the environment name if it is found. Inside that
method you can use the withConnection and
withWorkerPool methods to change the properties of
connections defined in configure and worker pools defined
in work:
component {
function configure() {
newConnection( "default" )
.setProvider( "DBProvider@cbq" );
newWorkerPool( "default", "default" )
.setTimeout( 5 )
.setMaxAttempts( 5 )
.setQuantity( 3 );
}
function development() {
withConnection( "default" )
.setProvider( "SyncProvider@cbq" );
withWorkerPool( "default" )
.setMaxAttempts( 1 )
.setQuantity( 1 );
}
}
Jobs are CFCs that extend
cbq.models.Jobs.AbstractJob. You need to define a
handle method that is ran when the Job is processed.
// GreetingJob.cfc
component extends="cbq.models.Jobs.AbstractJob" {
function handle() {
// this is ran when the job is processed
log.debug( "Hello world!" );
}
}
To dispatch a job to a queue to be worked, call the
dispatch method on a Job instance.
Dispatching a job serializes it and sends it to the configured
connection. It will later be picked up by a worker and processed.
getInstance( "GreetingJob" ).dispatch();
A Job is sent to the queue with any of its properties
serialized. You can set the properties of a job by calling the
setProperties method and passing a struct of properties.
getInstance( "GreetingJob" )
.setProperties( { "greeting": "Hello" } )
.dispatch();
Additional Job-level properties can be set before dispatching to override Worker Pool defaults on a per-Job basis.
getInstance( "GreetingJob" )
.setProperties( { "greeting": "Hello" } )
.setDelay( 10 ) // delay processing this job for 10 seconds
.dispatch();
A cbq model exists to make certain job- and dispatch-related actions
easier to perform. You can access it by injecting
cbq@cbq or simply @cbq. Here are the methods
available to you:
job
Creates a Job instance.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| job | string or Job instance | true | A job instance or mapping
string to a Job instance. Additionally, any string may be
provided here, even if it doesn't exist as a CFC. If so, cbq
will create a NonExecutableJob with the given
mapping. This can only be used if the instance dispatching the
jobs will never work the jobs. | |
| properties | struct | false | {}
| A struct of properties for the new Job. |
| chain | Job[] | false | []
| An array of Job instances to chain after this one. |
| queue | string | false | null
| The queue to run this Job on. Overrides the Job queue and the default queue, if provided. |
| backoff | numeric | false | null
| The backoff amount in seconds between Job attempts. Overrides the Job backoff and the default backoff, if provided. |
| timeout | numeric | false | null
| The timeout amount in seconds before a Job run is considered timed out. Overrides the Job timeout and the default timeout, if provided. |
| maxAttempts | numeric | false | null
| The maxAttempts amount before a Job run is considered failed. Overrides the Job maxAttempts and the default maxAttempts, if provided. |
dispatch
Creates a Job instance and immediately dispatches it.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| job | string or Job instance OR array of Job instances | true | A job instance or
mapping string to a Job instance. Additionally, any string may
be provided here, even if it doesn't exist as a CFC. If so, cbq
will create a NonExecutableJob with the given
mapping. This can only be used if the instance dispatching the
jobs will never work the jobs. If an array of Job instances are
passed, this forwards it on to chain and dispatches
the chain. | |
| properties | struct | false | {}
| A struct of properties for the new job. |
| chain | Job[] | false | []
| An array of Job instances to chain after this one. |
| queue | string | false | null
| The queue to run this Job on. Overrides the Job queue and the default queue, if provided. |
| backoff | numeric | false | null
| The backoff in seconds amount between Job attempts. Overrides the Job backoff and the default backoff, if provided. |
| timeout | numeric | false | null
| The timeout amount in seconds before a Job run is considered timed out. Overrides the Job timeout and the default timeout, if provided. |
| maxAttempts | numeric | false | null
| The maxAttempts amount before a Job run is considered failed. Overrides the Job maxAttempts and the default maxAttempts, if provided. |
chain
Creates a Job Chain and returns the first Job in the chain. To
dispatch the chain, you must call dispatch on the
returned Job.
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| chain | Job[] | false | []
| An array of Job instances to chain after this one. |
[email protected] accepts
batchSize from 1 to 100. The default is 1 and preserves
the existing order: announce onCBQJobAdded, reset the
job's attempt, then push that job before announcing the next.
transaction {
getInstance( "Dispatcher@cbq" ).bulkDispatch(
jobs = jobs,
connectionName = "db",
queueName = "mail",
batchSize = 100
);
}
With a larger batch size, each job is still announced separately and
becomes its own queue row, with its own payload and lifecycle. Events
for one chunk run before that chunk is persisted. Choose the default
if a job-added interceptor needs to read earlier jobs from the same
chunk. An explicit queue overrides job queues as before; otherwise
each job keeps its effective queue. As with existing
bulkDispatch, this does not use each job's backoff as an
initial delay.
The DB provider writes at most 100 rows (500 bindings) per insert.
Other providers continue through their ordinary push
implementation unless they implement pushMany(entries).
Entries contain the ordinary push arguments:
queueName, job, and optionally
delay and attempts.
DBProvider.pushMany also bounds direct calls to 100 rows
per insert. It uses the configured table, query options and datasource.
The caller owns transaction boundaries. Wrap the dispatch with related database changes when all chunks must commit or roll back together. Without a surrounding transaction, a later event, serialization or insert failure can leave earlier chunks persisted.
marshalJob future with the delay future
(dfd6102)failedJobIds except for incrementing failed jobs
(9ee8593)failedJobIds except for incrementing failed jobs
(9389595)failedJobIds except for incrementing failed jobs
(ac8eaf0)marshalJob future with the delay future
(dfd6102)failedJobIds except for incrementing failed jobs
(ac8eaf0)release to follow the new push method signature
(fc28483)shutdown pass through method to the QueueProvider
(1192c51)config variable in ModuleConfig
(1cff751)afterJobException and afterJobFailed lifecycle methods
(d263896)push (dcd077f)afterJobExpection provider-level method
(1262a67)providerContext field on the job (dae0d8e)shutdown on connections and workers on onUnload (fb7d808)afterConfigurationLoad to let all other modules load (1791bb9)lockForUpdate flag and add debug logging
(413760f)jobPattern (552e8ae)before and after lifecycle methods (8cf8390)originalId
(041f34b)release a job inside the handle method. (451b616)onFailure method, call it when the job fails
(6b091f7)cbq.job in case the job is not defined on this server
(14a6dfd)originalId
(041f34b)lockForUpdate flag and add debug logging
(413760f)jobPattern (552e8ae)before and after lifecycle methods (8cf8390)
$
box install cbq