Conductor tasks that are executed by remote workers communicate over HTTP endpoints/gRPC to poll for the task and update the status of the execution.
Conductor provides the following java clients to interact with the various APIs
|Metadata Client||Register / Update workflow and task definitions|
|Workflow Client||Start a new workflow / Get execution status of a workflow|
|Task Client||Poll for task / Update task result after execution / Get status of a task|
Conductor provides a framework to poll for tasks, manage the execution thread and update the status of the execution back to the server.
Implement the Worker interface to execute the task.
Manages the task workers thread pool and server communication (poll, task update and ack).
The WorkflowTaskCoordinator can be used to register the worker(s) and initialize the polling loop.
Use the Builder to create an instance of the WorkflowTaskCoordinator. The builder accepts the following parameters:
|withEurekaClient||EurekaClient is used to identify if the server is in discovery or not. When the server goes out of discovery, the polling is stopped. If passed null, discovery check is not done.||provided by platform|
|withTaskClient||TaskClient used to communicate to the Conductor server||(required)|
|withWorkers||Workers that will be used for polling work and task execution.||(required)|
|withThreadCount||Number of threads assigned to the workers. Should be at-least the size of taskWorkers to avoid starvation in a busy system.||Number of registered workers|
|withSleepWhenRetry||Time in milliseconds, for which the thread should sleep when task update call fails, before retrying the operation.||500|
|withUpdateRetryCount||Number of attempts to be made when updating task status when update status call fails.||3|
|withWorkerQueueSize||Worker queue size for the polled task.||100|
|withWorkerNamePrefix||String prefix that will be used for all the workers.||workflow-worker-|
Once an instance is created, call
init() method to initialize the executor service within the coordinator and begin the polling for tasks.
To ensure that the WorkflowTaskCoordinator stops polling for tasks when the instance becomes unhealthy, call the provided
shutdown() hook in a
Properties The worker behavior can be further controlled by using these properties:
|paused||boolean||If set to true, the worker stops polling.||false|
|pollCount||int||Number of tasks to poll for in a single poll request. Used for batched polling. Each task will be executed in a separate thread.||1|
|pollInterval||int||Interval in milliseconds at which the server should be polled for tasks.||1000|
|longPollTimeout||int||Time in milliseconds for long polling to Conductor server for tasks. Use a higher number here as opposed to more frequent polls to reduce excessive calls.||100|
Further, these properties can be set either by Worker implementation or by setting the following system properties in the JVM:
||Applies to ALL the workers in the JVM.|
||Applies to the specified worker. Overrides the global property.|
Follow the example as documented in the readme or take a look at kitchensink_workers.py
Python client is a community contribution. We encourage you to test it out and let us know the feedback. Pull Requests with fixes or enhancements are welcomed!