The external payload storage is currently only implemented to be used to by the Java client. Client libraries in other languages need to be modified to enable this.
Contributions are welcomed.
Conductor can be configured to enforce barriers on the size of workflow and task payloads for both input and output.
These barriers can be used as safeguards to prevent the usage of conductor as a data persistence system and to reduce the pressure on its datastore.
Conductor typically applies two kinds of barriers:
- Soft Barrier
- Hard Barrier
The soft barrier is used to alleviate pressure on the conductor datastore. In some special workflow use-cases, the size of the payload is warranted enough to be stored as part of the workflow execution.
In such cases, conductor externalizes the storage of such payloads to S3 and uploads/downloads to/from S3 as needed during the execution. This process is completely transparent to the user/worker process.
The hard barriers are enforced to safeguard the conductor backend from the pressure of having to persist and deal with voluminous data which is not essential for workflow execution. In such cases, conductor will reject such payloads and will terminate/fail the workflow execution with the reasonForIncompletion set to an appropriate error message detailing the payload size.
Conductor provides an implementation of Amazon S3 used to externalize large payload storage.
Set the following property in the JVM system properties:
This implementation assumes that S3 access is configured on the instance.
Set the following properties to the desired values in the JVM system properties:
|workflow.external.payload.storage.s3.bucket||S3 bucket where the payloads will be stored|
|workflow.external.payload.storage.s3.signedurlexpirationseconds||The expiration time in seconds of the signed url for the payload||5|
|conductor.workflow.input.payload.threshold.kb||Soft barrier for workflow input payload in KB||5120|
|conductor.max.workflow.input.payload.threshold.kb||Hard barrier for workflow input payload in KB||10240|
|conductor.workflow.output.payload.threshold.kb||Soft barrier for workflow output payload in KB||5120|
|conductor.max.workflow.output.payload.threshold.kb||Hard barrier for workflow output payload in KB||10240|
|conductor.task.input.payload.threshold.kb||Soft barrier for task input payload in KB||3072|
|conductor.max.task.input.payload.threshold.kb||Hard barrier for task input payload in KB||10240|
|conductor.task.output.payload.threshold.kb||Soft barrier for task output payload in KB||3072|
|conductor.max.task.output.payload.threshold.kb||Hard barrier for task output payload in KB||10240|
The payloads will be stored in the bucket configured above in a
UUID.json file at locations determined by the type of the payload. See here for information about how the object key is determined.