ADR-0041 State Tracking for Asynchronous Jobs

Publication Date2022-07-06
Last Update2022-07-06
StatusProposed

Context

For use case 2 (Dress Me Up App) both avatar creation and image composition trigger asynchronous processes in third party systems (QuantaCorp and Metail). We need to keep track of which processes are running and whether or not they complete successfully in order to unblock dependent steps or notify the user of job completion/failure.

When these asynchronous pipelines are called, the input includes pre-signed URLs that are used to write output back to the Asset Store. We need a similar mechanism for signalling errors.

Decision

We will add an extra input parameter to the Metail Avatar Creation and Image Composition step functions, whose value will be a pre-signed URL for writing error metadata to the Asset Store. If a pipeline fails to complete, an error message will be posted back to this URL.

We will add an extra callback parameter to the QuantaCorp API; if present, pipeline status will be written back to that URl. The pipeline status will conform to the AsyncScanStatusDTO schema documented in the QuantaCorp API.

We will add a new GCP function to handle events triggered when job output or error files are written to the Asset Store. This function will update job / asset status in the Firebase database so the application can communicate these events to the user.

Consequences

Third-party pipelines will be able to report failure states to eTryOn systems.

We will have to implement a new GCP function to handle the OBJECT_FINALIZE events of interest.

Each avatar and image composition request should have a unique id that appears in the path in Cloud Storage. Output and/or error metadata will be written under the same path prefix with an appropriate file extension, for example, the files created by the avatar pipeline would be:

  • qcscan>user>{userID}>scanatar-file-{assetID}.ply (created by Quantacorp pipeline)
  • qcscan>user>{userID}>scanatar-meta-{assetID}.json (created by Quantacorp pipeline)
  • avatar>user>{userID}>avatar-file-{assetID}.bin (created by Metail pipeline)
  • avatar>user>{userID}>avatar-error-{assetID}.json (created on pipeline error)

The userID and assetID must be enough information to identify the Firestore record to be updated on completion/error.

Metail Avatar Creation and Image Composition step functions will need to have a specification for the status messages added.