UC2 DressMeUp Metail Composition Service

The Metail Composition service will be invoked by starting an AWS Step Function execution.

The StartExecution API endpoint is documented here. The parameters of interest are:

Field Description
input JSON input data for the execution
name The name of the execution (a random UUID)
stateMachineArn ARN of the step function - will be known once the endpoint is in place

Input

The step function input JSON (serialized to a string in the input field above) must conform to the following schema:

{
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://etryon-h2020.eu/schema/metail/uc2-composition-service.json",
    "title": "Metail Composition Service",
    "description": "Step function input for composing 3D garments into 2D user photos",
    "type": "object",
    "properties": {
        "userAvatar": {
            "description": "Presigned GET URL for the Metail avatar in Google Cloud Storage",
            "type": "string",
            "format": "uri"
        },
        "userPhoto": {
            "description": "Presigned GET URL for the photo of the user that the garment will be composed into, in Google Cloud Storage",
            "type": "string",
            "format": "uri"
        },
        "garment": {
            "description": "Presigned GET URL for the browzwear file containing the garment to be composed, in Google Cloud Storage",
            "type": "string",
            "format": "uri"
        },
        "size": {
            "description": "Garment size selected by the user",
            "type": "string",
            "format": "string"
        },
        "warp_region": {
            "description": "Warp region for the garment type",
            "type": "string",
            "format": "string"
        },
        "colorway": {
            "description": "Colorway of the garment",
            "type": "string",
            "format": "string"
        },
        "avatarGender": {
            "description": "Gender of the user avatar",
            "type": "string",
            "format": "string"
        },
        "outputImage": {
            "description": "Presigned PUT URL for the output image in Google Cloud Storage",
            "type": "string",
            "format": "uri"
        },
        "outputError": {
            "description": "Presigned PUT URL for pipeline error metadata in Google Cloud Storage",
            "type": "string",
            "format": "uri"
        }
    },
    "required": ["userAvatar", "userPhoto", "garment", "size", "outputImage", "outputError"],
    "examples": [
        {
	        "userAvatar": "https://storage.googleapis.com/see-adr-0021",
            "userPhoto": "https://storage.googleapis.com/see-adr-0021",
            "garment": "https://storage.googleapis.com/see-adr-0021",
            "warp_region": "full",
            "size": "XL",
            "outputImage": "https://storage.googleapis.com/see-adr-0021",
            "outputError": "https://storage.googleapis.com/see-adr-0021"
        }
    ]
}

Error Reporting

If the pipeline encounters an error and cannot complete the request, a JSON payload will be saved to Cloud Storage using the pre-signed outputURL in the request. This will be a map containing the following fields:

Field Description
Execution Step function Execution Id, can be used for detailed debugging by Metail
ErrorCode An error code indicating which step of the pipeline failed (see table below)
Message A human-readable error message (high level, not informative)

The ErrorCode indicates which step of the pipeline failed, possible values are:

ErrorCode Description
ErrInit Could not read inputs from Cloud Storage (may work on retry)
ErrCreateAvatar Could not process user’s photo to create an aligned avatar
ErrGarmentRender VStitcher garment render failed, e.g. requested size/colour not available in Browzwear file
ErrComposeImages Error composing garment image onto user’s photo
ErrSaveOutput Error saving the completed image back to Cloud Storage (may work on retry)

Note on implementation

Developers rarely need to be concerned with the underlying REST API for AWS services - it is much more common to use one of the AWS SDKs. For example, to invoke the scanatar step function from a Python program you would use the boto3 step function client and your code would look something like:

import json
import uuid
import boto3

SFN_ARN = 'arn:aws:states:{region}:{accountId}:stateMachine:etryon-dress-me-up-composition'

client = boto3.client('stepfunctions')
response = client.start_execution(
    stateMachineArn=SFN_ARN,
    name=str(uuid.uuid4()),
    input=json.dumps({
        "userAvatar": "https://storage.googleapis.com/...",
        "userPhoto": "https://storage.googleapis.com/...",
        "garment": "https://storage.googleapis.com/...",
        "warp_region": "full",
        "size": "XL",
        "colorway": "red",
        "avatarGender": "male",
        "outputImage": "https://storage.googleapis.com/...",
        "outputError": "https://storage.googleapis.com/..." 
    })
)
print(f'Started execution {response["executionArn"]}')

Operation

See the sequence diagram for more details. A high level view of the operation of this service is as follows: -

  1. Download inputs
  2. Estimates user pose in photo.
  3. Compute pose offsets for VStitcher.
  4. Create avatar FBX file with 3D pose & VStitcher anchor points.
  5. Import avatar file in VStitcher & select pose
  6. Load garment file & select size
  7. Dress avatar pose
  8. Export V-Ray scene file
  9. Set camera, lights, and avatar offset.
  10. Render garment using V-Ray
  11. Morph rendered garment sprite to match user photo
  12. Compose morphed sprite & photo
  13. Upload result to presigned URL