ADR-0008 Use Obi Cloth for Real-time Cloth Physics
The main requirements considered for this decision are: accuracy, photorealism, and interactivity. There is always going to be a tradeoff between interactivity and the other two.
Requirements per use case
Overview of the different requirements depending on the use case:
(UC1) Designer app. For VR, interactive speeds are, so a game engine model works well. However, the domain also requires accuracy. If we go with a series of fixed animations and poses, we can store the 3D animations of the accurate garment simulation from VStitcher in Alembic format (support added to VStitcher in version 2021.1). For increased photorealism, we could have the option to ray-trace fixed camera angles, and even produce EcoShots imagery for those.
(UC2) DressMeUp app. The most important requirement is Photorealism because the final image must be worth sharing. We aren’t constrained by interaction speed. The simulation and rendering can happen offline. The cloth simulation can be done in VStitcher, and the rendering can be ray-traced with V-Ray. The final render can be composed with the user’s photograph using Metail’s EcoShot, which should give the most photorealistic look.
(UC3) Magic Mirror app. Interactivity is key, but we want to raise accuracy and photorealism as much as we can. Without the accuracy and photorealism requirements, we could target a platform like Snapchat’s Lens Studio, that provides AR body tracking.
This ADR focuses in UC3, although the decision affects UC1 as well.
Game engines offer what we need in terms of interactivity. In terms of photorealism, most engines have comparable results. In terms of accuracy of the cloth simulation, results differ. Although most engines use NvCloth under the hood, the capabilities can be quite different. Here is a summary of the findings.
Although Lens Studio does not offer any cloth simulation at the moment, it offers 3D body tracking out of the box. We can pre-simulate garments in VStitcher in some neutral pose and body shape, and then rig those garments. The skeleton in the rig needs to match the skeleton of Lens Studio. The garments will scale anisotropically based on the bone positions.
Unity is the strongest candidate for its ease-of-use, multiplatform support, and support for AR and VR. However, the in-built cloth simulation is mostly oriented to environment assets. See Getting Started with Cloth physics. For dressing up a character, we have different options:
- A fully-skinned garment, like in the Snapchat approach. See How to make clothes animate along a character.
- Have a mixture of skinned and unskinned meshes. See How to poncho as an example. The challenge of this approach is having colliders that represent the body accurately. For speed constraints, mesh colliders can’t be used in cloth simulation, so we need to approximate the geometry with capsules and spheres.
- For cloth pieces, you can paint the vertices of the cloth mesh to tell the engine how much you want it to be affected by physics, where a value of 0 keeps the vertex in place.
- Use Spring Joints, which join two rigid bodies together as if they were connected by a spring. Check Spring bone vs Unith cloth.
This is a plugin for Unity that provides a faster solver and it integrates the components mentioned above in a single framework. That is, you can have a garment file that combines skinned geometry (e.g. pre-simulated shirt torso), with unskinned pieces for pure cloth simulation (e.g. a skirt), and spring bones.
However, it doesn’t seem to add anything that can’t already be done with Unity.
Amazon Lumberyard has a few extra things useful for cloth simulation:
- Cloth simulation can be applied to skinned meshes as well, so we can have a hybrid model. Similar to Unity’s weight painting, you paint the weights of the motion constraints to define how far away a cloth vertex can move. In this case, how far from its position after skinning, so the vertex is also affected by skinning. The weights are normalised between 0 and 1, and then there’s a maximum distance value per cloth piece, defined in metres.
- The weights are stored as color channels in the FBX file, which it’s something we may want to consider if we want to export a single file.
- There is also a backstop value which is a cheap substitute for colliders. Instead of having colliders, we define how much the vertex can move in the normal direction of the mesh (a negative value for the vertex of a sleeve would allow it to penetrate the arm).
- Skinning in Lumberyard uses dual quaternion by default, which it’s useful to avoid bulging and candy-wrapper artefacts in skinned meshes. It is supported in mobile as well. Read Introduction to skinning and 3D animation.
- It automatically automatically stitches and creates a proxy geometry for simulation on garment import. The Level-of-Detail of the proxy geometry can be set as well. For rendering, the proxy geometry is replaced by the original geometry, so you can have multiple unstitched materials and a higher polygon count for photorealism.
Lumberyard does not support spring joints, but those are probably irrelevant. Spring joints are a cheap way to make bouncy skirts when cloth simulation is not avaible, or it’s unstable.
The main drawbacks for using Lumberyard are the lack of official AR support, the difficulty of building for multiple platforms (all major mobile and desktop platforms are supported, but the setup is not as straightforward as Unity), and build times.
Obi Cloth is a Unity plugin that adds support for the things mentioned above in the Lumberyard section, mainly support for skinned meshes, and cloth proxies. The motion constraints described earlier are called skin constraints.
The only disadvantage with respect to Lumberyard is that the skinning depends on Unity skinning, and Unity does not support dual-quaternions skinning.
However, we gain all the advantages of using Unity: fast development cycles, and good support for AR and VR.
We will use Unity for UC1 and UC3. For UC3 we will use Obi Cloth, which enables Unity with all the advantages seen in other engines like Amazon Lumberyard.
For UC1, we can use the same engine, but we also have the option of using Alembic 3D animation playback for increased accuracy, at the cost of some interactivity (we can still move freely inside the VR world and pause the animations at any point to inspect the garment).
We will also build a teaser app of UC3 using Lens Studio. This will use the same garment creation pipeline, but with a special skeleton for Lens Studio. It won’t require us to export any physics properties, so it can be created earlier.
Garment data needs to be prepared for Obi Cloth. In particular, we need to check the following:
- Do we need to pre-stitch garments beforehand, or can Obi do all the stitching?
- How do we provide the skin constraints weights?
- How do we convert other physics properties from VStitcher to Obi?