ADR-0021 Storage Path Scheme

Date2021-09-21
StatusAccepted
AmendsADR-0014 Scanatar Creation

Context

We need to define the storage paths of all use cases in the asset store (Cloud Storage) and database (Cloud Firestore) so data can be managed and filtered, taking into account the following concerns:

  • GDPR Compliance
  • Access Control
  • Event Pub/Sub

GDPR Compliance

We need to be able to identify all user-owned and PII-containing data so we can respond to subject access requests and right to be forgotten requests.

Access Control

Cloud Storage and Firestore use path-based rules for access control. For our use-cases, all data falls into one of three categories:

  • public data: curated by an administrator, can be read by anyone (including unauthenticated users)
  • shared data: curated by an administrator, can be read by any authenticated user
  • internal data: curated by an administrator, can be read and updated by any administrator but is not visible to other users
  • user data: this should only be read and written by the owning user

We need to be able to write path-based matching rules to define the desired access controls.

Note: the term “administrator” is used loosely here to mean a user with a role granting them elevated privileges, e.g. brand staff or marketing executives.

Event Pub/Sub

When certain assets and database records are created or updated, we need to be able to trigger a Cloud Function to run. This is done by configuring events to be published to a Pub/Sub topic, and subscribing a Cloud Function to the topic. The notifications are bucket-wide, but the subscriber can specify a filter to control which events they receive from the topic. The filter syntax is quite limited, allowing only exact or prefix match on the attributes in the event.

Decision

We will store public data under a prefix of /{assetType}/public/

We will store shared data under a prefix of /{assetType}/shared/

We will store internal data under a prefix of /{assetType}/internal/

We will store user data under a prefix of /{assetType}/user/{userID}/

There may be several levels of hierarchy within these prefixes.

Examples

  • image/public/logo.png
  • animation/user/enK0PQ8YY0hXDICJm1MKfjYTTvu1/1FYhdl4F9cWx0qT4EoVf.jpg
  • animation/shared/1FYhdl4F9cWx0qT4EoVf.jpg
  • texture/user/enK0PQ8YY0hXDICJm1MKfjYTTvu1/garment/1FYhdl4F9cWx0qT4EoVf/base.jpg

Conclusion

GDPR Compliance

We will be able to comply with GDPR requests as all user-owned data will match a path /{assetType}/user/{userID}/**.

Access Control

We will be able to write access control rules for user-owned files by writing a match rule with a condition requiring that the user id in the path matches that of the authenticated user. For example:

service firebase.storage {
  // Allow the requestor to read or delete any avatar on a path under the
  // user directory.
  match /avatar/user/{userId}/{anyUserFile=**} {
    allow read, delete: if request.auth != null && request.auth.uid == userId;
  }
}

For the shared data (admin read/write, user read) we could set up custom claims in Firebase Auth and use them in the rules:

service firebase.storage {
  // Allow authenticated users to read
  // Allow users with an `admin` flag set in the user's custom token to write
  match /animation/shared/{anySharedFile=**} {
    allow read: if request.auth != null;
    allow write: if request.auth.token.admin == true;
  }
}

Event Pub/Sub

Subscribers will be able to filter the events they see using a prefix match. For example, the filter attributes.eventType = "OBJECT_FINALIZE" AND hasPrefix(attributes.objectId, "/avatar/") would limit the events seen to avatar creation events.

Appendix

Storage Access Paths in Use Cases

Use Case 1

Filename Description Type Folder Path
animation-file-{timestamp} Mixamo animation file .fbx animation>shared
animation-preview-{timestamp} An image for preview purposes .png or .jpg animation>shared
avatar-file-{timestamp} VStitcher file .fbx avatar>shared
avatar-preview-{timestamp} An image for previewing purposes .png or .jpg avatar>shared
garment-file-{timestamp} 3D Object file .fbx garment>shared
garment-preview-{timestamp} An image for previewing purposes .png or .jpg garment>shared
textures-{documentID}-{timestamp} Zip file including all of the textures .zip textures>shared
scanatar-file-{timestamp} A temporary 3D Object .ply qcscan>shared
scanatar-meta-{timestamp} A JSON file that features details about the Scanatar .json qcmeta>shared

Use Case 2

Filename Description Type Folder Path
avatar-file-{timestamp} The avatar that is created for each user .fbx avatar>user>{userID}
garment-file-{timestamp} The garment 3D design file .bw garment>internal
user-media-{timestamp} A photo taken by the user .jpg collection>user>{userID}
output-{timestamp} The synthesized image, output from the platform .jpg collection>user>{userID}
scanatar-file-{timestamp} A temporary 3D Object .ply qcscan>user>{userID}
scanatar-meta-{timestamp} A JSON file that features details about the Scanatar .json qcmeta>user>{userID}

Use Case 3

Filename Description Type Folder Path
avatar-file-{timestamp} The avatar that is created for each user .fbx avatar>user>{userID}
garment-file-{timestamp} The garment 3D object file .fbx garment>shared
textures-{documentID}-{timestamp} Zip file including all of the textures .zip textures>shared
photo-{documentID}-{timestamp} Photos that are saved by the user when trying on a garment using the camera .jpg photos>user>{userID}
scanatar-file-{timestamp} A temporary 3D Object .ply qcscan>user>{userID}
scanatar-meta-{timestamp} A JSON file that features details about the Scanatar .json qcmeta>user>{userID}

Term Index

  • documentID: Google Firestore Database saves and handles data into Collections. Data entries written in a collection are assigned a unique id. This ID is used in Firebase Storage to connect assets to collection items.
  • userID: Each registered user has a unique identifier in Firestore Database. This user id is used in some parts in Firebase Storage path scheme to denote ownership of assets.
  • timestamp: The timestamp is expressed in milliseconds since the Unix Epoch.