ADR-0021 Storage Path Scheme
Publication Date | 2021-09-21 |
---|---|
Last Update | 2021-09-21 |
Status | Accepted |
Amends | ADR-0014 Scanatar Creation |
Context
We need to define the storage paths of all use cases in the asset store (Cloud Storage) and database (Cloud Firestore) so data can be managed and filtered, taking into account the following concerns:
- GDPR Compliance
- Access Control
- Event Pub/Sub
GDPR Compliance
We need to be able to identify all user-owned and PII-containing data so we can respond to subject access requests and right to be forgotten requests.
Access Control
Cloud Storage and Firestore use path-based rules for access control. For our use-cases, all data falls into one of three categories:
- public data: curated by an administrator, can be read by anyone (including unauthenticated users)
- shared data: curated by an administrator, can be read by any authenticated user
- internal data: curated by an administrator, can be read and updated by any administrator but is not visible to other users
- user data: this should only be read and written by the owning user
We need to be able to write path-based matching rules to define the desired access controls.
Note: the term “administrator” is used loosely here to mean a user with a role granting them elevated privileges, e.g. brand staff or marketing executives.
Event Pub/Sub
When certain assets and database records are created or updated, we need to be able to trigger a Cloud Function to run. This is done by configuring events to be published to a Pub/Sub topic, and subscribing a Cloud Function to the topic. The notifications are bucket-wide, but the subscriber can specify a filter to control which events they receive from the topic. The filter syntax is quite limited, allowing only exact or prefix match on the attributes in the event.
Decision
We will store public data under a prefix of /{assetType}/public/
We will store shared data under a prefix of /{assetType}/shared/
We will store internal data under a prefix of /{assetType}/internal/
We will store user data under a prefix of /{assetType}/user/{userID}/
There may be several levels of hierarchy within these prefixes.
Examples
image/public/logo.png
animation/user/enK0PQ8YY0hXDICJm1MKfjYTTvu1/1FYhdl4F9cWx0qT4EoVf.jpg
animation/shared/1FYhdl4F9cWx0qT4EoVf.jpg
texture/user/enK0PQ8YY0hXDICJm1MKfjYTTvu1/garment/1FYhdl4F9cWx0qT4EoVf/base.jpg
Conclusion
GDPR Compliance
We will be able to comply with GDPR requests as all user-owned data will match a path /{assetType}/user/{userID}/**
.
Access Control
We will be able to write access control rules for user-owned files by writing a match rule with a condition requiring that the user id in the path matches that of the authenticated user. For example:
service firebase.storage {
// Allow the requestor to read or delete any avatar on a path under the
// user directory.
match /avatar/user/{userId}/{anyUserFile=**} {
allow read, delete: if request.auth != null && request.auth.uid == userId;
}
}
For the shared data (admin read/write, user read) we could set up custom claims in Firebase Auth and use them in the rules:
service firebase.storage {
// Allow authenticated users to read
// Allow users with an `admin` flag set in the user's custom token to write
match /animation/shared/{anySharedFile=**} {
allow read: if request.auth != null;
allow write: if request.auth.token.admin == true;
}
}
Event Pub/Sub
Subscribers will be able to filter the events they see using a prefix match. For example, the filter attributes.eventType = "OBJECT_FINALIZE" AND hasPrefix(attributes.objectId, "/avatar/")
would limit the events seen to avatar creation events.
Appendix
Storage Access Paths in Use Cases
Use Case 1
Filename | Description | Type | Folder Path |
---|---|---|---|
animation-file-{timestamp} | Mixamo animation file | .fbx | animation >shared |
animation-preview-{timestamp} | An image for preview purposes | .png or .jpg | animation >shared |
avatar-file-{timestamp} | VStitcher file | .fbx | avatar >shared |
avatar-preview-{timestamp} | An image for previewing purposes | .png or .jpg | avatar >shared |
garment-file-{timestamp} | 3D Object file | .fbx | garment >shared |
garment-preview-{timestamp} | An image for previewing purposes | .png or .jpg | garment >shared |
textures-{documentID}-{timestamp} | Zip file including all of the textures | .zip | textures >shared |
scanatar-file-{timestamp} | A temporary 3D Object | .ply | qcscan >shared |
scanatar-meta-{timestamp} | A JSON file that features details about the Scanatar | .json | qcmeta >shared |
Use Case 2
Filename | Description | Type | Folder Path |
---|---|---|---|
avatar-file-{timestamp} | The avatar that is created for each user | .fbx | avatar >user >{userID} |
garment-file-{timestamp} | The garment 3D design file | .bw | garment >internal |
user-media-{timestamp} | A photo taken by the user | .jpg | collection >user >{userID} |
output-{timestamp} | The synthesized image, output from the platform | .jpg | collection >user >{userID} |
scanatar-file-{timestamp} | A temporary 3D Object | .ply | qcscan >user >{userID} |
scanatar-meta-{timestamp} | A JSON file that features details about the Scanatar | .json | qcmeta >user >{userID} |
Use Case 3
Filename | Description | Type | Folder Path |
---|---|---|---|
avatar-file-{timestamp} | The avatar that is created for each user | .fbx | avatar >user >{userID} |
garment-file-{timestamp} | The garment 3D object file | .fbx | garment >shared |
textures-{documentID}-{timestamp} | Zip file including all of the textures | .zip | textures >shared |
photo-{documentID}-{timestamp} | Photos that are saved by the user when trying on a garment using the camera | .jpg | photos >user >{userID} |
scanatar-file-{timestamp} | A temporary 3D Object | .ply | qcscan >user >{userID} |
scanatar-meta-{timestamp} | A JSON file that features details about the Scanatar | .json | qcmeta >user >{userID} |
Term Index
documentID
: Google Firestore Database saves and handles data into Collections. Data entries written in a collection are assigned a unique id. This ID is used in Firebase Storage to connect assets to collection items.userID
: Each registered user has a unique identifier in Firestore Database. This user id is used in some parts in Firebase Storage path scheme to denote ownership of assets.timestamp
: The timestamp is expressed in milliseconds since the Unix Epoch.