verity volumes: pre-load docker images and data into CVMs#752
Open
h4x3rotab wants to merge 8 commits into
Open
verity volumes: pre-load docker images and data into CVMs#752h4x3rotab wants to merge 8 commits into
h4x3rotab wants to merge 8 commits into
Conversation
h4x3rotab
changed the title
h4x3rotab
force-pushed
the
verity-volumes
branch
12 times, most recently
from
b0a463a to
109c584
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Add read-only verity volumes -- extra virtio-blk disks a CVM can mount
instead of pulling and unpacking their contents. A volume is declared in
app-compose.json as `{ verity_root, target }`, and attached at deploy time
with `--volume <name>`: the vmm looks the name up under cvm.volumes_dir and
attaches it read-only. Because verity_root is part of the measured compose,
the guest can check the bytes it's handed against it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Just before dockerd starts, dstack-prepare.sh runs the seeding helper. For each volume declared in the compose it finds the matching disk by opening it with veritysetup against the measured verity_root, then either seeds docker's overlay2 store (target "docker") so the images are already present, or mounts the volume at a path (a data volume). It's fail-safe throughout: a volume that's missing or doesn't verify is skipped, and its images just pull. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`dstack verity <image>...` builds a squashfs + dm-verity volume that pre-extracts docker images (or, with `--dir`, a plain directory) so a CVM can start without pulling or unpacking them. It fetches images itself through oci-client -- no docker daemon -- and lays out the overlay2 store deterministically (each layer's directory id is its chain-id, with a fixed timestamp and salt). The same inputs always produce the same verity_root, so anyone can recompute it from the pinned image digests and confirm what's in the volume without trusting the builder. `dstack deploy --volume` attaches the result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Starting a CVM is slow mostly because of image extraction — decompressing layers and writing millions of files to the encrypted disk. A verity volume skips that: a read-only, dm-verity-protected disk, built once, whose layers are already extracted. The CVM verifies blocks lazily as the app reads them, and one volume can back many CVMs.
How it fits together
dstack verity <image>…builds the volume throughoci-client, without a Docker daemon, and lays out the overlay2 store deterministically.--dirpacks a plain directory instead.volumes_dirand attaches them read-only throughdstack deploy --volume.verity_rootthroughapp-compose.jsonand therefore the measuredapp_id.Design and trade-offs are documented in
docs/verity-volumes.md.Rebased and relocated onto the unified monorepo layout from PR #770.
Validation
cargo test -p dstack-verity -p dstack-cli -p dstack-vmm -p dstack-guest-agent -p dstack-typesreuse lintprek run --all-files