Skip to content

[Feature] Supporting dstack-inside-dstack for dev environments and tasks#4023

Open
peterschmidt85 wants to merge 8 commits into
masterfrom
codex/task-server-access
Open

[Feature] Supporting dstack-inside-dstack for dev environments and tasks#4023
peterschmidt85 wants to merge 8 commits into
masterfrom
codex/task-server-access

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Problem

Tasks and dev environments may need to call dstack themselves, for example to
inspect runs, submit another run, or attach to it. This currently requires the
dstack server to accept direct connections from the run.

UX

Add dstack: true to task and dev-environment configurations.

type: task
image: dstackai/dstack
dstack: true
env:
  - DSTACK_TOKEN
commands:
  - dstack ps

With dstack: true, dstack exposes the server managing the run through
/run/dstack/server.sock inside each job. It sets DSTACK_SERVER_URL to the
corresponding http+unix URL and DSTACK_PROJECT to the current project,
allowing the CLI and API clients to use that connection.

Authentication remains explicit. dstack does not inject DSTACK_TOKEN; it can
be passed through the existing environment or secret mechanisms.

The CLI also accepts DSTACK_SERVER_URL, DSTACK_PROJECT, and DSTACK_TOKEN
as a complete configuration without requiring ~/.dstack/config.yml.

dstack is not supported for services.

Implementation

Before starting commands, the server opens an SSH connection to each job and
creates a remote Unix-socket forward. Connections to
/run/dstack/server.sock inside the job are forwarded to the HTTP port of the
server managing the run. DSTACK_SERVER_URL points clients to this socket.

This uses the existing job SSH path, so the server does not need a public URL
or gateway. Health checks use the same socket path end to end; if the SSH
connection or socket becomes stale, the server closes and recreates the
tunnel. The tunnel is removed when the job terminates.

@peterschmidt85 peterschmidt85 requested a review from un-def July 11, 2026 22:46
@peterschmidt85 peterschmidt85 changed the title Add server access to tasks and dev environments [Feature] Supporting dstack-inside-dstack for dev environments and tasks Jul 12, 2026
Comment thread src/dstack/_internal/core/services/api_client.py Outdated
Comment thread mkdocs/docs/concepts/dev-environments.md Outdated
Andrey Cheptsov and others added 5 commits July 13, 2026 12:36
Removing the job server connection on PROVISIONING/PULLING iterations
reset the failure time tracked by the pool, so retry_timed_out() could
never fire and jobs with a failing tunnel were never terminated by the
server. Remove the connection only when the job leaves RUNNING;
jobs_terminating covers the final cleanup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DSTACK_TOKEN alone now overrides the token of the project configured in
config.yml. DSTACK_SERVER_URL still takes effect only together with
DSTACK_TOKEN, and a missing token or project now produces an explicit
error instead of a silent fallback when no config is available.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enabling dstack access inside a run now reads like Docker-in-Docker
(`docker: true`): `dstack: true`. No migration needed: the field is
stored in run spec JSON only; pre-release DBs with the old field can be
reset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reverse forward targeted 127.0.0.1 while the server may be bound to
a specific address via DSTACK_SERVER_HOST, making the forward target
unreachable. Derive the target from the bind address and include it in
the error when the server is not reachable through the tunnel.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants