hydra

hydra docs

hydra wraps process-compose. One baseline stack runs in your main checkout. Each git worktree starts only the services it changed and points everything else at the baseline.

It exists for working on several branches at once, often with a coding agent in each worktree. Five copies of a full stack cost five times the memory and fight over the same ports. With hydra, a frontend-only branch runs one extra process.

Install

hydra needs git and process-compose. The Homebrew formula installs process-compose for you.

Homebrew
$ brew tap santiagotoscanini/tap
$ brew install --HEAD santiagotoscanini/tap/hydra
$ brew upgrade --fetch-HEAD santiagotoscanini/tap/hydra    # later, to update

To build from source, clone the repository and run make link. It puts that checkout's build on your PATH as hydra; make unlink removes it.

Quick start

1. Describe your services

Add x-hydra keys to the process-compose.yaml you already have: a block at the top level, and one on each process that a worktree may run. process-compose ignores keys it doesn't know, so the file still works without hydra.

process-compose.yaml
version: "0.5"

x-hydra:
  base_ref: main
  slot_stride: 200
  reserved_ports: [5432]
  global_paths: [pnpm-lock.yaml]

processes:
  db:
    command: docker compose up postgres
    x-hydra:
      shared: true
      paths: ["db/migrations/**"]

  deps:
    command: pnpm install
    x-hydra:
      follow: true

  api:
    command: go run ./services/api
    depends_on:
      db: { condition: process_healthy }
    x-hydra:
      port: 8080
      port_env: API_PORT
      paths: ["services/api/**", "packages/shared/**", "!**/*_test.go"]

  worker:
    command: go run ./services/worker
    depends_on:
      db: { condition: process_healthy }
    x-hydra:
      paths: ["services/worker/**", "packages/shared/**"]
      url: "jobs-{slot}"
      baseline_url: jobs
      upstreams:
        API_URL: api

  web:
    command: pnpm --filter web dev
    depends_on:
      deps: { condition: process_completed_successfully }
      api: { condition: process_started }
    x-hydra:
      port: 3000
      paths: ["apps/web/**", "packages/shared/**"]
      upstreams:
        API_URL: api
        JOBS_QUEUE: worker
        NEXT_PUBLIC_SITE_URL: web

Each service has to read its port and its upstreams from the environment variables named here. hydra sets those variables; it doesn't proxy traffic.

2. Start the baseline

In the main checkout, hydra runs the whole file, exactly as process-compose would.

$ cd ~/code/shop
$ hydra

3. Start a worktree

$ git worktree add ../shop-checkout -b feat/checkout
$ cd ../shop-checkout
$ hydra plan        # what would run here, and why
$ hydra             # run it

A fresh worktree has changed nothing, so nothing runs in it yet. Edit a file under apps/web and web starts on port 3200, pointed at the baseline API on 8080.

hydra plan
worktree, slot 1: ~/code/shop-checkout

SERVICE  RUNS      PORT  WHY
api      baseline  8080  unchanged
db       baseline  -     shared
deps     here      -     needed by web
web      here      3200  changed: apps/web/app/page.tsx
worker   baseline  -     unchanged

wiring:
  web  API_URL               -> api (baseline)     http://127.0.0.1:8080  default: api not running here
  web  JOBS_QUEUE            -> worker (baseline)  jobs                   default: worker not running here
  web  NEXT_PUBLIC_SITE_URL  -> web (local)        http://127.0.0.1:3200  default: web running here
Ignore hydra's files. Add .process-compose/ and .hydra.local.yaml to your .gitignore. If your process-compose.yaml is itself git-ignored, a worktree without one uses the main checkout's.

Concepts

TermMeaning
BaselineThe stack running in the main checkout. It runs every service on its normal port.
WorktreeAny other checkout made with git worktree add. It runs only what it changed.
SlotA number per worktree. The baseline is slot 0. Slot N listens on each port plus N × slot_stride.
UpstreamAn environment variable of one service that says where another service lives. hydra fills it with the local copy's address or the baseline's.
SharedA service that only ever runs in the baseline, such as a database or a tunnel.
FollowA service that runs in a worktree whenever a local service depends on it, such as pnpm install.
Stand-inA placeholder process for a baseline service that a local one depends on. It runs nothing and only waits for the baseline's copy to be ready.

How it works

hydra runs no services itself. It works out what should run in this checkout, writes an ordinary process-compose file for it, and hands that file to process-compose. In a worktree, hydra does this:

  1. Lists the files the worktree changed and picks the services to run here.
  2. Takes a slot for the worktree, which it keeps across runs.
  3. Writes .process-compose/hydra.yaml: those services only, on the slot's ports, with each upstream variable set.
  4. Starts process-compose on that file, on a unix socket that belongs to this worktree.

Because of the socket, hydra down, hydra attach and hydra process … in a worktree reach that worktree's instance and never the baseline's.

What counts as changed

A worktree changed a service when a file that differs from where its branch left main matches one of the service's paths. Committed, staged, unstaged and new files all count. Git-ignored files don't.

$ git diff --name-only $(git merge-base HEAD main origin/main)   # committed, staged, unstaged
$ git ls-files --others --exclude-standard                       # new files

The remote-tracking branch counts as well as the local one, because a local main is often behind. A branch cut from a freshly fetched origin/main forked there, and the commits merged in between aren't its work.

The baseline runs what the main checkout has checked out. Services left to the baseline run that code, so pull in the main checkout when a worktree needs newer baseline code.

Watching

The generated file has one extra process, hydra-watch. It starts with the services and stops with them. Every 2 seconds it lists the changed files again. When the set of services to run differs, it rewrites the file and calls process-compose project update, which touches only the processes that differ.

TransitionWhat happens
Unchanged → changedThe service starts in the worktree. Local services that point at it restart with its address.
Changed → unchangedThe service stops. Local services that pointed at it restart on the baseline's address.

Those are the only two things the watcher does. Further edits to a service that already runs here are left to the service's own reload, such as next dev, air or process-compose's watch.

The watcher polls git instead of watching the file system. Only git can answer "does this differ from main": a revert has to stop the service, and ignored files such as node_modules must not start one. It reads git without taking the index lock, so it doesn't collide with an agent committing in the same worktree.

Stacked branches

Worktrees can be stacked: b1 cut from main, b2 from b1, b3 from b2, each in its own worktree. Run hydra in b3 and it walks the chain down to main.

LayerCompared withIts diff holdsRuns
b1where it left mainb1's workapi, from b1's worktree
b2where it left b1b2's work, not b1'sworker, from b2's worktree
b3where it left b2b3's work onlyweb, from b3's worktree
  • A service runs from the topmost layer that changed it, in that layer's worktree. An agent editing b2's worktree reaches the worker through the worker's own reload.
  • Everything runs in b3's process-compose project: one slot, one set of logs. b1 and b2 don't need to run hydra themselves.
  • The watcher polls every worktree in the stack, so an edit in b1's worktree starts or stops services in b3's project.
  • Follow services run once per worktree that has a local service, so each one installs into its own node_modules.

How a parent is found

  1. parent: in the worktree's .hydra.local.yaml. The base branch there means "not stacked".
  2. The branch it was created from, read from its reflog, if that branch is checked out in a worktree and still shares commits with it past main. This holds after the parent gets new commits.
  3. Otherwise, the deepest checked-out branch whose tip it contains.

Branches that only share history are not parents. Two siblings cut from the same commit look, from ancestry alone, the same as a parent that moved on, so hydra doesn't guess. A branch made with git checkout -b b2 while on b1 records "Created from HEAD"; once b1 gets new commits, give b2 a parent: b1.

hydra plan prints the stack it found and why each service runs where it does.

The generated file

hydra render prints what hydra hands to process-compose. Compared with your file:

  • Only the services that run here are kept, with the x-hydra keys removed.
  • Each service's environment gains its port variable, its upstream variables, and HYDRA_SERVICE, HYDRA_SLOT and HYDRA_UPSTREAMS.
  • working_dir, log_location and watch paths become absolute, pointing into the worktree the service runs from.
  • Readiness and liveness probes on the baseline port move to the slot's port. Otherwise a worktree's probe would hit the baseline's copy and pass at once.
  • A stand-in replaces every baseline process that a local one depends on. It copies only the readiness probe or ready log line, so process_healthy waits for the baseline's database instead of starting a second one. Stopping the worktree never runs the baseline's shutdown commands.
  • A hydra-prepare-<service> step runs before each service that has prepare.

hydra relies on the process environment winning over .env files. That holds for Next.js and for godotenv; check it for other loaders.

Slots and ports

Slots live in hydra-slots.json inside the repository's shared .git directory, guarded by a file lock. A worktree keeps its slot across runs, and the entry of a deleted worktree is dropped.

When the config loads, hydra works out every service's port for every slot up to max_slots and fails if two of them meet or if one lands on a reserved port. Pick a slot_stride larger than the distance between your closest base ports, and list the ports of shared infrastructure in reserved_ports.

$ hydra slots              # which worktree has which slot
$ hydra slots free         # free this worktree's slot

Stack options

The top-level x-hydra block. A file without it runs unchanged.

KeyDefaultDescription
base_refstringmainThe branch a worktree is compared with. If it tracks a remote branch, that one counts too.
slot_strideinteger1000Port offset per slot. Slot N listens on port + N × slot_stride.
max_slotsinteger20How many worktrees can hold a slot. Port collisions are checked for all of them.
reserved_portslist of integers[]Ports used by shared services or anything else on the machine. No slot may land on one.
hoststring127.0.0.1The host used to build upstream addresses.
port_envstringPORTThe variable a service reads its port from, unless the process sets its own.
global_pathslist of globs[]Files that count as a change to every service that declares paths, such as a lockfile.

Process options

The x-hydra block on a process. A process without one is treated as shared and only runs in the baseline.

KeyDefaultDescription
portintegernoneThe port the baseline's copy listens on. Leave it out for services that don't listen, such as workers.
port_envstringstack'sThe variable this service reads its port from.
pathslist of globs[]Files that belong to the service, relative to the repository root. A change to one makes the service run in the worktree.
upstreamsmap{}Environment variable → the service it points at. Write API_URL: api, or API_PORT: { service: api, value: "{port}" } to change what the variable receives. A service may point at itself.
urlstringhttp://{host}:{port}What an upstream variable pointing at this service receives. It doesn't have to be a URL.
baseline_urlstringurlReplaces url for the baseline's copy, for values that don't come from a port.
sharedbooleanfalseAlways use the baseline's copy. A change to its paths is reported as a warning.
followbooleanfalseRun in a worktree whenever a local service depends on it.
preparestringnoneA shell command run once before the service starts in a worktree, for example to create the queue or namespace named by url.

Values that aren't URLs

"Where a service lives" is not always a port. A worker that reads jobs from a queue lives at a queue name. Give it a url built from the slot, and whoever enqueues jobs gets the right name:

worker:
  x-hydra:
    paths: ["services/worker/**"]
    url: "jobs-{slot}"        # slot 2 reads from jobs-2
    baseline_url: jobs        # the baseline reads from jobs
    prepare: "queue create {value}"

web:
  x-hydra:
    upstreams:
      JOBS_QUEUE: worker      # jobs-2 if the worker runs here, else jobs

Placeholders and globs

Placeholders

Available in url, baseline_url, prepare and an upstream's value.

PlaceholderBecomes
{host}The stack's host.
{port}The service's port in this slot.
{slot}The slot number. 0 for the baseline.
{value}In prepare only: the service's own rendered url.

Globs

Patterns in paths and global_paths match paths relative to the repository root, with / as the separator.

PatternMatches
*Anything within one path segment.
**Any number of segments.
?One character.
{a,b}Either alternative. Not nested.
dir/The same as dir/**.
!patternExcludes matching files, wherever it appears in the list.

Per-worktree settings

A git-ignored .hydra.local.yaml at the root of a worktree adjusts what that worktree runs. The watcher re-reads it, and the stack file, on every tick, so edits apply without a restart.

.hydra.local.yaml
parent: b1               # the branch this worktree is stacked on
with: [web]              # also run these here, changed or not
without: [worker]        # never run these here
routes:
  web: { api: baseline } # web's upstreams that point at api use the baseline's

What a worktree runs

  1. Services whose paths match a changed file, or global_paths for services that declare paths.
  2. Services named in with, or with hydra up NAME.
  3. Services with a route override, plus the target of a local override.
  4. Follow services that one of those depends on.
  5. Minus shared services and anything in without.

Routing

By default an upstream points at the worktree's copy if that service runs in the worktree, and at the baseline's otherwise. A route overrides that for one service and one target:

$ hydra plan --route web.api=baseline    # web here, but against the baseline API
$ hydra plan --route web.api=local       # also starts api here

The --route flag previews a route with plan and render. To run with one, put it under routes in .hydra.local.yaml.

Two frontends that share a backend each have their own upstream, so one can use the local API while the other stays on the baseline.

Command line

hydra stands in for process-compose. Arguments it doesn't know pass through.

CommandWhat it does
hydraStarts this checkout with the TUI. Everything in the main checkout, only what changed in a worktree.
hydra up -DStarts it in the background.
hydra up webAlso runs web here, changed or not. The watcher keeps running.
hydra attachOpens the TUI of this checkout's running instance.
hydra downStops this checkout's instance only.
hydra process …Any other process-compose command, sent to this checkout's instance.
hydra planShows what this checkout runs, on which ports, wired to what, and why.
hydra render [-o FILE]Prints or writes the generated process-compose file.
hydra slotsLists worktree slots.
hydra slots free [PATH]Frees a worktree's slot. The default is this worktree.

Flags for plan and render

FlagMeaning
-f FILEThe stack file. The default is process-compose.yaml in the checkout.
--root DIRThe checkout to plan for. The default is the one containing the current directory.
--slot NPlan as if this were slot N. 0 is the baseline.
--changed PATHSComma-separated changed paths, instead of asking git.
--with SVCAlso run this service here. Repeatable.
--without SVCNever run this service here. Repeatable.
--route S.U=TPoint service S's upstream U at T, which is baseline or local.

Set HYDRA_DRY_RUN=1 to print the process-compose command instead of running it.

Variables every local service receives

VariableValue
PORT or port_envThe service's port in this slot, if it has a port.
Each upstream variableThe address of the local or baseline copy of its target.
HYDRA_SERVICEThe service's name.
HYDRA_SLOTThe slot number.
HYDRA_UPSTREAMSThe names of its upstream variables, comma-separated.

Limits

  • The database is shared. A branch that adds a migration still uses the baseline's database, and hydra warns about it. A database per worktree is planned.
  • Scheduled jobs. A worker that sets up its own schedules on startup does so in every worktree it runs in.
  • Hard-coded ports. A service has to read its port and its upstreams from the environment. A port written into code or a dev script needs a small patch first.
  • Paths are written by hand. They aren't derived from a dependency graph yet, so a shared package has to be listed on each service that imports it.
  • Tested with process-compose v1.124 on macOS.

More: the source and the design notes, which cover open problems and decisions.