Everything Localhost Hides From You

magicimg was done. A Bun server that generates images via Flux and DALL-E, hashes the prompts deterministically, caches the results in .cache/, and serves them back by content hash. Locally, everything worked. bun run src/index.ts, hit the endpoint, get an image. Simple.

The goal was to get it live at magicimg.digitalpine.io on the Docker Swarm. What followed was a series of problems that all had the same shape: things that are invisible when you're on localhost become very visible when you're not.

The Container

The Dockerfile is a standard multi-stage Bun build — deps layer, build layer, runtime layer. The interesting bit is that Bun's --frozen-lockfile can read pnpm-lock.yaml natively and migrate it on the fly. No pnpm binary needed in the image. The trade-off: that migration takes ~280 seconds under QEMU amd64 emulation on Apple Silicon. The final bundle is 0.35MB, so the pain is front-loaded.

FROM oven/bun:1-alpine AS deps
COPY package.json pnpm-lock.yaml ./
RUN bun install --frozen-lockfile --production

FROM oven/bun:1-alpine AS build
COPY package.json pnpm-lock.yaml ./
RUN bun install --frozen-lockfile
COPY src/ ./src/
RUN bun build src/index.ts --outdir dist --target bun

FROM oven/bun:1-alpine
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
RUN mkdir -p /app/.cache/images && chown -R bun:bun /app/.cache
USER bun
CMD ["bun", "dist/index.js"]

Named volume at /app/.cache for image persistence across redeploys. Traefik labels route magicimg.digitalpine.io to port 3456. Node constraint pins it to the home PC (WSL) where the volume lives. All standard Swarm stuff.

The first deploy command: docker stack deploy -c docker-compose.yml magicimg. Fails immediately.

Problem 1: Your Credentials Don't Travel

"No such image: ghcr.io/digitalpine/magicimg:latest"

On localhost, docker pull uses your local credentials from docker login. In a Swarm, the node that actually pulls the image is a different machine — it doesn't have your credentials. This is obvious in retrospect, but nothing about the local dev experience surfaces it.

The fix is --with-registry-auth, which forwards your local Docker credentials to whatever node needs to pull:

docker stack deploy -c docker-compose.yml magicimg --with-registry-auth

Container starts. Health check passes. First image request fails.

Problem 2: The Secrets Gap

The compose file has BFL_API_KEY=${BFL_API_KEY} and OPENAI_API_KEY=${OPENAI_API_KEY}. Shell interpolation. Locally, these are in your .env and Bun auto-loads them. But docker stack deploy runs in a shell where those vars don't exist. They interpolate to empty strings. The container boots, the health check passes (it doesn't need API keys), and every actual image request returns:

{"error": "Image generation failed", "message": "OPENAI_API_KEY environment variable is required"}

Silent failure. The compose file doesn't warn you when a variable resolves to nothing.

First instinct: Docker secrets. The "proper" Swarm approach — mount secrets as files at /run/secrets/<name>, read them at startup. I wrote the reader, added the compose secrets: section, then realized that creating secrets means piping each key manually via docker secret create. For two API keys that already live in .env on the deploy machine, this felt like infrastructure ceremony for infrastructure ceremony's sake.

What actually worked: dotenv-cli. One line:

npx dotenv-cli -- docker stack deploy -c docker-compose.yml magicimg --with-registry-auth

It loads .env, exports the vars, then runs whatever follows --. Docker compose sees them, interpolates them into the container environment. Done.

But the code should be agnostic. What if I do switch to proper secrets later? The final env.ts resolves from both sources — Docker secrets first (check /run/secrets/), then fall back to process.env:

function resolve(name: string): string | undefined {
  const secretPath = `/run/secrets/${name.toLowerCase()}`;
  if (existsSync(secretPath)) {
    return readFileSync(secretPath, 'utf-8').trim();
  }
  return process.env[name];
}

const envSchema = z.object({
  PORT: z.coerce.number().default(3456),
  BFL_API_KEY: z.string().optional(),
  OPENAI_API_KEY: z.string().optional(),
}).refine((data) => data.BFL_API_KEY || data.OPENAI_API_KEY, {
  message: 'At least one provider key required',
});

export const env = envSchema.parse({
  PORT: resolve('PORT'),
  BFL_API_KEY: resolve('BFL_API_KEY'),
  OPENAI_API_KEY: resolve('OPENAI_API_KEY'),
});

The Zod refinement means the server crashes at startup if neither provider key exists — fail loud rather than serving 500s. Locally, Bun auto-loads .env so process.env always has them. In Swarm, they arrive via compose interpolation (from dotenv-cli) or Docker secrets (if you set them up). The application doesn't care which.

Problem 3: The Server Doesn't Know Its Own Protocol

Added a /demo route — a playground for testing image generation in the browser. The server-side template computed URLs like this:

const baseUrl = `${url.protocol}//${url.host}`;

Locally, this is http://localhost:3456. Fine. In production, the request path looks like this:

TLS terminates at Traefik. The request that actually reaches Bun is plain HTTP. So url.protocol is http:, the demo page renders fetch URLs as http://magicimg.digitalpine.io/image?..., and the browser blocks them: mixed content (page loaded over HTTPS, fetch target is HTTP).

This is another localhost illusion. Locally, there's no TLS termination layer — what you see is what you get. In production, your server is behind something, and it doesn't know the real protocol unless someone tells it (via X-Forwarded-Proto or similar).

The fix is to stop asking the server what protocol it's on:

// Fetch uses relative path — browser resolves against page's actual protocol
const res = await fetch('/image?' + params.toString());

// Display URL uses the browser's knowledge
const fullUrl = location.origin + '/image?' + params.toString();

The browser always knows the real protocol. The server doesn't need to.

Problem 4: Latest Doesn't Mean Latest

After pushing a new image and running the update:

docker service update --image ghcr.io/digitalpine/magicimg:latest \
  --with-registry-auth magicimg_magicimg

...the old version keeps serving. Docker sees that the tag is the same (latest) and decides nothing changed. It doesn't re-pull. This is mutable tag semantics — the name hasn't changed, so Docker assumes the content hasn't either.

You need --force to make it actually pull and recreate:

docker service update --image ghcr.io/digitalpine/magicimg:latest \
  --with-registry-auth --force magicimg_magicimg

The "correct" solution is content-addressed tags (SHA digests), but for a single-service deploy where you're the only person pushing, --force is the pragmatic answer.

The Deploy Pipeline

One last naming collision: pnpm deploy is a built-in pnpm command (monorepo workspace deployment). It shadows any script named "deploy". Renamed to ship:

"ship": "pnpm docker:build && pnpm docker:push && npx dotenv-cli -- docker service update --image ghcr.io/digitalpine/magicimg:latest --with-registry-auth --force magicimg_magicimg"

Build the amd64 image, push to ghcr.io, force-update the Swarm service with secrets from .env. One command from laptop to production.

The Pattern

Every problem in this deploy was something that worked perfectly on localhost and broke silently in production:

None of these are hard problems. They're all 1-line fixes once you understand what's happening. But they share a property: the local development experience actively hides them from you. The gap between "works on my machine" and "works in production" isn't about complexity — it's about visibility. Production has more layers, and each layer can silently transform or swallow information that localhost gives you for free.