I recently faced a problem when trying to build a Docker image using the @pulumi/docker provider.
In this Blog Post I explain my journey on how I use the Pulumi Operator to build Docker Images and what are the problems I encountered.
If you simply want the solution, click here to scroll down
import * as docker from '@pulumi/docker'
import * as kubernetes from '@pulumi/kubernetes'
const image = new docker.Image("app", {
build: {
context: "../",
args: {
BUILDKIT_INLINE_CACHE: "1",
},
cacheFrom: {
images: ["ghcr.io/kerwanp/pulumi-preview-test:latest"]
}
},
imageName: "ghcr.io/kerwanp/pulumi-preview-test:latest",
})
const appLabels = { app: "preview-test" }
const deployment = new kubernetes.apps.v1.Deployment("app", {
spec: {
replicas: 1,
selector: { matchLabels: appLabels },
template: {
metadata: { labels: appLabels },
spec: {
imagePullSecrets: [
{
name: 'regcred'
}
],
containers: [
{
name: "app",
image: image.imageName,
imagePullPolicy: 'Always',
}
]
}
}
}
})
And then deploy it by creating a stack using the Pulumi Operator:
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
name: app-stack
namespace: pulumi
spec:
envRefs:
PULUMI_ACCESS_TOKEN:
type: Secret
secret:
name: pulumi-secret
key: pulumiAccessToken
gitAuth:
basicAuth:
password:
type: Secret
secret:
name: pulumi-secret
key: githubToken
userName:
type: Literal
literal:
value: kerwanp
stack: kerwanp/preview-test/test
projectRepo: https://github.com/kerwanp/pulumi-preview-test
repoDir: deploy
branch: "refs/heads/main"
destroyOnFinalize: true
The problem
When the Pulumi Operator tries to deploy the stack it throws the following error:
failed to run update: exit status 255code: 255
stdout: Updating (test)
View Live: https://app.pulumi.com/kerwanp/preview-test/test/updates/1
+ pulumi:pulumi:Stack preview-test-test creating (0s)
@ Updating......
+ pulumi:pulumi:Stack preview-test-test creating (2s) panic: runtime error: invalid memory address or nil pointer dereference
+ pulumi:pulumi:Stack preview-test-test creating (2s) [signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x124befc]
+ pulumi:pulumi:Stack preview-test-test creating (2s) goroutine 27 [running]:
+ pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi-docker/provider/v4.dockerHybridProvider.Configure({{}, {0x21b7ca0, 0x26157, 0x26157}, {0x17dd3fc, 0x6}, {0x17fe470, 0xc0000d1400}, {0x17fe3c0, 0xc0000b0690}}, ...)
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/work/pulumi-docker/pulumi-docker/provider/hybrid.go:93 +0x17c
+ pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler.func1({0x17f1f08, 0xc00042e6f0}, {0x14c4e00?, 0xc0007d7900})
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:462 +0x78
+ pulumi:pulumi:Stack preview-test-test creating (2s) github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x17f1f08, 0xc0006ffe90}, {0x14c4e00, 0xc0007d7900}, 0xc000136ac0, 0xc00043eca8)
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8
+ pulumi:pulumi:Stack preview-test-test creating (2s) github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler({0x153bb40?, 0xc0007644b0}, {0x17f1f08, 0xc0006ffe90}, 0xc0005da8c0, 0xc000653960)
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:464 +0x138
+ pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).processUnaryRPC(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0xc00075cba0, 0x21f98e8, 0x0)
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
+ pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).handleStream(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0x0)
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
+ pulumi:pulumi:Stack preview-test-test creating (2s) google.golang.org/grpc.(*Server).serveStreams.func1.2()
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x98
+ pulumi:pulumi:Stack preview-test-test creating (2s) created by google.golang.org/grpc.(*Server).serveStreams.func1
+ pulumi:pulumi:Stack preview-test-test creating (2s) /home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
docker:index:Image app error: error reading from server: EOF
docker:index:Image app **failed** 1 error
+ pulumi:pulumi:Stack preview-test-test created (2s) 19 messages
Diagnostics:
docker:index:Image (app):
error: error reading from server: EOF
pulumi:pulumi:Stack (preview-test-test):
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x29 pc=0x124befc]
goroutine 27 [running]:
github.com/pulumi/pulumi-docker/provider/v4.dockerHybridProvider.Configure({{}, {0x21b7ca0, 0x26157, 0x26157}, {0x17dd3fc, 0x6}, {0x17fe470, 0xc0000d1400}, {0x17fe3c0, 0xc0000b0690}}, ...)
/home/runner/work/pulumi-docker/pulumi-docker/provider/hybrid.go:93 +0x17c
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvide
r_Configure_Handler.func1({0x17f1f08, 0xc00042e6f0}, {0x14c4e00?, 0xc0007d7900})
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:462 +0x78
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x17f1f08, 0xc0006ffe90}, {0x14c4e00, 0xc0007d7900}, 0xc000136ac0, 0xc00043eca8)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/grpc-opentracing@v0.0.0-20180507213350-8e809c8a8645/go/otgrpc/server.go:57 +0x3e8
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Configure_Handler({0x153bb40?, 0xc0007644b0}, {0x17f1f08, 0xc0006ffe90}, 0xc0005da8c0, 0xc000653960)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/v3@v3.64.0/proto/go/provider_grpc.pb.go:464 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0xc00075cba0, 0x21f98e8, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc000234000, {0x17fa100, 0xc000702b60}, 0xc0003c5e60, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
Resources:
+ 1 created
Duration: 4s
stderr: warning: A new version of Pulumi is available. To upgrade from version '3.66.0' to '3.68.0', visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
The Debuging Process
Replicating the issue
I opened a shell on the Pulumi Operator Pod to see if I can replicate the pulumi up inside the container.
I found that the Operator cloned my repository inside /tmp/pulumi-working/<stack-name> I jumped into it and simply ran pupumi up.
Problem...
getcwd() failed: No such file or directory
This error usually appears when the folder you are in does not exist anymore, so I decided to duplicate it and re-run pulumi up.
And voilà, error replicated!
Docker is simply not there..
If I try to build the Docker image by myself directly from the container I receive the following error:
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
It makes completely sense! If you do not have a Docker Daemon running, you cannot build a Docker image.
The solution
The idea is to add a docker:dind container inside the Pulumi Operator pod to be used as a Docker Daemon.
Starting in
18.09+the Docker Dind automatically generate TLS certificates. We have to mount a directory accross the containers to share thoses.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: pulumi-operator-fc266171
namespace: pulumi
spec:
replicas: 1
selector:
matchLabels:
name: pulumi-kubernetes-operator
template:
spec:
volumes:
- name: dind-storage
emptyDir: {}
- name: docker-certs-client
emptyDir: {}
containers:
- name: dind
image: docker:24-dind
resources: {}
volumeMounts:
- name: dind-storage
mountPath: /var/lib/docker
- name: docker-certs-client
mountPath: /certs/client
# ^^^^^^
# Mount volume to share certificates to the operator
securityContext:
privileged: true
# ^^^^
# Must be defined to run Docker
- name: pulumi-kubernetes-operator
image: pulumi/pulumi-kubernetes-operator:v1.12.0
args:
- "--zap-level=error"
- "--zap-time-encoding=iso8601"
env:
- name: DOCKER_HOST
value: 'tcp://localhost:2376'
# ^^^^
# Define address for remote Docker Daemon
- name: DOCKER_CERT_PATH
value: '/certs'
# ^^^^
# Define custom certificate directory (mounted volume)
- name: DOCKER_TLS_VERIFY
value: 1
# ^^^^
# Force Docker client to use TLS
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: OPERATOR_NAME
value: pulumi-kubernetes-operator
- name: GRACEFUL_SHUTDOWN_TIMEOUT_DURATION
value: 5m
- name: MAX_CONCURRENT_RECONCILES
value: "10"
resources: {}
volumeMounts:
- name: docker-certs-client
mountPath: /certs
# ^^^^^^
# Mount volume to get Docker certificates
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
You can actually check that it works by running docker info inside the Operator Pod.
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.4
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.17.3
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 24.0.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1677a17964311325ed1c31e2c0a3589ce6d5c30d
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
seccomp
Profile: builtin
Kernel Version: 5.15.90.1-microsoft-standard-WSL2
Operating System: Alpine Linux v3.18 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.32GiB
Name: pulumi-operator-anaxago-fc266171-6f896556dc-lxkvp
ID: 32511430-499e-4fa7-bbce-9b801ffb77ec
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Tada! 🎉 You can now build images using Pulumi and the Pulumi Operator!
I hope that this Blog Post helped you! If you have any questions, feel free to use the comment section! 💬
Oh and if you want more content like this, follow me: