back end dev, creative technology
Running Containers on k8s
Getting through the confusion of Kubernetes

Small Bits, Big Scale

For the past year, since working on a Kubernetes-based experience for Next 2018, [Kubernetes] and its sidekick [Istio] have been a huge mental block. Docker makes sense, and docker-compose is intuitive enough for someone like me to make work. But k8s is wrapped in jargon and huge API that makes it daunting, and that's a huge disservice. Kubernetes is a powerful tool for infrastructure automation and scaling, and after the initial legwork can make app dev much easier. No more shitty sysadmin bash scripts to help standup servers.

This is a collection of the process I used to stand up my first real "cluster," and some solid boilerplate yaml that I'll use moving forward. It's still rough, and it'll probably grow over time.

Currently, this only runs on Docker and GCP's GKE.

The gist of it is:

  • Write code and Dockerfiles
  • Push code to Google Source Repos
  • Tag containers with GCR and build with Cloud Build
  • Make k8s deployments and services, and expose a load balancer
  • Rinse, repeat

Setup

Kubernetes Setup

Create cluster:

gcloud beta container clusters create csp-city-test --project=$PROJECT_ID \
    --addons=Istio --istio-config=auth=MTLS_STRICT \
    --cluster-version=1.11.6-gke.6 \
    --machine-type=n1-standard-2 \
    --num-nodes=4 \
    --zone=us-west1-a

Generate creds:

gcloud container clusters get-credentials csp-city-test --zone=us-west1-a --project=$PROJECT_ID

Check services, get external IP:

kubectl get service -n istio-system
Start a Cloud Repo

The process is basically the same as git, create a repo here and add it as a remote for any project.

Create repo:

gcloud source repos create REPO_NAME

Commit to the Google Repo:

add .
commit "init"
git push --all google
Setup Docker-k8s Integration

This step is to automate the build process. We have to tell docker to use Google's Container Registry instead of Docker's, and use this for building images.

gcloud auth configure-docker
gcloud components install docker-credential-gcr
docker-credential-gcr configure-docker
cat creds.json | docker login -u _json_key --password-stdin https://gcr.io
Build Images

Build Image:

docker tag <repo> gcr.io/<gcp project name>/<repo>

gcloud builds submit "$(pwd)/<dockerfile dir>" --tag=gcr.io/<gcp project name>/<dockerfile dir>

Deploying to k8s

Create some config files to use, two deployments and two services. We'll use an nginx load balancer and a simple node server (the same as here), and expose the nginx container to the web.

Config YAML

Server

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.17.0 ()
  creationTimestamp: null
  labels:
    io.kompose.service: server
    app: server
  name: server
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: server
        app: server # We need this name to link it to nginx
      name: server
    spec:
      containers:
      - env:
        - name: MESSAGE
          value: no. 1
        image: <google container registry image> # UPDATE THIS!!!
        name: server
        resources: {}
        tty: true
      restartPolicy: Always
status: {}

Service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.17.0 ()
  creationTimestamp: null
  labels:
    io.kompose.service: server
  name: server
spec:
  selector:
    app: nginx # The name of the nginx container to link too, in this case "nginx"
  ports:
  - name: "8000"
    port: 8000 # The port the server is listening on
    targetPort: 8000
  selector:
    io.kompose.service: server
status:
  loadBalancer: {}
NGINX

Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.17.0 ()
  creationTimestamp: null
  labels:
    io.kompose.service: nginx
  name: nginx # The name used in the server configs
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: nginx
    spec:
      containers:
      - image: <google container registry image> # UPDATE THIS!!!
        name: nginx
        ports:
        - containerPort: 8080 # The port nginx is listening on
        resources: {}
        tty: true
      restartPolicy: Always
status: {}

Service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.17.0 ()
  creationTimestamp: null
  labels:
    io.kompose.service: nginx
  name: nginx # The name used in the server configs
spec:
  selector:
    app: server # The name of the server
  ports:
  - name: "8080"
    port: 80
    targetPort: 8080 # The port nginx is listening on
  selector:
    io.kompose.service: nginx
  type: LoadBalancer
status:
  loadBalancer: {}

Launch!

kubectl create -f <yaml>

Teardown

for ns in $(kubectl get ns --output=jsonpath={.items[*].metadata.name}); do kubectl delete ns/$ns; done;
kubectl delete deployments --all &&  kubectl delete services --all

gcloud container clusters delete csp-city-test --zone=us-west1-a