Kubernetes 1.14 was released in March 2019 and the release brought production support for Windows Containers on Windows Server nodes. Before moving on, I would like to highlight a few things from the previous link:

  1. Kubernetes control plane runs in Linux (and there is no plan to change that for a full Windows Kubernetes cluster)
  2. Versions supported for worker nodes and containers: Windows Server 1809/Windows Server 2019
  3. Windows containers have to be scheduled on Windows nodes

At the time this post was written (Nov 19), AKS, GKE and EKS offer some level of support for Windows based containers (EKS is the first to offer GA support for Windows based containers). The EKS documentation provides instructions on how to setup Windows nodes using eksctl and EKS. You might want to try that since it is GA. The following are the instructions for AKS 1.14+ (where Windows Containers are still under preview) but the example and instructions related to Jenkins we will use should be able to be used in a similar setup where Kubernetes has a Windows node pool.

Infrastructure Setup and Windows Nodepools

Jenkins has the ability to use Kubernetes pods as agents to build and deploy applications thanks to the Kubernetes plugin. In this blog post, we will create a simple declarative pipeline that has Linux and Windows containers as agents in AKS.

First, we need to follow Azure’s documentation to create the needed infrastructure to be able to deploy Linux and Windows based containers. Please make sure that you review AKS documentation and are aware of the limitations before running this in a production cluster. As explained here, the registration for the preview features cannot be unregistered at this moment.

The documentation at a high level goes through the following steps (using the Azure CLI):

  1. Install aks-preview CLI extension for Azure CLI
  2. Register the Windows preview feature needed for the Windows based containers
  3. Create a new resource group (if needed)
  4. Create an AKS cluster
    • You can use --nodepool-name with aks create cluster to name your control plane node pool (i.e default)
  5. Add a Windows Server node pool
    • This will be a node pool for Kubernetes pod Windows agents, we can name it --name winage

Node pools give us the possibility to extend our Kubernetes cluster with more types of machines depending on our use and budget (see available options and default values) for AKS. As an example, we are going to add two more identical pools (one for masters and one for Linux agents) but you can pick different machine sizes and node counts depending on your need (just make sure that the VMs used for Jenkins masters support Premium Storage as Jenkins requires high IOPS for better performance):

  • Linux Jenkins master pool example:

    az aks nodepool add \ 
--resource-group myResourceGroup \
--cluster-name eastUSAKS \
--os-type Linux \
--name masters \
--node-count 1 \
--kubernetes-version 1.14.6 \
--node-vm-size Standard_DS2_v2

  • Linux Jenkins agent pool example:

    az aks nodepool add \                                                                 --resource-group myResourceGroup \
--cluster-name eastUSAKS \
--os-type Linux \
--name linage \
--node-count 1 \
--kubernetes-version 1.14.6 \
--node-vm-size Standard_DS2_v2

    Once the pools are created, you can see them in the Azure portal:

Jenkins installation using Helm

Helm is the Kubernetes Package Manager and we can use it to install Jenkins using the official chart. If you haven’t installed Helm before, you can follow these instructions to install it. Using a nodeSelector in the values file (values.yaml) used by the chart will allow us to specify in which nodepool Jenkins will be installed (you can find master.nodeSelector option in the Jenkins chart link). For simplicity, we are only going to configure the values.yaml file so that it deploys Jenkins using such nodeSelector option but the file can include a lot more options. In this case, we need to make sure that Jenkins runs in the nodepool named masters (AKS assigns the nodepool name as the value of the agentpool tag, more on this in the next section)

  • values.yaml
master:
  nodeSelector:
    agentpool: masters

  • Installing the chart (add --namespace yourNamespace to the command if you want to deploy Jenkins in a specific namespace):

    helm install --name jenkins -f values.yaml stable/jenkins

The version that was installed in this example is 2.190.2.

Once installed, follow the “NOTES” section in the console that will allow you to get your Jenkins (user: admin) password and URL. It will include something similar to this:

printf $(kubectl get secret jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo 

export SERVICE_IP=$(kubectl get svc jenkins --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")

echo http://$SERVICE_IP:8080/login

You should be able to access Jenkins with the provided URL at this point.

Creating the pipeline with Windows and Linux Containers

Make sure to check the version of the Kubernetes plugin installed. This example uses the version 1.21.0 which supports the Windows container step.

  • Let’s create a pipeline by clicking "New Item", then enter a name for your pipeline job (i.e "win-lin-pipeline") and select "Pipeline" as your job type.

  • Select:

  • Click Save

Let’s take a look at the repository structure in https://github.com/mluyo3414org/pod-templates.git:


├── Jenkinsfile
├── README.md
├── linux
│   └── nodejs-pod.yaml
└── windows
    └── dotnet-pod.yaml

  • Both nodejs-pod.yaml and dotnet-pod.yaml are files describing the Kubernetes pod agents used in the Jenkinsfile.

  • The dotnet-pod.yaml has two container definitions: a Windows based jnlp(jenkins/jnlp-agent:latest-windows) and a windows-dotnet container (mcr.microsoft.com/dotnet/core/sdk:2.1). We need to overwrite the jnlp container in this pod since otherwise it will use the Linux based jnlp container defined under Jenkins --> Configuration --> Pod templates. In this case, Jenkins was automatically configured with the Linux pod: jenkins/jnlp-slave:3.27-1.

kind: Pod
metadata:
  name: windows
spec:
  containers:
  - name: jnlp
    image: jenkins/jnlp-agent:latest-windows
    tty: true
  - name: windows-dotnet
    image: mcr.microsoft.com/dotnet/core/sdk:2.1
    tty: true
  nodeSelector:
    agentpool: winage
  • The nodejs-pod.yaml has the node(nodeJS) container definition and will use the default Linux based jnlp (jenkins/jnlp-slave:3.27-1) mentioned before.
kind: Pod
metadata:
  name: nodejs-app
spec:
  containers:
  - name: nodejs
    image: node:slim
    command:
    - cat
    tty: true
  nodeSelector:
    agentpool: linage

  • Notice both pod yaml definitions use nodeSelector (nodeSelector) to decide where these pods should be scheduled. If this is not specified, the Kubernetes scheduler will provision the pods following the default behavior which could possibly start a pod in a node with the wrong OS. The tags used for the nodeSelectors are the default tags assigned by Azure when specifying the nodepool name: agentpool : nameOfNodePool. To find the node tags you can use the command: kubectl get nodes --show-labels. Other options to prevent scheduling errors are taints (more info).

    Scheduling error when not using nodeSelectors. Debug using kubectl describe podname

  • Jenkinsfile:

pipeline {
  agent none
  options { 
    buildDiscarder(logRotator(numToKeepStr: '2'))
    skipDefaultCheckout true
  }
  stages {
    stage('Test-linux') {
      agent {
        kubernetes {
          label 'nodejs-pod'
          yamlFile 'linux/nodejs-pod.yaml'
        }
      }
      steps {
        checkout scm
        container('nodejs') {
          echo 'Hello World!'   
          sh 'node --version'
        }
      }
    }
    stage('Test-windows') {
      agent {
        kubernetes {
	  label 'windows-pod'
          yamlFile 'windows/dotnet-pod.yaml'
        }
      }
      steps {
        bat 'dir'
        container(name:'windows-dotnet'){
          bat 'dotnet -h'
      } 
     }
    }
  }
}

  • This is a declarative pipeline using agent none so that we can specify agents per stage. yamlFile is used to read the pod template from a file location which is also in this repo. You can either define the pod template in another file, in the same Jenkinsfile or in the Jenkins Configuration page. More info about the syntax used in this pipeline can be found here.

  • The node --version command is executed inside the container step otherwise it will get executed inside the jnlp (Linux) container and fail as it doesn’t have node installed. Similarly bat 'dir' gets executed in the jnlp (Windows) container and bat 'dotnet -h' is executed inside the dotnet container.

  • Here is another example on how to use a Windows container using a scripted pipeline and tested in EKS.

Conclusion

In this post we were able to discuss how you can take advantage of Jenkins and Kubernetes to use Windows containers as part of your CI/CD pipelines. The architecture and process discussed in this post are depicted by this high-level diagram of the Kubernetes cluster and some of its components.

Listen to Windows Server Containers on the Kubernetes Podcast for more information.