Cloud-pipeline Azure installation and set up instructions

deploy/contents/k8s/azure-specific/Installation instructions.md

@@ -0,0 +1,177 @@
+# Installation Instructions
+**Note:** The instructions are provided for _azure.cloud-pipeline.com_ DNS name case.
+
+### AKS cluster
+Make sure cluster subnet security group allows exposure of cloud resources to the Internet
+
+### Database server
+Azure Database for PostgreSQL flexible server
+If it is a private server, create Private endpoint to allow from AKS virtual network to access the server 
+(Networking -> Create private endpoint)
+
+Put admin username and password to a key vault
+
+### Database
+Connect to postgres schema, admin user with admin password from the key vault
+```
+CREATE USER pipeline CREATEDB;
+ALTER USER pipeline WITH password 'pipeline';
+CREATE DATABASE pipeline OWNER pipeline;
+```
+Put connection string to config map: PSG_HOST key in cp-config-global.yaml
+
+### Label nodes
+```
+az aks nodepool update --resource-group <resource-group-name> --cluster-name <cluster-name> --name <node-pool-name> --labels cloud-pipeline/cp-edge="true" cloud-pipeline/cp-api-srv="true" cloud-pipeline/cp-idp="true" cloud-pipeline/region=<region-name> cloud-pipeline/cp-docker-registry="true" cloud-pipeline/scp-executor="true"
+```
+
+### Create Azure cloud credentials secret
+1. Create a new App registration (cloud-pipeline)
+```
+az ad sp create-for-rbac --name cloud-pipeline --role Contributor --scopes /subscriptions/<subscription-id> --sdk-auth > azureAuth.json
+```
+2. Make sure the file is in UTF-8
+3. Put the file to the secret (mounted as /root/.cloud, so the file path is /root/.cloud/azureAuth.json)
+```
+kubectl create secret generic cp-cloud-credentials --from-file="azureAuth.json"
+```
+
+### Create cluster ssh key secret
+```
+ssh-keygen -t rsa -b 4096 -f ./id_rsa_k8s -C "k8s-ssh-key"
+
+kubectl create secret generic cp-cluster-ssh-key --from-file=id_rsa=id_rsa_k8s --from-file=id_rsa.pub=id_rsa_k8s.pub
+```
+
+### Create config map
+1. Set CP_CLOUD_REGION_ID and PSG_HOST values
+2. Create config map
+```
+kubectl create configmap cp-config-global --from-env-file=cp-config-global.yaml
+```
+
+### Create persistent static volume for certificates
+```
+AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -n <storage-account-name> -g <resource-group-name> -o tsv)
+az storage share create -n cloud-pipeline-k8s --connection-string $AZURE_STORAGE_CONNECTION_STRING
+
+az storage account keys list --resource-group <resource-group-name> --account-name <storage-account-name> --query "[0].value" -o tsv
+kubectl create secret generic azure-secret --from-literal=azurestorageaccountname=<storage-account-name> --from-literal=azurestorageaccountkey=<storage-account-key>
+```
+change volumeHandle in azure-files-pv if needed
+```
+kubectl create -f azure-files/azure-files-pv.yaml
+kubectl apply -f azure-files/azure-files-mount-options-pvc.yaml
+kubectl get pvc azurefile	  
+az storage share show --name cloud-pipeline-k8s --account-name <storage-account-name> --account-key <storage-sccount-key> --output table
+```
+
+### Set cluster roles
+```
+kubectl apply -f cluster-role/cluster-role.yaml
+kubectl apply -f cluster-role/cluster-role-binding.yaml
+```
+
+### Deploy API
+```
+kubectl apply -f cp-api-srv/cp-api-srv-dpl.yam
+kubectl apply -f cp-api-srv/cp-api-srv-svc-ingress.yaml
+```
+Put CP_API_JWT_ADMIN to config map
+
+### Create Public IP
+Public IP name should be then specified in cp-edge-svc+lb.yaml under service.beta.kubernetes.io/azure-pip-name annotation
+currently public-IP-name = cloud_pipeline_public_IP
+```
+az network public-ip create --name <public-IP-name> --resource-group <cluster-resource-group-name> --allocation-method Static --sku Standard
+```
+Set all external hosts in config map and in dns table
+Example
+```
+<Public-IP> edge.azure.cloud-pipeline.com
+<Public-IP> idp.azure.cloud-pipeline.com
+<Public-IP> docker.azure.cloud-pipeline.com
+<Public-IP> git.azure.cloud-pipeline.com
+<Public-IP> azure.cloud-pipeline.com
+```
+
+### Setup kube coredns
+https://learn.microsoft.com/en-us/azure/aks/coredns-custom
+1. Modify core-dns-ms.yaml
+2. Apply
+```		  
+kubectl apply -f core-dns-ms.yaml
+```
+3. Restart coredns
+
+### Deploy CP Edge
+```
+kubectl apply -f cp-edge/cp-edge-dpl.yaml
+kubectl apply -f cp-edge/cp-edge-svc+lb.yaml
+```
+Make sure CP_EDGE_CLUSTER_RESOLVER is the same as
+```
+kubectl exec -it <cp-edge-pod-name> -- cat /etc/resolv.conf
+```
+Make sure cp-edge service has labels:  
+```
+kubectl label svc cp-edge "cloud-pipeline/external-host=edge.azure.cloud-pipeline.com"
+kubectl label svc cp-edge "cloud-pipeline/external-port=443"
+kubectl label svc cp-edge "cloud-pipeline/external-scheme=https"
+```
+
+### Deploy IDP
+```
+kubeclt apply -f cp-idp/cp-idp-dpl.yaml
+kubeclt apply -f cp-idp/cp-idp-svc-ingress.yaml
+```
+on cp-idp pod:
+```
+curl  "https://cp-idp.default.svc.cluster.local:32080/metadata" -o "cp-api-srv-fed-meta.xml"  -H "Host: idp.azure.cloud-pipeline.com:443" -s -k
+```
+put the file to /opt/api/sso for idp and api containers (azure DS), port should be 443 
+
++ idp_pd="${CP_IDP_PROFILE_DB:-/opt/idp/pdb/saml-idp-profiles.json}"
++ cert="/opt/api/pki/sso-public-cert.pem"
++ issuer="https://azure.cloud-pipeline.com:443/pipeline/"
+```
+saml-idp add-connection "https://azure.cloud-pipeline.com:443/pipeline/" -c "/opt/api/pki/sso-public-cert.pem" --profileDatabase "/opt/idp/pdb/saml-idp-profiles.json"
+```
+
+### Docker registry
+1. Deploy docker registry:
+```
+kubeclt apply -f cp-docker-registry/cp-docker-registry-dpl.yaml
+kubeclt apply -f cp-docker-registry/cp-docker-registry-svc-ingress.yaml
+```
+
+2. Make docker registry available from cluster virtual network on Azure dns resolver:
+
+Create private DNS Zone(azure.cloud-pipeline.com) and record (docker -> service cluster IP) and link to the cluster VNet(make sure the VNet was defined correctly)
+https://learn.microsoft.com/en-us/azure/dns/private-dns-getstarted-portal
++ DNS Zone: azure.cloud-pipeline.com
++ Record: docker, type: A, IP address: cp-docker-registry service cluster IP, 
+
+**Note:** For self-signed ssl certificate, add ssl certificate to a node trusted store:
+
+Put docker-public-cert.pem.crt content to the end of registry-ca-ds.yaml file
+```
+kubeclt apply -f registry-ca-ds.yaml
+```
+3. Register docker registry docker.azure.cloud-pipeline.com:443 in UI with pipe_admin jwt token as a password
+
+### Create Azure region
++ Name: <region-pretty-name>
++ Storage account: <storage-account-name>
++ Storage account key: <storage-account-key>
++ SSH Public Key Path: /opt/api/pki/id_rsa_k8s.pub
++ Meter Region Name: <region-name>
++ Azure API Url: https://management.azure.com/
++ Price Offer ID: 1
++ Resource group: <cluster-resource-group-name>
++ Auth file: /root/.cloud/azureAuth.json
+
+### System Preferences
++ base.pipe.distributions.url = https://azure.cloud-pipeline.com:443/pipeline/
++ base.api.host = https://azure.cloud-pipeline.com:443/pipeline/restapi/
++ base.api.host.external = https://azure.cloud-pipeline.com:443/pipeline/restapi/

deploy/contents/k8s/azure-specific/Prerequisites.md

@@ -0,0 +1,9 @@
+## Prerequisites
+1. SSL Certificates
+2. AKS cluster
+3. PostgreSQL flexible server
+4. Azure storage for persistent volume to keep secrets and certificates
+5. Service account(principal or App registration) - Azure service account and azure secret file to connect to Azure resources
+6. Public DNS names
+7. Public IP in the cluster resource group 
+8. Git repository

deploy/contents/k8s/azure-specific/cluster-role/cluster-role-binding.yaml

@@ -0,0 +1,13 @@
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: sa-cluster-reader
+  namespace: cloud-pipeline-test
+subjects:
+  - kind: ServiceAccount
+    name: <cp-service-account>
+    namespace: cloud-pipeline-test
+roleRef:
+  kind: ClusterRole
+  name: node-reader
+  apiGroup: rbac.authorization.k8s.io

deploy/contents/k8s/azure-specific/cluster-role/cluster-role.yaml

@@ -0,0 +1,9 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: node-reader
+  namespace: cloud-pipeline-test
+rules:
+- apiGroups: [""]
+  resources: ["nodes"]
+  verbs: ["get", "list", "watch"]

deploy/contents/k8s/azure-specific/core-dns-ms.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: coredns-custom
+  namespace: kube-system
+data:
+    test.override: |
+          hosts { 
+              <Cluster-IP> <cp-edge-external-host>
+              <Cluster-IP> <cp-api-srv-external-host>
+              <Cluster-IP> <Public-IP>
+              fallthrough
+          }

deploy/contents/k8s/azure-specific/cp-api-srv/cp-api-srv-dpl.yaml

@@ -0,0 +1,170 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: cp-api-srv
+  namespace: cloud-pipeline-test
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      cloud-pipeline/core-component: "Deployment"
+      cloud-pipeline/cp-api-srv: "true"
+  template:
+    metadata:
+      namespace: cloud-pipeline-test
+      labels:
+        cloud-pipeline/cp-api-srv: "true"
+        cloud-pipeline/core-component: "Deployment"
+        azure.workload.identity/use: "true"
+    spec:
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  cloud-pipeline/cp-api-srv: "true"
+              topologyKey: kubernetes.io/hostname
+      nodeSelector:
+        cloud-pipeline/cp-api-srv: "true"
+      tolerations:
+        - key: node-role.kubernetes.io/master
+          effect: NoSchedule
+      terminationGracePeriodSeconds: 50
+      serviceAccountName: <cp-service-account>
+      containers:
+        - name: cp-api-srv-leader-elector
+          image: quay.io/lifescience/cloud-pipeline:leader-elector-0.17
+          imagePullPolicy: "Always"
+          securityContext:
+            privileged: true
+          command: [ "/elector/init" ]
+          ports:
+            - containerPort: 4040
+          env:
+            - name: HA_ELECTION_SERVICE_NAME
+              value: "cp-api-srv"
+            - name: HA_ELECTION_PERIOD_SEC
+              value: "5"
+            - name: HA_VOTE_EXPIRATION_PERIOD_SEC
+              value: "7"
+          envFrom:
+            - configMapRef:
+                name: cp-config-global
+        - name: cp-api-srv
+          image: quay.io/lifescience/cloud-pipeline:api-srv-0.17
+          imagePullPolicy: "Always"
+          securityContext:
+            privileged: true
+          command: ["/init-api"]
+          ports:
+            - containerPort: 8080
+          env:
+            - name: CP_KUBE_EXTERNAL_HOST
+              value: ""
+            - name: CP_KUBE_EXTERNAL_PORT
+              value: "6443"
+            - name: CP_API_SRV_INTERNAL_HOST
+              value: cp-api-srv.default.svc.cluster.local
+            - name: CP_API_SRV_INTERNAL_PORT
+              value: "31080"
+            - name: CP_KUBE_MASTER_CHECK_URL
+              value: http://127.0.0.1:4040/
+            - name: CP_HA_DEPLOY_ENABLED
+              value: "true"
+            - name: CP_API_CURRENT_POD_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.name
+            - name: CP_API_CURRENT_NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+          envFrom:
+          - configMapRef:
+              name: cp-config-global
+          volumeMounts:
+            - mountPath: /opt/api/pki
+              subPath: api/pki
+              name: cp-system
+            - mountPath: /opt/api/sso
+              subPath: api/sso
+              name: cp-system
+            - mountPath: /var/log/cp-api
+              subPath: api/logs
+              name: cp-system
+            - mountPath: /opt/api/ext/templates/pipe-templates
+              subPath: api/templates/pipe-templates
+              name: cp-system
+            - mountPath: /opt/api/ext/templates/folder-templates
+              subPath: api/templates/folder-templates
+              name: cp-system
+            - mountPath: /opt/api/ext/templates/error-templates
+              subPath: api/templates/error-templates
+              name: cp-system
+            - mountPath: /opt/api/ext/templates/static
+              subPath: api/templates/static
+              name: cp-system
+            - mountPath: /opt/api/etc
+              subPath: api/etc
+              name: cp-system
+            - mountPath: /opt/idp/pki
+              subPath: idp/pki
+              name: cp-system
+            - mountPath: /opt/common/pki
+              subPath: common/pki
+              name: cp-system
+              readOnly: true
+            - mountPath: /opt/gitlab/pki
+              subPath: gitlab/pki
+              name: cp-system
+              readOnly: true
+            - name: cp-cloud-credentials
+              mountPath: "/root/.cloud"
+              readOnly: true
+              # Using /etc/cp_ssh instead of a "general" ~/.ssh/ because it may break some of the openssh-created files (e.g. known hosts)
+              # as this volumes is mounted "read only"
+            - name: cp-cluster-ssh-key
+              mountPath: "/etc/cp_ssh"
+              readOnly: true
+          readinessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - /liveness-api-srv.sh
+            initialDelaySeconds: 180
+            periodSeconds: 10
+          livenessProbe:
+            exec:
+              command:
+                - /bin/sh
+                - -c
+                - /liveness-api-srv.sh
+            # 5 min delay to make sure API is up and running and drop pod after 60 seconds (4*15s) if API didn't respond
+            initialDelaySeconds: 180
+            periodSeconds: 15
+            failureThreshold: 4
+          lifecycle:
+            preStop:
+              exec:
+                command: ["sleep", "10"]
+      dnsConfig:
+        options:
+          - name: ndots
+            value: "1"
+      volumes:
+        - name: cp-system
+          persistentVolumeClaim:
+            claimName: cp-system-pvc
+        - name: kube-config
+          hostPath:
+            path: /root/.kube
+        - name: cp-cloud-credentials
+          secret:
+            secretName: cp-cloud-credentials
+        - name: cp-cluster-ssh-key
+          secret:
+            secretName: cp-cluster-ssh-key
+            # Kube accepts mode in decimal. So 384 gives 600, which is required for SSH keys
+            defaultMode: 384
+