vCluster Platform Demo Repository

This repository template is used to create vCluster Platform Demo environments via GitOps and provides selectable demo use cases as code. The repository includes a vcluster-gitops directory that preconfigures a vCluster Platform Demo environment (a templated vCluster) and serves as an example of managing the vCluster Platform with GitOps using Argo CD.

vCluster Platform Integration Examples

• vCluster Platform GitOps: Mainly used to create and update vCluster Platform custom resources to preconfigure the vCluster Platform Demo environment. The resources are deployed to the vCluster Platform Demo vCluster via Argo CD.
• Argo CD: Argo CD is used for vCluster Platform Demo GitOps, to install additional selectable demo use cases as code.
• Using vNode with vCluster
• Crossplane
• External Secrets Operator
• vCluster Central Admission Control with Kyverno
• vCluster Integration with Flux
• MySQL Operator
• Postgres Operator
• vCluster Virtual Scheduler with Volcano Scheduler and KubeRay

Argo CD Integrations

vCluster Platform includes an Argo CD integration that will automatically add a vCluster instance, created with a virtual cluster template, to Argo CD as a target cluster of an Argo CD Application destination.

Example management.loft.sh/v1 VirtualClusterTemplate manifest (with unrelated configuration execluded - full version here) that enables the automatic syncing of the vCluster instance created with the template to Argo CD:

kind: VirtualClusterTemplate
apiVersion: management.loft.sh/v1
metadata:
  name: preview-template
spec:
  displayName: vCluster.Pro Preview Template
  template:
    metadata:
      labels:
        loft.sh/import-argocd: 'true'
...

The virtual cluster template integration requires the vCluster Platform Project, where the vCluster instance is created from said template, to have the Argo CD integration for Projects enabled.

Example management.loft.sh/v1 Project manifest (with unrelated configuration execluded - full version here) that enables the syncing of vCluster instances to Argo CD:

kind: Project
apiVersion: management.loft.sh/v1
metadata:
  name: api-framework
spec:
  displayName: API Framework
...
  argoCD:
    enabled: true
    cluster: loft-cluster
    namespace: argocd
    project:
      enabled: true

Important

The Argo CD instance must be in a vCluster Platform connected cluster or in a vCluster instance that is managed by vCluster Platform. More info is available here.

Example: ApplicationSet Cluster Generator

>[!IMPORTANT] >The vCluster Platform Argo CD integration, as described above, must be enabled on the vCluster Platform project the vCluster instance is created in, for the vCluster instance to be automatically added to Argo CD as an available `Application` `destination` cluster.

In addition to automatically adding/syncing vCluster instances to Argo CD, the vCluster Platform integration also syncs instanceTemplate labels of a virtual cluster template to the Argo CD cluster Secret generated by the integration discussed above. This integration allows the use of the labels as selectors with the Argo CD Cluster Generator for ApplciationSets.

Example management.loft.sh/v1 VirtualClusterTemplate manifest (with unrelated configuration execluded - full version here) that enables the automatic syncing of vCluster instances created with this template to Argo CD and adds the spec.versions.template.metadata.labels to the generate Argo CD Cluster Secret:

apiVersion: management.loft.sh/v1
kind: VirtualClusterTemplate
metadata:
  name: vcluster-pro-template
  labels:
    app.kubernetes.io/instance: loft-configuration
spec:
  displayName: Virtual Cluster Pro Template
...
  template:
...
  versions:
    - template:
        metadata:
          labels:
            loft.sh/import-argocd: 'true'
        instanceTemplate:
          metadata:
            labels:
              env: '{{ .Values.env }}'
              team: '{{ .Values.loft.project }}'
        pro:
          enabled: true
...
      parameters:
      ...
        - variable: env
          label: Deployment Environment
          description: Environment for deployments for this vCluster used as cluster label for Argo CD ApplicationSet Cluster Generator
          options:
            - dev
            - qa
            - prod
          defaultValue: dev
      version: 1.0.0
    - template:
        metadata: {}
        instanceTemplate:
          metadata: {}
      version: 0.0.0
...

In this example the value for the instanceTemplate.metadata.labels.env label is populated with the selected env parameter value, but the value also be hardcoded so that every vCluster instance created from this template had the same env label value. The team label is populated with the project vCluster Platform Parameter values as documented here.

The generated Argo CD Cluster Secret for a vCluster instance created in the api-framework project and using the above template:

apiVersion: v1
kind: Secret
metadata:
  name: loft-api-framework-vcluster-api-framework-dev
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    env: dev
    loft.sh/vcluster-instance-name: api-framework-dev
    loft.sh/vcluster-instance-namespace: loft-p-api-framework
    team: api-framework
  annotations:
    co-managed-by: loft.sh
    managed-by: argocd.argoproj.io
data:
  config: >-
    ...
  name: bG9mdC1hcGktZnJhbWV3b3JrLXZjbHVzdGVyLWFwaS1mcmFtZXdvcmstZGV2
  server: >-
    ...
type: Opaque

With all of that in place, you would then be able to create an Argo CD ApplicationSet that used the Cluster Generator as below (replacing necessary values with those for your Git repository):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: REPO_NAME-env-config
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            env: "dev"
    - clusters:
        selector:
          matchLabels:
            env: "qa"
    - clusters:
        selector:
          matchLabels:
            env: "prod"
  template:
    metadata:
      # {{name}} is the name of the kubernetes cluster as selected by the spec above
      name: REPO_NAME-{{name}}
    spec:
      destination:
        # {{server}} is the url of the 
        server: '{{server}}'
        # {{metadata.labels.env}} is the value of the env label that is being used to select kubernetes clusters 
        # and used as sub-folder in the target git repository
        namespace: hello-world-app-{{metadata.labels.env}}
      info:
        - name: GitHub Repo
          value: https://github.com/loft-demos/REPO_NAME/
      project: default
      source:
        path: k8s-manifests/{{metadata.labels.env}}/
        repoURL: https://github.com/loft-demos/REPO_NAME.git
        targetRevision: main
      syncPolicy:
        automated:
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

[!NOTE] The use of the env label as part of the spec.template.spec.source.path allowing vCluster instances with different env values to target different subdirectories in the GitHub repository for the Argo CD generated Application.

The resulting Argo CD Application for the hello-app-a1 repository:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-app-a1-config
  namespace: argocd
spec:
  destination:
    namespace: hello-world-app
    server: >-
      https://a1.us.demo.dev/kubernetes/project/api-framework/virtualcluster/api-framework-dev
  info:
    - name: GitHub Repo
      value: https://github.com/loft-demos/hello-app-a1/
  project: default
  source:
    path: k8s-manifests/dev/
    repoURL: https://github.com/loft-demos/hello-app-a1.git
    targetRevision: main
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Ephmeral Kubernetes Clusters for Pull Requests

Creating an ephemeral Kubernetes cluster for every GitHub pull request introduces several challenges across infrastructure, security, cost, and operational management. Here are some key considerations:

1. Infrastructure & Performance Challenges

• Cluster Provisioning Overhead: Spinning up a new Kubernetes cluster for every PR requires compute resources and may lead to long provisioning times.
• Resource Contention: In high-traffic repositories, multiple PRs could overwhelm the available infrastructure, impacting CI/CD pipeline performance.
• Storage Management: Persistent storage can be complex in ephemeral environments, especially for stateful applications requiring data retention.

2. Cost & Resource Management

• Compute Costs: Creating a cluster per PR consumes CPU, memory, and networking resources, leading to potential cost overruns if not properly managed.
• Idle Resource Waste: If PR clusters are not properly cleaned up after merging or closing, they may linger and increase cloud costs unnecessarily.

3. Security & Access Control

• Multi-Tenancy Risks: Running multiple ephemeral environments on shared infrastructure requires strict network isolation and RBAC policies to prevent unauthorized access.
• OIDC & Authentication Complexity: Ensuring secure authentication for every ephemeral instance (especially in a dynamic environment) adds operational overhead.
• GitHub Webhook Security: The automation that triggers ephemeral clusters must be secured to avoid unauthorized cluster creation or code execution.

4. CI/CD Pipeline Complexity

• State & Data Persistence: Testing in ephemeral clusters requires handling temporary databases, caching strategies, and external dependencies.
• PR Environment Lifecycle Management: Automating cleanup after PR closure is crucial but can be error-prone, leading to leftover resources.
• Secret Management: Each ephemeral cluster may need different secrets or environment variables, requiring a secure and automated way to inject them.

5. Integration with Argo CD & Crossplane

• Application Deployment Latency: Argo CD’s sync processes and Crossplane’s provisioning may introduce delays when spinning up environments.
• GitHub Rate Limits: Frequent interactions with GitHub (e.g., setting up webhooks, fetching repo state) can hit API rate limits, slowing down the process.
• Ingress & Networking: Exposing services from ephemeral clusters requires proper Ingress configuration, which can be tricky when scaling across multiple PR environments.

Mitigation Strategies

• Use vCluster Instead of Full Kubernetes Clusters: vCluster with vCluster Platform reduces resource overhead with features like Sleep Mode, while still providing isolation.
• Implement Auto-Cleanup & TTLs: Use vCluster Platform Auto Delete to automatically delete ephemeral virtual clusters after a set time to avoid resource waste.
• Optimize CI/CD Pipelines: Use caching and incremental builds to speed up provisioning and reduce redundant cluster setups.
• Enforce Strong RBAC & Network Policies: Secure ephemeral virtual clusters with least privilege access and strict namespace isolation with vCluster Platform Virtual Cluster Templates.
• Monitor & Optimize Costs: Leverage the vCluster Platfrom Cost Control Dashboard to track and optimize ephemeral vCluster expenses.