TLDR

The problem our app-of-apps setup required hidden knowledge (folder conventions, template patterns) that made it unintuitive for new developers.

What we evaluated:

  • Terraform modules
  • Helm charts
  • Terraform Operator
  • KRO

Each solved some of the problem but introduced new pain (slow plans, lock-in, operational complexity, missing features).

What worked Crossplane XRDs + Compositions to build XNamespace and XArgoApp abstractions. One YAML file per team, no hidden patterns. We ended up with faster ArgoCD syncs, smaller codebase, new user members productive on day one.

What i tried to solve

The existing deployment pipeline was built for a narrow usecase and we had outgrown it. It had too many hidden requirements and it wasnt clear what each piece actually did. Especially for developers who just wanted to ship their application. It was even hard to explain when you knew how it worked.

Some things I wanted to make it easier developers was to make it more intutive, because our current implemtation had alot of hidden requirements and it wasnt clear what actually did what things.

So i wrote down some pain points we would like to have fixed.

  1. Intuitive for GitOps newcomers - someone with basic git knowledge should be able to read a manifest and understand what it does.
  2. No hidden patterns - no implicit folder structures, no conventions you can only learn by reading the templates or just experience.
  3. Extensible - we know there will be use cases we havent thought of, so the system needs to leave the door open.

Terraform

I looked into terraform as i had good experience with it before and theoreticly it could do everything a user and we wanted to do.

Terraform modules: was a good fit, we could make a ArgoCD Application module that contains all the things we would need e.g.

  • ArgoCD
    • Application
    • Project
  • Kubernetes resources
    • Rolebindings
    • ResourceQuotas
    • NetworkPolicies
    • (and more)

As I wrote the logic of the modules we could basicly support anything that we wanted!

But… there were several issues. None were dealbreakers on their own, but when they added up it felt like I was forcing Terraform to do something it wasnt designed for.

For example:

  • The tfstate file grew and as many other people have experienced the time to run a plan + deploy takes to long
  • Spliting tfstate file could increase speed but came at a cost in complexity and would take away some of terraforms strenghts.

When using terraform, I essentialy just moved parameters to another place and that dosent solve anything. Here is a example of it could have looked like

# This could be really nice interface!
---
roles:
  redis-admin: admin
  redis-developer: editor
  redis-viewer: viewer

networkPolicies:
  allowAllInsideNamespace: true

sources:
  app:
    repository: https://github.com/org/repo
    chart: charts/app
    valueFiles:
      - values.yaml
      - values-dev.yaml
  redis:
    repository: https://charts.bitnami.com/bitnami
    chart: redis
    releaseName: single
    target_revision: 21.2.6
    values:
      global:
        redis:
          password: redis
      architecture: standalone

Abstractions and using the right tooling

Simplicity = Solve one thing really good, easy to use. Dosent scale beyond its simple usecase e.g docker compose

Complexity = Solves more things better, harder to use. Scales and solves more usecases e.g kubernetes

It felt like I just replaced helm based app-of-apps with terraform and didnt solve anything really :/

So i took a rethink and came up with this

  • Kubernetes as the control plane is the right way.
  • Everyone that works with kubernetes has touched helm.
  • We should use reconciliation as it will remove extra workflows to maintain.
  • We should still like to package multiple “things” we call “thing” as “one thing”
  • We where still using only simple tools, we needed more complex and powerfull tooling. Kinda the same principle as using a library in programing.

Contenders

Helm charts

Create a helm chart for different abstractions like what we call namespace and argocd application.

  • - More complex code is to hard to read.
  • - Requires us to build a artifact everytime.
  • - Still using the same tool for everything.
  • - Can only manage things in kubernetes.

Terraform operator

  • + Supports more complex code
  • + Can manage stuff outside of kubernetes!
  • + Could have super nice interface!
  • - Hard to read.
  • - Just replaces the tooling.
  • - terraform needs to have 100% controll, a big lock in.

KRO

  • - Too early in its lifecycle and had no loops or didnt handle deletion of old resources. Hard dealbreaker

Crossplane

  • + Supports multiple languages like gotemplating, python and there own patch based aproach.
  • + Kubernetes native, have their own crd that will live in kubernetes.
  • + Could manage stuff outside of kubernetes.
  • - People that have used it seem to have a great dislike for it.

Using crossplane

Crossplane Terminology when you get this its much easier to understand the docs.

TermDescription
CompositeResourceDefinition (XRD)Defines the schema for a composite basicly their version of the CRD
Composite Resource (XR)The same things as custom resource, and is just the name for the manifest you will create
CompositionThis contains the logic (pipeline of steps). Will take the XR and template your code
FunctionThink of this as what library or what language you want to use

After creating a simple resource for a namespace with the go-templating. I instantly fell in love, it was super intuitve after writing so many helm charts. Now i could create simple manifests that handled all the complexity for me.

Namespace

Now to create whats calle a “namespace” actually contains what we would expect a namespace to contain!

---
# Minimal version
apiVersion: exempel.se/v1
kind: XNamespace
metadata:
  name: my-namespace
spec:
  owner:
    team: my-team
    email: team@exempel.se
    systemOwner: owner@exempel.se
  roles:
    my-team: edit
---
# A more realistic example
apiVersion: exempel.se/v1
kind: XNamespace
metadata:
  name: my-namespace
spec:
  owner:
    team: my-team
    email: team@exempel.se
    systemOwner: owner@exempel.se
  roles:
    my-team: edit
  quota:
    resources: large
    storage: 100Gi

ArgoCD Applications

Now when you talk about using ArgoCD you usually just talk about the applications crd. Now we have a actuall manifest that configures what you intuitvely describe!

---
apiVersion: example.se/v1
kind: XArgoApp
metadata:
  name: my-app
spec:
  groups:
    my-app-group: admin
  source:
    repoURL: https://github.com/example/my-app
    path: deploy/my-app/dev  # Enviroment specific path
    valueFiles:
      - values.yaml

The implementation creates a application and project in background and makes the interface much easier for a developer that will for 99% of the time only interact with the application crd from the ui.

App-of-apps

Combine this with removing all the resource creation from the app-of-apps helm chart. Im left with this super simple single file chart :D

{{- if not .Values.cluster }}
  {{- fail "cluster variable must be set" }}
{{- end }}


{{- $argoApps := dict }}
{{- $namespaces := dict }}

{{- range $path, $_ := .Files.Glob (printf "app-of-apps/%s/**.yaml" .Values.cluster) }}
  {{/*
  Get the content of the file and split it into documents
  */}}

  {{- $content := $.Files.Get $path }}
  {{- $documents := splitList "---" $content }}

  {{/*
  Now loop over each manifest present in the file.
  */}}

  {{- range $documents }}
    {{- $yamlToVars := . | fromYaml }}

    {{/*
    1. Check what kind of manifest it is
    2. If it is a manifest we want to include we do the following:
      2.1. Check if it has any duplicates inside the dictionary, if it has fail!
      2.2. If it has no duplicates, add it to the dictionary.
    */}}
    {{- if and $yamlToVars (eq $yamlToVars.kind "XNamespace") }}
      {{- if hasKey $namespaces $yamlToVars.metadata.name }}
        {{- fail (printf "Duplicate XNamespace found: %s in %s" $yamlToVars.metadata.name $path) }}
      {{- end }}
      {{- $_ := set $namespaces $yamlToVars.metadata.name . }}
    {{- end }}

    {{- if and $yamlToVars (eq $yamlToVars.kind "XArgoApp") }}
      {{- if hasKey $argoApps $yamlToVars.metadata.name }}
        {{- fail (printf "Duplicate XArgoApp found: %s in %s" $yamlToVars.metadata.name $path) }}
      {{- end }}
      {{- if $yamlToVars.metadata.namespace }}
        {{- fail (printf "metadata.namespace is not allowed for XArgoApp: %s" $yamlToVars.metadata.name) }}
      {{- end }}
      {{- $_ := set $argoApps $yamlToVars.metadata.name . }}
    {{- end }}

  {{- end }}{{/* end of range documents */}}
{{- end}}{{/* end of range files */}}


{{- range $argoAppName, $content := $argoApps }}
{{ $content  }}
---
{{- end }}

{{- range $namespaceName, $content := $namespaces }}
{{ $content  }}
---
{{- end }}

And how do we install it the the cluster? Simple like this!

---
apiVersion: kim.karolinska.se/v1
kind: XArgoApp
metadata:
  name: app-of-apps
  namespace: argocd
spec:
  roles:
    ocp-admin: admin
  source:
    repoURL: https://github.com/org/app-of-apps
    path: ArgoCD
    values:
      cluster: production
  clusterResourceWhitelist:
    - group: "kim.karolinska.se"
      kind: "XNamespace"
    - group: "kim.karolinska.se"
      kind: "XArgoApp"

Directory structure

One super nice thing about this approach is if we install this chart under ArgoCD/ in the root of the repository. There is no hard requirements for directories after ArgoCD/app-of-apps/<cluster>.

ArgoCD/
  Chart.yaml
  templates/
    app-of-apps.yaml  # app-of-apps chart
  app-of-apps/
    <cluster>/        # Directorie after this point is purely for better organization
      <organization>/
        <team-or-domain>/
          <application-name>.yaml

I want a namespace

Create some subdirectories for organizational purposes

export ORG=name-of-org
export TEAM=name-of-team
export CLUSTER=name-of-cluster
mkdir -p ArgoCD/app-of-apps/${CLUSTER}/${ORG}/${TEAM}

Place a file under the created directory.

---
apiVersion: example.se/v1
kind: XNamespace
metadata:
  name: namespace-name
spec:
  owner:
    team: name-of-team
    systemOwner: first lastname <first.lastname@example.se>
    email: contact@example.se # Example a shared inbox
  roles:
    my-team: edit
  quota:
    resources: small
    storage: 50G

I also want to deploy my application using ArgoCD

---
apiVersion: example.se/v1
kind: XNamespace
metadata:
  name: namespace-name
  labels:
    openshift.io/user-monitoring: "true"
spec:
  owner:
    team: name-of-team
    systemOwner: first lastname <first.lastname@example.se>
    email: contact@example.se # Example a shared inbox
  roles:
    my-team: edit
---
apiVersion: example.se/v1
kind: XArgoApp
metadata:
  name: namespace-name
spec:
  roles:
    my-team: edit
  source:
    repoURL: https://github.com/org/my-app
    path: deploy/my-app/dev
    valueFiles:
      - values.yaml

Result

  • Faster ArgoCD because we dont have to keep track of N templates created from the top down, only one XRD per resource. The instance frequently used to OOM now it never comes close.
  • Intuitive developer experince. We spend barely a day explaining the preferd gitops strategy
  • Smaller codebase refactored into easy to understand segments. Before the app-of-apps tried to do everything and made us scared of making improvement as it was hard to predict how it will happen.
  • Flexibility for both helm, kustomize and totaly manual if required. Before you where locked into a domain specific directory structure with mandatory workflows. Now we directly template whats in the developers self-managed git repositories.

Lesson learned!

Using more complex tools can make it simpler when used in the right places!