[{"content":"TLDR The problem our app-of-apps setup required hidden knowledge (folder conventions, template patterns) that made it unintuitive for new developers.\nWhat we evaluated:\nTerraform modules Helm charts Terraform Operator KRO Each solved some of the problem but introduced new pain (slow plans, lock-in, operational complexity, missing features).\nWhat worked Crossplane XRDs + Compositions to build XNamespace and XArgoApp abstractions. One YAML file per team, no hidden patterns. We ended up with faster ArgoCD syncs, smaller codebase, new user members productive on day one.\nWhat i tried to solve The existing deployment pipeline was built for a narrow usecase and we had outgrown it. It had too many hidden requirements and it wasnt clear what each piece actually did. Especially for developers who just wanted to ship their application. It was even hard to explain when you knew how it worked.\nSome things I wanted to make it easier developers was to make it more intutive, because our current implemtation had alot of hidden requirements and it wasnt clear what actually did what things.\nSo i wrote down some pain points we would like to have fixed.\nIntuitive for GitOps newcomers - someone with basic git knowledge should be able to read a manifest and understand what it does. No hidden patterns - no implicit folder structures, no conventions you can only learn by reading the templates or just experience. Extensible - we know there will be use cases we havent thought of, so the system needs to leave the door open. Terraform I looked into terraform as i had good experience with it before and theoreticly it could do everything a user and we wanted to do.\nTerraform modules: was a good fit, we could make a ArgoCD Application module that contains all the things we would need e.g.\nArgoCD Application Project Kubernetes resources Rolebindings ResourceQuotas NetworkPolicies (and more) As I wrote the logic of the modules we could basicly support anything that we wanted!\nBut\u0026hellip; there were several issues. None were dealbreakers on their own, but when they added up it felt like I was forcing Terraform to do something it wasnt designed for.\nFor example:\nThe tfstate file grew and as many other people have experienced the time to run a plan + deploy takes to long Spliting tfstate file could increase speed but came at a cost in complexity and would take away some of terraforms strenghts. When using terraform, I essentialy just moved parameters to another place and that dosent solve anything. Here is a example of it could have looked like\n# This could be really nice interface! --- roles: redis-admin: admin redis-developer: editor redis-viewer: viewer networkPolicies: allowAllInsideNamespace: true sources: app: repository: https://github.com/org/repo chart: charts/app valueFiles: - values.yaml - values-dev.yaml redis: repository: https://charts.bitnami.com/bitnami chart: redis releaseName: single target_revision: 21.2.6 values: global: redis: password: redis architecture: standalone Abstractions and using the right tooling Simplicity = Solve one thing really good, easy to use. Dosent scale beyond its simple usecase e.g docker compose\nComplexity = Solves more things better, harder to use. Scales and solves more usecases e.g kubernetes\nIt felt like I just replaced helm based app-of-apps with terraform and didnt solve anything really :/\nSo i took a rethink and came up with this\nKubernetes as the control plane is the right way. Everyone that works with kubernetes has touched helm. We should use reconciliation as it will remove extra workflows to maintain. We should still like to package multiple \u0026ldquo;things\u0026rdquo; we call \u0026ldquo;thing\u0026rdquo; as \u0026ldquo;one thing\u0026rdquo; We where still using only simple tools, we needed more complex and powerfull tooling. Kinda the same principle as using a library in programing. Contenders Helm charts\nCreate a helm chart for different abstractions like what we call namespace and argocd application.\n- More complex code is to hard to read. - Requires us to build a artifact everytime. - Still using the same tool for everything. - Can only manage things in kubernetes. Terraform operator\n+ Supports more complex code + Can manage stuff outside of kubernetes! + Could have super nice interface! - Hard to read. - Just replaces the tooling. - terraform needs to have 100% controll, a big lock in. KRO\n- Too early in its lifecycle and had no loops or didnt handle deletion of old resources. Hard dealbreaker Crossplane\n+ Supports multiple languages like gotemplating, python and there own patch based aproach. + Kubernetes native, have their own crd that will live in kubernetes. + Could manage stuff outside of kubernetes. - People that have used it seem to have a great dislike for it. Using crossplane Crossplane Terminology when you get this its much easier to understand the docs.\nTerm Description CompositeResourceDefinition (XRD) Defines the schema for a composite basicly their version of the CRD Composite Resource (XR) The same things as custom resource, and is just the name for the manifest you will create Composition This contains the logic (pipeline of steps). Will take the XR and template your code Function Think of this as what library or what language you want to use After creating a simple resource for a namespace with the go-templating. I instantly fell in love, it was super intuitve after writing so many helm charts. Now i could create simple manifests that handled all the complexity for me.\nNamespace Now to create whats calle a \u0026ldquo;namespace\u0026rdquo; actually contains what we would expect a namespace to contain!\n--- # Minimal version apiVersion: exempel.se/v1 kind: XNamespace metadata: name: my-namespace spec: owner: team: my-team email: team@exempel.se systemOwner: owner@exempel.se roles: my-team: edit --- # A more realistic example apiVersion: exempel.se/v1 kind: XNamespace metadata: name: my-namespace spec: owner: team: my-team email: team@exempel.se systemOwner: owner@exempel.se roles: my-team: edit quota: resources: large storage: 100Gi ArgoCD Applications Now when you talk about using ArgoCD you usually just talk about the applications crd. Now we have a actuall manifest that configures what you intuitvely describe!\n--- apiVersion: example.se/v1 kind: XArgoApp metadata: name: my-app spec: groups: my-app-group: admin source: repoURL: https://github.com/example/my-app path: deploy/my-app/dev # Enviroment specific path valueFiles: - values.yaml The implementation creates a application and project in background and makes the interface much easier for a developer that will for 99% of the time only interact with the application crd from the ui.\nApp-of-apps Combine this with removing all the resource creation from the app-of-apps helm chart. Im left with this super simple single file chart :D\n{{- if not .Values.cluster }} {{- fail \u0026#34;cluster variable must be set\u0026#34; }} {{- end }} {{- $argoApps := dict }} {{- $namespaces := dict }} {{- range $path, $_ := .Files.Glob (printf \u0026#34;app-of-apps/%s/**.yaml\u0026#34; .Values.cluster) }} {{/* Get the content of the file and split it into documents */}} {{- $content := $.Files.Get $path }} {{- $documents := splitList \u0026#34;---\u0026#34; $content }} {{/* Now loop over each manifest present in the file. */}} {{- range $documents }} {{- $yamlToVars := . | fromYaml }} {{/* 1. Check what kind of manifest it is 2. If it is a manifest we want to include we do the following: 2.1. Check if it has any duplicates inside the dictionary, if it has fail! 2.2. If it has no duplicates, add it to the dictionary. */}} {{- if and $yamlToVars (eq $yamlToVars.kind \u0026#34;XNamespace\u0026#34;) }} {{- if hasKey $namespaces $yamlToVars.metadata.name }} {{- fail (printf \u0026#34;Duplicate XNamespace found: %s in %s\u0026#34; $yamlToVars.metadata.name $path) }} {{- end }} {{- $_ := set $namespaces $yamlToVars.metadata.name . }} {{- end }} {{- if and $yamlToVars (eq $yamlToVars.kind \u0026#34;XArgoApp\u0026#34;) }} {{- if hasKey $argoApps $yamlToVars.metadata.name }} {{- fail (printf \u0026#34;Duplicate XArgoApp found: %s in %s\u0026#34; $yamlToVars.metadata.name $path) }} {{- end }} {{- if $yamlToVars.metadata.namespace }} {{- fail (printf \u0026#34;metadata.namespace is not allowed for XArgoApp: %s\u0026#34; $yamlToVars.metadata.name) }} {{- end }} {{- $_ := set $argoApps $yamlToVars.metadata.name . }} {{- end }} {{- end }}{{/* end of range documents */}} {{- end}}{{/* end of range files */}} {{- range $argoAppName, $content := $argoApps }} {{ $content }} --- {{- end }} {{- range $namespaceName, $content := $namespaces }} {{ $content }} --- {{- end }} And how do we install it the the cluster? Simple like this!\n--- apiVersion: kim.karolinska.se/v1 kind: XArgoApp metadata: name: app-of-apps namespace: argocd spec: roles: ocp-admin: admin source: repoURL: https://github.com/org/app-of-apps path: ArgoCD values: cluster: production clusterResourceWhitelist: - group: \u0026#34;kim.karolinska.se\u0026#34; kind: \u0026#34;XNamespace\u0026#34; - group: \u0026#34;kim.karolinska.se\u0026#34; kind: \u0026#34;XArgoApp\u0026#34; Directory structure One super nice thing about this approach is if we install this chart under ArgoCD/ in the root of the repository. There is no hard requirements for directories after ArgoCD/app-of-apps/\u0026lt;cluster\u0026gt;.\nArgoCD/ Chart.yaml templates/ app-of-apps.yaml # app-of-apps chart app-of-apps/ \u0026lt;cluster\u0026gt;/ # Directorie after this point is purely for better organization \u0026lt;organization\u0026gt;/ \u0026lt;team-or-domain\u0026gt;/ \u0026lt;application-name\u0026gt;.yaml I want a namespace Create some subdirectories for organizational purposes\nexport ORG=name-of-org export TEAM=name-of-team export CLUSTER=name-of-cluster mkdir -p ArgoCD/app-of-apps/${CLUSTER}/${ORG}/${TEAM} Place a file under the created directory.\n--- apiVersion: example.se/v1 kind: XNamespace metadata: name: namespace-name spec: owner: team: name-of-team systemOwner: first lastname \u0026lt;first.lastname@example.se\u0026gt; email: contact@example.se # Example a shared inbox roles: my-team: edit quota: resources: small storage: 50G I also want to deploy my application using ArgoCD --- apiVersion: example.se/v1 kind: XNamespace metadata: name: namespace-name labels: openshift.io/user-monitoring: \u0026#34;true\u0026#34; spec: owner: team: name-of-team systemOwner: first lastname \u0026lt;first.lastname@example.se\u0026gt; email: contact@example.se # Example a shared inbox roles: my-team: edit --- apiVersion: example.se/v1 kind: XArgoApp metadata: name: namespace-name spec: roles: my-team: edit source: repoURL: https://github.com/org/my-app path: deploy/my-app/dev valueFiles: - values.yaml Result Faster ArgoCD because we dont have to keep track of N templates created from the top down, only one XRD per resource. The instance frequently used to OOM now it never comes close. Intuitive developer experince. We spend barely a day explaining the preferd gitops strategy Smaller codebase refactored into easy to understand segments. Before the app-of-apps tried to do everything and made us scared of making improvement as it was hard to predict how it will happen. Flexibility for both helm, kustomize and totaly manual if required. Before you where locked into a domain specific directory structure with mandatory workflows. Now we directly template whats in the developers self-managed git repositories. Lesson learned! Using more complex tools can make it simpler when used in the right places!\n","permalink":"https://blog.timlind.cloud/posts/app-of-apps/","summary":"\u003ch2 id=\"tldr\"\u003eTLDR\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eThe problem\u003c/strong\u003e our app-of-apps setup required hidden knowledge (folder conventions, template patterns) that made it unintuitive for new developers.\u003c/p\u003e\n\u003cp\u003eWhat we evaluated:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eTerraform modules\u003c/li\u003e\n\u003cli\u003eHelm charts\u003c/li\u003e\n\u003cli\u003eTerraform Operator\u003c/li\u003e\n\u003cli\u003eKRO\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eEach solved some of the problem but introduced new pain (slow plans, lock-in, operational complexity, missing features).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhat worked\u003c/strong\u003e Crossplane XRDs + Compositions to build XNamespace and XArgoApp abstractions. One YAML file per team, no hidden patterns. We ended up with faster ArgoCD syncs, smaller codebase, new user members productive on day one.\u003c/p\u003e","title":"ArgoCD + Crossplane = ❤️"}]