Appearance
GitOps cluster layout
Context
ADR#6561361204 places what a controller reconciles under devops/infra/gitops/clusters/<cluster>/<kind>/<name>/ and leaves <kind> undefined. It also says "one Kustomization per directory", which is ambiguous: kustomize has a kustomization.yaml, and Flux has a Kustomization custom resource that points at a directory holding one.
A cluster has to apply things in an order the tree can show: a CustomResourceDefinition before the objects that use it, a controller before its configuration, configuration before the workloads that need it. Flux documents several repository shapes and chooses none. They converge on the same tiers and on a cluster being a directory (Flux repository structure).
Pruning is the other risk. A Flux Kustomization with prune: true deletes whatever leaves the tree, which is how a namespace, a CustomResourceDefinition or a volume disappears with a careless commit.
Resolution
Clusters and kinds
You MUST make each cluster one directory,
devops/infra/gitops/clusters/<cluster>/.<cluster>MUST equal the name of the Terragrunt unit that creates the cluster, so one name finds both. An environment is a cluster, never a branch.You MUST group units under one of these kinds, and MUST NOT add another. A unit that fits none is an
applicationsunit.Kind Holds crdsCustomResourceDefinitions and nothing else controllersOperators and controllers, with their namespaces configsCustom resources that a controller's CRDs define applicationsWorkloads that people or other systems use
Units
You MUST give each unit its own directory,
<kind>/<name>/, where<name>is the kebab-case product or upstream name.You MUST put in a unit
ks.yaml, the FluxKustomizationobjects that reconcile it, andapp/, the payload.app/MUST hold akustomization.yamlthat lists every file, and MUST NOT encode order in file names. A unit whose order matters inside MAY add a sibling directory per stage, such asdatabase/beforeapp/, each stage with its ownKustomizationchained bydependsOn.You MUST list each unit's
ks.yamlin<kind>/kustomization.yaml, and the four kinds inclusters/<cluster>/kustomization.yaml.txtdevops/infra/gitops/clusters/<cluster> ├── kustomization.yaml # lists the four kinds └── <kind>/ ├── kustomization.yaml # lists each unit's ks.yaml └── <name>/ ├── ks.yaml # Flux Kustomization(s) └── app/ ├── kustomization.yaml # kustomize └── ... # the payloadYou MUST name the Flux
Kustomizationof theapp/stage<kind>-<name>, such ascrds-cert-managerorapplications-memorizer, and a stage other thanapp/<kind>-<name>-<stage>. Flux object names are unique per namespace, and a controller and its configuration share<name>; leading with the kind keeps a unit namedcert-manager-crdsin one kind from colliding withcert-managerin another. You MUST keep Flux objects influx-system.
Ordering
- You MUST order units only with
dependsOn, naming each dependency explicitly. The kind alone orders nothing. - A unit MUST NOT depend on a unit of a later kind in the order
crds,controllers,configs,applications, and the graph MUST be acyclic. - A unit that others depend on MUST set
wait: trueand atimeout, so "ready" means healthy and not only applied.
Delete safety
- You MUST create each new
Kustomizationwithprune: false, and MAY change it totrueonly in a later change, after a render shows what a prune would delete. - You MUST add
kustomize.toolkit.fluxcd.io/prune: disabledto every Namespace, CustomResourceDefinition, PersistentVolumeClaim and database cluster, so that removing it from Git, or its unit withprune: true, never deletes it. A unit made only of those, such as everycrdsunit, MAY keepprune: falsefor good. - The annotation stops Flux pruning only. Removing a
HelmReleaseruns a Helm uninstall, which deletes what the chart created whatever Flux says. You MUST also sethelm.sh/resource-policy: keepon every such object a chart creates, through the chart values or a post-renderer, and MUST NOT remove aHelmReleasethat holds data in the same change that changes anything else about it. - Removing such an object is a deliberate act: remove the annotations in one change, and the object in the next.
Bootstrap and sources
- You MUST have the Flux Operator and its
FluxInstanceinstalled by Terraform (ADR#6561361204), withsync.pathset todevops/infra/gitops/clusters/<cluster>on the default branch,provider: githuband a GitHub App credential in thepullSecret. You MUST NOT use a personal token. Flux then reconciles itself from the tree, and the tree MUST NOT hold the operator or theFluxInstanceagain. - You MUST deliver a Helm chart that a cluster runs from the team's own OCI registry, pinned to an exact version (ADR#6497710059).
- You MUST pull a chart through an
OCIRepositorywhoserefsets both thetagand thedigest, and point theHelmReleaseat it withchartRef. AHelmReleasechart.spec.versioncannot carry a digest, so it does not meet the deployment pin rule. - You MAY isolate teams with Flux multi-tenancy lockdown later, by a follow-up ADR. Until then every unit is reconciled from
flux-systemas cluster-admin.
Migration
- You MAY keep a legacy GitOps tree, such as an existing Argo CD one, outside this layout while units move into it one at a time. You MUST NOT require a single change that moves everything.
- A unit MUST live in exactly one place, and exactly one controller MUST reconcile it at any time. Disabling deletion stops pruning but not reconciliation, and two controllers do not see one commit at the same moment, so a single change that moves a unit is not a handover. Hand it over in order:
- Stop the old controller for the unit: turn off its sync and deletion, then release the unit without deleting its objects.
- Verify the old controller no longer reconciles or owns the unit.
- Add the unit to the new tree with
prune: false, after a render shows no change to the live objects.
Links
- ADR#6561361204: Live infrastructure
- ADR#6497710059: OCI artifact paths and tags
- Flux repository structure
- Flux Kustomization
- Flux Operator FluxInstance