From ECS to a Cloud-Native Platform: the Five Pillars
I drafted the proposal, won leadership buy-in, and set the technical direction that took Nexus from ~75 services on ECS to 250+ on EKS.
When Nexus had grown to roughly 75 microservices, ECS was straining to manage it. I wrote the proposal to move to Kubernetes, took it to leadership, and, once it was accepted, owned the transition end to end. Being in charge meant I could set the technical direction and standards for Armor's new stack, so I anchored it on five pillars from day one: security, cost, observability, GitOps, and self-healing. Each was a decision made at the start, not retrofitted once the platform was already big.
1 · Security from the start
The security posture came straight from my time at CloudCover building secure-by-default landing zones and DevSecOps pipelines: make the secure choice the default, then prove it at every stage. So the platform gates a change from the moment it is committed to the moment it runs, with the foundation underneath scanned just as hard.
Every layer has an owner. Static analysis and dependency scanning run before a build; Trivy scans the built image; the SBOM is Cosign-signed with in-toto attestations; Kyverno decides what is allowed to admit; Falco (soon Tetragon) watches it at runtime; kube-bench audits the cluster; tfsec scans the Terraform and Terragrunt modules; and OpenBao, the External Secrets Operator, and Reloader keep secrets in one place, rotated, and never sprawled across the estate.
A gate at every stage, from a commit to a running pod, plus the foundation underneath. Tap a gate.
Once the image is built, Trivy scans the assembled container for CVEs and misconfigurations.
2 · Cost-aware by design
About 80% of the platform runs on spot, and it stays reliable because a reclaim is engineered to be a non-event. Karpenter watches the native SQS interruption queue and pre-provisions replacements; the parts that must not move, its own controller on a small managed NodeGroup along with CoreDNS, the Cilium controllers, gateways, and databases, stay on on-demand; and every workload carries PodDisruptionBudgets, spread constraints, anti-affinity, and proper SIGTERM handling so a node can vanish without dropping a request.
On top of that, the platform is sized to what it actually uses. OpenCost breaks spend down per namespace, workload, and pod, so Resources, Limits, and LimitRanges get tuned from data rather than guesswork. KEDA-managed HPAs scale most workloads on real signals like RabbitMQ depth (including scale-to-zero for idle connector workers), VPA right-sizes what cannot scale horizontally, a Cluster Proportional Autoscaler tracks platform tooling to cluster size, and cert-manager provisions and rotates certificates across the cluster.
Cheap capacity only works if a reclaim is a non-event. These are the layered defenses that make it one. Tap a defense.
Karpenter watches the native SQS queue for the 2-minute spot interruption notice and provisions a replacement node proactively, so capacity is already there when the old node goes.
3 · Observe everything
Nothing runs blind. Application telemetry goes through the LGTM stack (Loki, Tempo, Mimir, Pyroscope, Grafana), shipped by a single Alloy collector that packages Beyla, with a Grafana MCP so the operator LLMs can query it too. The LLM stack has its own lens: Langfuse traces every call into the Envoy AI Gateway, per harness, capturing tokens per second, latency, cost, and failures.
The mesh moved from Kiali to Hubble when we moved from Istio to Cilium, and runtime security is moving from Falco to Tetragon for the same eBPF datapath and the ability to enforce, not just audit. Alerts tie it together: Prometheus rules fire into AlertManager, Robusta enriches each with a HolmesGPT investigation, and OpsGenie pages the human, who already holds a probable root cause.
Observability is split by what it watches: apps, the LLM stack, the mesh, runtime, and the alerting path. Tap a domain.
The full LGTM stack, shipped by a single Alloy OpenTelemetry collector that packages Beyla for zero-instrumentation eBPF telemetry. A Grafana MCP server lets the LLMs query it too.
4 · GitOps everywhere
Nothing reaches the cluster except through Git. Argo CD reconciles the whole platform, ApplicationSets generate apps automatically from matching repositories, and promotion from develop to production happens entirely through GitHub, with no manual kubectl in the path. The mechanics have their own case study.
No pipeline pushes to the cluster. Click a stage to see what carries the change forward.
- Open a pull request and add the genivi-deploy label. A Merge ApplicationSet, MERGE(SCM + MATRIX(SCM x PR)), folds the PR into that repo's single app: an image tagged test-<commit> is built and the app flips over to the PR, deploying the preview to Dev.
- Remove the label and the preview tears itself down; the app reverts to the last stable state on master. No kubectl, no console, no leftover environments.
5 · Self-healing infrastructure and apps
The last pillar is the payoff of the other four: the platform fixes most of its own problems. A bad deploy never becomes an incident because Argo Rollouts ships it as a canary and the AnalysisRuns auto-abort on elevated 5xx. Configuration drift never accumulates because Argo CD reverts anything that diverges from Git. An unhealthy node is drained and replaced by Node Problem Detector and Karpenter, a rotated secret restarts exactly the pods that use it, and a load spike is absorbed by KEDA and VPA rather than paged to a human.
Every failure mode has a closed loop that catches it and fixes it on its own. Tap a loop.
A new version ships as a canary. AnalysisRuns watch the 5xx rate against the baseline; if it climbs past the threshold, the rollout aborts itself and returns to the last good version. No human in the loop.