Nops: a semi-automatic GitOps controller for Nomad

A while ago I published the full list of personal projects I’ve started and successfully finished. The list was short. It was “Zero.”

I’m happy to announce that the number is now one.

It’s called Nops, and it’s a semi-automatic GitOps controller for HashiCorp Nomad. It watches a git repository of Nomad jobs and deploys them according to a policy you set per job: automatically, after a person approves, or never.

Why I built it#

I run a small self-hosted Nomad cluster for personal use. I wanted the usual GitOps promise: the jobs live in git, a commit is a deployment, and the history tells me what changed and when.

But I didn’t want everything to deploy on its own. A frontend can roll out the moment I push. A database, or anything with a volume, should wait until I’ve looked at the diff and said yes. And some jobs I still want to manage by hand, with git only telling me when the cluster has drifted.

gerrowadat/nomad-gitops already did a lot of what I wanted, and Nops borrows many of its best ideas: Nomad parses the HCL, the plan’s diff drives both the decision and what you see, registers use CAS, and secrets are redacted from the diff. But nomad-gitops is stateless by design, and approvals and hooks need state. So Nops keeps its own, in SQLite.

How it works#

1. Nops reads the job files from git and has Nomad parse them.
2. It plans each managed job and, if there’s a difference, creates a deployment.
3. The job’s policy decides what happens next:

4. Nops registers the job only if nobody changed it since the plan.

Every deployment goes through the same states:

detected → [pending_approval] → [pre_hook] → applying → [post_hook] → completed
                                     │            │           │
                                     └────────────┴───────────┴──→ failed

The policy only decides who says go. After that, every deployment runs the same way.

Around this there’s a dashboard to review diffs and approve or reject deployments, plus an API and a command-line client for when a browser is too much effort. Access goes through tokens with ACL policies per namespace, so “who can approve a deployment of the database” is an actual rule, not a hope.

Configuring a job#

There’s no separate config file per job. Policy and hooks live in the job’s own HCL, as flat meta keys:

job "api" {
  meta {
    nops_managed   = "true"
    nops_policy    = "approval"                # auto | approval | none
    nops_pre_hook  = "api-backup,api-migrate"  # parameterized batch jobs, run in this order
    nops_post_hook = "api-smoke"
  }
  # ...
}

Hooks are plain Nomad jobs, parameterized batch jobs, that run before or after a deployment. A backup before touching the database, a migration before the new version starts, a smoke test after it’s up. If a hook fails, the deployment fails, and Nops tells you.

A couple of other things that turned out to matter in practice:

• Sync windows limit when an auto job deploys on its own, for example only on weekdays during working hours.
• Pause stops Nops from undoing a manual fix during an incident, for one job or for all of them at once. Nops records who paused and when.

Every key is in the meta keys reference, and the hooks guide shows how to write one.

The rules it never breaks#

A tool that deploys things on its own has to be boring and predictable. So Nops has a short list of invariants that no change is allowed to break, and a test file whose only job is to try to break them.

• Plan before every write. Nops registers a job only after a fresh plan shows a difference. No useless job versions, and the plan’s diff is exactly what the dashboard shows.
• CAS on every register. Nops registers with the job’s modify index read at detection, so Nomad refuses the write if anyone changed the job in the meantime. Time passes between detection and apply, especially while a deployment waits for approval. Nops never overwrites a manual change.
• Never auto-apply under approval. Only an authenticated request moves a deployment past pending_approval, and the approval is valid only for the exact diff it saw. If git or the live job changes first, Nops replaces the deployment with a new one to approve.
• Git is the source of truth for Nops’s behaviour, and Nops never writes to git. Policy and hooks come from the HCL in git, never from the live job. You change what Nops does with a reviewed commit.
• State is persisted before acting. Nops saves what it’s about to do before it touches Nomad. After a crash, it knows where it was and resumes safely.

And on top of that: fail loud. Nops never ignores a Nomad or SQLite error. When in doubt it takes the safe path: invalid meta means policy none, a missing hook fails the deployment, a CAS conflict fails it too. Every error says what failed and where. If you’ve read my post on errors in Go, this won’t surprise you.

Built with Claude#

I didn’t build Nops alone. I built it together with Claude, Anthropic’s AI coding agent.

The split was roughly this: I decided what Nops is, what it does, and what it must never do. The invariants, the policies, the choice of keeping state in SQLite, and the rule that “a feature is built when the cluster needs it, never for a scale it doesn’t have yet”. Claude wrote a large part of the code, the tests (including the ones that try to break the invariants), and the documentation. I reviewed, pushed back, changed my mind, and pushed back again.

It went from the first commit to v0.10.0 in about three weeks. That’s not something I could have done alone in my spare time, and it’s probably the main reason the counter in the intro moved at all.

The part that surprised me most is that the speed didn’t come at the cost of rigour. Having the rules written down, and tests that enforce them, meant I could move fast and still trust what came out.

Try it#

Nops is ready to use, but it’s still at v0: a minor release may change the configuration or the database, and its release notes say how to upgrade. It’s tested against Nomad 2.0.7, and other 2.0.x releases should work.

• Getting started: about fifteen minutes on one machine.
• Documentation: configuration, dashboard, API, CLI, ACL, metrics.
• Releases

It’s licensed under Apache 2.0. Issues and contributions are welcome.

The counter is at one.

Stay tuned folks.