Self-destructing hotfixes with updatecli

Table of Contents

podman 5.8.7 broke every uses: step on our Forgejo runners (podman#29805). The hotfix moved the runners to Docker. We also added two updatecli policies: one to propose reverting the hotfix, and one to remove both policies afterwards.

We already use updatecli to keep dependencies up to date. Its sources, conditions, targets and PR actions work for this too: read the installed version, check upstream, then propose a git revert.

When to revert #

The policies run daily:

Read podman’s version from the nixpkgs locked in our nix repo.
If it is newer than 5.8.7 and the upstream issue is closed, open a PR reverting the Docker workaround.
When the cleanup policy detects that the workaround is gone, open a PR deleting both policies and their workflow entries.

The version comes from our pinned nixpkgs, because an upstream release alone would not change what our runners use:

command: nix eval --raw --impure --expr '(builtins.getFlake (toString ./.)).inputs.nixpkgs.legacyPackages.x86_64-linux.podman.version'

These are the revert conditions, with the repository wiring omitted:

upstream-closed:
  kind: json
  disablesourceinput: true
  spec:
    file: https://api.github.com/repos/podman-container-tools/podman/issues/29805
    key: state
    value: closed
fixed:
  kind: shell
  sourceid: podman
  spec:
    command: sh -c '[ "$(printf "%s\n" 5.8.7 "$1" | sort -V | tail -n1)" != 5.8.7 ]' --
workaround-present:
  kind: file
  disablesourceinput: true
  spec:
    file: modules/forgejo-runner.nix
    matchpattern: 'virtualisation\.docker'

A newer version and a closed issue are enough to ask for a review, but they do not prove that our package contains the fix. The PR body asks the reviewer to check the release notes for the buildah copier change before merging, then re-run an ubuntu-latest job using actions/checkout after deployment.

Reverting and cleaning up #

The revert target runs this shell command in the nix repo:

sh -c '
  set -- $(git log -1 -F --grep="(#255)" --format="%H %P")
  [ $# -gt 0 ] || { echo "o27/nix#255 is not on the branch" >&2; exit 2; }
  commit=$1; mainline=""
  [ $# -gt 2 ] && mainline="-m 1"
  [ "$DRY_RUN" = true ] && { echo "would revert $commit"; exit 0; }
  git revert --no-commit $mainline "$commit" || { git revert --abort; exit 2; }
  git diff --cached --quiet && exit 1
  exit 0
'

It finds the hotfix by the (#255) marker in its commit message. Reverting that commit also removes the comments explaining the workaround. Both shell targets check DRY_RUN, so PR checks report the proposed changes without applying them.

The cleanup policy checks the nix repo for two things: the hotfix’s commit message is in the history, and modules/forgejo-runner.nix no longer contains virtualisation.docker. Checking only for Docker’s absence would also pass before the hotfix merged.

That is our proxy for a merged revert; it does not inspect the revert PR’s status. The cleanup target then opens a PR in the gitops repo, deleting both policy files and removing their entries from the updatecli workflow.

Full updatecli policies

These use our existing nix_ci values and credentials from the environment.

forgejo-runner-podman.yaml

# Temporary: o27/nix#255 moved the Forgejo runners to Docker because podman 5.8.7
# cannot copy actions into /var/run/act (podman#29805). This proposes reverting
# that PR once the pinned nixpkgs carries a newer podman;
# forgejo-runner-podman-cleanup.yaml then deletes both policies.
name: Revert the Forgejo runner Docker workaround
{{ if .nix_ci.enabled }}
scms:
  nix:
    kind: gitea
    spec:
      url: '{{ .nix_ci.url }}'
      owner: '{{ .nix_ci.owner }}'
      repository: '{{ .nix_ci.repository }}'
      branch: '{{ .nix_ci.branch }}'
      token: '{{ requiredEnv "GITEA_TOKEN" }}'
      user: '{{ requiredEnv "UPDATECLI_GIT_USER" }}'
      email: '{{ requiredEnv "UPDATECLI_GIT_EMAIL" }}'
      gpg:
        signingkey: {{ requiredEnv "UPDATECLI_GPG_PRIVATE_KEY" | quote }}
        passphrase: {{ env "UPDATECLI_GPG_PASSPHRASE" | quote }}
      commitmessage:
        type: revert
        scope: forgejo-runner
        hidecredit: true
        footers: 'Signed-off-by: {{ requiredEnv "UPDATECLI_GIT_USER" }} <{{ requiredEnv "UPDATECLI_GIT_EMAIL" }}>'
{{ end }}

sources:
  podman:
    name: podman version in the nixpkgs pinned by o27/nix
    kind: shell
{{ if .nix_ci.enabled }}
    scmid: nix
{{ end }}
    spec:
      command: nix eval --raw --impure --expr '(builtins.getFlake (toString ./.)).inputs.nixpkgs.legacyPackages.x86_64-linux.podman.version'
      environments:
        - name: PATH
        - name: HOME

conditions:
  # A newer podman alone does not prove the buildah copier change was relaxed.
  upstream-closed:
    name: podman#29805 is closed
    kind: json
    disablesourceinput: true
    spec:
      file: https://api.github.com/repos/podman-container-tools/podman/issues/29805
      key: state
      value: closed
  fixed:
    name: the pinned podman is newer than 5.8.7
    kind: shell
    sourceid: podman
    spec:
      command: sh -c '[ "$(printf "%s\n" 5.8.7 "$1" | sort -V | tail -n1)" != 5.8.7 ]' --
      environments:
        - name: PATH
  # Without it the revert would be proposed again, and fail, once it has merged.
  workaround-present:
    name: o27/nix still runs the runners on Docker
    kind: file
    disablesourceinput: true
{{ if .nix_ci.enabled }}
    scmid: nix
{{ end }}
    spec:
      file: modules/forgejo-runner.nix
      matchpattern: 'virtualisation\.docker'

targets:
  revert:
    name: revert o27/nix#255
    kind: shell
    disablesourceinput: true
{{ if .nix_ci.enabled }}
    scmid: nix
{{ end }}
    spec:
      # Reverting the merge rather than editing the module also drops the comments that justified the workaround.
      # updatecli runs this under DRY_RUN=true in diff mode, so it only reports there.
      command: |
        sh -c '
          set -- $(git log -1 -F --grep="(#255)" --format="%H %P")
          [ $# -gt 0 ] || { echo "o27/nix#255 is not on the branch" >&2; exit 2; }
          commit=$1; mainline=""
          [ $# -gt 2 ] && mainline="-m 1"
          [ "$DRY_RUN" = true ] && { echo "would revert $commit"; exit 0; }
          git revert --no-commit $mainline "$commit" || { git revert --abort; exit 2; }
          git diff --cached --quiet && exit 1
          exit 0
        '        
      environments:
        - name: PATH
        - name: HOME
      changedif:
        kind: exitcode
        spec:
          warning: 0
          success: 1
          failure: 2
{{ if .nix_ci.enabled }}
actions:
  nix:
    kind: gitea/pullrequest
    scmid: nix
    title: 'revert(forgejo-runner): go back to podman {{ source "podman" }}'
    spec:
      url: '{{ .nix_ci.url }}'
      owner: '{{ .nix_ci.owner }}'
      repository: '{{ .nix_ci.repository }}'
      body: |
        Reverts o27/nix#255. The pinned nixpkgs now carries podman {{ source "podman" }}, newer than the 5.8.7 that broke action copies into /var/run/act (podman#29805).

        A newer version is not proof of the fix: check the podman release notes for the buildah copier change before merging. After the deploy, re-run an ubuntu-latest job that uses actions/checkout.

        Once this merges, forgejo-runner-podman-cleanup.yaml in o27/gitops opens the PR that deletes both policies.        
{{ end }}

forgejo-runner-podman-cleanup.yaml

# Temporary: deletes itself and forgejo-runner-podman.yaml once o27/nix has
# reverted the #255 Docker workaround.
name: Remove the Forgejo runner podman policies
{{ if .nix_ci.enabled }}
scms:
  nix:
    kind: gitea
    spec:
      url: '{{ .nix_ci.url }}'
      owner: '{{ .nix_ci.owner }}'
      repository: '{{ .nix_ci.repository }}'
      branch: '{{ .nix_ci.branch }}'
      token: '{{ requiredEnv "GITEA_TOKEN" }}'
  gitops:
    kind: gitea
    spec:
      url: '{{ .nix_ci.url }}'
      owner: '{{ .nix_ci.owner }}'
      repository: gitops
      branch: main
      token: '{{ requiredEnv "GITEA_TOKEN" }}'
      user: '{{ requiredEnv "UPDATECLI_GIT_USER" }}'
      email: '{{ requiredEnv "UPDATECLI_GIT_EMAIL" }}'
      gpg:
        signingkey: {{ requiredEnv "UPDATECLI_GPG_PRIVATE_KEY" | quote }}
        passphrase: {{ env "UPDATECLI_GPG_PASSPHRASE" | quote }}
      commitmessage:
        type: chore
        scope: updatecli
        hidecredit: true
        footers: 'Signed-off-by: {{ requiredEnv "UPDATECLI_GIT_USER" }} <{{ requiredEnv "UPDATECLI_GIT_EMAIL" }}>'
{{ end }}

conditions:
  # "Workaround gone" alone is also true before #255 merges, which would delete the policies before they ran.
  reverted:
    name: o27/nix merged PR 255 and no longer runs the runners on Docker
    kind: shell
    disablesourceinput: true
{{ if .nix_ci.enabled }}
    scmid: nix
{{ end }}
    spec:
      command: sh -c 'git log -1 -F --grep="(#255)" --format=%H | grep -q . && ! grep -q "virtualisation\.docker" modules/forgejo-runner.nix'
      environments:
        - name: PATH

targets:
  remove:
    name: delete both policies and their workflow entries
    kind: shell
    disablesourceinput: true
{{ if .nix_ci.enabled }}
    scmid: gitops
{{ end }}
    spec:
      command: |
        sh -c '
          [ -e updatecli/nix-ci.d/forgejo-runner-podman.yaml ] || exit 1
          [ "$DRY_RUN" = true ] && { echo "would delete the forgejo-runner-podman policies"; exit 0; }
          git rm -q updatecli/nix-ci.d/forgejo-runner-podman.yaml updatecli/nix-ci.d/forgejo-runner-podman-cleanup.yaml || exit 2
          sed -i "/nix-ci\.d\/forgejo-runner-podman/d" .forgejo/workflows/updatecli.yaml || exit 2
          exit 0
        '        
      environments:
        - name: PATH
      changedif:
        kind: exitcode
        spec:
          warning: 0
          success: 1
          failure: 2
{{ if .nix_ci.enabled }}
actions:
  gitops:
    kind: gitea/pullrequest
    scmid: gitops
    title: 'chore(updatecli): remove the Forgejo runner podman policies'
    spec:
      url: '{{ .nix_ci.url }}'
      owner: '{{ .nix_ci.owner }}'
      repository: gitops
      body: |
        o27/nix has merged the revert of #255, so the runners are back on podman and these two policies have nothing left to do.        
{{ end }}