{{- /*
Catalyst overlay: detect-and-recover CronJob for HelmReleases stuck in
Ready=Unknown after a helm-controller leader-election loss OR after an
apiserver flap on a slow secondary CP dropped the Ready=True write.

WHY THIS EXISTS — TWO FAILURE MODES
------------------------------------

(A) ISSUE #925 — leader-election loss recovery
    -------------------------------------------
helm-controller's reconcile loop has a known soft bug: when leadership
is lost mid-install (transient kube-apiserver pressure / pod restart /
OOMKill), the new leader takes over an in-flight Helm install whose
release secret was already committed (Status=deployed). The new leader's
"is the release in storage?" short-circuit returns yes, so it never
re-runs the install — but the previous leader's last write to the HR's
Ready condition was `Unknown`, and `Unknown` is treated by the controller
as a transient state that "something else" will eventually flip.
Nothing does. The HR is permanently stuck.

Live evidence (otech113.omani.works, 2026-05-05): bp-vpa stuck in
Ready=Unknown for 10+ minutes → blocked bootstrap-kit Kustomization →
blocked sovereign-tls Kustomization → cilium-gateway never deployed →
every HTTPRoute returned TLS handshake failure.

Recovery: suspend-toggle. Detects via `helm release secret label
status=deployed` + Ready=Unknown for >threshold.

(B) TBD-A66 / ISSUE #1989 — slow secondary CP apiserver-flap recovery
    -----------------------------------------------------------------
On embedded-etcd k3s secondary CPs (nbg1-2 / hel1-1 in the t37
canonical walk) apiserver flaps during the bootstrap cascade. helm-
controller completes the install — the HR's OWN
`.status.history[0].status` transitions to "deployed" — but the
apiserver write that flips `.status.conditions[type=Ready]` from
Unknown → True is lost. suspend-toggling (branch A) does NOT recover
this state because helm-controller's "is the release in storage?"
short-circuit returns yes on the next reconcile, so it never re-
evaluates Ready.

Recovery: direct status-subresource patch flipping Ready=True, stamped
with `stuck-hr-recovery.openova.io/auto-corrected-at=<RFC3339>` for
audit + idempotency.

The primary mitigation (extended leader-election lease durations +
memory headroom in values.yaml) reduces the frequency of the trigger.
This CronJob is the recovery safety net: every 2 minutes it scans every
HelmRelease in the cluster and applies whichever branch matches.

WHY A CRONJOB AND NOT A WEBHOOK / OPERATOR
------------------------------------------
A webhook would need to watch HRs and Helm secrets cluster-wide and run
inside an event loop — that's another stateful controller competing with
helm-controller for leadership. A CronJob is stateless, idempotent, and
cheap (kubectl-only, runs in <5s when there's nothing to do). When the
upstream bug is fixed in helm-controller, this CronJob becomes a no-op
because the trigger condition will never match — operators can drop it
by setting .Values.catalyst.stuckHelmReleaseRecovery.enabled=false.

Per docs/INVIOLABLE-PRINCIPLES.md #4, every parameter is operator
overridable.
*/}}
{{- $cfg := default (dict) (index .Values "catalyst" "stuckHelmReleaseRecovery") -}}
{{- /* `default true X` treats `X=false` as empty and returns true, so use
    an explicit hasKey check to honour an operator-set enabled=false
    override (well-known Helm gotcha). */ -}}
{{- $enabled := true -}}
{{- if hasKey $cfg "enabled" -}}{{- $enabled = $cfg.enabled -}}{{- end -}}
{{- if $enabled -}}
{{- $fluxNs := default "flux-system" (index .Values "catalyst" "fluxNamespace") -}}
{{- $schedule := default "*/2 * * * *" (index $cfg "schedule") -}}
{{- $stuckThreshold := default "5m" (index $cfg "stuckThreshold") -}}
{{- /* Fix #158: bitnamilegacy/kubectl (Bitnami deleted versioned tags
       from docker.io/bitnami/kubectl 2025-08; bitnamilegacy is the
       deprecation-fallback path). */ -}}
{{- $image := default "bitnamilegacy/kubectl:1.31.4" (index $cfg "image") -}}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bp-flux-stuck-hr-recovery
  namespace: {{ $fluxNs }}
  labels:
    app.kubernetes.io/name: bp-flux
    app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
    app.kubernetes.io/part-of: catalyst
    app.kubernetes.io/component: stuck-hr-recovery
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    catalyst.openova.io/blueprint: bp-flux
  annotations:
    catalyst.openova.io/issue: "https://github.com/openova-io/openova/issues/925"
    catalyst.openova.io/issue-tbd-a66: "https://github.com/openova-io/openova/issues/1989"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bp-flux-stuck-hr-recovery
  labels:
    app.kubernetes.io/name: bp-flux
    app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
    app.kubernetes.io/part-of: catalyst
    app.kubernetes.io/component: stuck-hr-recovery
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    catalyst.openova.io/blueprint: bp-flux
rules:
  # Read every HelmRelease cluster-wide to find stuck ones.
  - apiGroups: ["helm.toolkit.fluxcd.io"]
    resources: ["helmreleases"]
    verbs: ["get", "list", "watch"]
  # Patch spec.suspend on stuck HRs to force re-evaluation (issue #925
  # branch). Cluster-wide because tenant HRs live in tenant namespaces.
  # Also `patch` here covers metadata-only patches like our audit
  # annotation `stuck-hr-recovery.openova.io/auto-corrected-at`.
  - apiGroups: ["helm.toolkit.fluxcd.io"]
    resources: ["helmreleases"]
    verbs: ["patch"]
  # TBD-A66 branch: patch .status.conditions.Ready directly when the
  # HR's own .status.history[0].status proves the install completed but
  # apiserver flap on a slow secondary CP dropped the Ready=True write.
  # The status subresource is a separate RBAC target from the main
  # resource — without this rule `kubectl patch --subresource=status`
  # returns 403.
  - apiGroups: ["helm.toolkit.fluxcd.io"]
    resources: ["helmreleases/status"]
    verbs: ["patch", "update"]
  # Read Helm release secrets to confirm Status=deployed (the precondition
  # for the issue #925 fallback branch). The TBD-A66 branch uses the HR's
  # own .status.history[0].status instead, so secret reads aren't needed
  # for it — but the fallback path still uses them.
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bp-flux-stuck-hr-recovery
  labels:
    app.kubernetes.io/name: bp-flux
    app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
    app.kubernetes.io/part-of: catalyst
    app.kubernetes.io/component: stuck-hr-recovery
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    catalyst.openova.io/blueprint: bp-flux
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: bp-flux-stuck-hr-recovery
subjects:
  - kind: ServiceAccount
    name: bp-flux-stuck-hr-recovery
    namespace: {{ $fluxNs }}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: bp-flux-stuck-hr-recovery-script
  namespace: {{ $fluxNs }}
  labels:
    app.kubernetes.io/name: bp-flux
    app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
    app.kubernetes.io/part-of: catalyst
    app.kubernetes.io/component: stuck-hr-recovery
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    catalyst.openova.io/blueprint: bp-flux
data:
  stuck-threshold: {{ $stuckThreshold | quote }}
  recover.sh: |
    #!/bin/sh
    # Detect HelmReleases stuck in Ready=Unknown and recover them.
    #
    # Two distinct stuck-state failure modes are handled:
    #
    # (A) Leader-election loss recovery (issue #925)
    #     ─────────────────────────────────────────
    #     helm-controller lost leadership mid-install. The Helm release
    #     SECRET shows Status=deployed (storage write committed by the
    #     previous leader) but the Ready condition was never flipped.
    #     Fix: toggle spec.suspend=true → false to force re-evaluation.
    #
    # (B) Slow-secondary-CP apiserver-flap recovery (TBD-A66, issue #1989)
    #     ─────────────────────────────────────────────────────────
    #     On embedded-etcd k3s secondary CPs (nbg1-2/hel1-1 in t37 walk)
    #     apiserver flaps during the bootstrap cascade. helm-controller
    #     completes the install — the HR's OWN .status.history[0].Status
    #     transitions to "deployed" — but the apiserver write that flips
    #     .status.conditions[Ready] from Unknown → True is lost. suspend-
    #     toggling (branch A) does NOT recover this state because
    #     helm-controller's "is the release in storage?" short-circuit
    #     returns yes on the next reconcile, so it never re-evaluates
    #     Ready. The only safe recovery is a direct status-subresource
    #     patch that flips Ready=True and stamps an audit annotation.
    #
    # Algorithm:
    #   1. List every HelmRelease cluster-wide.
    #   2. For each: read .status.conditions[type=Ready] + history[0].
    #      - If Ready.status != "Unknown" → skip.
    #      - If Ready.lastTransitionTime within threshold → skip
    #        (still might be a real in-flight install).
    #      - If already annotated stuck-hr-recovery.openova.io/
    #        auto-corrected-at → skip (idempotent re-runs).
    #   3. Decision tree:
    #      (B) If .status.history[0].status == "deployed" → status-patch
    #          Ready=True with audit annotation. PREFERRED branch — the
    #          HR itself confirms install succeeded.
    #      (A) Else fall back: read the Helm release secret in the HR's
    #          storageNamespace. If secret label status=deployed →
    #          suspend-toggle (legacy issue #925 recovery).
    #      Else skip (HR really IS still installing; not stuck).
    #
    # Guardrail: if more than 10 HRs would be acted on in a single run,
    # log and exit non-zero — that signals a cluster-wide problem (real
    # outage) where touching everything would mask the cause.
    set -eu
    THRESHOLD_HUMAN="$(cat /config/stuck-threshold)"
    # Convert "5m" / "30s" / "1h" to seconds for date math.
    case "$THRESHOLD_HUMAN" in
      *s) THRESHOLD_SEC="${THRESHOLD_HUMAN%s}" ;;
      *m) THRESHOLD_SEC=$(( ${THRESHOLD_HUMAN%m} * 60 )) ;;
      *h) THRESHOLD_SEC=$(( ${THRESHOLD_HUMAN%h} * 3600 )) ;;
      *)  THRESHOLD_SEC="$THRESHOLD_HUMAN" ;;
    esac
    NOW="$(date -u +%s)"
    NOW_ISO="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
    ACTED=0
    SCANNED=0

    # Build a list of
    # "ns|name|storageNs|releaseName|readyStatus|readyTransition|historyStatus|autoCorrectedAt"
    # in a single kubectl call (cheap — JSONPath does the work server-side).
    # NOTE: jsonpath dot-escape on the annotation key — `\.` segments are
    # required because the annotation contains literal dots (RFC1123 host
    # name segment style). kubectl jsonpath supports `\.` as a literal
    # dot since v1.16.
    HRS="$(kubectl get hr -A -o jsonpath='{range .items[*]}{.metadata.namespace}|{.metadata.name}|{.spec.storageNamespace}|{.spec.releaseName}|{range .status.conditions[?(@.type=="Ready")]}{.status}|{.lastTransitionTime}{end}|{.status.history[0].status}|{.metadata.annotations.stuck-hr-recovery\.openova\.io/auto-corrected-at}{"\n"}{end}')" || HRS=""

    echo "$HRS" | while IFS='|' read -r NS NAME STORAGE_NS RELEASE_NAME READY_STATUS READY_TRANSITION HISTORY_STATUS AUTO_CORRECTED_AT; do
      [ -z "$NAME" ] && continue
      SCANNED=$((SCANNED+1))
      # Only act on Ready=Unknown.
      [ "$READY_STATUS" = "Unknown" ] || continue
      [ -n "$READY_TRANSITION" ] || continue
      # Idempotency: if we've already auto-corrected this HR (annotation
      # present), skip — either the correction stuck and Ready won't be
      # Unknown anymore (so we wouldn't be here), or helm-controller
      # re-flipped it back to Unknown which means there's a genuine
      # ongoing problem and operator should investigate.
      [ -z "$AUTO_CORRECTED_AT" ] || { echo "skip $NS/$NAME: already auto-corrected at $AUTO_CORRECTED_AT"; continue; }
      # Has the Unknown state persisted past the threshold?
      TRANSITION_SEC="$(date -u -d "$READY_TRANSITION" +%s 2>/dev/null || echo 0)"
      [ "$TRANSITION_SEC" -gt 0 ] || continue
      AGE=$((NOW - TRANSITION_SEC))
      [ "$AGE" -ge "$THRESHOLD_SEC" ] || continue

      # ── Branch B (TBD-A66): HR's own history says deployed ─────────
      # The HR controller already wrote history[0].status=deployed,
      # which proves the install completed. Ready=Unknown is therefore
      # a lost-write on the conditions subresource (apiserver flap on
      # slow secondary CP). Direct status patch is the safe recovery.
      if [ "$HISTORY_STATUS" = "deployed" ]; then
        ACTED=$((ACTED+1))
        if [ "$ACTED" -gt 10 ]; then
          echo "GUARDRAIL: more than 10 stuck HRs detected — refusing to mass-recover. Investigate cluster-wide outage."
          exit 1
        fi
        # TBD-A66-followup (issue #1995): structured [A66] log lines so
        # operators / agents can grep CronJob output to confirm the
        # branch fired and whether the patch landed. Previously the
        # whole branch was a single human-readable line and the patch
        # outcome swallowed stderr (`2>&1`), so silent failures looked
        # identical to silent successes — t38 / agent ae9d7638 lost
        # ~10min diagnosing exactly this.
        echo "[A66] HR $NS/$NAME Ready=Unknown for ${AGE}s, history[0]=deployed → attempting patch"
        # Step 1: stamp audit annotation on the object's metadata so
        # idempotency works regardless of whether the status patch
        # below races against a helm-controller reconcile.
        kubectl annotate hr -n "$NS" "$NAME" --overwrite "stuck-hr-recovery.openova.io/auto-corrected-at=${NOW_ISO}" >/dev/null
        # Step 2: patch the status subresource. We use a JSON-merge
        # patch on the entire status object replacing only conditions
        # — the helm-controller's next reconcile will re-render the
        # rest of status from the HR spec + history without loss.
        # `--subresource=status` requires kubectl ≥ v1.24 (we ship
        # 1.31.4 in this CronJob image, so safe).
        #
        # Stderr capture (TBD-A66-followup, issue #1995): keep stderr
        # separate from stdout — we LOG it on failure rather than
        # discard it. Stdout is redirected to /dev/null because a
        # successful kubectl patch echoes "helmrelease.helm.toolkit.fluxcd.io/<name> patched"
        # which would interleave with our structured log lines. Stderr
        # is captured to a temp file (mktemp under /tmp, the writable
        # emptyDir mount) so multi-line apiserver errors survive intact.
        PATCH_BODY='{"status":{"conditions":[{"type":"Ready","status":"True","reason":"ReconciliationSucceeded","message":"auto-corrected from deployed-but-unknown-Ready by stuck-hr-recovery (TBD-A66)","lastTransitionTime":"'"$NOW_ISO"'"}]}}'
        PATCH_ERR_FILE="$(mktemp /tmp/a66-patch-err.XXXXXX)"
        if kubectl patch hr -n "$NS" "$NAME" --subresource=status --type=merge -p "$PATCH_BODY" >/dev/null 2>"$PATCH_ERR_FILE"; then
          echo "[A66] HR $NS/$NAME patched to Ready=True"
        else
          PATCH_ERR="$(tr '\n' ' ' <"$PATCH_ERR_FILE")"
          echo "[A66] HR $NS/$NAME patch FAILED: ${PATCH_ERR:-<no stderr captured>}"
          # Roll back the annotation so the next run isn't blocked by
          # idempotency guard on a half-applied recovery. Capture stderr
          # here too — if annotation rollback fails the operator needs
          # to know (a stuck idempotency annotation would mask repeat
          # patch attempts in the next CronJob run).
          if ! kubectl annotate hr -n "$NS" "$NAME" "stuck-hr-recovery.openova.io/auto-corrected-at-" >/dev/null 2>>"$PATCH_ERR_FILE"; then
            ROLLBACK_ERR="$(tail -1 "$PATCH_ERR_FILE" 2>/dev/null || true)"
            echo "[A66] HR $NS/$NAME annotation rollback also failed: ${ROLLBACK_ERR:-<no stderr captured>}"
          fi
        fi
        rm -f "$PATCH_ERR_FILE"
        continue
      fi

      # ── Branch A (issue #925): fall back to helm-secret + suspend toggle ──
      # Resolve storage ns + release name (defaults match upstream HR semantics).
      EFFECTIVE_STORAGE_NS="${STORAGE_NS:-$NS}"
      EFFECTIVE_RELEASE="${RELEASE_NAME:-$NAME}"

      # Find the most recent Helm release secret for this release. Helm
      # secrets are named sh.helm.release.v1.<release>.v<n>; we list and
      # take the highest-version one.
      LATEST_SECRET="$(kubectl get secret -n "$EFFECTIVE_STORAGE_NS" -l "owner=helm,name=$EFFECTIVE_RELEASE" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null | tr ' ' '\n' | sort -t v -k 4 -n | tail -1)"
      [ -n "$LATEST_SECRET" ] || { echo "skip $NS/$NAME: no helm secret found"; continue; }
      SECRET_STATUS="$(kubectl get secret -n "$EFFECTIVE_STORAGE_NS" "$LATEST_SECRET" -o jsonpath='{.metadata.labels.status}' 2>/dev/null || echo "")"
      # Only act if Helm itself thinks the release is deployed; if it's
      # pending-install / failed / etc the HR really IS still in flight.
      [ "$SECRET_STATUS" = "deployed" ] || { echo "skip $NS/$NAME: helm secret status=$SECRET_STATUS history=$HISTORY_STATUS"; continue; }

      ACTED=$((ACTED+1))
      if [ "$ACTED" -gt 10 ]; then
        echo "GUARDRAIL: more than 10 stuck HRs detected — refusing to mass-recover. Investigate cluster-wide outage."
        exit 1
      fi

      echo "RECOVER $NS/$NAME (Ready=Unknown for ${AGE}s, helm secret $LATEST_SECRET status=deployed) — toggling spec.suspend (issue #925 branch)"
      kubectl patch hr -n "$NS" "$NAME" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
      sleep 5
      kubectl patch hr -n "$NS" "$NAME" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
      echo "RECOVERED $NS/$NAME (issue #925 branch)"
    done

    echo "scan complete: scanned=$SCANNED acted=$ACTED threshold=${THRESHOLD_SEC}s"
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: bp-flux-stuck-hr-recovery
  namespace: {{ $fluxNs }}
  labels:
    app.kubernetes.io/name: bp-flux
    app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
    app.kubernetes.io/part-of: catalyst
    app.kubernetes.io/component: stuck-hr-recovery
    helm.sh/chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    catalyst.openova.io/blueprint: bp-flux
  annotations:
    catalyst.openova.io/issue: "https://github.com/openova-io/openova/issues/925"
    catalyst.openova.io/issue-tbd-a66: "https://github.com/openova-io/openova/issues/1989"
spec:
  schedule: {{ $schedule | quote }}
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  startingDeadlineSeconds: 60
  jobTemplate:
    spec:
      backoffLimit: 0
      ttlSecondsAfterFinished: 300
      template:
        metadata:
          labels:
            app.kubernetes.io/name: bp-flux
            app.kubernetes.io/component: stuck-hr-recovery
        spec:
          serviceAccountName: bp-flux-stuck-hr-recovery
          restartPolicy: Never
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
            runAsGroup: 65534
            fsGroup: 65534
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: recover
              image: {{ $image | quote }}
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "/config/recover.sh"]
              securityContext:
                allowPrivilegeEscalation: false
                readOnlyRootFilesystem: true
                capabilities:
                  drop: ["ALL"]
              resources:
                requests: { cpu: 10m, memory: 32Mi }
                limits:   { memory: 96Mi }
              volumeMounts:
                - name: script
                  mountPath: /config
                - name: tmp
                  mountPath: /tmp
                - name: home
                  mountPath: /.kube
          volumes:
            - name: script
              configMap:
                name: bp-flux-stuck-hr-recovery-script
                defaultMode: 0555
            - name: tmp
              emptyDir: {}
            - name: home
              emptyDir: {}
{{- end -}}
