Issue with Server side apply on argocd

When I try to deploy this helm chart, with server side apply because the crds are too fat.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: prometheus-operator-crds
  namespace: infra-monitoring
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-99"
spec:
  project: infra
  source:
    repoURL: "https://prometheus-community.github.io/helm-charts"
    targetRevision: "9.0.1"
    chart: prometheus-operator-crds
  destination:
    name: in-cluster
    namespace: infra-monitoring
  syncPolicy:
    automated: {}
    syncOptions:
      - ServerSideApply=true

We have:

ComparisonError
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp [fde2:eeaa:xxx:yyy::bbb]:8081: connect: connection refused"

or

runtime error: invalid memory address or nil pointer dereference

for this last error, I have to click click to purge the cluster cache then https://github.com/argoproj/argo-cd/issues/14098

More infos https://argo-cd.readthedocs.io/en/stable/proposals/server-side-apply/