Skip to content

spark-operator: Dataflow

Controller Watches

Kubernetes resources this controller monitors for changes. Each watch triggers reconciliation when the watched resource is created, updated, or deleted.

Type GVK Source
For api/v1alpha1/SparkConnect internal/controller/sparkconnect/reconciler.go:97
For scheduling/v1alpha1/ElasticQuota .gomod-cache/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/elasticquota_controller.go:175
For scheduling/v1alpha1/ElasticQuota .gopath-loader/pkg/mod/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/elasticquota_controller.go:175
For scheduling/v1alpha1/PodGroup .gomod-cache/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/podgroup_controller.go:193
For scheduling/v1alpha1/PodGroup .gopath-loader/pkg/mod/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/podgroup_controller.go:193
Watches /v1/Pod .gomod-cache/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/elasticquota_controller.go:174
Watches /v1/Pod .gomod-cache/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/podgroup_controller.go:192
Watches /v1/Pod .gopath-loader/pkg/mod/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/elasticquota_controller.go:174
Watches /v1/Pod .gopath-loader/pkg/mod/sigs.k8s.io/scheduler-plugins@v0.32.7/pkg/controllers/podgroup_controller.go:192

Programmatic Resource Operations

Verb Kind Group Condition
delete SparkApplication api

Reconciliation Flow

How the controller interacts with the Kubernetes API during reconciliation.

sequenceDiagram
    %% Static dataflow for spark-operator

    participant KubernetesAPI as Kubernetes API
    participant controller_manager as controller-manager
    participant peaks as peaks
    participant spark_operator_controller as spark-operator-controller
    participant spark_operator_webhook as spark-operator-webhook
    participant the_deployment as the-deployment

    KubernetesAPI->>+controller_manager: Watch SparkConnect (reconcile)
    KubernetesAPI->>+controller_manager: Watch ElasticQuota (reconcile)
    KubernetesAPI->>+controller_manager: Watch ElasticQuota (reconcile)
    KubernetesAPI->>+controller_manager: Watch PodGroup (reconcile)
    KubernetesAPI->>+controller_manager: Watch PodGroup (reconcile)
    KubernetesAPI-->>+controller_manager: Watch Pod (informer)
    KubernetesAPI-->>+controller_manager: Watch Pod (informer)
    KubernetesAPI-->>+controller_manager: Watch Pod (informer)
    KubernetesAPI-->>+controller_manager: Watch Pod (informer)

    Note over controller_manager: Exposed Services
    Note right of controller_manager: spark-operator-webhook-svc:443/TCP [webhook]
    Note right of controller_manager: the-service:8666/TCP []
    Note right of controller_manager: the-service:8666/TCP []

    Note over KubernetesAPI: Defined CRDs
    Note right of KubernetesAPI: SparkConnect (sparkoperator.k8s.io/v1alpha1)
    Note right of KubernetesAPI: ScheduledSparkApplication (sparkoperator.k8s.io/v1beta2)
    Note right of KubernetesAPI: SparkApplication (sparkoperator.k8s.io/v1beta2)

Webhooks

Name Type Path Failure Policy Service Overlays Enable Condition Sources
conversion-unknown conversion /convert system/webhook-service config/crd/patches/webhook_in_sparkapplications.yaml, .gomod-cache/sigs.k8s.io/controller-runtime@v0.20.4/pkg/builder/webhook.go, .gopath-loader/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/builder/webhook.go
mutate-pod.sparkoperator.k8s.io mutating /mutate--v1-pod Fail opendatahub/spark-operator-webhook-svc config/overlays/odh config/webhook/manifests.yaml, kustomize:config/overlays/odh (spark-operator-webhook)
mutate-scheduledsparkapplication.sparkoperator.k8s.io mutating /mutate-sparkoperator-k8s-io-v1beta2-scheduledsparkapplication Fail opendatahub/spark-operator-webhook-svc config/overlays/odh config/webhook/manifests.yaml, kustomize:config/overlays/odh (spark-operator-webhook)
mutate-sparkapplication.sparkoperator.k8s.io mutating /mutate-sparkoperator-k8s-io-v1beta2-sparkapplication Fail opendatahub/spark-operator-webhook-svc config/overlays/odh config/webhook/manifests.yaml, kustomize:config/overlays/odh (spark-operator-webhook)
validate-scheduledsparkapplication.sparkoperator.k8s.io validating /validate-sparkoperator-k8s-io-v1beta2-scheduledsparkapplication Fail opendatahub/spark-operator-webhook-svc config/overlays/odh config/webhook/manifests.yaml, kustomize:config/overlays/odh (spark-operator-webhook)
validate-sparkapplication.sparkoperator.k8s.io validating /validate-sparkoperator-k8s-io-v1beta2-sparkapplication Fail opendatahub/spark-operator-webhook-svc config/overlays/odh config/webhook/manifests.yaml, kustomize:config/overlays/odh (spark-operator-webhook)

HTTP Endpoints

Method Path Source
* / .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:46
* / .gopath-loader/pkg/mod/github.com/google/pprof@v0.0.0-20250403155104-27863c87afa6/internal/driver/webui.go:212
* / .gomod-cache/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:46
* / .gomod-cache/golang.org/x/tools@v0.37.0/cmd/present/dir.go:23
* / .gomod-cache/golang.org/x/net@v0.44.0/webdav/litmus_test_server.go:83
* / .gomod-cache/github.com/google/pprof@v0.0.0-20250403155104-27863c87afa6/internal/driver/webui.go:212
* / .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/webdav/litmus_test_server.go:83
* / .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/cmd/present/dir.go:23
GET / .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/group.go:57
GET / .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/version.go:67
GET / .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/version.go:44
GET / .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/root.go:154
GET / .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/legacy.go:59
GET / .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/aggregated/wrapper.go:62
GET / .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/version.go:44
GET / .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/aggregated/wrapper.go:62
GET / .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/group.go:57
GET / .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/legacy.go:59
GET / .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/version.go:67
GET / .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/endpoints/discovery/root.go:154
* /abort .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:63
* /abort .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:63
* /aggregated-nonprimary-procs-report .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:60
* /aggregated-nonprimary-procs-report .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:60
* /before-suite-completed .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:57
* /before-suite-completed .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:57
* /before-suite-state .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:58
* /before-suite-state .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:58
* /compile .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/playground/playground.go:23
* /compile .gomod-cache/golang.org/x/tools@v0.37.0/playground/playground.go:23
* /counter .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:61
* /counter .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:61
* /debug/flags .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:55
* /debug/flags .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:55
* /debug/flags/ .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:56
* /debug/flags/ .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:56
* /debug/pprof .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:44
* /debug/pprof .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:44
* /debug/pprof/ .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:45
* /debug/pprof/ .gopath-loader/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:316
* /debug/pprof/ .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:45
* /debug/pprof/ .gomod-cache/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:316
* /debug/pprof/cmdline .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:46
* /debug/pprof/cmdline .gomod-cache/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:317
* /debug/pprof/cmdline .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:46
* /debug/pprof/cmdline .gopath-loader/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:317
* /debug/pprof/profile .gomod-cache/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:318
* /debug/pprof/profile .gopath-loader/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:318
* /debug/pprof/profile .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:47
* /debug/pprof/profile .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:47
* /debug/pprof/symbol .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:48
* /debug/pprof/symbol .gomod-cache/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:319
* /debug/pprof/symbol .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:48
* /debug/pprof/symbol .gopath-loader/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:319
* /debug/pprof/trace .gomod-cache/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:320
* /debug/pprof/trace .gopath-loader/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/manager/internal.go:320
* /debug/pprof/trace .gomod-cache/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:49
* /debug/pprof/trace .gopath-loader/pkg/mod/k8s.io/apiserver@v0.32.5/pkg/server/routes/debugsocket.go:49
* /did-run .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:49
* /did-run .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:49
* /emit-output .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:51
* /emit-output .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:51
* /have-nonprimary-procs-finished .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:59
* /have-nonprimary-procs-finished .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:59
* /main.css .gomod-cache/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:48
* /main.css .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:48
* /main.js .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:47
* /main.js .gomod-cache/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:47
* /play.js .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/cmd/present/play.go:43
* /play.js .gomod-cache/golang.org/x/tools@v0.37.0/cmd/present/play.go:43
* /progress-report .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:52
* /progress-report .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:52
* /report-before-suite-completed .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:55
* /report-before-suite-completed .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:55
* /report-before-suite-state .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:56
* /report-before-suite-state .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:56
* /select.json .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:49
* /select.json .gomod-cache/golang.org/x/tools@v0.37.0/go/types/internal/play/play.go:49
* /socket .gomod-cache/golang.org/x/tools@v0.37.0/cmd/present/play.go:59
* /socket .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/cmd/present/play.go:59
* /static/ .gopath-loader/pkg/mod/golang.org/x/tools@v0.37.0/cmd/present/main.go:98
* /static/ .gomod-cache/golang.org/x/tools@v0.37.0/cmd/present/main.go:98
* /suite-did-end .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:50
* /suite-did-end .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:50
* /suite-will-begin .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:48
* /suite-will-begin .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:48
* /ui/ .gopath-loader/pkg/mod/github.com/google/pprof@v0.0.0-20250403155104-27863c87afa6/internal/driver/webui.go:211
* /ui/ .gomod-cache/github.com/google/pprof@v0.0.0-20250403155104-27863c87afa6/internal/driver/webui.go:211
* /up .gomod-cache/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:62
* /up .gopath-loader/pkg/mod/github.com/onsi/ginkgo/v2@v2.27.2/internal/parallel_support/http_server.go:62
GET /{user-id} .gomod-cache/github.com/emicklei/go-restful/v3@v3.12.1/doc.go:19
GET /{user-id} .gopath-loader/pkg/mod/github.com/emicklei/go-restful/v3@v3.12.1/doc.go:83
GET /{user-id} .gopath-loader/pkg/mod/github.com/emicklei/go-restful/v3@v3.12.1/doc.go:19
GET /{user-id} .gomod-cache/github.com/emicklei/go-restful/v3@v3.12.1/doc.go:83
* header .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:267
* header .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:267
* header .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:165
* header .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:165
* header .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:187
* header .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:211
* header .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:211
* header .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:187
* raw .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:217
* raw .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:217
* raw .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:193
* raw .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:193
* raw .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:172
* raw .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:172
* shm .gopath-loader/pkg/mod/github.com/containerd/containerd@v1.7.29/pkg/cri/server/sandbox_run_linux.go:302
* shm .gopath-loader/pkg/mod/github.com/containerd/containerd@v1.7.29/pkg/cri/sbserver/podsandbox/sandbox_run_linux.go:285
* shm .gomod-cache/github.com/containerd/containerd@v1.7.29/pkg/cri/server/sandbox_run_linux.go:302
* shm .gomod-cache/github.com/containerd/containerd@v1.7.29/pkg/cri/sbserver/podsandbox/sandbox_run_linux.go:285
* vantage_point .gomod-cache/golang.org/x/net@v0.44.0/quic/qlog.go:96
* vantage_point .gopath-loader/pkg/mod/golang.org/x/net@v0.44.0/quic/qlog.go:96

Configuration

ConfigMaps and Helm values that control this component's runtime behavior.

Helm

Chart: scheduler-plugins v0.32.7