distributed-workloads: Dataflow¶
Controller Watches¶
Kubernetes resources this controller monitors for changes. Each watch triggers reconciliation when the watched resource is created, updated, or deleted.
Reconciliation Flow¶
How the controller interacts with the Kubernetes API during reconciliation.
sequenceDiagram
%% Static dataflow for distributed-workloads
participant KubernetesAPI as Kubernetes API
participant controller_manager as controller-manager
participant kubeflow_trainer_controller_manager as kubeflow-trainer-controller-manager
participant kuberay_operator as kuberay-operator
participant kueue_controller_manager as kueue-controller-manager
participant training_operator as training-operator
KubernetesAPI->>+controller_manager: Watch Pod (reconcile)
KubernetesAPI->>+controller_manager: Watch Pod (reconcile)
KubernetesAPI->>+controller_manager: Watch Pod (reconcile)
KubernetesAPI->>+controller_manager: Watch Pod (reconcile)
KubernetesAPI->>+controller_manager: Watch Pod (reconcile)
KubernetesAPI->>+controller_manager: Watch Pod (reconcile)
KubernetesAPI->>+controller_manager: Watch StatefulSet (reconcile)
KubernetesAPI->>+controller_manager: Watch StatefulSet (reconcile)
KubernetesAPI->>+controller_manager: Watch ClusterOperator (reconcile)
KubernetesAPI->>+controller_manager: Watch ClusterOperator (reconcile)
KubernetesAPI->>+controller_manager: Watch JobSet (reconcile)
KubernetesAPI->>+controller_manager: Watch JobSet (reconcile)
KubernetesAPI->>+controller_manager: Watch AdmissionCheck (reconcile)
KubernetesAPI->>+controller_manager: Watch AdmissionCheck (reconcile)
KubernetesAPI->>+controller_manager: Watch Workload (reconcile)
KubernetesAPI->>+controller_manager: Watch Workload (reconcile)
KubernetesAPI->>+controller_manager: Watch LeaderWorkerSet (reconcile)
KubernetesAPI->>+controller_manager: Watch LeaderWorkerSet (reconcile)
KubernetesAPI->>+controller_manager: Watch Operator (reconcile)
KubernetesAPI->>+controller_manager: Watch Operator (reconcile)
KubernetesAPI->>+controller_manager: Watch ClusterServiceVersion (reconcile)
KubernetesAPI->>+controller_manager: Watch ClusterServiceVersion (reconcile)
KubernetesAPI->>+controller_manager: Watch ClusterServiceVersion (reconcile)
KubernetesAPI->>+controller_manager: Watch ClusterServiceVersion (reconcile)
KubernetesAPI->>+controller_manager: Watch Subscription (reconcile)
KubernetesAPI->>+controller_manager: Watch Subscription (reconcile)
KubernetesAPI->>+controller_manager: Watch OperatorCondition (reconcile)
KubernetesAPI->>+controller_manager: Watch OperatorCondition (reconcile)
KubernetesAPI->>+controller_manager: Watch RayCluster (reconcile)
KubernetesAPI->>+controller_manager: Watch RayCluster (reconcile)
KubernetesAPI->>+controller_manager: Watch RayJob (reconcile)
KubernetesAPI->>+controller_manager: Watch RayJob (reconcile)
KubernetesAPI->>+controller_manager: Watch RayService (reconcile)
KubernetesAPI->>+controller_manager: Watch RayService (reconcile)
KubernetesAPI->>+controller_manager: Watch TrainJob (reconcile)
KubernetesAPI->>+controller_manager: Watch TrainJob (reconcile)
controller_manager->>KubernetesAPI: Create/Update Pod
controller_manager->>KubernetesAPI: Create/Update Pod
controller_manager->>KubernetesAPI: Create/Update Secret
controller_manager->>KubernetesAPI: Create/Update Secret
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update Service
controller_manager->>KubernetesAPI: Create/Update ProvisioningRequest
controller_manager->>KubernetesAPI: Create/Update ProvisioningRequest
controller_manager->>KubernetesAPI: Create/Update Job
controller_manager->>KubernetesAPI: Create/Update Job
controller_manager->>KubernetesAPI: Create/Update Job
controller_manager->>KubernetesAPI: Create/Update Job
controller_manager->>KubernetesAPI: Create/Update JobSet
controller_manager->>KubernetesAPI: Create/Update JobSet
controller_manager->>KubernetesAPI: Create/Update Workload
controller_manager->>KubernetesAPI: Create/Update Workload
controller_manager->>KubernetesAPI: Create/Update Workload
controller_manager->>KubernetesAPI: Create/Update Workload
controller_manager->>KubernetesAPI: Create/Update Workload
controller_manager->>KubernetesAPI: Create/Update Workload
controller_manager->>KubernetesAPI: Create/Update RayCluster
controller_manager->>KubernetesAPI: Create/Update RayCluster
controller_manager->>KubernetesAPI: Create/Update RayCluster
controller_manager->>KubernetesAPI: Create/Update RayCluster
KubernetesAPI-->>+controller_manager: Watch ConfigMap (informer)
KubernetesAPI-->>+controller_manager: Watch ConfigMap (informer)
KubernetesAPI-->>+controller_manager: Watch ConfigMap (informer)
KubernetesAPI-->>+controller_manager: Watch ConfigMap (informer)
KubernetesAPI-->>+controller_manager: Watch LimitRange (informer)
KubernetesAPI-->>+controller_manager: Watch LimitRange (informer)
KubernetesAPI-->>+controller_manager: Watch Namespace (informer)
KubernetesAPI-->>+controller_manager: Watch Namespace (informer)
KubernetesAPI-->>+controller_manager: Watch Namespace (informer)
KubernetesAPI-->>+controller_manager: Watch Namespace (informer)
KubernetesAPI-->>+controller_manager: Watch Namespace (informer)
KubernetesAPI-->>+controller_manager: Watch Namespace (informer)
KubernetesAPI-->>+controller_manager: Watch Pod (informer)
KubernetesAPI-->>+controller_manager: Watch Pod (informer)
KubernetesAPI-->>+controller_manager: Watch Pod (informer)
KubernetesAPI-->>+controller_manager: Watch Pod (informer)
KubernetesAPI-->>+controller_manager: Watch Pod (informer)
KubernetesAPI-->>+controller_manager: Watch Pod (informer)
KubernetesAPI-->>+controller_manager: Watch Secret (informer)
KubernetesAPI-->>+controller_manager: Watch Secret (informer)
KubernetesAPI-->>+controller_manager: Watch Secret (informer)
KubernetesAPI-->>+controller_manager: Watch Secret (informer)
KubernetesAPI-->>+controller_manager: Watch Service (informer)
KubernetesAPI-->>+controller_manager: Watch Service (informer)
KubernetesAPI-->>+controller_manager: Watch ServiceAccount (informer)
KubernetesAPI-->>+controller_manager: Watch ServiceAccount (informer)
KubernetesAPI-->>+controller_manager: Watch ServiceAccount (informer)
KubernetesAPI-->>+controller_manager: Watch ServiceAccount (informer)
KubernetesAPI-->>+controller_manager: Watch CustomResourceDefinition (informer)
KubernetesAPI-->>+controller_manager: Watch CustomResourceDefinition (informer)
KubernetesAPI-->>+controller_manager: Watch CustomResourceDefinition (informer)
KubernetesAPI-->>+controller_manager: Watch CustomResourceDefinition (informer)
KubernetesAPI-->>+controller_manager: Watch APIService (informer)
KubernetesAPI-->>+controller_manager: Watch APIService (informer)
KubernetesAPI-->>+controller_manager: Watch APIService (informer)
KubernetesAPI-->>+controller_manager: Watch APIService (informer)
KubernetesAPI-->>+controller_manager: Watch Deployment (informer)
KubernetesAPI-->>+controller_manager: Watch Deployment (informer)
KubernetesAPI-->>+controller_manager: Watch Deployment (informer)
KubernetesAPI-->>+controller_manager: Watch Deployment (informer)
KubernetesAPI-->>+controller_manager: Watch Deployment (informer)
KubernetesAPI-->>+controller_manager: Watch Deployment (informer)
KubernetesAPI-->>+controller_manager: Watch AdmissionCheck (informer)
KubernetesAPI-->>+controller_manager: Watch AdmissionCheck (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterQueue (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterQueue (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterQueue (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterQueue (informer)
KubernetesAPI-->>+controller_manager: Watch LocalQueue (informer)
KubernetesAPI-->>+controller_manager: Watch LocalQueue (informer)
KubernetesAPI-->>+controller_manager: Watch ProvisioningRequestConfig (informer)
KubernetesAPI-->>+controller_manager: Watch ProvisioningRequestConfig (informer)
KubernetesAPI-->>+controller_manager: Watch ProvisioningRequestConfig (informer)
KubernetesAPI-->>+controller_manager: Watch ProvisioningRequestConfig (informer)
KubernetesAPI-->>+controller_manager: Watch ResourceFlavor (informer)
KubernetesAPI-->>+controller_manager: Watch ResourceFlavor (informer)
KubernetesAPI-->>+controller_manager: Watch Workload (informer)
KubernetesAPI-->>+controller_manager: Watch Workload (informer)
KubernetesAPI-->>+controller_manager: Watch Workload (informer)
KubernetesAPI-->>+controller_manager: Watch Workload (informer)
KubernetesAPI-->>+controller_manager: Watch Workload (informer)
KubernetesAPI-->>+controller_manager: Watch Workload (informer)
KubernetesAPI-->>+controller_manager: Watch RuntimeClass (informer)
KubernetesAPI-->>+controller_manager: Watch RuntimeClass (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterServiceVersion (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterServiceVersion (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterServiceVersion (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterServiceVersion (informer)
KubernetesAPI-->>+controller_manager: Watch InstallPlan (informer)
KubernetesAPI-->>+controller_manager: Watch InstallPlan (informer)
KubernetesAPI-->>+controller_manager: Watch InstallPlan (informer)
KubernetesAPI-->>+controller_manager: Watch InstallPlan (informer)
KubernetesAPI-->>+controller_manager: Watch Subscription (informer)
KubernetesAPI-->>+controller_manager: Watch Subscription (informer)
KubernetesAPI-->>+controller_manager: Watch Subscription (informer)
KubernetesAPI-->>+controller_manager: Watch Subscription (informer)
KubernetesAPI-->>+controller_manager: Watch OperatorCondition (informer)
KubernetesAPI-->>+controller_manager: Watch OperatorCondition (informer)
KubernetesAPI-->>+controller_manager: Watch OperatorCondition (informer)
KubernetesAPI-->>+controller_manager: Watch OperatorCondition (informer)
KubernetesAPI-->>+controller_manager: Watch OperatorCondition (informer)
KubernetesAPI-->>+controller_manager: Watch OperatorCondition (informer)
KubernetesAPI-->>+controller_manager: Watch RayCluster (informer)
KubernetesAPI-->>+controller_manager: Watch RayCluster (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRole (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRole (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRole (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRole (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch ClusterRoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch Role (informer)
KubernetesAPI-->>+controller_manager: Watch Role (informer)
KubernetesAPI-->>+controller_manager: Watch Role (informer)
KubernetesAPI-->>+controller_manager: Watch Role (informer)
KubernetesAPI-->>+controller_manager: Watch Role (informer)
KubernetesAPI-->>+controller_manager: Watch Role (informer)
KubernetesAPI-->>+controller_manager: Watch RoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch RoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch RoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch RoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch RoleBinding (informer)
KubernetesAPI-->>+controller_manager: Watch RoleBinding (informer)
Note over controller_manager: Exposed Services
Note right of controller_manager: kuberay-operator:8080/TCP [monitoring-port]
Note right of controller_manager: kuberay-operator:8080/TCP [monitoring-port]
Note right of controller_manager: training-operator:8080/TCP [monitoring-port]
Note right of controller_manager: training-operator:8080/TCP [monitoring-port]
Note right of controller_manager: visibility-server:443/TCP [https]
Note right of controller_manager: visibility-server:443/TCP [https]
Note right of controller_manager: webhook-service:443/TCP []
Note right of controller_manager: webhook-service:443/TCP []
Note right of controller_manager: webhook-service:443/TCP []
Note right of controller_manager: webhook-service:443/TCP []
Note right of controller_manager: webhook-service:443/TCP []
Note right of controller_manager: webhook-service:443/TCP []
Webhooks¶
HTTP Endpoints¶
Configuration¶
ConfigMaps and Helm values that control this component's runtime behavior.
Helm¶
Chart: kubeflow-trainer v2.1.0