Application distribution status and FAQ

Updated at:

This topic explains how application distribution works, describes distribution statuses, and provides troubleshooting for common issues.

How it works

image
  1. Resource matching: The system matches application resources with distribution policies. If a distribution policy selects an existing application resource in the cluster, a ResourceBinding object is created. This object has a one-to-one correspondence with the application resource. The ResourceBinding object is named using the ${resource name}-${resource Kind} format, allowing you to query it directly by name. For cluster-scoped application resources, the corresponding object is a ClusterResourceBinding.

  2. Multi-cluster scheduling: The system schedules the ResourceBinding object based on the created distribution policy. After scheduling is complete, the SCHEDULED condition in the status of the ResourceBinding is set to True.

  3. Differentiated rendering: If you create a differentiated policy for the application resource, the policy takes effect at this stage.

  4. Application resource creation on sub-clusters: The application resource is created on the target sub-clusters. After successful creation, the FULLYAPPLIED condition in the status of the ResourceBinding is set to True.

  5. Application resources run on sub-clusters: When workload-type application resources run on a sub-cluster, their status changes to Ready. When an application resource is ready, the ALLAVAILABLE condition in the status of the ResourceBinding is set to True.

Application distribution to multiple clusters

Step 1: Create resources and policies

  1. Use the following sample YAML to create a resource and a distribution policy.

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-demo
    spec:
      replicas: 5
      selector:
        matchLabels:
          app: web-demo
      template:
        metadata:
          labels:
            app: web-demo
        spec:
          containers:
          - name: nginx
            image: registry-cn-hangzhou.ack.aliyuncs.com/acs/web-demo:0.5.0
            ports:
            - containerPort: 80
    ---
    apiVersion: policy.one.alibabacloud.com/v1alpha1
    kind: PropagationPolicy
    metadata:
      name: web-demo
    spec:
      # If this parameter is set to true, resources on sub-clusters are retained when you delete the resource from the Fleet instance. To also delete the resources on sub-clusters, set this parameter to false.
      preserveResourcesOnDeletion: true
      resourceSelectors:
      - apiVersion: apps/v1
        kind: Deployment
        name: web-demo
      placement:
        clusterAffinity:
          clusterNames:
          - ${cluster1-id} # Your cluster ID.
          - ${cluster2-id} 
        replicaScheduling:
          replicaSchedulingType: Duplicated
  2. (Optional) Use the following sample YAML to create a differentiated policy.

    apiVersion: one.alibabacloud.com/v1alpha1
    kind: OverridePolicy
    metadata:
      name: demo
    spec:
      resourceSelectors:
        - apiVersion: apps/v1
          kind: Deployment
          name: web-demo
      overrideRules:
        - targetCluster:
            clusterNames:
              - ${cluster1-id} # Your cluster ID.
          overriders:
            plaintext:
              - operator: replace
                path: /spec/replicas
                value: 3

Step 2: Check distribution status

  1. Run the following command to check the distribution status.

    kubectl get rb web-demo-deployment

    Expected output:

    NAME                  SCHEDULED   FULLYAPPLIED  OVERRIDDEN  ALLAVAILABLE   AGE
    web-demo-deployment   True        True          True        True           17h

    Parameter

    Description

    SCHEDULED

    Indicates whether the resource was successfully scheduled.

    FULLYAPPLIED

    Indicates whether the resource was successfully created on the sub-clusters.

    OVERRIDDEN

    Indicates whether the differentiated policy was applied. This status is True by default and becomes False only if a defined policy fails to apply.

    ALLAVAILABLE

    Indicates whether the resource has reached its desired state on the sub-clusters.

  2. (Optional) Verify the differentiated policy.

    • Run the kubectl get rb web-demo-deployment command and check the OVERRIDDEN column in the output. If the value is True, the differentiated policy has taken effect.

    • Run the kubectl describe deploy web-demo command and check the events to determine if the differentiated policy was applied. The following output indicates success:

        Normal   ApplyOverridePolicySucceed  2m37s (x3 over 49m)     override-manager                    Apply override policy(default/demo) for cluster(cxxxxxxx) succeed.

Step 3: View specific resource status

Run the following command to view the status of specific application resources.

kubectl get rb web-demo-deployment -oyaml

Expected output:

...
status:
  aggregatedStatus:
  - applied: true
    appliedMessage: 'Applied Condition Message: Apply manifest complete; AvailableCondition
      Message:Resource is available'
    clusterName: ${cluster1-id}
    health: Healthy
    status:
      availableReplicas: 3
      generation: 7
      observedGeneration: 7
      readyReplicas: 3
      replicas: 3
      resourceTemplateGeneration: 2
      updatedReplicas: 3
  - applied: true
    appliedMessage: 'Applied Condition Message: Apply manifest complete; AvailableCondition
      Message:Resource is available'
    clusterName: ${cluster2-id}
    health: Healthy
    status:
      availableReplicas: 5
      generation: 1
      observedGeneration: 1
      readyReplicas: 5
      replicas: 5
      resourceTemplateGeneration: 2
      updatedReplicas: 5

The aggregatedStatus section shows the status of the deployment on both clusters, including the cluster name and the corresponding deployment status details. Note that the differentiated policy set the replica count for Cluster1 to 3.

FAQ

Why is no ResourceBinding or ClusterResourceBinding created?

This issue usually occurs when the resourceSelector parameter in the distribution policy does not match the application resource. Check whether the conditions declared in ResourceSelectors match your application resource.

How to handle scheduling failures?

status:
  conditions:
  - lastTransitionTime: "2025-03-03T02:06:21Z"
    message: '0/2 clusters are available: 2 cluster(s) did not match the placement
      cluster affinity constraint.'
    reason: NoClusterFit
    status: "False"
    type: Scheduled

You can run the following command to view the details of the ResourceBinding:

kubectl get rb xxx -oyaml -n xxx

Scheduling typically fails because the clusterAffinity parameter in your distribution policy does not match any of your sub-clusters. Ensure that you specify the correct cluster IDs.

How to handle a failed differentiated policy?

First, check if the resourceSelector parameter of the differentiated policy selects the intended application resource. You can troubleshoot this issue in one of the following ways:

  • Run the following command to check the message field in the ResourceBinding for failure details.

    kubectl get rb ${resource name}-${resource kind} -oyaml -n ${namespace}

    Expected output:

    - lastTransitionTime: "2025-03-04T08:46:46Z"
      message: 'Failed to apply OverridePolicy/ClusterOverridePolicy, err is: failed
        to apply overrides(default/demo) for resource(default/web-demo), error: replace
        operation does not apply: doc is missing key: /spec/replicas: missing value'
      reason: OverriddenFailed
      status: "False"
      type: Overridden
  • Run the following command to view the events related to the resource. Look for key events like ApplyOverridePolicyFailed and SyncWorkFailed. The message field provides details about the failure.

    kubectl describe ${resource-kind} ${resource-name} -n ${resource-namespace}

A common reason a differentiated policy fails is an incorrect path configuration, such as specifying a path that does not exist in the resource.

Why does scheduling succeed but creation fails?

If scheduling is successful but the FULLYAPPLIED status is False, view the status of specific application resources. Check the appliedMessage field for each cluster to find the reason for the failure.

This failure often occurs when you try to distribute an unsupported application resource. You can submit a ticket for assistance.

Why is a created resource not ready?

If an application resource is created on a sub-cluster but the ALLAVAILABLE status is False, you can view the status of specific application resources. Check the appliedMessage field for each cluster to find out why the resource is not ready.

Typically, resources like ConfigMaps and Secrets are considered ready as soon as they are created. However, for workload-type application resources such as Deployments and StatefulSets, the ready state can be affected by various factors, including insufficient underlying resources or scheduling failures on the sub-cluster. You can connect to your sub-cluster or use AMC to investigate and resolve the issue.