EAS FAQ

Q: The service has been stuck in Waiting state for a long time. How do I fix this?

After you deploy a service, it enters the Waiting state while resources are scheduled and instances start. When all instances start successfully, the service transitions to Running state. The following scenarios may cause a prolonged Waiting state:

[2025-07-18 18:50:46] WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
[2025-07-18 18:50:50]  * Serving Flask app 'service'
[2025-07-18 18:50:50]  * Debug mode: off
[2025-07-18 18:50:50] WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
[2025-07-18 18:50:50]  * Running on all addresses (0.0.0.0)
[2025-07-18 18:50:50]  * Running on http://127.0.0.1:7000
[2025-07-18 18:50:50]  * Running on http://xxx:7000
[2025-07-18 18:50:50] Press CTRL+C to quit

Q: The service is in Failed state. How do I fix this?

A service enters the Failed state in two situations:

• During deployment: If a resource specified at deployment time (such as a model path) doesn't exist, the service status shows the error reason. The error message usually indicates why the deployment failed.

• During startup: If the service fails after deployment succeeds and scheduling begins, the status shows: Instance <network-test-5ff76448fd-h9dsn> not healthy: Instance crashed, please inspect instance log.

  This message means one or more instances failed to start. Check the Service instances list on the service Overview page to identify the specific cause. Common failure causes include:

  • The instance ran out of memory during startup and the system killed it (OOMKilled). Redeploy the service with a higher memory allocation. In the instance list, the Last status column shows OOMKilled(247), indicating the container was terminated for exceeding the memory limit. Check the Restart count and Last exit reason columns for further confirmation.

  • A code error caused a crash during startup. In this case, Last status shows Error(error code). Click Log in the Actions column of the instance to review the service logs and identify the startup failure.

  • The service image failed to pull. See What do I do if image pull fails (ImagePullBackOff)?

Q: What do I do if image pull fails (ImagePullBackOff)?

If the Last exit reason column in the service instance list shows ImagePullBackOff, the image pull has failed. If an icon appears in the Status column, click it to view the specific reason.

Common image pull failure causes are listed below:

Based on the regions where the image registry and EAS service are located, use the following guidance:

• Same region: Pull images using the private image address.

• Cross-region: ACR Personal Edition supports public image addresses only. For ACR Enterprise Edition, choose based on your requirements:

  • For higher security and reliability, use a private image address. This requires connecting the VPCs via CEN. See Access ACR Enterprise Edition instances cross-region or from IDC.

  • For simpler scenarios or when private network connectivity isn't available yet, use a public image address as a temporary option. Public network downloads are slower.

Additional notes for ACR Enterprise Edition:

• Configure access control for both VPC and public network access as needed.

• If the repository doesn't allow anonymous pulls, cross-region pulls over the public address require configuring the image registry username and password in the EAS service.

Q: The EAS service restarts automatically after being stopped. What's happening?

Problem description: An EAS service restarts on its own after being stopped for a period.

Root cause:

The service has auto-scaling configured with a minimum instance count of 0. When traffic stops for a period of time, the instance count automatically scales down to 0. If a request arrives when no instances are available due to scale-down, EAS triggers a scale-up automatically — this doesn't require the configured scale-up threshold to be reached.

Check the deployment events for auto scaling messages to confirm whether auto scale-up occurred. A Service is now auto scaling entry in the information column indicates the service was triggered to scale up automatically, after which the event status transitions from Waiting through Scaling to Running, and the available instance count changes from 0 to 1.

Solution:

• If the service is no longer needed, delete it.

• To keep the service but prevent automatic restart, stop it through the console or by calling the StopService API. A manually stopped service won't scale up when traffic arrives.

• To prevent auto-scaling from stopping the service, don't set the minimum instance count to 0.

• Disable auto-scaling entirely if unexpected traffic triggering a scale-up is a concern.

Q: PAI-EAS startup fails with IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" })

Problem description:

[2024-10-21 20:59:33] serialize_file(_flatten(tensors), filename, metadata=metadata)

[2024-10-21 20:59:33] safetensors_rust.SafetensorError: Error while serializing: IoError(Os { code: 28, kind: StorageFull, message: "No space left on device" })

[2024-10-21 20:59:35] time="2024-10-21T12:59:35Z" level=info msg="program stopped with status:exit status 1" program=/bin/sh

Root cause: The EAS instance's system disk is full because of large model files. This prevents the service from starting.

Solutions:

Option 1: Expand the system disk for the EAS instance.

Option 2: If the model files are too large, store them in external storage (OSS, NAS, or similar) and read them using Storage mounts.

Q: Deployment fails with fail to start program with error: fork/exec /bin/sh: exec format error

The exec format error means the operating system can't run the target binary. The most common cause is a CPU architecture mismatch between the executable or container image and the host machine.

Try a different resource specification.

Q: Error: Invalid GPU count 6, only supported: [0 1 2 4 8 16]

The number of GPUs per service must be a power of 2 to maximize inter-GPU communication efficiency.

Supported GPU counts per service: 0, 1, 2, 4, 8, or 16.

Q: An EAS instance fails to delete, or clicking Stop has no effect. What do I do?

This issue has two common scenarios:

1. Instance deletion fails or returns an error: If the delete operation reports an error such as the associated workspace not existing, first check whether the associated workspace is in a normal state. You can also go to the service list in the PAI console and click Delete directly to release the instance.

2. Clicking Stop has no effect (the instance stays in Running state):

   • Browser-side checks: Refresh the page, clear the browser cache, or retry in a different browser or in incognito/private mode.

   • Request diagnosis: Open the browser developer tools (F12), go to the Network panel, click Stop, and check whether the API request is sent successfully and inspect the response status code.

   • Log check: Check the Operation logs or Cloud Monitor in the PAI console for a stop record and the specific failure reason.

   • API alternative: If the console operation still doesn't work, use OpenAPI Explorer to call the StopService API to force-stop the service.

Q: Nothing happens when I add a model link in the Shared Edition. What do I do?

The Shared Edition doesn't support custom model links, so clicking the entry does nothing. This is a feature limitation, not an error.

Use one of the following workarounds:

• Deploy the Dedicated Edition and add custom model links there.

• Download the model files manually, upload them to OSS or NAS, and load the model through a storage mount when you deploy the service.

Call errors

Troubleshoot based on the returned HTTP status code. For details, see Service status codes and errors.

HTTPS and custom domains

Q: Does EAS support HTTPS calls?

Yes. Replace http:// with https:// in the service endpoint URL to enable encrypted transport. If the client (such as the Python requests library) reports an SSL certificate verification error, this is a client-side configuration issue, not an EAS issue.

Q: How do I force HTTPS-only access?

• Shared gateway: HTTPS redirect is not supported.

• Dedicated gateway: Supported. Enable HTTPS redirect in the dedicated gateway configuration. Once enabled, all HTTP requests are automatically redirected to HTTPS.

Q: Can I call a service using a custom domain?

Yes. Create and use a fully managed dedicated gateway, then configure your custom domain in the gateway settings. For details, see Use a dedicated gateway.

Token management

Q: Does the service token expire or change?

No. The Token generated when a service is deployed is long-lived. Restarting, updating (unless you manually change the authentication method), or scaling the service doesn't change the Token. The Token is invalidated only when you manually reset it or delete the service.

Q: Can I create multiple tokens for one service?

No. Each EAS service instance supports a single authentication Token. For multi-user permission management or separate usage tracking, use Alibaba Cloud RAM-based authentication or a similar access control solution.

Other call issues

Q: How do I enable streaming responses for an LLM service?

EAS doesn't have a global streaming toggle. Specify streaming in each API request body individually. For example, when calling an OpenAI-compatible LLM service, include "stream": true in the JSON request body.

Q: What's the difference between VPC address calls and VPC direct connect?

• VPC address calls: Requests go through an internal SLB plus a gateway (public address calls go through a public SLB plus a gateway). This is the standard request path. Each request passes through layer-4 forwarding at the SLB and layer-7 forwarding at the gateway before reaching the service instance. Under high concurrency and heavy traffic, this forwarding adds latency overhead. The gateway also has a bandwidth ceiling (1 Gbps by default).

• VPC direct connect: EAS provides a high-speed direct connect mode that addresses both performance and scalability without additional cost. Enabling VPC direct connect establishes a network path between your VPC and the EAS service VPC. Your requests use EAS service discovery to locate the service, then perform client-side software load balancing. This requires the EAS SDK and setting endpoint_type to DIRECT.

  For example, in the Python SDK scenario, add the following line to your code to switch from gateway calls to direct connect:

  • client = PredictClient('http://pai-eas-vpc.cn-hangzhou.aliyuncs.com', 'mnist_saved_model_example')
    client.set_token('M2FhNjJlZDBmMzBmMzE4NjFiNzZhMmUxY2IxZjkyMDczNzAzYjFi****')
    client.set_endpoint_type(ENDPOINT_TYPE_DIRECT) # Direct link
    client.init()

Q: Why can't a RAM user automatically create or delete the EAS service-linked role?

Only users with specific permissions can create or delete AliyunServiceRoleForPaiEas automatically. If a RAM user can't perform this action, grant them the required policy as follows:

1. Create the following policy as a custom policy using the script-based configuration method. For details, see Create a custom policy.

   Policy for creating or deleting a service-linked role

   {
     "Statement": [
       {
         "Action": "ram:CreateServiceLinkedRole",
         "Resource": "*",
         "Effect": "Allow",
         "Condition": {
           "StringEquals": {
             "ram:ServiceName": "eas.pai.aliyuncs.com"
           }
         }
       }
     ],
     "Version": "1"
   }

2. Attach the custom policy created in the previous step to the target RAM user. For details, see Manage RAM user permissions.

Q: How does an EAS service access the internet?

EAS services don't have internet access by default. To enable internet access, configure a VPC with internet connectivity for the EAS service. See EAS access to public and private resources.

Q: Does EAS support SSH access to instances?

No. EAS is a managed service and doesn't provide SSH access into containers. To run specific commands when the container starts, specify them in the Run Command field of the service configuration.

Q: What service statuses does EAS have?

EAS services have the following statuses. You can also view the status in the status column on the Model Serving (EAS) page.

• Creating

• Waiting — waiting for instances to start

• Stopped

• Failed

• Updating — instances are being updated

• Stopping

• HotUpdate — updating without replacing instances

• Starting

• DeleteFailed

• Running

• Scaling — instances are scaling

• Pending — waiting for processing

• Deleting

• Completed

• Preparing

Q: How do I find out which RAM user created a service?

Query events in the ActionTrail console. Set the event name filter to CreateService. For details, see Query events in the ActionTrail console.

Q: Can I download PAI official images from the internet?

No. PAI official images are platform-internal images and can only be used within the PAI platform. They can't be downloaded outside of platform containers.

Q: What are the uniqueness rules for PAI-EAS service names and service IDs?

• Service name uniqueness: EAS service names are globally unique within a single region. If you see a "service name already exists" error when creating a service but the name doesn't appear in the Model Serving (EAS) list, another user in the same region has taken that name. Choose a different name and try again.

• Service ID uniqueness: PAI-EAS service IDs are globally unique and assigned automatically by the system. You can't specify them manually.

Q: Can the service name be modified after the service is deployed?

No. The service name cannot be modified after it is created. As the unique identifier for API calls and management operations, the service name field is disabled in the Update form in the console. If you need to change the service name, create a new service and delete the original service. Note that this will result in a service interruption and an IP address change.

Q: Do I need to repurchase EAS after disabling the module? What if I can't find my service instances?

• No repurchase needed: Once activated, PAI-EAS remains available indefinitely. To use it again, go to the PAI console, select or create a workspace, and create an EAS instance in that workspace. No additional purchase or reactivation is required.

• If you can't find your instances:

  1. Log in to the PAI console and verify that the Workspace shown at the top of the page is the one you're looking for.

  2. If the workspace was deleted or switched, look for the Model Serving (EAS) list from the Overview page or within the target workspace.

  3. If the list shows no data and no instances, no billable EAS instances have been created in that workspace. Activating the module alone doesn't incur charges and requires no additional cleanup.

Q: When do storage mount configuration changes take effect? Do I need to redeploy the service?

Storage mount settings are part of the deployment configuration of a service and don't take effect dynamically. To apply the changes, update the service. After the update completes, the service instances restart with the new configuration and the new mounts take effect.

Q: What do I do if mounted files disappear after I update the service configuration?

This issue is usually caused by one of the following:

• The update removed the original storage mount settings or changed the mount path, so instances no longer mount the original storage. The files aren't actually lost. They remain in the original OSS or NAS.

• The files were written to the temporary local disk inside the container instead of the mounted directory. Local disk data is cleared when an instance restarts or is updated.

In the first case, restore the mount settings to the pre-update values and update the service again. The files will reappear. In the second case, the data cannot be recovered. Always write data that must persist to the OSS or NAS mount directory, and verify that the mount settings are complete and correct before you update the service.

Q: Why can't I select an OSS bucket when deploying an EAS service?

When deploying an EAS service, you can mount models and code by specifying storage locations. Make sure the OSS bucket and NAS file system are in the same region as the EAS service. Cross-region storage can't be selected.

Q: How is OSS storage billed when mounted to an EAS service? How do I estimate the required capacity?

EAS itself doesn't charge for storage capacity. When you mount OSS, OSS bills the storage based on your actual usage and storage class. EAS charges only for the compute resources of the service. To estimate capacity, use the actual sizes of your model, code, and data files. OSS storage scales elastically with usage, so there is no need to purchase a fixed capacity in advance.

Q: Q: EAS fails to access OSS via the default public endpoint. How to resolve?

When accessing OSS resources through a custom domain, if the OSS region is in mainland China, the domain must complete ICP filing as required by regulations. Domains without ICP filing cannot be used for access.

Use the following alternatives:

• Use the internal endpoint.

• For public network access, use a custom domain.When accessing OSS resources through a custom domain, if the OSS region is in mainland China, the domain must complete ICP filing as required by regulations. Domains without ICP filing cannot be used for access.

Q: How do I switch an EAS service from pay-as-you-go to subscription billing?

• EAS services support switching from pay-as-you-go to subscription billing.

  Log in to the PAI console, go to the Model Serving (EAS) page, and click Update in the Actions column for the target service. In the Resource info section, change Resource type from Public resource to EAS resource group. On the model deployment configuration page, the Resource info section provides three resource type options: Public resource, EAS resource group, and Resource quota. Configure the Resource group, Instance count, and deployment resources such as CPU (cores), Memory (GB), and GPU (cards).

• Dedicated EAS resource group resources don't support switching from pay-as-you-go to subscription billing.

Q: TensorFlow issues

For details, see TensorFlow FAQ.