Managing the infrastructure for machine learning model serving is complex and resource-intensive. KServe runs your trained models — including TensorFlow Serving (TFServing), TorchServe, and Triton Inference Server — as Kubernetes Custom Resource Definitions (CRDs) on ACS, with built-in auto scaling and scale-to-zero support. Deploy KServe through the ACS console to enable serverless model serving on your cluster.
How KServe works
The KServe Controller manages InferenceService custom resources and creates Knative Services to automate resource scaling.
When traffic increases, the KServe Controller scales the Deployment of a Knative Service accordingly. When no requests arrive, it scales the Service pods to zero. This auto scaling mechanism maximizes resource efficiency and reduces idle costs.
Model serving runtimes
KServe includes two built-in model serving runtimes:
|
Runtime |
Description |
|
ModelServer |
A Python-based runtime that implements KServe prediction protocol v1 |
|
MLServer |
A runtime that implements KServe prediction protocol v2 with both REST and gRPC support |
Both runtimes provide out-of-the-box model serving. For more complex use cases, build a custom model server using KServe's API primitives or tools such as BentoML.
Serverless features
After deploying models with a KServe InferenceService, the following serverless features are available:
Scale to zero
Auto scaling based on requests per second (RPS), concurrency, and CPU and GPU metrics
Version management
Traffic management
Security authentication
Out-of-the-box metrics
Deploy KServe
KServe on ACS runs in Knative-backed mode, which uses Knative's serverless infrastructure to handle auto scaling and traffic routing. Knative must be running on your cluster before you install KServe.
Prerequisites
Before you begin, ensure that you have:
Knative deployed in your ACS cluster. For more information, see Deploy Knative
Install KServe from the console
Log on to the ACS console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the target cluster and click its ID. In the left-side navigation pane, choose Applications > Knative.
On the Components tab, find KServe and click Deploy in the Actions column. Click Confirm in the dialog box. The deployment may take several minutes to complete.
Verify the deployment
After the deployment completes, check the Status column of the KServe component on the Components tab. A status of Deployed confirms that the component is installed.
To verify from the command line, run the following command and confirm that the KServe controller pods are running:
kubectl get pods -n kserveExpected output:
NAME READY STATUS RESTARTS AGE
kserve-controller-manager-xxxxxxxxxx-xxxxx 2/2 Running 0 2mIf the Status column does not show Deployed, or the pods are not in Running state, check the controller logs for errors: run kubectl logs -n kserve -l control-plane=kserve-controller-manager. Also verify that all Knative components are healthy before reinstalling KServe.