全部产品

使用镜像缓存 CRD 加速创建 Pod

更新时间:2020-01-06 11:20:30

背景

ECI容器组实例创建过程中,大部分时间消耗在镜像下载阶段,为加速ECI容器组创建速度,ECI提供镜像缓存功能,用户事先将需要用到的镜像作为云盘快照缓存,在创建ECI容器组实例时基于快照创建,避免或减少镜像层下载,从而提升ECI容器组实例创建速度。经实测,基于dockerhub的flink镜像(386.26MB)创建Pod,普通创建eci过程中,镜像准备阶段需要耗费50s,使用镜像快照后镜像准备阶段仅需5s。因此在创建eci过程中,镜像准备阶段节省时间45s。具体的提升速度跟Pod使用的镜像个数,镜像大小和镜像仓库网络因素决定。

为方便kubernetes用户也可以使用到ECI的镜像缓存功能,因此以CRD的方式将ECI镜像缓存功能暴露给kubernetes用户。重点强调:ImageCache CRD在kubernetes集群中为Cluster级别的资源,类似于PV,被集群内所有namespace共享。

准备工作

验证您的kubernetes集群是否支持ImageCache:

  1. $ kubectl get crd/imagecaches.eci.alibabacloud.com
  2. NAME CREATED AT
  3. imagecaches.eci.alibabacloud.com 2019-09-27T01:15:07Z

输出以上信息,证明您的kubernetes集群已经支持ImageCache。

  1. kubectl get crd/imagecaches.eci.alibabacloud.com
  2. Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "imagecaches.eci.alibabacloud.com" not found

输出以上信息,证明您的kubernetes集群不支持ImageCache,需要升级virtual-kubelet实现支持ImageCache。

virtual-kubelet升级方式:

  • 阿里云 serverless kubernetes集群:由管理员统一负责升级
  • 阿里云 托管 kubernetes集群:用户更新
  • 阿里云 专有 kubernetes集群:用户更新
  • 阿里云 自建 kubernetes集群:用户更新

准备工作完成后,您可以在您的kubernetes集群中,创建ImageCache资源,并且可以使用ImageCache创建Pod。

注:如果您的集群的ImageCache CRD不是最新的,不支持ImageCache CRD新增的参数,建议升级完virtual-kubelet后,执行如下命令:

  1. kubectl apply -f imagecache-crd-sample.yaml

imagecache-crd-sample.yaml:

  1. apiVersion: apiextensions.k8s.io/v1beta1
  2. kind: CustomResourceDefinition
  3. metadata:
  4. name: imagecaches.eci.alibabacloud.com
  5. spec:
  6. group: eci.alibabacloud.com
  7. version: v1
  8. names:
  9. kind: ImageCache
  10. plural: imagecaches
  11. shortNames:
  12. - ic
  13. categories:
  14. - all
  15. scope: Cluster
  16. subresources:
  17. status: {}
  18. validation:
  19. openAPIV3Schema:
  20. required:
  21. - spec
  22. properties:
  23. spec:
  24. type: object
  25. required:
  26. - images
  27. properties:
  28. imagePullSecrets:
  29. type: array
  30. items:
  31. type: string
  32. images:
  33. minItems: 1
  34. type: array
  35. items:
  36. type: string
  37. imageCacheSize:
  38. type: integer
  39. retentionDays:
  40. type: integer
  41. additionalPrinterColumns:
  42. - name: Age
  43. type: date
  44. JSONPath: .metadata.creationTimestamp
  45. - name: CacheId
  46. type: string
  47. JSONPath: .status.imageCacheId
  48. - name: Phase
  49. type: string
  50. JSONPath: .status.phase
  51. - name: Progress
  52. type: string
  53. JSONPath: .status.progress

ImageCache基本操作

创建 CustomResourceDefinition 对象后,您可以操作ImageCache资源。

ImageCache全参数YAML文件:

  1. apiVersion: eci.alibabacloud.com/v1
  2. kind: ImageCache
  3. metadata:
  4. name: imagecache-sample
  5. spec:
  6. images:
  7. - centos:latest
  8. - busybox:latest
  9. imagePullSecrets:
  10. - default:secret1
  11. - default:secret2
  12. - kube-system:secret3
  13. imageCacheSize:
  14. 25
  15. retentionDays:
  16. 7

参数说明:

名称 类型 必选 描述
spec.images []string Y 镜像列表
spec.imagePullSecrets []string N 镜像仓库对应的Secret列表, 格式:。如果images列表中有用户私有仓库镜像,需要用户为私有镜像仓库创建secret,然后根据namespace:secretName格式,设置此参数。如若images中均是共有镜像,不需要设置此参数
spec.imageCacheSize int N 镜像缓存使用的快照盘大小,默认为20GB,取值范围为:[20, 32768],单位为GB
spec.retentionDays int N 镜像缓存保留的时间,取值范围[1,65536],默认永不过期。

创建ImageCache

创建ImageCahce:

  1. # kubectl create -f imagecache-secrets-test.yaml
  2. apiVersion: eci.alibabacloud.com/v1
  3. kind: ImageCache
  4. metadata:
  5. name: imagecache-sample-liu
  6. spec:
  7. images:
  8. - centos:latest
  9. - busybox:latest
  10. imagePullSecrets:
  11. - default:secret1
  12. - default:secret2
  13. - kube-system:secret3
  14. imageCacheSize:
  15. 25
  16. retentionDays:
  17. 7

查看ImageCache状态:

  1. liumihustdeMacBook-Pro:vk-debug liumihust$ kubectl get imagecache imagecache-sample-liu
  2. NAME AGE CACHEID PHASE PROGRESS
  3. imagecache-sample-liu 20h imc-2zeditzeoemfhqor4rf8 Ready 100%

查询ImageCache

获取Cluster下全部ImageCache列表:

  1. kubectl get imagecache imagecache-sample-liu -o yaml

查看ImageCache详情(包括属性、状态以及事件):

  1. $ kubectl get imagecache/imagecache-secrets-test -o yaml
  2. apiVersion: eci.alibabacloud.com/v1
  3. kind: ImageCache
  4. metadata:
  5. creationTimestamp: "2019-10-24T13:40:56Z"
  6. generation: 1
  7. name: imagecache-sample-liu
  8. resourceVersion: "212484735"
  9. selfLink: /apis/eci.alibabacloud.com/v1/imagecaches/imagecache-sample-liu
  10. uid: e83a1c53-f663-11e9-a0d4-fa13fc175b32
  11. spec:
  12. imageCacheSize: 25
  13. images:
  14. - busybox:latest
  15. - nginx
  16. status:
  17. events:
  18. - count: 1
  19. eventTime: null
  20. firstTimestamp: "2019-10-24T13:41:40Z"
  21. involvedObject: {}
  22. lastTimestamp: "2019-10-24T13:41:40Z"
  23. message: Success pull image busybox:latest
  24. metadata:
  25. creationTimestamp: null
  26. name: busybox:latest
  27. reportingComponent: ""
  28. reportingInstance: ""
  29. source:
  30. component: kubelet
  31. host: eci
  32. type: Normal
  33. - count: 1
  34. eventTime: null
  35. firstTimestamp: "2019-10-24T13:42:04Z"
  36. involvedObject: {}
  37. lastTimestamp: "2019-10-24T13:42:04Z"
  38. message: Success pull image nginx
  39. metadata:
  40. creationTimestamp: null
  41. name: nginx
  42. reportingComponent: ""
  43. reportingInstance: ""
  44. source:
  45. component: kubelet
  46. host: eci
  47. type: Normal
  48. - count: 1
  49. eventTime: null
  50. firstTimestamp: "2019-10-24T13:42:06Z"
  51. involvedObject: {}
  52. lastTimestamp: "2019-10-24T13:42:06Z"
  53. message: Success pull image registry-vpc.cn-beijing.aliyuncs.com/acs/pause-amd64:3.0
  54. metadata:
  55. creationTimestamp: null
  56. name: registry-vpc.cn-beijing.aliyuncs.com/acs/pause-amd64:3.0
  57. reportingComponent: ""
  58. reportingInstance: ""
  59. source:
  60. component: kubelet
  61. host: eci
  62. type: Normal
  63. imageCacheId: imc-2zeditzeoemfhqor4rf8
  64. expireDateTime: "2019-10-25T13:40:54Z"
  65. lastUpdatedTime: "2019-10-24T13:44:17Z"
  66. phase: Ready
  67. progress: 100%
  68. startTime: "2019-10-24T13:40:58Z"

ImageCache快速创建Pod

ImageCache资源是Cluster级别的,所以在不同的namespace下创建Pod都可以使用ImageCache实现创建Pod。

使用ImageCache快速创建Pod有两种使用方式:

  • 明确指定方式:在创建pod时,明确的指定使用哪个ImageCacheId创建Pod
  • 自动匹配方式:在创建Pod时,根据匹配策略自动选择最优的ImageCache创建Pod

明确指定方式

这种使用方式,有一个前提条件就是ImageCache状态必须为Ready状态,其他状态的ImageCache会导致Pod创建失败。

Deployment使用方式:

在PodTemplate增加Annotation关联ImageCache:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deployment
  5. labels:
  6. app: nginx
  7. spec:
  8. replicas: 2
  9. selector:
  10. matchLabels:
  11. app: nginx
  12. template:
  13. metadata:
  14. labels:
  15. app: nginx
  16. annotations:
  17. k8s.aliyun.com/eci-image-snapshot-id: imc-2ze5tm5gehgtiigaz5jh
  18. spec:
  19. nodeName: virtual-kubelet
  20. containers:
  21. - name: nginx
  22. image: nginx:1.7.9
  23. imagePullPolicy: IfNotPresent

Pod使用方式:

在Pod的Yaml文件中增加Annotation关联ImageCache:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. annotations:
  5. k8s.aliyun.com/eci-image-snapshot-id: imc-2ze5tm5gehgtiigaz5jh
  6. name: nginx-imagecache-id
  7. spec:
  8. containers:
  9. - image: nginx:1.7.9
  10. imagePullPolicy: IfNotPresent
  11. name: nginx
  12. resources:
  13. limits:
  14. cpu: 300m
  15. memory: 200Mi
  16. requests:
  17. cpu: 200m
  18. memory: 100Mi
  19. nodeName: virtual-kubelet

自动匹配方式

在创建Pod时,用户通过Annotation方式,声明使用自动匹配ImageCache方式快速创建Pod,ECI管控基于用户已有的ImageCache列表依据策略进行匹配,匹配出最优的ImageCache创建Pod。如果没有匹配到合适的ImageCache,执行正常下载镜像创建Pod流程。

目前匹配策略的优先级从高到底:

  • 镜像匹配度
  • 匹配的镜像大小
  • 创建时间

Deployment使用方式:

在PodTemplate增加Annotation关联ImageCache,告知ECI管控在创建Pod时是否启用自动匹配ImageCache:

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deployment
  5. labels:
  6. app: nginx
  7. spec:
  8. replicas: 2
  9. selector:
  10. matchLabels:
  11. app: nginx
  12. template:
  13. metadata:
  14. labels:
  15. app: nginx
  16. annotations:
  17. k8s.aliyun.com/eci-image-cache: "true"
  18. spec:
  19. nodeName: virtual-kubelet
  20. containers:
  21. - name: nginx
  22. image: nginx:1.7.9
  23. imagePullPolicy: IfNotPresent

Pod使用方式:

基于Annotation方式,在Pod的Yaml文件中增加Annotation:

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. annotations:
  5. k8s.aliyun.com/eci-image-cache: "true"
  6. name: nginx-auto-match
  7. spec:
  8. containers:
  9. - image: nginx:1.7.9
  10. imagePullPolicy: IfNotPresent
  11. name: nginx
  12. resources:
  13. limits:
  14. cpu: 300m
  15. memory: 200Mi
  16. requests:
  17. cpu: 200m
  18. memory: 100Mi
  19. nodeName: virtual-kubelet

使用建议

  1. Pod中容器的image建议采用ImageCache中镜像,提高匹配度。
  2. Pod中容器的ImagePullPolicy建议设置为IfNotPresent,避免镜像层重复下载。

Tip: 在创建Pod时,k8s.aliyun.com/eci-image-snapshot-idk8s.aliyun.com/eci-image-cache同时设置时,明确指定方式优先级高于自动匹配方式

注:如果是deployment,使用cache的Annotation应该放在pod级别,而不是放在deployment级别,否则会设置失效。

删除ImageCache

删除ImageCache时,不影响基于ImageCache创建成功的Pod实例。

操作如下:

  1. $ kubectl get imagecache/imagecache-secrets-test
  2. NAME AGE CACHEID PHASE PROGRESS
  3. imagecache-secrets-test 14h imc-2ze5tm5gehgtjru0kfm8 Ready 100%
  4. $ kubectl delete imagecache/imagecache-secrets-test
  5. imagecache.eci.alibabacloud.com " imagecache-secrets-test" deleted
  6. $ kubectl get imagecache/ imagecache-secrets-test
  7. Error from server (NotFound): imagecaches.eci.alibabacloud.com " imagecache-secrets-test" not found

kubernetes ImageCache API文档

Group Version Kind
eci.alibabacloud.com v1 ImageCache

ImageCache v1 eci.alibabacloud.com

ImageCacheList:

Field Type Description
apiVersion string api版本: eci.alibabacloud.com/v1
kind string 资源类别: ImageCacheList
metadata ListMeta Standard list metadata.
Items []ImageCache 镜像缓存列表

ImageCache:

Field Type Description
apiVersion string api版本: eci.alibabacloud.com/v1
kind string 资源类别: ImageCache
metadata ObjectMeta
spec ImageCacheSpec 镜像缓存详细信息
status ImageCacheStatus 镜像缓存状态信息

ImageCacheSpec:

Field Type Request Description
images []string Y 镜像缓存的镜像列表
imagePullSecrets []string N 私有镜像仓库对应的Secret列表,格式:namespace:secretName
imageCacheSize int N 镜像缓存的大小,默认为20GB,取值范围为:[20, 32768],单位为GB
retentionDays int N 镜像缓存保留的时间,取值范围[1,65536],默认永不过期。

ImageCacheStatus:

Field Type Description
phase string 镜像缓存状态
progress string 标示制作过程中进度
imageCacheId string 镜像缓存ID,调用Provdier的Create操作返回,然后更新到ImageCache对象
startTime Time 镜像缓存对象创建时间
lastUpdatedTime Time 镜像缓存最后更新时间
events []Event 制作镜像cache过程中的事件信息
expireDateTime Time 镜像缓存的过期时间,有两种来源:1、用户设置了retentionDays;2、制作失败的镜像缓存自动在24小时后过期。

CreateImageCache

创建镜像缓存对象

Http Request

POST /apis/eci.alibabacloud.com/v1/imagecaches

curl request example

  1. curl -X POST -H 'Content-Type: application/yaml' --data '
  2. apiVersion: eci.alibabacloud.com/v1
  3. kind: ImageCache
  4. metadata:
  5. name: imagecache-secrets-test
  6. spec:
  7. images:
  8. - registry.cn-shanghai.aliyuncs.com/baz/nginx:1.0
  9. imagePullSecrets:
  10. - default:acr-test
  11. ' http://127.0.0.1:8001/apis/eci.alibabacloud.com/v1/imagecaches

curl response example

  1. {
  2. "apiVersion": "eci.alibabacloud.com/v1",
  3. "kind": "ImageCache",
  4. "metadata": {
  5. "creationTimestamp": "2019-10-10T03:50:47Z",
  6. "generation": 1,
  7. "name": "imagecache-secrets-test",
  8. "resourceVersion": "647052328",
  9. "selfLink": "/apis/eci.alibabacloud.com/v1/imagecaches/imagecache-secrets-test11",
  10. "uid": "2506ba37-eb11-11e9-9576-f2ef49bf1a08"
  11. },
  12. "spec": {
  13. "imagePullSecrets": [
  14. "default:acr-test"
  15. ],
  16. "images": [
  17. "registry.cn-shanghai.aliyuncs.com/baz/nginx:1.0"
  18. ]
  19. }
  20. }

ListImageCache

获取镜像缓存列表

Http Request

GET /apis/eci.alibabacloud.com/v1/imagecaches

curl request example

  1. curl -X GET 'http://127.0.0.1:8001/apis/eci.alibabacloud.com/v1/imagecaches'

curl response example

  1. {
  2. "apiVersion": "eci.alibabacloud.com/v1",
  3. "items": [
  4. {
  5. "apiVersion": "eci.alibabacloud.com/v1",
  6. "kind": "ImageCache",
  7. "metadata": {
  8. "annotations": {
  9. "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"eci.alibabacloud.com/v1\",\"kind\":\"ImageCache\",\"metadata\":{\"annotations\":{},\"name\":\"imagecache-secrets-test\"},\"spec\":{\"imagePullSecrets\":[\"default:acr-test\"],\"images\":[\"registry.cn-shanghai.aliyuncs.com/baz/nginx:1.0\"]}}\n"
  10. },
  11. "creationTimestamp": "2019-10-10T01:43:48Z",
  12. "generation": 1,
  13. "name": "imagecache-secrets-test",
  14. "resourceVersion": "646375979",
  15. "selfLink": "/apis/eci.alibabacloud.com/v1/imagecaches/imagecache-secrets-test",
  16. "uid": "67460461-eaff-11e9-9774-b2c06862f69a"
  17. },
  18. "spec": {
  19. "imagePullSecrets": [
  20. "default:acr-test"
  21. ],
  22. "images": [
  23. "registry.cn-shanghai.aliyuncs.com/baz/nginx:1.0"
  24. ]
  25. },
  26. "status": {
  27. "imageCacheId": "imc-2ze5xnx4yiply807zo0o",
  28. "lastUpdatedTime": "2019-10-10T01:47:14Z",
  29. "phase": "Ready",
  30. "progress": "100%",
  31. "startTime": "2019-10-10T01:43:49Z"
  32. }
  33. }
  34. ],
  35. "kind": "ImageCacheList",
  36. "metadata": {
  37. "continue": "",
  38. "resourceVersion": "647062964",
  39. "selfLink": "/apis/eci.alibabacloud.com/v1/imagecaches"
  40. }
  41. }

GetImageCache

获取某一个镜像缓存的相信信息

Http Request

GET /apis/eci.alibabacloud.com/v1/imagecaches/{name}

curl request example

  1. curl -X GET 'http://127.0.0.1:8001/apis/eci.alibabacloud.com/v1/imagecaches/imagecache-secrets-test'

curl response example

  1. {
  2. "apiVersion": "eci.alibabacloud.com/v1",
  3. "kind": "ImageCache",
  4. "metadata": {
  5. "annotations": {
  6. "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"eci.alibabacloud.com/v1\",\"kind\":\"ImageCache\",\"metadata\":{\"annotations\":{},\"name\":\"imagecache-secrets-test\"},\"spec\":{\"imagePullSecrets\":[\"default:acr-test\"],\"images\":[\"registry.cn-shanghai.aliyuncs.com/baz/nginx:1.0\"]}}\n"
  7. },
  8. "creationTimestamp": "2019-10-10T01:43:48Z",
  9. "generation": 1,
  10. "name": "imagecache-secrets-test",
  11. "resourceVersion": "646375979",
  12. "selfLink": "/apis/eci.alibabacloud.com/v1/imagecaches/imagecache-secrets-test",
  13. "uid": "67460461-eaff-11e9-9774-b2c06862f69a"
  14. },
  15. "spec": {
  16. "imagePullSecrets": [
  17. "default:acr-test"
  18. ],
  19. "images": [
  20. "registry.cn-shanghai.aliyuncs.com/baz/nginx:1.0"
  21. ]
  22. },
  23. "status": {
  24. "imageCacheId": "imc-2ze5xnx4yiply807zo0o",
  25. "lastUpdatedTime": "2019-10-10T01:47:14Z",
  26. "phase": "Ready",
  27. "progress": "100%",
  28. "startTime": "2019-10-10T01:43:49Z"
  29. }
  30. }

DeleteImageCache

删除指定镜像缓存

Http Request

DELETE /apis/eci.alibabacloud.com/v1/imagecaches/{name}

curl request example

  1. curl -X DELETE -H 'Content-Type: application/yaml' 'http://127.0.0.1:8001/apis/eci.alibabacloud.com/v1/imagecaches/imagecache-secrets-test'

curl response example

  1. "kind": "Status",
  2. "apiVersion": "v1",
  3. "metadata": {},
  4. "status": "Success",
  5. "details": {
  6. "name": "imagecache-secrets-test",
  7. "group": "eci.alibabacloud.com",
  8. "kind": "imagecaches",
  9. "uid": "67460461-eaff-11e9-9774-b2c06862f69a"
  10. }
  11. }