文档

计算巢服务容器场景支持上报源为Prometheus的计量上报方式

本文介绍服务商在容器场景下的计量上报流程信息。

背景信息

计算巢服务售卖时依托云市场。云市场商品依据售卖模式不同,分为按周期付费和按量付费,本文场景为按量付费。按量付费的使用方式为:服务商预先在云市场定义好计量项及其单价,同时云市场会提供上报接口PushMeteringData,服务商通过调用此接口推送计量项数据,结合预定义好的价格,便可在用户的账号中出账收费。

针对依赖硬件资源计费的软件(如底层服务器的规格越好、数量越多,则其费用越高),需要获取云资源数据以计量计费。但是软件获取云资源数据存在一定难度,且不同厂商需要针对上报逻辑定制化开发软件,存在一定开发成本。此外,全托管服务由于资源是创建在服务商账号下的,服务商需要收取资源费用,但是这部分费用在现有流程下较难收取。特别是容器场景下,服务资源费用的计量数据更加难以获取。

前提条件

  1. 当前服务为容器部署场景(新建ACK/已有ACK场景均可以),且服务模板中必须为标准化部署方式。

  2. 服务商发布计算巢服务至云市场,并定义按量付费收费模式。

操作步骤

  1. 配置计量项。

    服务详情页,单击前往编辑,进入云市场计量项编辑页面。2024-03-05_17-57-13.png

  2. 云市场页的计量上报页签中,配置服务上报方式。

    • 对于VirtualCPUMemoryNetworkInNetworkOut计量项,其界面如下图所示。

      计算巢将获取ACK集群对应的Namespace(详情请参见附录2)收集计量数据。计量项的详细指标信息,请参见附录12024-03-06_14-19-27.png

    • 对于Unit计量项,您可以选择平均Pod数自定义指标。计量项的详细指标信息,请参见附录1

      若选择自定义指标,则输入自定义的PromQL语句,计算巢将读取此语句,并至ACK集群对应Namespace收集计量数据,请确保自定义语句的准确性。2024-03-06_15-37-50.png

  3. 配置商品绑定。

    单击商品绑定,将需要上报的计量项与具体的模板或套餐进行绑定。

    说明

    若不进行绑定操作,计算巢将不会收集此指标的数据。

    10.png

  4. 用户完成服务实例的创建后,将自动完成上报。

    说明

    上报方式选择计算巢上报上报源选择Prometheus时,上报频率为1小时1次。

附录1:计量项对应PromQL语句

不同的计量项对应不同的PromQL语句(Namespace参数计算巢将会依据不同服务实例替换,示例默认给出default)。计算巢获取指标值后将会适配云市场上报单位,详细信息请参见各计量项描述。

Unit - 平均Pod数:

avg_over_time(count(kube_pod_info{namespace="default"})[1h:1m])

PromQL单位:个。

云市场单位:个。

Memory - 平均内存

avg_over_time(sum(container_memory_working_set_bytes{namespace=~"default"}) [1h:10s])

PromQL单位:bytes。

云市场单位:GB(PromQL返回值/1024/1024/1024)。由于云市场必须为整数,因此计算巢上报时会进行四舍五入处理。

VirtualCpu - 平均虚拟CPU

avg_over_time(sum(rate(container_cpu_usage_seconds_total{namespace=~"arms-prom"}[2m]))[1h:10s])

PromQL单位:核数。

云市场单位:核数。

NetworkIn - 流入总流量

返回当前整点时间与前一小时整点时间做差值,若前一小时整点无数据,则返回当前整点时间值,整点时间值获取语句如下。

sum(container_network_receive_bytes_total{image!="",namespace=~"default"})

PromQL单位:bytes。

云市场单位:bits(PromQL返回值 * 8)。

NetworkOut - 流出总流量

返回当前整点时间与前一小时整点时间做差值,若前一小时整点无数据,则返回当前整点时间值,整点时间值获取语句如下。

sum(container_network_transmit_bytes_total{image!="",namespace=~"default"})

PromQL单位:bytes。

云市场单位:bits(PromQL返回值 * 8)。

附录2:容器标准化部署模板

新建ACK场景

说明
  1. namespace为资源WordpressComputenestHelmApplication中的Namespace

  2. 容器必须开启Prometheus组件(ManagedKubernetesCluster中的Addons参数,模板中默认开启(Name:arms-prometheus))。

ROSTemplateFormatVersion: '2015-09-01'
Description:
  en: 新建ack部署wordpress
  zh-cn: new ack deploy wordpress
Parameters:
  CreateAck:
    Type: Boolean
    Description:
      en: An existing ack cluster can be deployed by entering the cluster id. If there is no current cluster, create a new ack cluster before deploying
      zh-cn: 已有ack集群输入集群id即可部署,当前无集群先新建ack集群再进行部署
    Label:
      en: Wether create ack cluster
      zh-cn: 是否新建ack集群
    Default: true
  ClusterId:
    Type: String
    Description:
      en: The ID of Kubernetes ClusterId in which application deployed.
      zh-cn: 部署应用程序的K8s集群ID
    AllowedPattern: '[0-9a-z]+$'
    Default: null
    Required: true
    Label:
      en: Kubernetes ClusterId
      zh-cn: K8s集群ID
    AssociationProperty: 'ALIYUN::CS::Cluster::ClusterId'
    AssociationPropertyMetadata:
      RegionId: '${RegionId}'
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - false
  PayType:
    Type: String
    Label:
      en: ECS Instance Charge Type
      zh-cn: 付费类型
    Default: PostPaid
    AllowedValues:
      - PostPaid
      - PrePaid
    AssociationProperty: ChargeType
    AssociationPropertyMetadata:
      LocaleKey: InstanceChargeType
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
  PayPeriodUnit:
    Type: String
    Label:
      en: Pay Period Unit
      zh-cn: 购买资源时长周期
    Default: Month
    AllowedValues:
      - Month
      - Year
    AssociationProperty: PayPeriodUnit
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::And:
            - Fn::Equals:
                - ${CreateAck}
                - true
            - Fn::Not:
                Fn::Equals:
                  - ${PayType}
                  - PostPaid
  PayPeriod:
    Type: Number
    Description:
      en: When the resource purchase duration is Month, the value of Period ranges from 1 to 9, 12, 24, 36, 48, or 60. <br><b><font color='red'> When ECS instance types are PrePaid valid </b></font>
      zh-cn: 当购买资源时长为Month时,Period取值:1~9 <br><b><font color='red'>当ECS实例类型为PrePaid有效</b></font>
    Label:
      en: Period
      zh-cn: 购买资源时长
    Default: 1
    AllowedValues:
      - 1
      - 2
      - 3
      - 4
      - 5
      - 6
      - 7
      - 8
      - 9
    AssociationProperty: PayPeriod
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::And:
            - Fn::Equals:
                - ${CreateAck}
                - true
            - Fn::Not:
                Fn::Equals:
                  - ${PayType}
                  - PostPaid
  ZoneId:
    Type: String
    AssociationProperty: ALIYUN::ECS::Instance:ZoneId
    Label:
      en: Zone ID
      zh-cn: 可用区
    Default: cn-hangzhou-h
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
  VpcId:
    Type: String
    Label:
      en: VPC ID
      zh-cn: 专有网络VPC实例ID
    Description:
      en: >-
        Please search the ID starting with (vpc-xxx) from console-Virtual
        Private Cloud
      zh-cn: 现有虚拟专有网络的实例ID
    Default: ''
    AssociationProperty: 'ALIYUN::ECS::VPC::VPCId'
    AssociationPropertyMetadata:
      RegionId: '${RegionId}'
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
  VSwitchId:
    Type: String
    Label:
      en: VSwitch ID
      zh-cn: 交换机实例ID
    Description:
      en: >-
        Instance ID of existing business network switches, console-Virtual
        Private Cloud-VSwitches under query
      zh-cn: 现有业务网络交换机的实例ID
    Default: ''
    AssociationProperty: 'ALIYUN::ECS::VSwitch::VSwitchId'
    AssociationPropertyMetadata:
      VpcId: '${VpcId}'
      ZoneId: '${ZoneId}'
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
  LoginPassword:
    NoEcho: true
    Type: String
    Description:
      en: Server login password, Length 8-30, must contain three(Capital letters, lowercase letters, numbers, ()`~!@#$%^&*_-+=|{}[]:;<>,.?/ Special symbol in)
      zh-cn: 服务器登录密码,长度8-30,必须包含三项(大写字母、小写字母、数字、 ()`~!@#$%^&*_-+=|{}[]:;<>,.?/ 中的特殊符号)
    Label:
      en: Instance Password
      zh-cn: 实例密码
    ConstraintDescription:
      en: Length 8-30, must contain three(Capital letters, lowercase letters, numbers, ()`~!@#$%^&*_-+=|{}[]:;<>,.?/ Special symbol in)
      zh-cn: 长度8-30,必须包含三项(大写字母、小写字母、数字、 ()`~!@#$%^&*_-+=|{}[]:;<>,.?/ 中的特殊符号)
    AssociationProperty: ALIYUN::ECS::Instance::Password
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
    AllowedPattern: ^[a-zA-Z0-9-\(\)\`\~\!\@\#\$\%\^\&\*\_\-\+\=\|\{\}\[\]\:\;\<\>\,\.\?\/]*$
    MinLength: 8
    MaxLength: 30
    Default: Lzq123456
  WorkerInstanceType:
    Type: String
    Label:
      en: Worker Nodes Types
      zh-cn: Worker节点规格
    AssociationProperty: ALIYUN::ECS::Instance::InstanceType
    AssociationPropertyMetadata:
      ZoneId: ${ZoneId}
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
    Default: ecs.g6.large
  WorkerSystemDiskCategory:
    Type: String
    AllowedValues:
      - cloud_efficiency
      - cloud_ssd
      - cloud_essd
    AssociationPropertyMetadata:
      LocaleKey: DiskCategory
      InstanceType: ${WorkerInstanceType}
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
    Label:
      en: Worker System Disk Category
      zh-cn: Worker 系统盘磁盘类型
    Default: cloud_essd
  WorkerSystemDiskSize:
    Type: Number
    Label:
      en: Worker System Disk Size(GB)
      zh-cn: Worker节点系统盘大小(GB)
    MinValue: 1
    Default: 120
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
  AckNetworkPlugin:
    Type: String
    Label:
      en: ack plugin network
      zh-cn: ack网络插件
    AllowedValues:
      - Flannel
      - Terway
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
    Default: Flannel
  PodCidr:
    Type: String
    Description:
      zh-cn: 请填写有效的私有网段,即以下网段及其子网:10.0.0.0/8,172.16-31.0.0/12-16,192.168.0.0/16<br>不能与 VPC 及 VPC 内已有 Kubernetes 集群使用的网段重复。<font color='blue'><b>创建成功后不能修改</b></font>
      en: 'Please fill in a valid private segment, i.e. the following segments and their subnets: 10.0.0.0/8, 172.16-31.0.0/12-16, 192.168.0.0/16<br> which cannot duplicate the network segments already used by clusters in VPC and VPC Kunetberes. <font color=''blue''><b>Cannot be modified after successful creation</b></font>'
    Label:
      zh-cn: Pod 网络 CIDR
      en: Pod Network CIDR
    AssociationProperty: ALIYUN::CS::ManagedKubernetesCluster::PodCidr
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::And:
            - Fn::Equals:
                - ${CreateAck}
                - true
            - Fn::Equals:
                - ${AckNetworkPlugin}
                - Flannel
    Default: 10.0.0.0/16
  PodVswitchId:
    Type: String
    Label:
      en: VSwitch ID
      zh-cn: pod交换机实例ID
    Description:
      en: >-
        Instance ID of existing business network switches, console-Virtual
        Private Cloud-VSwitches under query
      zh-cn: 建议选择网段掩码不大于 19 的虚拟交换机
    Default: ''
    AssociationProperty: 'ALIYUN::ECS::VSwitch::VSwitchId'
    AssociationPropertyMetadata:
      VpcId: '${VpcId}'
      ZoneId: '${ZoneId}'
      Visible:
        Condition:
          Fn::And:
            - Fn::Equals:
                - ${CreateAck}
                - true
            - Fn::Equals:
                - ${AckNetworkPlugin}
                - Terway
  ServiceCidr:
    Type: String
    Description:
      zh-cn: 可选范围:10.0.0.0/16-24,172.16-31.0.0/16-24,192.168.0.0/16-24<br>不能与 VPC 及 VPC 内已有 Kubernetes 集群使用的网段重复。<font color='blue'><b>创建成功后不能修改</b></font>
      en: 'Optional range: 10.0.0.0/16-24, 172.16-31.0.0/16-24, 192.168.0.0/16-24<br> cannot duplicate segments already used by existing Kubernetes clusters in VPC and VPC.<font color=''blue''><b>Cannot be modified after successful creation</b></font>'
    Label:
      zh-cn: Service CIDR
      en: Service CIDR
    AssociationProperty: ALIYUN::CS::ManagedKubernetesCluster::ServiceCidr
    AssociationPropertyMetadata:
      Visible:
        Condition:
          Fn::Equals:
            - ${CreateAck}
            - true
    Default: 172.16.0.0/16
  WordpressUsername:
    Type: String
    Label:
      zh-cn: wordpress用户名
      en: wordpress username
    Default: user
  WordpressPassword:
    NoEcho: true
    Type: String
    Label:
      zh-cn: wordpress密码
      en: wordpress password
    Default: Lzq123456
Conditions:
  CreateAck:
    Fn::Equals:
      - true
      - Ref: CreateAck
  FlannelPluginCondition:
    Fn::Equals:
      - Ref: AckNetworkPlugin
      - Flannel
  TerwayPluginCondition:
    Fn::Equals:
      - Ref: AckNetworkPlugin
      - Terway
Resources:
  VpcsDataSource:
    Type: DATASOURCE::VPC::Vpcs
    Properties:
      VpcIds:
        - Ref: VpcId
  EcsSecurityGroup:
    Type: ALIYUN::ECS::SecurityGroup
    Condition: CreateAck
    Properties:
      SecurityGroupName:
        Ref: ALIYUN::StackName
      VpcId:
        Ref: VpcId
      SecurityGroupEgress:
        - PortRange: '-1/-1'
          Priority: 1
          IpProtocol: all
          DestCidrIp: 0.0.0.0/0
          NicType: intranet
      SecurityGroupIngress:
        Fn::If:
          - FlannelPluginCondition
          - - PortRange: '-1/-1'
              Priority: 1
              IpProtocol: all
              SourceCidrIp:
                Ref: PodCidr
              Description: pod网络访问开放
              NicType: intranet
            - PortRange: '-1/-1'
              Priority: 1
              IpProtocol: all
              SourceCidrIp:
                Fn::Jq:
                  - First
                  - .[].CidrBlock
                  - 'Fn::GetAtt':
                      - VpcsDataSource
                      - Vpcs
              Description: vpc网络访问开放
              NicType: intranet
            - PortRange: '-1/-1'
              Priority: 1
              IpProtocol: icmp
              SourceCidrIp: 0.0.0.0/0
              Description: icmp协议端口放开
              NicType: intranet
          - - PortRange: '-1/-1'
              Priority: 1
              IpProtocol: all
              SourceCidrIp:
                Fn::Jq:
                  - First
                  - .[].CidrBlock
                  - 'Fn::GetAtt':
                      - VpcsDataSource
                      - Vpcs
              Description: vpc网络访问开放
              NicType: intranet
            - PortRange: '-1/-1'
              Priority: 1
              IpProtocol: icmp
              SourceCidrIp: 0.0.0.0/0
              Description: icmp协议端口放开
              NicType: intranet
  ManagedKubernetesCluster:
    Type: ALIYUN::CS::ManagedKubernetesCluster
    Condition: CreateAck
    Properties:
      Name:
        Ref: ALIYUN::StackName
      ChargeType:
        Ref: PayType
      Period:
        Ref: PayPeriod
      PeriodUnit:
        Ref: PayPeriodUnit
      VSwitchIds:
        - Ref: VSwitchId
      VpcId:
        Ref: VpcId
      WorkerInstanceTypes:
        - Ref: WorkerInstanceType
      NumOfNodes: 3
      ClusterSpec: ack.pro.small
      ContainerCidr:
        Fn::If:
          - FlannelPluginCondition
          - Ref: PodCidr
          - Ref: ALIYUN::NoValue
      ServiceCidr:
        Ref: ServiceCidr
      PodVswitchIds:
        Fn::If:
          - TerwayPluginCondition
          - - Ref: PodVswitchId
          - Ref: ALIYUN::NoValue
      ZoneIds:
        - Ref: ZoneId
      SecurityGroupId:
        Ref: EcsSecurityGroup
      WorkerSystemDiskCategory:
        Ref: WorkerSystemDiskCategory
      WorkerSystemDiskSize:
        Ref: WorkerSystemDiskSize
      LoginPassword:
        Ref: LoginPassword
      SnatEntry: true
      Addons:
        Fn::If:
          - FlannelPluginCondition
          - - Name: flannel
              Config: ''
            - Name: arms-prometheus
          - - Name: terway-eniip
              Config: ''
            - Name: arms-prometheus
  WordpressComputenestHelmApplication:
    Type: MODULE::ACS::ComputeNest::FluxOciHelmDeploy
    Version: v1
    Properties:
      ClusterId:
        Fn::If:
          - CreateAck
          - Fn::GetAtt:
              - ManagedKubernetesCluster
              - ClusterId
          - Ref: ClusterId
      HelmChartUrl: oci://registry-1.docker.io/bitnamicharts/wordpress:15.4.1
      ChartValues:
        mariadb:
          primary:
            persistence:
              enabled: true
              storageClass: alicloud-disk-essd
              size: 100Gi
        persistence:
          enabled: false
        wordpressUsername:
          Ref: WordpressUsername
        wordpressPassword:
          Ref: WordpressPassword
      Namespace:
        Ref: 'ALIYUN::StackName'
      ReleaseName: wordpress
      WaitUntil:
        - Kind: Service
          Name: wordpress
          Namespace:
            Ref: ALIYUN::StackName
          JsonPath: $.status.loadBalancer.ingress[0].ip
          Operator: NotEmpty
          FirstMatch: true
          Timeout: 300
  # 获取service信息,输出到output中
  ClusterApplicationResources:
    Type: DATASOURCE::CS::ClusterApplicationResources
    DependsOn:
      - WordpressComputenestHelmApplication
    Properties:
      ClusterId:
        Fn::If:
          - CreateAck
          - Fn::GetAtt:
              - ManagedKubernetesCluster
              - ClusterId
          - Ref: ClusterId
      Kind: Service
      Name: wordpress
      Namespace:
        Ref: ALIYUN::StackName
      JsonPath: $.status.loadBalancer.ingress[0].ip
      FirstMatch: true
Outputs:
  # 将公网ip做为http返回的地址显示在控制台
  Endpoint:
    Description:
      zh-cn: 对外暴露的公网IP地址
      en: Public IP Addresses
    Value:
      Fn::Sub:
        - "http://${ServerAddress} \n http://${ServerAddress}/admin"
        - ServerAddress:
            Fn::GetAtt:
              - ClusterApplicationResources
              - Response
Metadata:
  ALIYUN::ROS::Interface:
    ParameterGroups:
      - Parameters:
          - CreateAck
          - ClusterId
        Label:
          en: Whether create ack
          zh-cn: 是否新建ack集群
      - Parameters:
          - PayType
          - PayPeriodUnit
          - PayPeriod
        Label:
          en: PayType Configuration
          zh-cn: 付费类型配置
      - Parameters:
          - ZoneId
          - VpcId
          - VSwitchId
          - LoginPassword
        Label:
          en: Basic Configuration
          zh-cn: 基础配置
      - Parameters:
          - WorkerInstanceType
          - WorkerSystemDiskCategory
          - WorkerSystemDiskSize
          - AckNetworkPlugin
          - PodCidr
          - PodVswitchId
          - ServiceCidr
        Label:
          en: Kubernetes
          zh-cn: Kubernetes配置
      - Parameters:
          - WordpressUsername
          - WordpressPassword
        Label:
          en: Wordpress Config
          zh-cn: Wordpress配置

已有ACK场景

说明
  1. namespace为资源ClusterApplication中的DefaultNamespace

  2. 容器必须开启Prometheus组件,请至集群详情页的左侧运维管理 > Prometheus监控,然后单击开启

ROSTemplateFormatVersion: '2015-09-01'
Description:
  en: A simple demo that deploys a RDS instance and a container-base app into ack. The app use RDS instance as a persistence storage.
  zh-cn: 新建一个RDS实例并向ACK中部署容器应用。容器应用使用RDS实例作为数据库。
Parameters:
  ClusterId:
    Type: String
    Description:
      en: >-
        The ID of Kubernetes ClusterId in which application deployed.
      zh-cn: >-
        部署应用程序的K8s集群ID
    AllowedPattern: '[0-9a-z]+$'
    Label:
      en: Kubernetes ClusterId
      zh-cn: K8s集群ID
    ConstraintDescription:
      en: >-
        must be lowercase letters or numbers
      zh-cn: '集群ID必须由小写字母或者数字组成'
Resources:
  ClusterApplication:
    Type: ALIYUN::CS::ClusterApplication
    Properties:
      ClusterId:
        Ref: ClusterId
      DefaultNamespace:
        Ref: ALIYUN::StackName
      YamlContent: |
        apiVersion: v1
        data: 
          .dockerconfigjson: {{ computenest::acr::dockerconfigjson }}
        kind: Secret
        metadata:
          name: computenestrepo
        type: kubernetes.io/dockerconfigjson
        ---
        apiVersion: v1
        kind: Pod
        metadata:
          name: mysql
          labels:
            tier: backend
        spec:
          containers:
            - name: mysql
              image: mysql:8.0.25
              imagePullPolicy: IfNotPresent
              ports:
                - containerPort: 3306
              env:
                - name: MYSQL_ROOT_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: credentials
                      key: MYSQL_ROOT_PASSWORD
                - name: MYSQL_DATABASE
                  value: wordpress
          restartPolicy: Always
        ---
        apiVersion: v1
        kind: Service
        metadata:
          name: mysql
          labels:
            tier: backend
        spec:
          type: ClusterIP
          ports:
            - port: 3306
              targetPort: 3306
          selector:
            tier: backend
        ---
        apiVersion: v1
        kind: Secret
        metadata:
          name: credentials
        data:
          MYSQL_ROOT_PASSWORD: ZGItcGFzc3dvcmQ=
        ---
        apiVersion: v1
        kind: Pod
        metadata:
          name: wordpress
          labels:
            tier: frontend
        spec:
          containers:
            - name: wordpress
              image: {{ computenest::acrimage::springbootdemo }}
              imagePullPolicy: Always
              ports:
                - containerPort: 80
              env:
                - name: WORDPRESS_DB_HOST
                  value: mysql
                - name: WORDPRESS_DB_USER
                  value: root
                - name: WORDPRESS_DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: credentials
                      key: MYSQL_ROOT_PASSWORD
                - name: WORDPRESS_DB_NAME
                  value: wordpress
          restartPolicy: Always
          imagePullSecrets:
            - name: computenestrepo
        ---
        apiVersion: v1
        kind: Service
        metadata:
          name: wordpress
          labels:
            tier: frontend
        spec:
          type: LoadBalancer
          ports:
            - port: 80
              targetPort: 80
          selector:
            tier: frontend
Metadata:
  ALIYUN::ROS::Interface:
    ParameterGroups:
      - Parameters:
          - ClusterId
        Label:
          en: ACK Configuration
          zh-cn: ACK参数配置

  • 本页导读 (0)
文档反馈