Higress Ingress高级用法

在Kubernetes集群中,Higress Ingress对集群服务(Service)中的外部可访问的API对象进行管理,提供七层负载均衡能力。本文介绍Higress Ingress的高级用法,方便您对集群入口流量进行治理。

灰度发布

Higress Ingress提供复杂的路由处理能力,支持基于Header、Query Parameter、Cookie以及权重的灰度发布功能。灰度发布功能可以通过设置注解来实现,为了启用灰度发布功能,需要设置注解nginx.ingress.kubernetes.io/canary: "true",通过不同注解可以实现不同的灰度发布功能。

说明

当多种方式同时配置时,灰度方式选择优先级为:基于Header | 基于Query Parameter > 基于Cookie > 基于权重(从高到低)。

基于Header灰度发布

  • 只配置nginx.ingress.kubernetes.io/canary-by-header:基于Request Header的流量切分,当配置的header值为always时,请求流量会被分配到灰度服务入口;其他情况时,请求流量不会分配到灰度服务。

  • 同时配置nginx.ingress.kubernetes.io/canary-by-header-value和nginx.ingress.kubernetes.io/canary-by-header:当请求中的header和header-value与设置的值匹配时,请求流量会被分配到灰度服务;其他情况下,请求流量不会被分配到灰度服务。

说明

相比Nginx Ingress和ALB Ingress灰度发布时最多只支持两个版本服务,Higress Ingress灰度发布时支持多个版本服务(无上限)。

例如:

  • 请求Header为mse:always时会访问灰度服务demo-service-canary;其他情况将访问正式服务demo-service。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact          

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 请求Header为mse:v1时将访问灰度服务demo-service-canary-v1;请求Header为mse:v2时将访问灰度服务demo-service-canary-v2;其他情况将访问正式服务demo-service。配置如下。

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary-v1
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
      name: demo-canary-v2
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary-v2
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary-v1
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
      name: demo-canary-v2
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary-v2
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80

基于Query Parameter灰度发布

  • 仅配置higress.ingress.kubernetes.io/canary-by-query

    基于URL Query Parameter的流量切分,当请求的URL中Query Parameter的Key为该参数配置且Value为always时,请求流量会被分配到灰度服务入口。其他情况下,请求流量不会分配到灰度服务。

  • 同时配置higress.ingress.kubernetes.io/canary-by-query-value和higress.ingress.kubernetes.io/canary-by-query

    当请求中的query parameter keyquery parameter value与设置的值匹配时,请求流量会被分配到灰度服务。其他情况下,请求流量不会分配到灰度服务。

    说明

    基于Header的灰度发布可以和基于Query Parameter的灰度发布一起使用,同时满足匹配条件,请求流量才会被分配到灰度服务。

示例:

  • 请求URL的Query Parameter为canary:gray时会访问灰度服务demo-service-canary,其他情况将访问正式服务demo-service。相关配置如下。

    1.19及之后版本集群

    apiVersion:networking.k8s.io/v1 
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        higress.ingress.kubernetes.io/canary-by-query: "canary"
        higress.ingress.kubernetes.io/canary-by-query-value: "gray"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact 

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        higress.ingress.kubernetes.io/canary-by-query: "canary"
        higress.ingress.kubernetes.io/canary-by-query-value: "gray"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 请求URL的Query Parameter为canary:gray,同时请求Header包含x-user-id: test时,会访问灰度服务demo-service-canary,其他情况将访问正式服务demo-service。相关配置如下。

    1.19及之后版本集群

    apiVersion:networking.k8s.io/v1 
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        higress.ingress.kubernetes.io/canary-by-query: "canary"
        higress.ingress.kubernetes.io/canary-by-query-value: "gray"
        nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
        nginx.ingress.kubernetes.io/canary-by-header-value: "test"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact 

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        higress.ingress.kubernetes.io/canary-by-query: "canary"
        higress.ingress.kubernetes.io/canary-by-query-value: "gray"
        nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
        nginx.ingress.kubernetes.io/canary-by-header-value: "test"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80

基于Cookie灰度发布

nginx.ingress.kubernetes.io/canary-by-cookie:基于Cookie的流量切分,当配置的cookie值为always时,请求流量会被分配到灰度服务;其他情况时,请求流量将不会分配到灰度服务。

说明

基于Cookie的灰度发布不支持设置自定义值,配置的cookie值只能为always

例如,请求的Cookie为demo=always时会访问灰度服务demo-service-canary;其他情况将访问正式服务demo-service。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
  name: demo-canary
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
  name: demo-canary
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

基于权重灰度发布

注解

说明

nginx.ingress.kubernetes.io/canary-weight

设置请求到指定服务的百分比(值为0~100的整数)。

nginx.ingress.kubernetes.io/canary-weight-total

设置权重总和,默认为100。

例如,配置灰度服务demo-service-canary-v1的权重为30%,配置灰度服务demo-service-canary-v2的权重为20%,配置正式服务demo-service的权重为50%。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  name: demo-canary-v1
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v1
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
  name: demo-canary-v2
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v2
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  name: demo-canary-v1
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v1
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
  name: demo-canary-v2
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v2
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

服务Subset

服务Subset适用于一个Service关联多个Deployment的场景,通过Ingress将请求转发至该Service下Pod集合的子集,常见情况是将请求转发至某个Service下含有某个Label的Pod集合,有如下两种配置方式:

使用Higress Ingress约定的Pod Label

通过注解higress.ingress.kubernetes.io/service-subset设置Service版本。默认情况下,Higress Ingress约定配置的服务版本与Pod Label中以opensergo.io/canary为前缀的Label有对应关系。该注解含义如下:

  • 当配置为""或者base时,请求会被转发到Label中含有opensergo.io/canary: ""或不含有任何opensergo.io/canary为前缀的Label Key的Pod集合,即Label上打了空标或未打标的Pod集合。

  • 当配置为其他值,请求会被转发到Label中含有opensergo.io/canary-{其他值}: {其他值}的Pod集合。例如当配置为gray,请求会被转发到Label中含有opensergo.io/canary-gray: gray的Pod集合。

例如存在一个K8s Service go-httpbin关联了两个Deployment,其中一个Deployment管理的Pod不含有任何opensergo.io/canary为前缀的Label Key,另一个Deployment管理的Pod含有灰度标opensergo.io/canary-gray: gray,配置如下:

# go-httpbin k8s service
apiVersion: v1
kind: Service
metadata:
  name: go-httpbin
  namespace: default
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: go-httpbin
---
# go-httpbin base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-base
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=base"
          imagePullPolicy: Always
          name: go-httpbin
---
# go-httpbin gray deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-gray
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
        opensergo.io/canary-gray: gray
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=gray"
          imagePullPolicy: Always
          name: go-httpbin

如果期望对于example.com/test请求,若请求Header包含x-user-id: test,则转发到go-httpbin-gray;否则转发到go-httpbin-base,配置如下:

1.19及之后版本集群

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # 转发请求到含有灰度标opensergo.io/canary-gray: gray的Pod集合
    higress.ingress.kubernetes.io/service-subset: gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # 转发请求到不含有以opensergo.io/canary为前缀的Label的Pod集合
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact 

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # 转发请求到含有灰度标opensergo.io/canary-gray: gray的Pod集合
    higress.ingress.kubernetes.io/service-subset: gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # 配置服务为go-httpbin,但在注解中指定版本
              serviceName: go-httpbin
              servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # 转发请求到不含有以opensergo.io/canary为前缀的Label的Pod集合
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # 配置服务为go-httpbin,但在注解中指定版本
              serviceName: go-httpbin
              servicePort: 8080

使用自定义Label

通过同时配置注解higress.ingress.kubernetes.io/service-subsethigress.ingress.kubernetes.io/subset-labels,设置自定义Label来定义Subset所属Pod集合。

说明

此时该subset不再与opensergo.io/canary为前缀的Label有对应关系。

例如存在一个K8s Service go-httpbin关联了两个Deployment,其中一个Deployment管理的Pod不含有任何opensergo.io/canary为前缀的Label Key,另一个Deployment管理的Pod含有灰度标version: gray,配置如下:

# go-httpbin k8s service
apiVersion: v1
kind: Service
metadata:
  name: go-httpbin
  namespace: default
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: go-httpbin
---
# go-httpbin base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-base
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=base"
          imagePullPolicy: Always
          name: go-httpbin
---
# go-httpbin base gray
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-gray
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
        version: gray
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=gray"
          imagePullPolicy: Always
          name: go-httpbin

如果期望对于example.com/test的请求,若请求Header包含x-user-id: test,则转发到go-httpbin-gray;否则转发到go-httpbin-base。

1.19及之后版本集群

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # 转发请求到含有灰度标version: gray的Pod集合
    higress.ingress.kubernetes.io/service-subset: gray
    higress.ingress.kubernetes.io/subset-labels: version gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # 转发请求到不含有以opensergo.io/canary为前缀的Label的Pod集合
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact 

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # 转发请求到含有灰度标version: gray的Pod集合
    higress.ingress.kubernetes.io/service-subset: gray
    higress.ingress.kubernetes.io/subset-labels: version gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # 配置服务为go-httpbin,但在注解中指定版本
              serviceName: go-httpbin
              servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # 转发请求到不含有以opensergo.io/canary为前缀的Label的Pod集合
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # 配置服务为go-httpbin,但在注解中指定版本
              serviceName: go-httpbin
              servicePort: 8080

跨域

跨域资源共享CORS(Cross-Origin Resource Sharing)是指允许Web应用服务器进行跨域访问控制,从而实现跨域数据安全传输。关于跨域的更多信息,请参见跨源资源共享(CORS)

注解

说明

nginx.ingress.kubernetes.io/enable-cors

开启或关闭跨域。

nginx.ingress.kubernetes.io/cors-allow-origin

允许的第三方站点,第三方站点之间使用英文逗号分隔,支持通配符*。默认值为*,即允许所有第三方站点。

nginx.ingress.kubernetes.io/cors-allow-methods

允许的请求方法,如GET、POST、PUT等,请求方法之间使用英文逗号分隔,支持通配符*。默认值为GET、PUT、POST、DELETE、PATCH、OPTIONS。

nginx.ingress.kubernetes.io/cors-allow-headers

允许的请求Header,Header之间使用英文逗号分隔,支持通配符*。默认值为DNT、X-CustomHeader、Keep-Alive、User-Agent、X-Requested-With、If-Modified-Since、Cache-Control、Content-Type、Authorization。

nginx.ingress.kubernetes.io/cors-expose-headers

允许暴露给浏览器的响应Header,响应Header之间使用英文逗号分隔。

nginx.ingress.kubernetes.io/cors-allow-credentials

是否允许携带凭证信息。默认允许。

nginx.ingress.kubernetes.io/cors-max-age

预检结果的最大缓存时间,单位为秒。默认值为1728000秒。

例如,跨域请求被限制为只能来自example.com域的请求,并且HTTP的请求方法只能是GET和POST,允许的请求头部为X-Foo-Bar,不允许携带凭证信息。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

正则匹配

标准的K8s Ingress只支持精确匹配和前缀匹配,Higress Ingress额外支持正则匹配,您可以通过注解nginx.ingress.kubernetes.io/use-regex: true使Ingress Spec中定义的Path匹配变为正则匹配。

如期望域名为example.com,请求Path以/app或/test开头的请求转发至服务demo,配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name: regex-match
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo
                port: 
                  number: 8080
            path: /(app|test)/(.*)
            pathType: Prefix

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name: regex-match
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /(app|test)/(.*)
            backend:
              serviceName: demo
              servicePort: 8080

Rewrite重写Path和Host

在请求转发给目标后端服务之前,重写可以修改原始请求的路径(Path)和主机域(Host)。

注解

说明

nginx.ingress.kubernetes.io/rewrite-target

重写Path,支持捕获组(Capture Group)。

nginx.ingress.kubernetes.io/upstream-vhost

重写Host。

Rewrite重写Path

  1. 将请求example.com/test转发至后端服务之前,重写为example.com/dev。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/dev"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/dev"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /test
                pathType: Exact
                backend:
                  serviceName: demo-service
                  servicePort: 80
  2. 将请求example.com/v1/xxx,即以/v1/为前缀的任意Path,转发至后端服务之前,去掉Path前缀/v1,重写为example.com/xxx。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /v1/(.*)
                pathType: Prefix

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /v1/(.*) 
              	pathType: Prefix
                backend:
                  serviceName: demo-service
                  servicePort: 80
  3. 将请求example.com/v1/xxx,即以/v1/为前缀的任意Path,转发至后端服务之前,将Path前缀/v1更改为/v2,重写为example.com/v2/xxx。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /v1/(.*)
                pathType: Prefix

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /v1/(.*)
                pathType: Prefix
                backend:
                  serviceName: demo-service
                  servicePort: 80

Rewrite重写Host

例如,把请求example.com/test在转发至后端服务之前,重写为test.com/test。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

重定向

通过重定向可以把原始客户端请求更改为目标请求。

配置HTTP重定向至HTTPS

注解

说明

nginx.ingress.kubernetes.io/ssl-redirect

HTTP重定向到HTTPS

nginx.ingress.kubernetes.io/force-ssl-redirect

HTTP重定向到HTTPS

说明

Higress Ingress对于以上两个注解不区分对待,都是强制将HTTP重定向到HTTPS。

例如,将请求http://example.com/test重定向为https://example.com/test。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

永久重定向

注解

说明

nginx.ingress.kubernetes.io/permanent-redirect

永久重定向的目标URL,必须包含Scheme(HTTP或HTTPS)。

nginx.ingress.kubernetes.io/permanent-redirect-code

永久重定向的HTTP状态码,默认值为301。

例如,把请求http://example.com/test永久重定向为http://example.com/app。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

临时重定向

nginx.ingress.kubernetes.io/temporal-redirect:临时重定向的目标URL,必须包含Scheme(HTTP或者HTTPS)。

例如,将请求http://example.com/test临时重定向为http://example.com/app。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Header控制

通过Header控制,您可以在请求转发到后端服务之前对请求Header进行增删改,在收到响应转发给客户端时对响应Header进行增删改。

请求Header控制

注解

说明

higress.ingress.kubernetes.io/request-header-control-add

请求在转发给后端服务时,添加指定Header。若该Header存在,则其值拼接在原有值后面。语法如下:

  • 单个Header:Key Value。

  • 多个Header:使用YAML特殊符号|,每对Key Value单独处于一行。

higress.ingress.kubernetes.io/request-header-control-update

请求在转发给后端服务时,修改指定Header。若该Header存在,则其值覆盖原有值。语法如下:

  • 单个Header:Key Value。

  • 多个Header:使用YAML特殊符号|,每对Key Value单独处于一行。

higress.ingress.kubernetes.io/request-header-control-remove

请求在转发给后端服务时,删除指定Header。语法如下:

  • 单个Header:Key。

  • 多个Header:使用英文逗号分隔。

例如:

  • 对于请求example.com/test添加两个Header,分别是foo: bar和test: true。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/request-header-control-add: |
          foo bar
          test true
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/request-header-control-add: |
          foo bar
          test true
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com 
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Header控制可以结合灰度发布,对灰度流量进行染色。请求Header为mse:v1时将访问灰度服务demo-service-canary-v1,并添加Header(stage: gray);其他情况将访问正式服务demo-service,并添加Header(stage: production)。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
        higress.ingress.kubernetes.io/request-header-control-add: "stage gray"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary-v1
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/request-header-control-add: "stage production"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
        higress.ingress.kubernetes.io/request-header-control-add: "stage gray"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary-v1
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/request-header-control-add: |
          foo bar
          test true
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com 
          http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80

响应Header控制

注解

说明

higress.ingress.kubernetes.io/response-header-control-add

请求在收到后端服务响应之后并且转发响应给客户端之前,添加指定Header。若该Header存在,则其值拼接在原有值后面。语法如下:

  • 单个Header:Key Value。

  • 多个Header:使用YAML特殊符号|,每对Key Value单独处于一行。

higress.ingress.kubernetes.io/response-header-control-update

请求在收到后端服务响应之后并且转发响应给客户端之前,修改指定Header。若该Header存在,则其值覆盖原有值。语法如下:

  • 单个Header:Key Value。

  • 多个Header:使用YAML特殊符号|,每对Key Value单独处于一行。

higress.ingress.kubernetes.io/response-header-control-remove

请求在收到后端服务响应之后并且转发响应给客户端之前,删除指定Header。语法如下:

  • 单个Header:Key。

  • 多个Header:使用英文逗号分隔。

例如,对于请求example.com/test的响应删除Header:req-cost-time。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

重试

Higress Ingress提供路由级别的重试设置,可以自动为出错的请求进行重试。您可以按需设置重试条件,例如建立连接失败、后端服务不可用或者对指定HTTP状态码的响应等进行请求重试。

注解

说明

nginx.ingress.kubernetes.io/proxy-next-upstream-tries

请求的最大重试次数。默认为3次。

nginx.ingress.kubernetes.io/proxy-next-upstream-timeout

请求重试的超时时间,单位秒。默认未配置超时时间。

nginx.ingress.kubernetes.io/proxy-next-upstream

请求重试条件,使用英文逗号作为分隔。默认值为error,timeout,合法值如下:

  • error:建立连接失败,请求出错5xx。

  • timeout:建立连接超时,请求出错5xx。

  • invalid_header:请求出错5xx。

  • http_xxx:针对具体响应状态码的情况进行重试。例如:http_502、http_403。

  • non_idempotent:对于非幂等请求出错时进行重试。默认情况下,Higress Ingress针对非幂等POST、PATCH请求出错时不会进行重试;如果配置non_idempotent,可以开启重试。

  • off:关闭重试。

例如,设置example/test请求的最大重试次数为2次,重试超时时间为5秒,只有在响应状态码为502才重试,并且开启非幂等重试。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

IP黑白名单访问控制

Higress Ingress提供域名级和路由级的IP黑/白名单访问控制,且路由级的优先级高于域名级。

路由级IP访问控制

注解

说明

nginx.ingress.kubernetes.io/whitelist-source-range

指定路由上的IP白名单,支持IP地址或CIDR地址块,以英文逗号分隔。

higress.ingress.kubernetes.io/blacklist-source-range

指定路由上的IP黑名单,支持IP地址或CIDR地址块,以英文逗号分隔。

例如:

  • 仅允许客户端IP为1.1.xx.xx访问example.com/test。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com 
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 拒绝客户端IP为2.2.xx.xx访问example.com/test。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

域名级IP访问控制

注解

说明

higress.ingress.kubernetes.io/domain-whitelist-source-range

指定域名上的IP白名单,域名优先级低于路由级别,支持IP地址或CIDR地址块,IP之间以英文逗号分隔。

higress.ingress.kubernetes.io/domain-blacklist-source-range

指定域名上的IP黑名单,域名优先级低于路由级别,支持IP地址或CIDR地址块,IP之间以英文逗号分隔。

例如:

  • 仅允许客户端IP为1.1.xx.xx和2.2.xx.xx可以访问example.com域名下所有路由。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact
             - backend:
                  service:
                    name: app-service
                    port: 
                      number: 80
                path: /app
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
              - path: /app
                backend:
                  serviceName: app-service
                  servicePort: 80
  • 域名级和路由级IP访问控制可以结合使用,仅允许客户端IP为1.1.xx.xx和2.2.xx.xx可以访问example.com域名下所有路由,但对于example.com/order这条路由,仅允许客户端IP为3.3.xx.xx可以访问。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo-domain
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact
             - backend:
                  service:
                    name: app-service
                    port: 
                      number: 80
                path: /app
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
      name: demo-route
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /order
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo-domain
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
              - path: /app
                backend:
                  serviceName: app-service
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
      name: demo-route
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /order
                backend:
                  serviceName: demo-service
                  servicePort: 80

单机限流

Higress Ingress支持针对路由级别的单机限流策略,在设定的时间周期内,限制每个网关副本匹配在某个路由上的请求数量不大于阈值。

说明

该限流是针对单机级别,即配置的阈值在每个网关实例进行流控。如果希望限制某个路由在网关集群上的全局流量,请使用全局限流控制。

注解

说明

higress.ingress.kubernetes.io/route-limit-rpm

该Ingress定义的路由在每个网关实例上每分钟最大请求次数。瞬时最大请求次数为该值乘以limit-burst-multiplier。

触发限流时,响应Body内容为local_rate_limited,响应状态码说明:

  • 网关版本小于1.2.23:状态码为503。

  • 网关版本1.2.23及以上:状态码为429。

higress.ingress.kubernetes.io/route-limit-rps

该Ingress定义的路由在每个网关实例上每秒最大请求次数。瞬时最大请求次数为该值乘以limit-burst-multiplier。

触发限流时,响应Body内容为local_rate_limited,响应状态码说明:

  • 网关版本小于1.2.23:状态码为503。

  • 网关版本1.2.23及以上:状态码为429。

higress.ingress.kubernetes.io/route-limit-burst-multiplier

瞬时最大请求次数的因子,默认为5。

例如:

  • 限制example.com/test的请求每分钟最大请求数为100,瞬时请求数为200。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/route-limit-rpm: "100"
        higress.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/route-limit-rpm: "100"
        higress.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 限制example.com/test的请求每秒最大请求数为10,瞬时请求数50。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/route-limit-rps: "10"
        # 默认为5
        # higress.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        higress.ingress.kubernetes.io/route-limit-rps: "10"
        # 默认为5
        # higress.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

全局限流控制

Higress Ingress与Sentinel集成,提供路由级别的网关集群全局限流,即限制某个路由在网关集群全局的每秒最大请求数。

说明

该功能要求Higress Ingress网关的版本至少为1.2.25。

通过注解higress.ingress.kubernetes.io/rate-limit设置路由在网关集群全局上每秒最大请求数。当触发限流时,请求的响应结果的默认行为为:响应状态码为429,响应Body为sentinel rate limited。目前Higress Ingress提供两种方式自定义限流行为:自定义响应和重定向,这两种方式只能二选一。

自定义响应

  • higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-code:触发限流时的响应状态码,默认为429。

  • higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body-type:触发限流时的响应Body格式,默认为text

    • 配置为text时:响应的Content-Type值为text/plain; charset=UTF-8

    • 配置为json时:响应的Content-Type的值为application/json; charset=UTF-8

  • higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body:触发限流时的响应Body,默认为sentinel rate limited

样例一:期望限制example.com/test请求在网关集群上每秒最大请求数为100,保持默认的限流行为,配置如下。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

样例二:期望限制example.com/test请求在网关集群上每秒最大请求数为100,触发限流时,响应状态码为503,响应体为server is overload。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

重定向

  • higress.ingress.kubernetes.io/rate-limit-fallback-redirect-url:触发限流时的重定向地址。

样例一:期望限制example.com/test请求在网关集群上每秒最大请求数为100,触发限流时,重定向到example.com/fallback。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

全局并发控制

Higress Ingress与Sentinel集成,提供路由级别的网关集群全局并发控制,即限制某个路由在网关集群全局的最大正在处理的请求数。

说明

该功能要求Higress Ingress网关的版本至少为1.2.25。

通过注解higress.ingress.kubernetes.io/concurrency-limit设置路由在网关集群全局上最大处理请求数。当触发全局并发控制时,请求响应状态码为429,Body为sentinel rate limited。目前Higress Ingress提供两种方式可以自定义并发行为:自定义响应和重定向,这两种方式只能二选一。

自定义响应

  • higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code:触发并发控制时的响应状态码,默认为429。

  • higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body-type:触发并发控制时的响应Body的格式,默认为text

    • 配置为text时:响应的Content-Type值为text/plain; charset=UTF-8

    • 配置为json时:响应的Content-Type的值为application/json; charset=UTF-8

  • higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body:触发并发控制时的响应Body,默认为sentinel rate limited

样例一:期望限制example.com/test的请求在网关集群全局上最大处理请求数为1000,保持默认的并发行为。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

样例二:期望限制example.com/test的请求在网关集群全局最大处理请求数为1000,触发并发控制时,响应状态码为503,响应体为server is overloaded

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

重定向

  • higress.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url:触发并发控制时的重定向地址。

期望限制example.com/test请求在网关集群全局上最大处理请求数为1000,触发并发控制时,重定向到example.com/fallback。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

流量镜像

通过配置流量镜像,可以复制流量到指定服务,常用于操作审计和流量测试等场景。

  • higress.ingress.kubernetes.io/mirror-target-service:复制流量转发到指定镜像服务。服务格式为:namespace/name:port。

    • namespace: K8s Service所在的命名空间,可选,默认为Ingress所在的命名空间。

    • name:K8s Service的名称,必选。

    • port:待转发至K8s Service的端口,可选,默认为第一个端口。

  • higress.ingress.kubernetes.io/mirror-percentage:复制流量的比例。可配置的值的范围为:0~100,默认100。

说明

复制的流量在转发给目标服务时,原始请求中的Host会被自动加上-shadow后缀。

例如,将example.com/test的流量复制并转发到目标服务:命名空间为test,服务名为app,端口为8080。

说明

本示例中,复制的流量在转发给目标服务时,Host会被自动改写为example.com-shadow。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

例如,将example.com/test的流量复制并转发到目标服务:命名空间为test,服务名为app,端口为8080,且复制比例为10%。

说明

本示例中,复制的流量在转发给目标服务时,Host会被自动改写为example.com-shadow。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
    higress.ingress.kubernetes.io/mirror-percentage: 10
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
    higress.ingress.kubernetes.io/mirror-percentage: 10
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

配置后端服务协议:HTTPS或gRPC

Higress Ingress默认使用HTTP协议转发请求到后端业务容器。当您的业务容器为HTTPS协议时,可以通过使用注解nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"来转发请求到后端业务容器;当您的业务容器为gRPC服务时,可以通过使用注解nginx.ingress.kubernetes.io/backend-protocol: "GRPC"来转发请求到后端业务容器。

说明

相比Nginx Ingress的优势,如果您的后端服务所属的K8s Service资源中关于Port Name的定义为gRPC或HTTP2,您无需配置注解nginx.ingress.kubernetes.io/backend-protocol: "GRPC",Higress Ingress会自动使用gRPC或者HTTP2。

例如:

  • 请求example/test转发至后端服务使用HTTPS协议。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 请求example/test转发至后端服务使用gRPC协议。此处列举两种做法,如下:

    • 方法1:通过注解,配置如下:

      1.19及之后版本集群

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        annotations:
          nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - backend:
                    service:
                      name: demo-service
                      port: 
                        number: 80
                  path: /test
                  pathType: Exact

      1.19版本之前集群

      apiVersion: networking.k8s.io/v1beta1
      kind: Ingress
      metadata:
        annotations:
          nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - path: /test
                  backend:
                    serviceName: demo-service
                    servicePort: 80
    • 方法2:通过Service Port Name,配置如下:

      1.19及之后版本集群

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - backend:
                    service:
                      name: demo-service
                      port: 
                        number: 80
                  path: /order
                  pathType: Exact
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: demo-service
      spec:
        ports:
          - name: grpc
            port: 80
            protocol: TCP
        selector:
          app: demo-service

      1.19版本之前集群

      apiVersion: networking.k8s.io/v1beta1
      kind: Ingress
      metadata:
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - path: /test
                  backend:
                    serviceName: demo-service
                    servicePort: 80
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: demo-service
      spec:
        ports:
          - name: grpc
            port: 80
            protocol: TCP
        selector:
          app: demo-service

配置后端服务的负载均衡算法

负载均衡决定着网关在转发请求至后端服务时如何选择节点。

普通负载均衡算法

nginx.ingress.kubernetes.io/load-balance:后端服务的普通负载均衡算法。默认为round_robin。合法值如下:

  • round_robin:基于轮询的负载均衡。

  • least_conn:基于最小请求数的负载均衡。

  • random:基于随机的负载均衡。

重要

云原生网关不支持EWMA算法,若配置为EWMA算法,会回退到Round Robin算法。

例如,设置后端服务demo-service的负载均衡算法为least_conn。设置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: "least_conn"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /order
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: "least_conn"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

基于一致性Hash的负载均衡算法

基于一致性Hash的负载均衡算法具备请求亲和性,具有相同特征的请求会始终负载到相同节点上。Higress Ingress支持基于部分Nginx变量的请求Header和请求路径参数作为Hash Key。

nginx.ingress.kubernetes.io/upstream-hash-by:基于一致性Hash的负载均衡算法,云原生网关支持以下几种形式:

  • 云原生网关支持配置部分nginx变量:

    • $request_uri:请求的Path(包括路径参数)作为Hash Key。

    • $host:请求的Host作为Hash Key。

    • $remote_addr:请求的客户端IP作为Hash Key。

  • 基于请求Header的一致性Hash。您只需配置为$http_headerName。

  • 基于请求路径参数的一致性Hash。您只需配置为$arg_varName。

例如:

  • 基于请求的客户端IP作为Hash Key,同一个客户端IP的请求始终负载到同一个节点。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 基于请求Header x-stage作为Hash key,带有x-stage头部的请求且值相同的请求始终负载到同一个节点。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 基于请求路径参数 x-stage作为Hash key,带有路径参数x-stage的请求且值相同的请求始终负载到同一个节点。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

服务预热(无损上线)

服务预热可以保证新节点上线时,流量在指定预热窗口内是逐步调大,充分保证新节点完成预热。

higress.ingress.kubernetes.io/warmup:服务预热时间,单位为秒。默认不开启。

说明

服务预热依赖于所选的负载均衡算法,目前仅支持Round Robin和least_conn。

例如,对于后端服务demo-service开启预热,预热窗口为30s。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/warmup: "30"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/warmup: "30"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Cookie亲和性(会话保持)

具备相同Cookie的请求会被网关始终负载到同一个节点,并且如果第一次访问携带Cookie,Higress Ingress会在第一次响应时为客户端生成一个Cookie,用来保证后续的请求被网关始终负载到相同节点。

注解

说明

nginx.ingress.kubernetes.io/affinity

亲和性种类,目前只支持Cookie,默认为Cookie。

nginx.ingress.kubernetes.io/affinity-mode

亲和性模式,云原生网关目前只支持Balanced模式,默认为Balanced模式。

nginx.ingress.kubernetes.io/session-cookie-name

配置指定Cookie的值作为Hash Key,默认为INGRESSCOOKIE。

nginx.ingress.kubernetes.io/session-cookie-path

当指定Cookie不存在,生成的Cookie的Path值,默认为/。

nginx.ingress.kubernetes.io/session-cookie-max-age

当指定Cookie不存在,生成的Cookie的过期时间,单位为秒,默认为Session会话级别。

nginx.ingress.kubernetes.io/session-cookie-expires

当指定Cookie不存在,生成的Cookie的过期时间,单位为秒,默认为Session会话级别。

例如:

  • 开启Cookie亲和性,利用Higress Ingress的默认配置,即Cookie的名字为INGRESSCOOKIE,Path为/,Cookie的生命周期为Session会话级别。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • 开启Cookie亲和性,Cookie的名字为test,Path为/,Cookie的过期时间为10s。配置如下:

    1.19及之后版本集群

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
        nginx.ingress.kubernetes.io/session-cookie-name: "test"
        nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    1.19版本之前集群

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
        nginx.ingress.kubernetes.io/session-cookie-name: "test"
        nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

网关与后端服务之间的连接池配置

通过在网关侧对指定服务进行连接池配置,可以控制网关与后端服务之间的连接数量,有效防止后端服务过载,提高后端服务的稳定性和高可用。

  • higress.ingress.kubernetes.io/connection-policy-tcp-max-connection:网关与后端服务之间可以建立连接的最大数量。

  • higress.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint:网关与后端服务的单个节点之间可以建立连接的最大数量。

  • higress.ingress.kubernetes.io/connection-policy-http-max-request-per-connection:网关与后端服务之间单个连接上的最大请求数。

例如,对后端服务demo-service配置,网关与后端服务之间可以建立连接的最大数量为10,网关与后端服务的单个节点之间可以建立连接的最大数量为2。

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/connection-policy-tcp-max-connection:10
  	higress.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint:2
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/connection-policy-tcp-max-connection:10
  	higress.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint:2
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

配置客户端到网关之间的TLS版本以及加密套件

目前,Higress Ingress默认最小TLS版本为TLSv1.0,默认最大TLS版本为TLSv1.3,默认加密套件为:

  • ECDHE-ECDSA-AES128-GCM-SHA256

  • ECDHE-RSA-AES128-GCM-SHA256

  • ECDHE-ECDSA-AES128-SHA

  • ECDHE-RSA-AES128-SHA

  • AES128-GCM-SHA256

  • AES128-SHA

  • ECDHE-ECDSA-AES256-GCM-SHA384

  • ECDHE-RSA-AES256-GCM-SHA384

  • ECDHE-ECDSA-AES256-SHA

  • ECDHE-RSA-AES256-SHA

  • AES256-GCM-SHA384

  • AES256-SHA

您可以通过以下注解为特定的域名设置最小或者最大TLS版本以及加密套件。

注解

说明

higress.ingress.kubernetes.io/tls-min-protocol-version

指定TLS的最小版本,默认值为TLSv1.0。合法值如下:

  • TLSv1.0

  • TLSv1.1

  • TLSv1.2

  • TLSv1.3

higress.ingress.kubernetes.io/tls-max-protocol-version

指定TLS的最大版本,默认值为TLSv1.3。

nginx.ingress.kubernetes.io/ssl-cipher

指定TLS的加密套件,可以指定多个英文冒号分隔,仅当TLS握手时采用TLSv1.0~1.2生效。

例如,对于域名example.com,设置TLS最小版本为TLSv1.2,最大版本为TLSv1.2。配置如下:

1.19及之后版本集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
    higress.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
    higress.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

网关与后端服务双向认证 (mTLS)

Higress Ingress默认使用HTTP协议转发请求到后端业务容器。您可以通过使用注解nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"配置Higress Ingress访问后端服务使用HTTPS协议,但这是单向TLS,也就是说只有Higress Ingress会验证后端服务提供的证书,且一般后端服务使用的证书需要是权威CA(Certificate Authority)签发的。另一种更安全的模式是零信任,网关会验证后端服务的证书是否合法,同样后端服务也会验证网关提供的证书是否合法,这就是MTLS,网关与后端服务进行双向认证。

注解

说明

nginx.ingress.kubernetes.io/proxy-ssl-secret

网关使用的客户端证书,用于后端服务对网关进行身份认证,格式为secretNamespace/secretName。

nginx.ingress.kubernetes.io/proxy-ssl-name

TLS握手期间使用的SNI。

nginx.ingress.kubernetes.io/proxy-ssl-server-name

开启或关闭TLS握手期间使用的SNI。

例如,网关与后端服务进行双向认证,网关使用的secret name为gateway-cert,命名空间为default。配置如下:

1.19版本之后集群

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/ateway-cert"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

1.19版本之前集群

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/ateway-cert"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80