将TLS应用于工作入口会导致默认后端-404

提问者：小点点

将TLS应用于工作入口会导致默认后端-404

我已经通过helm在入口命名空间中安装了Nginx入口控制器。

helm ls --namespace ingress
NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                   APP VERSION
nginx-ingress   ingress     1           2020-03-15 10:47:51.143159 +0530 IST    deployed    nginx-ingress-1.34.2    0.30.0

服务和部署如下

apiVersion: v1
kind: Service
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
spec:
  type: LoadBalancer
  ports:
    - port: 8080
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: test-service
    app.kubernetes.io/instance: test-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: test-service
      app.kubernetes.io/instance: test-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: test-service
        app.kubernetes.io/instance: test-service
    spec:
      containers:
        - name: test-service
          image: "<acr-url>/test-service:c93c58c0bd4918de06d46381a89b293087262cf9"
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /devops/health/liveness
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /devops/health/readiness
              port: 8080
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          env:
            - name: test-serviceClientId
              valueFrom:
                secretKeyRef:
                  key: test-serviceClientId
                  name: test-service-133
            - name: test-serviceClientSecret
              valueFrom:
                secretKeyRef:
                  key: test-serviceClientSecret
                  name: test-service-133
            - name: test-serviceTenantClientId
              valueFrom:
                secretKeyRef:
                  key: test-serviceTenantClientId
                  name: test-service-133
            - name: test-serviceTenantClientSecret
              valueFrom:
                secretKeyRef:
                  key: test-serviceTenantClientSecret
                  name: test-service-133
          resources:
            limits:
              cpu: 800m
            requests:
              cpu: 300m

通过重写在服务上配置入口，如下所示

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
    - apiexample.centralus.cloudapp.azure.com
    secretName: tls-secret
  rules:
    - host: "apiexample.centralus.cloudapp.azure.com"
      http:
        paths:
          - path: /testservice(/|$)(.*)
            backend:
              serviceName: test-service
              servicePort: 8080

tls-秘密已使用

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"

$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt

在入口中应用tls配置之前，我能够从apiendpoint获得响应。apiendpoint使用oauth保护。

APIendpoint：

http://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint

在入口上应用TLS配置后，并点击

https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint

我得到默认后端404。

我已经使用另一个示例服务（没有oauth保护）测试了入口的TLS，它似乎适用于该服务。这是其他服务的配置

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tea
spec:
  replicas: 3
  selector:
    matchLabels:
      app: tea
  template:
    metadata:
      labels:
        app: tea
    spec:
      containers:
      - name: tea
        image: nginxdemos/nginx-hello:plain-text
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: tea-svc
  labels:
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: tea

服务的入口配置如下

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
    - apiexample.centralus.cloudapp.azure.com
    secretName: tls-secret
  rules:
  - host: apiexample.centralus.cloudapp.azure.com
    http:
      paths:
      - path: /teaprefix(/|$)(.*)
        backend:
          serviceName: tea-svc
          servicePort: 80

endpoint

https://apiexample.centralus.cloudapp.azure.com/teaprefix/someurl

工作正常。

如果我的配置中缺少任何东西，以及我可能忽略的任何潜在问题，请告诉我。

注意：服务和入口部署在默认命名空间中，入口控制器在入口命名空间中运行Nginx入口控制器在2个Pod中运行

来自入口控制器的日志，具有TLS配置

Pod1

 10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 302 0 "-" "PostmanRuntime/7.23.0" 1495 0.004 [default-test-service-8080] [] 10.244.0.7:8080 0 0.004 302 f4671ede2f95148220c21fe44de6fdad
 10.244.0.1 - - [22/Mar/2020:06:57:13 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 404 21 "http://apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1563 0.001 [upstream-default-backend] [] 10.244.0.225:8080 21 0.004 404 ed41b36bc6b89b60bc3f208539a0d44c

Pod2

    10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 308 171 "https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1580 0.000 [upstream-default-backend] [] - - - - ce955b7bb5118169e99dd4051060c897

没有TLS配置的入口控制器日志

10.244.0.1 - - [22/Mar/2020:07:04:34 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 200 276 "-" "PostmanRuntime/7.23.0" 1495 2.165 [default-test-service-8080] [] 10.244.0.4:8080 548 2.168 200 e866f277def90c398df4e509e45718b2

更新

禁用后端服务（test-service）上的身份验证也会导致相同的行为。不应用TLS，能够使用超文本传输协议命中endpoint，而无需任何承载令牌。应用TLS后，当我使用https/超文本传输协议命中endpoint时，获得默认后端-404

更新

通过ClusterIP公开服务，而无需

service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'

注释代替LoadBalancer似乎也没有帮助。endpoint无需TLS工作，并应用TLS，获取默认后端-404

更新

test-service是一个Spring Boot应用程序，具有以下WebSecurityConfiguration

@Component
@EnableResourceServer
public class WebSecurityConfiguration extends ResourceServerConfigurerAdapter {

  private static final Logger LOGGER = LoggerFactory.getLogger(WebSecurityConfiguration.class);

  private final HealthCheckWebSecurity healthCheckWebSecurity = new HealthCheckWebSecurity();

  private final Oauth2Settings oauth2Settings;
  private final JwtTokenStore jwtTokenStore;
  private final TenantService tenantService;
  private final TransportGuaranteeWebSecurity transportGuaranteeWebSecurity;

  @Autowired
  public WebSecurityConfiguration(
      Oauth2Settings oauth2Settings,
      JwtTokenStore jwtTokenStore,
      TenantService tenantService,
      TransportGuaranteeWebSecurity transportGuaranteeWebSecurity) {
    this.oauth2Settings = oauth2Settings;
    this.jwtTokenStore = jwtTokenStore;
    this.tenantService = tenantService;
    this.transportGuaranteeWebSecurity = transportGuaranteeWebSecurity;
  }

  @Override
  public void configure(ResourceServerSecurityConfigurer resources) throws Exception {
    String resourceId = oauth2Settings.getResource("default").getResourceId();
    LOGGER.info("Resource service id: {}", resourceId);

    resources.resourceId(resourceId).tokenStore(jwtTokenStore);
  }

  @Override
  public void configure(HttpSecurity http) throws Exception {
    http.requestMatchers().anyRequest();
    http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
    http.csrf().disable();

    healthCheckWebSecurity.configure(http);
    transportGuaranteeWebSecurity.configure(http);

    http.authorizeRequests().anyRequest().permitAll();
    http.addFilterAfter(buildTenancyContextFilter(), ChannelProcessingFilter.class);
    http.addFilterAfter(buildLongUrlFilter(), ChannelProcessingFilter.class);
  }

  private TenancyContextFilter buildTenancyContextFilter() {
    return new TenancyContextFilter(tenantService,
        new PathVariableTenantExtractor(Arrays.asList("/{tenantAlias}/api/**")));
  }

  private LongRequestHttpFilter buildLongUrlFilter() {
    return new LongRequestHttpFilter();
  }
}

public final class TransportGuaranteeWebSecurity {

  private TransportGuaranteeSettings transportGuaranteeSettings;

  TransportGuaranteeWebSecurity(TransportGuaranteeSettings transportGuaranteeSettings) {
    this.transportGuaranteeSettings = transportGuaranteeSettings;
  }


  public void configure(HttpSecurity httpSecurity) throws Exception {
    if (httpsRequired()) {
      httpSecurity.requiresChannel().anyRequest().requiresSecure();
    } else {
      httpSecurity.requiresChannel().anyRequest().requiresInsecure();
    }
  }

  private boolean httpsRequired() {
    final String transportGuarantee = transportGuaranteeSettings.getTransportGuarantee();
    return !TransportGuaranteeSettings.TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee);
  }

}


@ConfigurationProperties(prefix = "web.security")
public class TransportGuaranteeSettings {
  static final String TRANSPORT_GUARANTEE_NONE = "NONE";
  static final String TRANSPORT_GUARANTEE_CONFIDENTIAL = "CONFIDENTIAL";

  private static final Logger LOGGER = LoggerFactory.getLogger(TransportGuaranteeSettings.class);

  private static final String TRANSPORT_GUARANTEE_PROPERTY = "web.security.transportGuarantee";

  private String transportGuarantee;

  public String getTransportGuarantee() {
    return transportGuarantee;
  }

  public void setTransportGuarantee(String transportGuarantee) {
    this.transportGuarantee = transportGuarantee.trim();
    logUnexpectedValue();
  }

  private void logUnexpectedValue() {
    if (!TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee)
        && !TRANSPORT_GUARANTEE_CONFIDENTIAL.equalsIgnoreCase(transportGuarantee)) {
      LOGGER.debug(
          "Unknown value '{}' for property '{}' (expected '{}' or '{}'). Defaulted to '{}'.",
          transportGuarantee, TRANSPORT_GUARANTEE_PROPERTY, TRANSPORT_GUARANTEE_NONE, TRANSPORT_GUARANTEE_CONFIDENTIAL,
          TRANSPORT_GUARANTEE_CONFIDENTIAL);
    }
  }
}

在我的application. yaml中，

web.security.transportGuarantee: NONE

租户上下文过滤器从URL中提取租户信息并设置ThreadLocal。这应该没有任何问题，因为我可以在没有TLS配置的情况下访问endpoint。出于同样的原因，我也没有看到TransportGu保证WebSecurity有任何问题。

更多用于调试的日志

kubectl获取pods-owide--namespace入口

NAME                                             READY   STATUS    RESTARTS   AGE   IP             NODE                                NOMINATED NODE   READINESS GATES
nginx-ingress-controller-5fcbccd545-bdh25        1/1     Running   1          15d   10.244.0.22    aks-agentpool-44086776-vmss000000   <none>           <none>
nginx-ingress-controller-5fcbccd545-ptx6j        1/1     Running   0          15d   10.244.0.21    aks-agentpool-44086776-vmss000000   <none>           <none>
nginx-ingress-default-backend-554d7bd77c-zxzlf   1/1     Running   0          15d   10.244.0.225   aks-agentpool-44086776-vmss000000   <none>           <none>

kubectl获取svc

test-service                          LoadBalancer   10.0.231.35    13.89.111.39    8080:31534/TCP               14d
tea-svc                                   ClusterIP      10.0.12.216    <none>          80/TCP                       17d

kubectl获取ing

test-service         apiexample.centralus.cloudapp.azure.com   10.240.0.4   80, 443      15d

共1个答案

匿名用户

我已经在我的GCP帐户中重现了您的场景，但没有得到相同的结果，因此我发布了对每个组件进行故障排除的步骤，以确保所有组件都正常工作。在简历中，似乎主要问题是应用程序如何处理路径或主机。

库伯内特斯： 1.15.3（GKE）Nginx入口：安装在官方文档之后

根据您的yaml，我删除了就绪和活性探针和要测试的环境环境，并将图像更改为nginx图像（在端口80上）：

apiVersion: v1
kind: Service
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: test-service
    app.kubernetes.io/instance: test-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: test-service
      app.kubernetes.io/instance: test-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: test-service
        app.kubernetes.io/instance: test-service
    spec:
      containers:
        - name: test-service
          image: nginx
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 80
              protocol: TCP

应用后，我们可以在应用入口规范之前检查（部署和服务）是否按预期运行。

要检查这一点，我们可以使用curl图像来通过官方kubernetes文档来curl目标或dnsutil容器。在这种情况下，我使用了curlimages/curl做测试：

apiVersion: v1
kind: Pod
metadata:
  name: curl
  namespace: default
spec:
  containers:
  - name: curl
    image: curlimages/curl
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

随着curl容器的运行，我们可以首先检查我们的nginx映像的容器是否正确运行，并直接IP响应请求“curling”。

下面的命令将从带有标签的pod创建一个名为$pod的变量app.kubernetes.io/name=test-service带有pod的ip。

$ pod=$(kubectl get pods -ojsonpath='{.items[*].status.podIP}' -l app.kubernetes.io/name=test-service)

$ echo $pod
192.168.109.12

使用前面的curlpod create，我们可以检查pod是否正在处理请求：

$ kubectl exec curl -- curl -Is $pod

HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Tue, 24 Mar 2020 09:08:21 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
Connection: keep-alive
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

请参阅响应HTTP/1.1 200 OK，让我们继续测试服务：

$ kubectl exec curl -- curl -Is test-service
HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Tue, 24 Mar 2020 09:11:13 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
Connection: keep-alive
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

这里也一样，HTTP/1.1 200 OK服务。

让我们更进一步，在没有TLS的情况下部署nginx入口，进行前后测试：

生成和应用证书：

$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"
...
$ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
secret/tls-secret created

没有TLS进入（我已经更改为端口80以匹配我的nginx映像）：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
    - host: "apiexample.centralus.cloudapp.azure.com"
      http:
        paths:
          - path: /testservice(/|$)(.*)
            backend:
              serviceName: test-service
              servicePort: 80

使用GCP提供的IP（省略）从我的桌面测试到互联网：

$ curl -ILH "Host: apiexample.centralus.cloudapp.azure.com" http://34.77.xxx.xx/testservice
HTTP/1.1 200 OK
Server: nginx/1.17.8
Date: Tue, 24 Mar 2020 10:41:21 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 03 Mar 2020 14:32:47 GMT
ETag: "5e5e6a8f-264"
Accept-Ranges: bytes

直到这里一切正常。我们现在可以将TLS添加到入口规范，然后重试：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-service
  labels:
    app.kubernetes.io/name: test-service
    helm.sh/chart: test-service-0.1.0
    app.kubernetes.io/instance: test-service
    app.kubernetes.io/managed-by: Helm
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  tls:
  - hosts:
    - apiexample.centralus.cloudapp.azure.com
    secretName: tls-secret
  rules:
    - host: "apiexample.centralus.cloudapp.azure.com"
      http:
        paths:
          - path: /testservice(/|$)(.*)
            backend:
              serviceName: test-service
              servicePort: 80

使用curl测试：

curl -ILH "Host: apiexample.centralus.cloudapp.azure.com" https://34.77.147.74/testservice -k
HTTP/2 200 
server: nginx/1.17.8
date: Tue, 24 Mar 2020 10:45:25 GMT
content-type: text/html
content-length: 612
vary: Accept-Encoding
last-modified: Tue, 03 Mar 2020 14:32:47 GMT
etag: "5e5e6a8f-264"
accept-ranges: bytes
strict-transport-security: max-age=15724800; includeSubDomains

好的，这就是TLS的工作，因此基于此，我们可以得出结论，您的yaml规范正在工作，也许您在入口定义和应用程序中遇到了path的一些问题：

您正在使用注释nginx.ingress.kubernetes.io/rewrite-target：/2美元并在路径/testservice(/|$)(.*)中

这意味着，由(.*)捕获的任何字符都将分配给占位符2美元，然后将其用作rewrite-target注释中的参数。

根据您的入口路径正则表达式：

apiexample.centralus.cloudapp.azure.com/testservice重写为apiexample.centralus.cloudapp.azure.com/

apiexample.centralus.cloudapp.azure.com/testservice/重写为apiexample.centralus.cloudapp.azure.com/

apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint重写为apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint

在nginx pod日志中检查时，您可以在重写后看到请求的url：

2020/03/24 10:59:33 [error] 7#7: *186 open() "/usr/share/nginx/html/tenant/api/v1/endpoint" failed (2: No such file or directory), client: 10.20.1.61, server: localhost, request: "HEAD /tenant/api/v1/endpoint HTTP/1.1", host: "apiexample.centralus.cloudapp.azure.com"

所以，通过这个测试，我可以得出结论，您的部署、服务和入口正在工作，没有任何错别字或格式问题。所以我的建议是仔细检查应用程序

确保您的应用程序正确处理路径；
如果您的应用程序正在执行一些URL验证，请确保它可以处理超文本传输协议和https；
如果您已启用CORS，请按照此处所述调整入口。

由于您没有发布任何应用程序作为示例来重现，我的测试仅限于作为后端的通用应用程序。如果您可以提供有关后端应用程序的更多详细信息，或者一些产生相同行为的通用应用程序，请告诉我，并乐意通过更多详细信息改进我的答案。

参考资料：

https://kubernetes.github.io/ingress-nginx/deploy/

https://kubernetes.github.io/ingress-nginx/user-guide/ingress-path-matching/