pre-shared-cert ordering (documentation/implementation mismatch)

[hermanbanken]

The annotation pre-shared-cert as documented on [1] is supposed to be ordered:

Which certificate is presented?

The load balancer chooses a certificate according to these rules:
[...]
For pre-shared certificates listed in the annotation, the primary certificate is the first certificate in the list.

However, this is not the case. If I apply the YAML [2], then the annotation is rewritten by the controller. The last-applied annotation correctly reflects the order cert-2, cert-1, but the annotation becomes cert-1,cert-2.

This is troublesome, because some client might not support SNI. The order of the certificates in GCP Load Balancers is therefore (critically) important. We switched from Kubernetes-managed certificates to pre-shared-certs to more easily have the same certificates shared by multiple clusters each in their own LB. We would be able to have our Security officer deliver us a new certificate name, which we could apply to all clusters automatically using Kustomize & Flux (GitOps). However, that is impacted by this reordering bug. As a workaround, we'll need to name the primary certificate with a lexicographically-first name.

1 = https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-multi-ssl#which_certificate_is_presented is
2 = Ingress resource

# file: ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.gcp.kubernetes.io/pre-shared-cert: cert-2, cert-1
    kubernetes.io/ingress.allow-http: "false"
  name: services
spec:
  rules:
  - host: subdomain2.example.org
    http:
      paths:
      - backend:
          serviceName: service2
          servicePort: 80
  - host: subdomain1.example.org
    http:
      paths:
      - backend:
          serviceName: service1
          servicePort: 80

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rsevat]

/reopen
/remove-lifecycle rotten

I can confirm that this is still the case. It gets alphabetically reordered. No matter what order you submit it in. As such my controller was continuously fighting with the controller that kept reordering it. The documentation that describes that the order matters is correct, the first certificate on the list does get shown as primary certificate if there is no SNI match.

Due to this bug the only way we can currently control which certificate gets shown is if we need to use self managed certificates where we can choose what the name is of the certificate and via that we can ensure that it's alphabetically either before or after the google managed certificates.

For example:

We use a single google managed certificate with a random generated name by google: mcrt-780449c3-edc2-4989

With multiple subject alternative names:

app.abc.org
app2.abc.org

Now we want to add app3.abc.org, but that domain is live in our old datacenter. To migrate it to GCP we'll add a wildcard certificate named: wildcard-abc-org to our ingress with as domain name: *.abc.org. So that we have a 0 downtime migration while the new google managed certificate gets created in the background.

We add it to our ingress as:
ingress.gcp.kubernetes.io/pre-shared-cert: wildcard-abc-org, mcrt-780449c3-edc2-4989

Expected behavior:

Users that connect to app3.abc.org have no SNI match, thus get shown the primary certificate (first in the list)

Actual behavior:

Under the surface the pre-shared-cert annotation gets alphabetically rewritten to:

ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-780449c3-edc2-4989, wildcard-abc-org

Users that connect to app3.abc.org have no SNI match, thus get shown the primary certificate (first in the list). Which is a invalid certificate because it only matches for app.abc.org and app2.abc.org. Which leads to certificate errors.

The only way for us to mitigate this is:

1. Ensure self managed certificates have an alphabetically earlier name like: a-wildcard-abc-org
2. Delete the google managed certificate and only show the wildcard certificate during the migration while we wait for the new google managed certificate that includes all domain names to be ready

We'll be going with option 2 for now. But ideally this gets fixed so that it does not get reordered and so the user can actually control which certificate gets shown first.

[k8s-ci-robot]

@rsevat: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

I can confirm that this is still the case. It gets alphabetically reordered. No matter what order you submit it in. As such my controller was continuously fighting with the controller that kept reordering it. The documentation that describes that the order matters is correct, the first certificate on the list does get shown as primary certificate if there is no SNI match.

Due to this bug the only way we can currently control which certificate gets shown is if we need to use self managed certificates where we can choose what the name is of the certificate and via that we can ensure that it's alphabetically either before or after the google managed certificates.

For example:

We use a single google managed certificate with a random generated name by google: mcrt-780449c3-edc2-4989

With multiple subject alternative names:

app.abc.org
app2.abc.org

Now we want to add app3.abc.org, but that domain is live in our old datacenter. To migrate it to GCP we'll add a wildcard certificate named: wildcard-abc-org to our ingress with as domain name: *.abc.org. So that we have a 0 downtime migration while the new google managed certificate gets created in the background.

We add it to our ingress as:
ingress.gcp.kubernetes.io/pre-shared-cert: wildcard-abc-org, mcrt-780449c3-edc2-4989

Expected behavior:

Users that connect to app3.abc.org have no SNI match, thus get shown the primary certificate (first in the list)

Actual behavior:

Under the surface the pre-shared-cert annotation gets alphabetically rewritten to:

ingress.gcp.kubernetes.io/pre-shared-cert: mcrt-780449c3-edc2-4989, wildcard-abc-org

Users that connect to app3.abc.org have no SNI match, thus get shown the primary certificate (first in the list). Which is a invalid certificate because it only matches for app.abc.org and app2.abc.org. Which leads to certificate errors.

The only way for us to mitigate this is:

1. Ensure self managed certificates have an alphabetically earlier name like: a-wildcard-abc-org
2. Delete the google managed certificate and only show the wildcard certificate during the migration while we wait for the new google managed certificate that includes all domain names to be ready

We'll be going with option 2 for now. But ideally this gets fixed so that it does not get reordered and so the user can actually control which certificate gets shown first.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hermanbanken]

/reopen

[k8s-ci-robot]

@hermanbanken: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hermanbanken]

reopening for @rsevat

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[swetharepakula]

/lifecycle frozen

To keep the issue alive

[swetharepakula]

The Ingress Controller does not write to this annotation. This is actually a consequence of using the Managed Certificates controller which writes to the annotation to add the Google Managed Cert: https://github.com/GoogleCloudPlatform/gke-managed-certs/blob/master/pkg/controller/sync/sync.go#L152-L162.

The managed cert controller has to add the managed cert to the pre-shared-cert annotation for the Ingress controller to add it the SSL cert. When adding the cert, the managed cert controller always sorts the certs.