fix: make compatibility versions test use dynamic n-1 branch for test (from hardcoded release-1.31) #33837

aaron-prindle · 2024-11-25T06:28:42Z

This PR modifies experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh to fetch the latest release-X.Y branch from https://github.com/kubernetes/kubernetes.git and use that as the n-1 version for compatibility versions testing. This way this test is dynamic and updates the n-1 version as new releases are made - updating automatically over time vs the current hardcoded release-1.31 value currently present in the test.

k8s-ci-robot · 2024-11-25T06:28:49Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aaron-prindle
Once this PR has been reviewed and has the lgtm label, please assign wojtek-t for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

config/OWNERS
~~experiment/compatibility-versions/OWNERS~~ [aaron-prindle]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

aaron-prindle · 2024-11-25T19:54:28Z

/cc @BenTheElder

BenTheElder · 2024-11-25T20:16:38Z

experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

-  # Clone the specific Kubernetes release branch
-  clone_kubernetes_release
+  # Clone the previous versions Kubernetes release branch
+  # TODO(aaron-prindle) extend the branches to test from n-1 -> n-1..3 as more k8s releases are done that support compatibility versions


We should also think about which branches we're trying to test, if what we're trying to test is the api-server compatibility in master, it might better suit us to use the latest stable tagged patch release in the previous branch for the other components. In practice there's not usually a big distinction anyhow though.

Also, if we did that we could fetch binaries instead of compiling both branches from source.

Makes sense, given that it doesn't seem to big a distinction atm perhaps we can keep this as is for now? I created #33594 to track using precompiled binaries so perhaps we can address that in a seperate PR.

Yeah, I just wanted to point that out, with the precompiled builds we have a bunch of different marker files for the subtle variations depending on what you're actually focusing on testing and how stable you want the other end of the skew.

For now 🤷, let's just get it working, but later we'll want to decide.

BenTheElder · 2024-11-25T20:18:13Z

experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

-  git clone --single-branch --branch "${KUBE_RELEASE_BRANCH}" https://github.com/kubernetes/kubernetes.git
+get_latest_release_version() {
+    # Fetch all branch names
+    git ls-remote --heads https://github.com/kubernetes/kubernetes.git | \


we should probably prefer using local git state?

we can list branches or tags locally and determine this, and it will work in offline development

actually, further, to make this work with N branches, it should look at something like git describe, get the current release branch from that and then compute the N-... branch.

Because we'll want to be running this on each release going forward and those can't just assume the target is the latest release at all.

Does the k8s repo that Prow pulls have all of the other branches available? If so, definitely +1 - just want to confirm that. We pass in base_ref: master as part of the k8s/k8s repo config so wasn't sure if all branches were pulled down.

BenTheElder · 2024-11-25T20:18:49Z

experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

-  # Clone the specific Kubernetes release branch
-  clone_kubernetes_release
+  # Clone the previous versions Kubernetes release branch
+  # TODO(aaron-prindle) extend the branches to test from n-1 -> n-1..3 as more k8s releases are done that support compatibility versions


Also, if we did that we could fetch binaries instead of compiling both branches from source.

BenTheElder · 2024-11-25T20:19:49Z

experiment/compatibility-versions/e2e-k8s-compatibility-versions.sh

+  # Clone the previous versions Kubernetes release branch
+  # TODO(aaron-prindle) extend the branches to test from n-1 -> n-1..3 as more k8s releases are done that support compatibility versions
+  export PREV_RELEASE_BRANCH="release-${EMULATED_VERSION}"
+  git clone --single-branch --branch "${PREV_RELEASE_BRANCH}" https://github.com/kubernetes/kubernetes.git "${PREV_RELEASE_BRANCH}"


Do we really need another full clone? (also if we're going to clone, we should add some performance related flags, cloning kubernetes in full is really expensive with the 10 years of history)

If we need to use git, let's use a worktree. But I think we can fetch binaries for the old release and for the new release we should already have the checkout in order, presumably.

+1, I'm familiar with some of the flags to pull branch with minimal history (--depth=1, --single-branch) which I've changed this to use now. Did you have any other suggestions for flags there?

Can you explain how a worktree would help in this context or give an example for what you are suggesting with If we need to use git, let's use a worktree.? Thanks

+1, I'm familiar with some of the flags to pull branch with minimal history (--depth=1, --single-branch) which I've changed this to use now. Did you have any other suggestions for flags there?

--depth=1 will break git describe (it has to walk back to find the last tag), in prow we use --filter=blob:none instead

Ah got it, I've updated this to use --filter=blob:none now:

git clone --filter=blob:none --single-branch --branch "${PREV_RELEASE_BRANCH}" https://github.com/kubernetes/kubernetes.git "${PREV_RELEASE_BRANCH}"

BenTheElder · 2024-11-25T20:24:00Z

TLDR I think:

let's determine which release we're on currently by using hack/print-workspace-status.sh (so single source of truth about kube versioning from the main repo), that way this script "just works" for 1.32, 1.33, 1.34, ... (instead of floating with the current latest)
then we can compute the N-1 or N-2 release minor version (maybe we should make the constant minor back-compat offset a variable with a default of 1 or 2?)
we can then do kind build node image v1.N-1.(latest) (TODO: we can get latest from git tag | grep 'v1.N-1.' ?)
create kind cluster from this image, ensure --compat-version is set to N-1
then we can build and patch in api-server from the current branch
run tests from N-1 ? (we can also download these the same as kind downloading the component release binaries)

aaron-prindle · 2024-11-26T00:29:44Z

let's determine which release we're on currently by using hack/print-workspace-status.sh (so single source of truth about kube versioning from the main repo), that way this script "just works" for 1.32, 1.33, 1.34, ... (instead of floating with the current latest)
Ah I didn't know about hack/print-workspace-status.sh, thanks! I've updated the logic here to use that script vs looking at the git remote information.

then we can compute the N-1 or N-2 release minor version (maybe we should make the constant minor back-compat offset a variable with a default of 1 or 2?)
Currently I've updated this PR to compute only the N-1 release minor version as v1.31-v1.32 are the only versions support for Compatibility Versions currently

Currently the PR is focused on fixing #33573. I've created #33849 to track the issue with the kubelet version being incorrect. I will address the below points in a separate PR to fix #33849 if that is ok. Can also do them both in the PR, currently my plan is to do them in 2 PRs though.

we can then do kind build node image v1.N-1.(latest) (TODO: we can get latest from git tag | grep 'v1.N-1.' ?)

create kind cluster from this image, ensure --compat-version is set to N-1

then we can build and patch in api-server from the current branch

run tests from N-1 ? (we can also download these the same as kind downloading the component release binaries)

… (from hardcoded release-1.31)

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 25, 2024

k8s-ci-robot requested review from chases2 and rjsadow November 25, 2024 06:28

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/config Issues or PRs related to code in /config area/jobs sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Nov 25, 2024

k8s-ci-robot requested a review from BenTheElder November 25, 2024 19:54

BenTheElder reviewed Nov 25, 2024

View reviewed changes

aaron-prindle force-pushed the fix-33573 branch from 40b1238 to 7ff6885 Compare November 26, 2024 00:29

aaron-prindle force-pushed the fix-33573 branch from 7ff6885 to b6498a0 Compare November 26, 2024 00:46

fix: make compatibility versions test use dynamic n-1 branch for test…

a396d12

… (from hardcoded release-1.31)

aaron-prindle force-pushed the fix-33573 branch from b6498a0 to a396d12 Compare November 27, 2024 19:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: make compatibility versions test use dynamic n-1 branch for test (from hardcoded release-1.31) #33837

fix: make compatibility versions test use dynamic n-1 branch for test (from hardcoded release-1.31) #33837

aaron-prindle commented Nov 25, 2024 •

edited

Loading

k8s-ci-robot commented Nov 25, 2024

aaron-prindle commented Nov 25, 2024

BenTheElder Nov 25, 2024

BenTheElder Nov 25, 2024

aaron-prindle Nov 26, 2024 •

edited

Loading

BenTheElder Nov 27, 2024

BenTheElder Nov 25, 2024

BenTheElder Nov 25, 2024

aaron-prindle Nov 26, 2024 •

edited

Loading

BenTheElder Nov 25, 2024

BenTheElder Nov 25, 2024

aaron-prindle Nov 26, 2024 •

edited

Loading

BenTheElder Nov 27, 2024 •

edited

Loading

aaron-prindle Nov 27, 2024

BenTheElder commented Nov 25, 2024 •

edited

Loading

aaron-prindle commented Nov 26, 2024 •

edited

Loading

fix: make compatibility versions test use dynamic n-1 branch for test (from hardcoded release-1.31) #33837

Are you sure you want to change the base?

fix: make compatibility versions test use dynamic n-1 branch for test (from hardcoded release-1.31) #33837

Conversation

aaron-prindle commented Nov 25, 2024 • edited Loading

k8s-ci-robot commented Nov 25, 2024

aaron-prindle commented Nov 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaron-prindle Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaron-prindle Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaron-prindle Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

BenTheElder Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenTheElder commented Nov 25, 2024 • edited Loading

aaron-prindle commented Nov 26, 2024 • edited Loading

aaron-prindle commented Nov 25, 2024 •

edited

Loading

aaron-prindle Nov 26, 2024 •

edited

Loading

aaron-prindle Nov 26, 2024 •

edited

Loading

aaron-prindle Nov 26, 2024 •

edited

Loading

BenTheElder Nov 27, 2024 •

edited

Loading

BenTheElder commented Nov 25, 2024 •

edited

Loading

aaron-prindle commented Nov 26, 2024 •

edited

Loading