Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubeflow sync release pipeline #469

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

atheo89
Copy link
Member

@atheo89 atheo89 commented Nov 25, 2024

Related to: https://issues.redhat.com/browse/RHOAIENG-15453

Description

This PR incorporates the idea to automate completely the kubeflow release procedure

Key Points:

  • Branch sync: the sync-brances.yaml updated to be compatible as callable action from another. Additionally, ensures that changes in the main branch are reflected in the release branch.
  • Image Availability Check: Polls for the availability of notebooks' images on Quay.io before proceeding, set a a timeout for 15mins
  • PR Creation and Merging: Automates updating the notebook controller image tags and waits for the related PR to be merged in order to proceed to the final release, set a timeout as well for 1h.
  • Release Creation: Finally, triggers the release process once all previous steps are complete. For now the create-relase.yaml is just a mockup in order to go on with the dev, This workload will be developed through RHOAIENG-15391
  • Main Tags: Changed the image tags on main branch to point to the main tag since it is the latest. This way we don't need to update everytime the main branch with latest githash. (Not part of the workflow)

Investigation Item:
Investigate how the PR generated from the update-release-images step can be auto-merged in case all the tests pass, without user innervation. Currently, needs user approval in order to go the the last step.

How Has This Been Tested?

Tested locally:
Run: https://github.com/atheo89/kubeflow/actions/runs/12007990443

Video Demo:
With set create-new-release true & false
https://drive.google.com/file/d/1uXC5xuTvoR9FWvfMYs2H13zIHa_Usrps/view?usp=sharing

image

Merge criteria:

  • The commits are squashed in a cohesive manner and have meaningful messages.
  • Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
  • The developer has manually tested the changes and verified that the changes work

Copy link

openshift-ci bot commented Nov 25, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link

openshift-ci bot commented Nov 25, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from atheo89. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@atheo89 atheo89 marked this pull request as ready for review November 25, 2024 11:51
@atheo89 atheo89 removed the request for review from paulovmr November 25, 2024 11:58
@@ -0,0 +1,24 @@
---
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @andyatmiami! This is only a mockup gha, it prints a Eyo, World for now 🤣 It used to work on the logic of the kubeflow release.

Copy link

@andyatmiami andyatmiami Nov 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, fwiw... and I'm happy to change what I did with the create release PoC I did.. but I was condititioning the release creation logic on the push of a tag matching a specific pattern...

So it wouldn't need to be explicitly invoked here.. but if that "trigger off tag" won't work for reasons I am not appreciating - I am happy to adapt!

@jiridanek
Copy link
Member

Looks good. I would not mind merging this even with the placeholder Hello World create release part. Submitting the GitHub release is just a cherry on top.

@atheo89
Copy link
Member Author

atheo89 commented Nov 25, 2024

/test odh-notebook-controller-e2e

@jiridanek
Copy link
Member

/lgtm

@jiridanek
Copy link
Member

/override ci/prow/odh-notebook-controller-e2e

Copy link

openshift-ci bot commented Nov 25, 2024

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/odh-notebook-controller-e2e

In response to this:

/override ci/prow/odh-notebook-controller-e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@jiridanek
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added lgtm and removed lgtm labels Nov 25, 2024
@atheo89
Copy link
Member Author

atheo89 commented Nov 27, 2024

/override ci/prow/odh-notebook-controller-e2e

Copy link

openshift-ci bot commented Nov 27, 2024

@atheo89: Overrode contexts on behalf of atheo89: ci/prow/odh-notebook-controller-e2e

In response to this:

/override ci/prow/odh-notebook-controller-e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@jiridanek
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Nov 27, 2024
Copy link
Member

@jstourac jstourac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this, Adriana! Nice work, the changes seem good. I put couple of comments - some are relevant, some probably not. Otherwise LGTM!

.github/workflows/kubeflow-release.yaml Show resolved Hide resolved
# PR to look for
PR_TITLE="[GHA-${{ github.run_id }}]"
# Fetch matching PRs
gh pr list --repo atheo89/kubeflow --state all --search "$PR_TITLE" --json number,title > pr_list.json
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
gh pr list --repo atheo89/kubeflow --state all --search "$PR_TITLE" --json number,title > pr_list.json
gh pr list --repo "$REPO_OWNER/kubeflow" --state all --search "$PR_TITLE" --json number,title > pr_list.json

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can also use the --jq option on the gh CLI call to avoid needing to dump this to file. And while presently we specify in --json to return the title - I don't see us actually using that anywhere..

for (( i=1; i<=MAX_ATTEMPTS; i++ )); do
echo "Checking if PR #$PR_NUMBER is merged (Attempt $i/$MAX_ATTEMPTS)..."
PR_STATE=$(gh pr view --repo atheo89/kubeflow $PR_NUMBER --json mergedAt --jq '.mergedAt')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
PR_STATE=$(gh pr view --repo atheo89/kubeflow $PR_NUMBER --json mergedAt --jq '.mergedAt')
PR_STATE=$(gh pr view --repo "$REPO_OWNER/kubeflow" $PR_NUMBER --json mergedAt --jq '.mergedAt')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If --jq is specified.. I don't think the --json argument does anything... (but I could be wrong...)

steps:
- name: Poll for images availability
id: check-images
run: |
Copy link

@andyatmiami andyatmiami Nov 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, personal preference here (so fine to ignore) - but if we are going to inline scripts that aren't a couple lines... pulling that out into a proper external script seems easier to work with (i.e. don't need to worry about the lurking concern of YAML indentation, etc)

However, doing so requires the actions/checkout@v4 workflow.. and I'm not clear on the potential performance concerns (if any) of introducing that.

@@ -68,12 +97,12 @@ jobs:
- name: Update related files
id: apply-changes
run: |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic of this complexity could be warranted to pull out into a separate script for ease of maintenance (just a suggestion!)

git config user.email "github-actions[bot]@users.noreply.github.com"

- name: Merge source branch into target
run: |

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull this logic out into a separate script to be invoked (?)

@andyatmiami
Copy link

Nice job.. even a "relative newcomer" like me could follow the logical flow through the work 💯

I added various comments through the PR - but nothing that I would consider a blocking issue that:

  • impacts functionaliy
  • requires being addressed

so...

/lgtm

@andyatmiami
Copy link

ℹ️ On one last quick scan through the code.. it occurred to me that in a couple places we have hard-coded the 1.9 release value... This would then obviously require code changes once we move to v2.x (or any other non 1.9 release version)

Given we seem to be reading input for some of these jobs already... might be good to think about how we can parameterize this value to keep the script logic version-agnostic.

However, that can certainly just be handled as technical debt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants