What To Do With Merged Branches #55
Comments
|
What remote are you using? The tomono script creates an entirely new repository, but it sounds like you're reusing an existing remote. I'm not sure what to even expect from that setup, so I don't know what to say other than "create a new repository on github and push to that, instead". Git is saying there's already a |
|
I probably have made the process more complicated by accident then. I originally created a totally new repository, added a new Whenever I would try again I would delete all existing branches besides Ill try making a new repository again and see what happens. Current steps:
|
|
I tried eliminating a few potential issues, I deleted the old repository, and created a totally new one. I also switched to running it the way recommended in the README instead of passing lines through stdin. repos.txt run.sh I have tried running the script directly / line by line in my terminal and I see 634 behind, 4 ahead still. I made sure to run the script from a folder that is not a git repository so it won't conflict in any way. I also made sure that the branch it is trying to merge is only 1 commit ahead on the auth remote. To make it easier to debug the steps I followed were:
|
|
Sorry I just now realize we may be talking about a different UI: what exactly is telling you the "behind" part? I thought it was github but maybe it's a local tool? |
|
Sorry for the delay in replying, it is actually coming from Github. Is there a way to help debug these issues further? 
|
|
Ok but that makes sense, right? At this point I'm not sure what the bug is. What did you expect to see? master will contain commits that aren't present in the feature branches, namely those of the new repos, so it makes sense github will show that the branch is behind compared to master. |
|
That make sense, I didn't see anything in the readme about what to expect the merged branches to look like. The merged branches are intended to be for reference not to be mergable after the process is run? This seems to be a similar but different issue than #21 so I wanted to clarify. Is there a recommended way to handle these branches in that case? Rebasing off of master to remove the extra commits or anything like that? |
C-Howard
changed the title
|
Have a look at this blog post see if that helps clarify things: https://syslog.ravelin.com/multi-to-mono-repository-c81d004df3ce lmk if it's still unclear. Also see the git log graph in the readme, and the opening paragraph (emphasis added):
|
|
@C-Howard if the blog helped explain something that was previously unclear, I'd love to pick your brain if you got a moment and ask what helped clarify things. You're probably not the first to be confused, just the first to actually go through the trouble of opening a ticket about it, so if I can clarify something in the README with your help that'd be great. |
|
I apologize for the delay in responding, the blog is helpful for explaining all the theory and the process used which is excellent for understanding. It seems like it might be wise to mention the blog post in the readme as well. From a tool user standpoint it does not cover initial or ongoing maintenance however, while using the tool I am already past the stage the blog post covers. My concerns at this point would be how do I get my changes out of the newly merged branches in a way that is not painful. I guess a lot of what Im asking for is a rough idea of the "Dos and Do Nots" of this approach. What types of branches you want to clean up on a repository before trying to merge it in and things like that. The most ideal setup for us would be to take all of our current package repositories and merge them together so we can move the builds to the repository for each package and archive the old ones. Right now both rebasing and merges run into rename conflicts:
Specifically this branch was to rename files in the repository to remove a .Net47 suffix from file names. If I attempt to merge in newer changes following the readme https://github.com/hraban/tomono?tab=readme-ov-file#pull-in-new-changes-from-a-remote it also fails. The auth origin has already had this branch merged in so I am not able to pull in those changes as is.
|
|
I'd love to reproduce this somehow because I've heard others report something similar but I can never manage to actually reproduce it. Here's a test script you can run that merges SBCL and Git from 1 year ago, then merges in 1 entire year's worth of changes from both repos into the monorepo using that same subtree merge: https://github.com/hraban/tomono/blob/test-merge/test-merge.sh That works fine locally, and I can't think of any other way to try and break it. Do you have a reproducible case somewhere I can try? |
|
I can confirm that test script works as is and works even when adding support for using the third parameter. I wanted to make sure you could put both git and sbcl into the lib folder to see if that was related. i can not reproduce the error so far so I am going to try to create a set of test repos that can replicate the issue. #!/usr/bin/env bash
set -euo pipefail
set -x
# The tomono script is tangled right next to the test script
export PATH="$PWD:$PATH"
d="$(mktemp -d)"
cd "$d"
# Two reasonably complex repos with lots of changes but not too many. And both
# with a ‘master’ branch.
>remotes.txt cat <<EOF
https://github.com/git/git.git git lib/git
https://git.code.sf.net/p/sbcl/sbcl sbcl lib/sbcl
EOF
cat remotes.txt | while read url origin dir ; do
# Manual folder is not set
if [[ $dir == "" ]]; then
dir="$origin"
fi
git clone "$url" "$dir"
(
cd "$dir"
git log --before 1.year.ago -1 --format=%h | xargs git reset --hard
)
# Build set of entries for local repositories
if [[ $dir == "" ]]; then
echo "$PWD/$dir $dir" >> locals.txt
else
echo "$PWD/$dir $origin $dir" >> locals.txt
fi
done
cat locals.txt
<locals.txt tomono
# Now we have a repo core with SBCL and Git, both where they were 1y ago. Update
# those and merge in the changes.
cat remotes.txt | while read _ origin dir ; do
(
# Manual folder is not set
if [[ $dir == "" ]]; then
dir="$origin"
fi
cd "$dir"
git reset --hard HEAD@{1}
)
done
cd core
git fetch --all
git checkout master
cat ../remotes.txt | while read _ origin dir; do
# Manual folder is not set
if [[ $dir == "" ]]; then
dir="$origin"
fi
git merge --no-edit -X subtree="$dir/" "$origin/master"
done
echo "Successfully merged in 1 year worth of changes from both remotes into $PWD" |
|
I have run through several tests with a few different results so far:
So my overall observations are that when you run the script the resulting branch works correctly, if you fetch all remotes after the merge you should be able to cleanly merge for each origin, but you will need to manually merge / rebase all branches that come from the merge. This is not an issue with the test scripts as it works just against the master branches and those are not having issues. The fetch is required for any future subtree merges which I can verify by running the test script using my list of repositories. Because the branches are only merged on the first run of the tool and you subtree merge anything new it explains why nothing else appears to conflict. It would only be an issue if you planned to try to merge branches later on. It may may sense to not pull over any branches at all in some cases, merge it into the upstream, and then subtree merge the changes into the monorepo. So for copied branches the safest step to take is to merge the default branch (ex: I did also notice by accident a small detail about how newlines are being handled. If you for example generate your repo list the way the test script does it creates a newline: >remotes.txt cat <<EOF
https://github.com/git/git.git git
https://git.code.sf.net/p/sbcl/sbcl sbcl
EOFThis file reads all entries: This file reads just git: If you create the .txt by hand without a newline it will not parse the last line of the file when parsing the file with cat. So it may make sense to note the file must end a newline or the last repo will not be included. This appears to be a Unix standard around |
|
Interesting points, thank you for getting this deep into the weeds. Yes I will say: once you start making independent changes in the monorepo versus the source repos, later merging can get tricky. This is definitely something you want to avoid. You can keep both in sync easily, but if you start making independent changes then at some point you'll get into regular old git territory and will have to deal with merge conflicts, like any other situation where you have to diverging sources. But is that what you were doing when you encountered those conflicts? What were you encountering actual problems with? I thought you were just pulling in new changes into an otherwise unchanged monorepo--had you been making changes to the monorepo before fetching those updated sources? Regarding the newlines: yes that's just Unix convention I'm afraid. This is how tools do it across the board. Some are more gracious but that's best effort. There's no expectation of handling lines without terminating newline, in general. Newlines aren't separators but terminators. |
|
Roughly the TLDR is that there are no actual errors, the script works really well but you need to make sure git remotes are updated and branches get updated to have things work as expected. I think the confusion can come down to several things depending on how you merge things. The test script you provided actually has slightly different behavior than running the tool directly because it clones the repositories, resets to a year old commit, merges, then subtree merges the new changes. Technically that approach ignores other branches you would see this issues with as it only uses branches locally checked out. In theory if you pulled down a branch in each repository it should show the branch issue but not the subtree issue. I am using the generated repository from tomono as is without making any changes in the repository itself. To successfully merge multiple repositories the process is the following:
What I have seen is if you miss step 3 the subtree merge finds no changes. I have a version of the test script that reverts two commits and then subtree merges the changes back in and it always says things are up to date without the fetch. I can't say exactly what state my repository was in when I saw the error but it was missing data needed to merge successfully so it showed the rename conflicts. So if you run your test script with https://github.com/hraban/tomono/blob/test-merge/test-merge.sh#L42 commented out you will run into a situation where while it didn't error, it also didn't actually merge any changes. This is more of a gotcha than the scripts fault, it's just part of the process you need to know about. You will be missing a years worth of changes just by missing that single line. And the 5th step is required because Github will always show merge conflicts in branches that Git can correct locally. You run the script, make sure to fetch and update all your repository state it should be possible to merge, and then fix merge conflicts in branches. I don't know if it makes sense to try to rebase as part of the tomono script but that would produce cleaner results when it is successful, though that means you need to know the default branch that will be used. It might be better to include as "cleanup instructions". Ill get a pr written shortly to get a section added to the README about that, and Im happy to tweak it to match your style. |
|
Curious to see your take on it, looking forward to the PR. |
|
Sorry for the delay, I have pushed up my first draft for what information to include for the cleanup section. I believe I got it roughly in the right spot in the documentation but Im open to any suggestions or questions. |
Specs
OS:
M3 Mac running Sonoma 14.5Terminal:
iTerm2Git:
2.45.2Tomono Commit: 8052bd8
Issue
I am working on getting a series of older packages all consolidated to the new folder. That has worked correctly, however when I
git push --allto the repository all the branches besidesmastershow something like 800 commits behind, 4 commits ahead. For example there is aproject-cleanupbranch that shows 795 commits behind and 4 ahead with those commits being the following:The first commit is the from the auth-sdk repository used in the script while the other 3 come from tomono. Is this behavior expected? I have verified that my default branch setting in git matches the branch used across all the repositories
masterand themasterbranch shows all expected folders.My tomono script is the following:
And I have tested a version with no subfolders:
With the same results.
The text was updated successfully, but these errors were encountered: