pack-objects: Create an alternative name hash algorithm (recreated) #1823
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
derrickstolee
force-pushed
the
full-name
branch
2 times, most recently
from
d7b67b7
to
ab341dd
Compare
|
/submit |
|
Submitted as [email protected] To fetch this version into To fetch this version to local tag |
|
This patch series was integrated into seen via git@a4c5ed7. |
|
This patch series was integrated into seen via git@4f1ca50. |
|
This patch series was integrated into seen via git@38d672e. |
|
This patch series was integrated into seen via git@e81b3ef. |
|
This patch series was integrated into seen via git@27dd7da. |
|
This patch series was integrated into seen via git@3feb04a. |
|
This patch series was integrated into seen via git@f70d861. |
|
This patch series was integrated into seen via git@960de39. |
|
This patch series was integrated into seen via git@db680f6. |
|
This patch series was integrated into seen via git@d9378c7. |
|
This patch series was integrated into seen via git@27b487d. |
|
This patch series was integrated into seen via git@becc917. |
|
This patch series was integrated into seen via git@6ac0572. |
|
This patch series was integrated into seen via git@f6a46ca. |
|
This patch series was integrated into seen via git@e8ae937. |
![gitgitgadget[bot]](https://avatars.githubusercontent.com/in/12836?s=60&v=4){kind=small}
| @@ -15,7 +15,8 @@ SYNOPSIS | |||
| [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[email protected]>
>
> The pack_name_hash() method has not been materially changed since it was
> introduced in ce0bd64299a (pack-objects: improve path grouping
> heuristics., 2006-06-05). The intention here is to group objects by path
> name, but also attempt to group similar file types together by making
> the most-significant digits of the hash be focused on the final
> characters.
>
> Here's the crux of the implementation:
>
> /*
> * This effectively just creates a sortable number from the
> * last sixteen non-whitespace characters. Last characters
> * count "most", so things that end in ".c" sort together.
> */
> while ((c = *name++) != 0) {
> if (isspace(c))
> continue;
> hash = (hash >> 2) + (c << 24);
> }
Hah. I like that the existing implementation is small enough to fit (in
its entirety!) into the commit message!
> As the comment mentions, this only cares about the last sixteen
> non-whitespace characters. This cause some filenames to collide more
> than others. Here are some examples that I've seen while investigating
> repositories that are growing more than they should be:
>
> * "/CHANGELOG.json" is 15 characters, and is created by the beachball
> [1] tool. Only the final character of the parent directory can
> differntiate different versions of this file, but also only the two
s/differntiate/differentiate ;-).
> most-significant digits. If that character is a letter, then this is
> always a collision. Similar issues occur with the similar
> "/CHANGELOG.md" path, though there is more opportunity for
> differences in the parent directory.
>
> * Localization files frequently have common filenames but differentiate
> via parent directories. In C#, the name "/strings.resx.lcl" is used
> for these localization files and they will all collide in name-hash.
>
> [1] https://github.com/microsoft/beachball
>
> I've come across many other examples where some internal tool uses a
> common name across multiple directories and is causing Git to repack
> poorly due to name-hash collisions.
>
> It is clear that the existing name-hash algorithm is optimized for
> repositories with short path names, but also is optimized for packing a
> single snapshot of a repository, not a repository with many versions of
> the same file. In my testing, this has proven out where the name-hash
> algorithm does a good job of finding peer files as delta bases when
> unable to use a historical version of that exact file.
I'm not sure I entirely agree with the suggestion that the existing hash
function is only about packing repositories with short pathnames. I
think an important part of the existing implementation is that tries to
group similar files together, regardless of whether or not they appear
in the same tree.
As you have shown, this can be a problem when the fact two files that
happen to end in "CHANGELOG.json" end up in vastly different trees and
*aren't* related. I don't think that nailing all of these details in the
commit message is necessary, but I do think it's worth adjusting what
the original commit message says in terms of what the existing algorithm
is optimized for.
> However, for repositories that have many versions of most files and
> directories, it is more important that the objects that appear at the
> same path are grouped together.
>
> Create a new pack_full_name_hash() method and a new --full-name-hash
> option for 'git pack-objects' to call that method instead. Add a simple
> pass-through for 'git repack --full-name-hash' for additional testing in
> the context of a full repack, where I expect this will be most
> effective.
>
> The hash algorithm is as simple as possible to be reasonably effective:
> for each character of the path string, add a multiple of that character
> and a large prime number (chosen arbitrarily, but intended to be large
> relative to the size of a uint32_t). Then, shift the current hash value
> to the right by 5, with overlap. The addition and shift parameters are
> standard mechanisms for creating hard-to-predict behaviors in the bits
> of the resulting hash.
>
> This is not meant to be cryptographic at all, but uniformly distributed
> across the possible hash values. This creates a hash that appears
> pseudorandom. There is no ability to consider similar file types as
> being close to each other.
I think you hint at this in the series' cover letter, but I suspect that
this pseduorandom behavior hurts in some small number of cases and that
the full-name hash idea isn't a pure win, e.g., when we really do want
to delta two paths that both end in CHAGNELOG.json despite being in
different parts of the tree.
You have some tables here below that demonstrate a significant
improvement with the full-name hash in use, which I think is good worth
keeping in my own opinion. It may be worth updating those to include the
new examples you highlighted in your revised cover letter as well.
> In a later change, a test-tool will be added so the effectiveness of
> this hash can be demonstrated directly.
>
> For now, let's consider how effective this mechanism is when repacking a
> repository with and without the --full-name-hash option. Specifically,
Is this repository publicly available? If so, it may be worth mentioning
here.
> let's use 'git repack -adf [--full-name-hash]' as our test.
>
> On the Git repository, we do not expect much difference. All path names
> are short. This is backed by our results:
>
> | Stage | Pack Size | Repack Time |
> |-----------------------|-----------|-------------|
> | After clone | 260 MB | N/A |
> | Standard Repack | 127MB | 106s |
> | With --full-name-hash | 126 MB | 99s |
Ahh. Here's a great example of it helping to a smaller extent. Thanks
for including this as part of demonstrating the full picture (both the
benefits and drawbacks).
> This example demonstrates how there is some natural overhead coming from
> the cloned copy because the server is hosting many forks and has not
> optimized for exactly this set of reachable objects. But the full repack
> has similar characteristics with and without --full-name-hash.
Good.
> However, we can test this in a repository that uses one of the
> problematic naming conventions above. The fluentui [2] repo uses
> beachball to generate CHANGELOG.json and CHANGELOG.md files, and these
> files have very poor delta characteristics when comparing against
> versions across parent directories.
>
> | Stage | Pack Size | Repack Time |
> |-----------------------|-----------|-------------|
> | After clone | 694 MB | N/A |
> | Standard Repack | 438 MB | 728s |
> | With --full-name-hash | 168 MB | 142s |
>
> [2] https://github.com/microsoft/fluentui
>
> In this example, we see significant gains in the compressed packfile
> size as well as the time taken to compute the packfile.
Amazing!
> Using a collection of repositories that use the beachball tool, I was
> able to make similar comparisions with dramatic results. While the
> fluentui repo is public, the others are private so cannot be shared for
> reproduction. The results are so significant that I find it important to
> share here:
>
> | Repo | Standard Repack | With --full-name-hash |
> |----------|-----------------|-----------------------|
> | fluentui | 438 MB | 168 MB |
> | Repo B | 6,255 MB | 829 MB |
> | Repo C | 37,737 MB | 7,125 MB |
> | Repo D | 130,049 MB | 6,190 MB |
>
> Future changes could include making --full-name-hash implied by a config
> value or even implied by default during a full repack.
>
> It is important to point out that the name hash value is stored in the
> .bitmap file format, so we must disable the --full-name-hash option when
> bitmaps are being read or written. Later, the bitmap format could be
> updated to be aware of the name hash version so deltas can be quickly
> computed across the bitmapped/not-bitmapped boundary.
Agreed.
> Signed-off-by: Derrick Stolee <[email protected]>
> ---
> Documentation/git-pack-objects.txt | 3 ++-
> builtin/pack-objects.c | 25 ++++++++++++++++++++-----
> builtin/repack.c | 5 +++++
> pack-objects.h | 21 +++++++++++++++++++++
> t/t5300-pack-object.sh | 15 +++++++++++++++
> 5 files changed, 63 insertions(+), 6 deletions(-)
>
> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt
> index e32404c6aae..93861d9f85b 100644
> --- a/Documentation/git-pack-objects.txt
> +++ b/Documentation/git-pack-objects.txt
> @@ -15,7 +15,8 @@ SYNOPSIS
> [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
> [--cruft] [--cruft-expiration=<time>]
> [--stdout [--filter=<filter-spec>] | <base-name>]
> - [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>
> + [--shallow] [--keep-true-parents] [--[no-]sparse]
> + [--full-name-hash] < <object-list>
OK, I see that --full-name-hash is now listed in the synopsis, but I
don't see a corresponding description of what the option does later on
in this file. I took a look through the remaining patches in this series
and couldn't find any further changes to git-pack-objects(1) either.
> DESCRIPTION
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 08007142671..85595dfcd88 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -266,6 +266,14 @@ struct configured_exclusion {
> static struct oidmap configured_exclusions;
>
> static struct oidset excluded_by_config;
> +static int use_full_name_hash;
> +
> +static inline uint32_t pack_name_hash_fn(const char *name)
> +{
> + if (use_full_name_hash)
> + return pack_full_name_hash(name);
> + return pack_name_hash(name);
> +}
>
> /*
> * stats
> @@ -1698,7 +1706,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,
> return 0;
> }
>
> - create_object_entry(oid, type, pack_name_hash(name),
> + create_object_entry(oid, type, pack_name_hash_fn(name),
> exclude, name && no_try_delta(name),
> found_pack, found_offset);
> return 1;
> @@ -1912,7 +1920,7 @@ static void add_preferred_base_object(const char *name)
> {
> struct pbase_tree *it;
> size_t cmplen;
> - unsigned hash = pack_name_hash(name);
> + unsigned hash = pack_name_hash_fn(name);
>
> if (!num_preferred_base || check_pbase_path(hash))
> return;
> @@ -3422,7 +3430,7 @@ static void show_object_pack_hint(struct object *object, const char *name,
> * here using a now in order to perhaps improve the delta selection
> * process.
> */
> - oe->hash = pack_name_hash(name);
> + oe->hash = pack_name_hash_fn(name);
> oe->no_try_delta = name && no_try_delta(name);
>
> stdin_packs_hints_nr++;
> @@ -3572,7 +3580,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type
> entry = packlist_find(&to_pack, oid);
> if (entry) {
> if (name) {
> - entry->hash = pack_name_hash(name);
> + entry->hash = pack_name_hash_fn(name);
> entry->no_try_delta = no_try_delta(name);
> }
> } else {
> @@ -3595,7 +3603,7 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type
> return;
> }
>
> - entry = create_object_entry(oid, type, pack_name_hash(name),
> + entry = create_object_entry(oid, type, pack_name_hash_fn(name),
> 0, name && no_try_delta(name),
> pack, offset);
> }
> @@ -4429,6 +4437,8 @@ int cmd_pack_objects(int argc,
> OPT_STRING_LIST(0, "uri-protocol", &uri_protocols,
> N_("protocol"),
> N_("exclude any configured uploadpack.blobpackfileuri with this protocol")),
> + OPT_BOOL(0, "full-name-hash", &use_full_name_hash,
> + N_("optimize delta compression across identical path names over time")),
> OPT_END(),
> };
>
> @@ -4576,6 +4586,11 @@ int cmd_pack_objects(int argc,
> if (pack_to_stdout || !rev_list_all)
> write_bitmap_index = 0;
>
> + if (write_bitmap_index && use_full_name_hash) {
> + warning(_("currently, the --full-name-hash option is incompatible with --write-bitmap-index"));
> + use_full_name_hash = 0;
> + }
> +
Good, we determine this early on in the command, so we don't risk
computing different hash functions within the same process.
I wonder if it's worth guarding against mixing the hash functions within
the pack_name_hash() and pack_full_name_hash() functions themselves. I'm
thinking something like:
static inline uint32_t pack_name_hash(const char *name)
{
if (use_full_name_hash)
BUG("called pack_name_hash() with --full-name-hash")
/* ... */
}
and the inverse in pack_full_name_hash(). I don't think it's strictly
necessary, but it would be a nice guard against someone calling, e.g.,
pack_full_name_hash() directly instead of pack_name_hash_fn().
The other small thought I had here is that we should use the convenience
function die_for_incompatible_opt3() here, since it uses an existing
translation string for pairs of incompatible options.
(As an aside, though that function is actually implemented in the
_opt4() variant, and it knows how to handle a pair, trio, and quartet of
mutually incompatible options, there is no die_for_incompatible_opt2()
function. It may be worth adding one here since I'm sure there are other
spots which would benefit from such a function).
> diff --git a/builtin/repack.c b/builtin/repack.c
> index d6bb37e84ae..ab2a2e46b20 100644
> --- a/builtin/repack.c
> +++ b/builtin/repack.c
I'm surprised to see the new option plumbed into repack in this commit.
I would have thought that it'd appear in the subsequent commit instead.
The implementation below looks good, I just imagined it would be placed
in the next commit instead of this one.
The remaining parts of this change look good to me.
Thanks,
TaylorThere was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote:
> The remaining parts of this change look good to me.
Oops, one thing I forgot (which reading Peff's message in [1] reminded
me of) is that I think we need to disable full-name hashing when we're
reusing existing packfiles as is the case with try_partial_reuse().
There we're always looking at classic name hash values, so mixing the
two would be a mistake. I think that amounts to:
--- 8< ---
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 762949e4c8..7e370bcfc9 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)
if (!(bitmap_git = prepare_bitmap_walk(revs, 0)))
return -1;
+ use_full_name_hash = 0;
+
if (pack_options_allow_reuse())
reuse_partial_packfile_from_bitmap(bitmap_git,
&reuse_packfiles,
--- >8 ---
Thanks,
Taylor
[1]: https://lore.kernel.org/git/[email protected]/There was a problem hiding this comment.
On the Git mailing list, Junio C Hamano wrote (reply to this):
Taylor Blau <[email protected]> writes:
> On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote:
>> The remaining parts of this change look good to me.
>
> Oops, one thing I forgot (which reading Peff's message in [1] reminded
> me of) is that I think we need to disable full-name hashing when we're
> reusing existing packfiles as is the case with try_partial_reuse().
>
> There we're always looking at classic name hash values, so mixing the
> two would be a mistake. I think that amounts to:
>
> --- 8< ---
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 762949e4c8..7e370bcfc9 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)
> if (!(bitmap_git = prepare_bitmap_walk(revs, 0)))
> return -1;
>
> + use_full_name_hash = 0;
Hmph, is this early enough, or has some other code path already
computed the name hashes for the paths for the files to be packed?
... Goes and looks ...
This is called from get_object_list() which
- is not called under --stdin-packs,
- is not called in cruft mode,
- is not called when reading object list from --stdin
so we are looking at the bog-standard "objects to be packed are
given in the form of rev-list command line options from our command
line". And in the function, we walk the history near the end, which
makes show_object calls that adds object-entry with the name-hash.
So the call to get_object_list_from_bitmap() happens way before the
first use of the name-hash function, so this is probably safe.
And obviously get_object_list_from_bitmap() is the only place we
select objects to be packed from an existing pack and a bitmap file,
so even if we gain new callers in the future, it is very likely that
the new callers would benefit from this change.
OK. Nicely done.
> if (pack_options_allow_reuse())
> reuse_partial_packfile_from_bitmap(bitmap_git,
> &reuse_packfiles,
> --- >8 ---
>
> Thanks,
> Taylor
>
> [1]: https://lore.kernel.org/git/[email protected]/There was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 4:35 PM, Taylor Blau wrote:
> On Thu, Nov 21, 2024 at 03:08:09PM -0500, Taylor Blau wrote:
>> The remaining parts of this change look good to me.
> > Oops, one thing I forgot (which reading Peff's message in [1] reminded
> me of) is that I think we need to disable full-name hashing when we're
> reusing existing packfiles as is the case with try_partial_reuse().
> > There we're always looking at classic name hash values, so mixing the
> two would be a mistake. I think that amounts to:
> > --- 8< ---
> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
> index 762949e4c8..7e370bcfc9 100644
> --- a/builtin/pack-objects.c
> +++ b/builtin/pack-objects.c
> @@ -4070,6 +4070,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs)
> if (!(bitmap_git = prepare_bitmap_walk(revs, 0)))
> return -1;
> > + use_full_name_hash = 0;
> +
Thanks. I have applied this code change with a comment detailing
the context around the bitmap file storing only the default name-hash
(for now) but that it can change in the future.
Thanks,
-Stolee
There was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 3:08 PM, Taylor Blau wrote:
> On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <[email protected]>
>> It is clear that the existing name-hash algorithm is optimized for
>> repositories with short path names, but also is optimized for packing a
>> single snapshot of a repository, not a repository with many versions of
>> the same file. In my testing, this has proven out where the name-hash
>> algorithm does a good job of finding peer files as delta bases when
>> unable to use a historical version of that exact file.
> > I'm not sure I entirely agree with the suggestion that the existing hash
> function is only about packing repositories with short pathnames. I
> think an important part of the existing implementation is that tries to
> group similar files together, regardless of whether or not they appear
> in the same tree.
I'll be more explicit about the design for "hash locality" earlier in
the message, but also pointing out that the locality only makes sense as
a benefit when there are not enough versions of a file in history, since
it's nearly always better to choose a previous version of the same file
instead of a different path with a name-hash collision. Directory renames
are on place where this is a positive decision, but those are typically
rare compared to the full history of a large repo.
>> This is not meant to be cryptographic at all, but uniformly distributed
>> across the possible hash values. This creates a hash that appears
>> pseudorandom. There is no ability to consider similar file types as
>> being close to each other.
> > I think you hint at this in the series' cover letter, but I suspect that
> this pseduorandom behavior hurts in some small number of cases and that
> the full-name hash idea isn't a pure win, e.g., when we really do want
> to delta two paths that both end in CHAGNELOG.json despite being in
> different parts of the tree.
I mention that this doesn't work well in all cases when operating under
a 'git push' or in a shallow clone. Shallow clones are disabled in a later
commit and we don't have the necessary implementation to make this hash
function be selected within 'git push'.
> You have some tables here below that demonstrate a significant
> improvement with the full-name hash in use, which I think is good worth
> keeping in my own opinion. It may be worth updating those to include the
> new examples you highlighted in your revised cover letter as well.
I'll try to remember to move the newer examples to the cover letter.
>> In a later change, a test-tool will be added so the effectiveness of
>> this hash can be demonstrated directly.
>>
>> For now, let's consider how effective this mechanism is when repacking a
>> repository with and without the --full-name-hash option. Specifically,
> > Is this repository publicly available? If so, it may be worth mentioning
> here.
Here, by "when repacking a repository" I mean "we are going to test
repacking a number of example repositories, that will be listed in detail
in the coming tables".
>> Using a collection of repositories that use the beachball tool, I was
>> able to make similar comparisions with dramatic results. While the
>> fluentui repo is public, the others are private so cannot be shared for
>> reproduction. The results are so significant that I find it important to
>> share here:
>>
>> | Repo | Standard Repack | With --full-name-hash |
>> |----------|-----------------|-----------------------|
>> | fluentui | 438 MB | 168 MB |
>> | Repo B | 6,255 MB | 829 MB |
>> | Repo C | 37,737 MB | 7,125 MB |
>> | Repo D | 130,049 MB | 6,190 MB |
These repos B, C, and D are _not_ publicly available, though.
>> diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt
>> index e32404c6aae..93861d9f85b 100644
>> --- a/Documentation/git-pack-objects.txt
>> +++ b/Documentation/git-pack-objects.txt
>> @@ -15,7 +15,8 @@ SYNOPSIS
>> [--revs [--unpacked | --all]] [--keep-pack=<pack-name>]
>> [--cruft] [--cruft-expiration=<time>]
>> [--stdout [--filter=<filter-spec>] | <base-name>]
>> - [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list>
>> + [--shallow] [--keep-true-parents] [--[no-]sparse]
>> + [--full-name-hash] < <object-list>
> > OK, I see that --full-name-hash is now listed in the synopsis, but I
> don't see a corresponding description of what the option does later on
> in this file. I took a look through the remaining patches in this series
> and couldn't find any further changes to git-pack-objects(1) either.
I'll fix that. Thanks. As well as moving the 'git repack' changes out
of this patch. I'll adjust the commit message to say "packing all objects'
instead of 'git repack' to be clear that this can be done with a direct
call to 'git pack-objects' instead of needing 'git repack'.
>> + if (write_bitmap_index && use_full_name_hash) {
>> + warning(_("currently, the --full-name-hash option is incompatible with --write-bitmap-index"));
>> + use_full_name_hash = 0;
>> + }
>> +
> > Good, we determine this early on in the command, so we don't risk
> computing different hash functions within the same process.
> > I wonder if it's worth guarding against mixing the hash functions within
> the pack_name_hash() and pack_full_name_hash() functions themselves. I'm
> thinking something like:
> > static inline uint32_t pack_name_hash(const char *name)
> {
> if (use_full_name_hash)
> BUG("called pack_name_hash() with --full-name-hash")
> /* ... */
> }
> > and the inverse in pack_full_name_hash(). I don't think it's strictly
> necessary, but it would be a nice guard against someone calling, e.g.,
> pack_full_name_hash() directly instead of pack_name_hash_fn().
I think this is interesting defensive programming for future contributions.
We essentially want the methods to only be called by pack_name_hash_fn()
and don't have method privacy. We could extract it to its own header file
but then would need to modify the prototype to include the signal for
which hash type to use, but that would cause us to lose our ability to
check for a bug like this.
It may be even better to store a static value for the value of
use_full_name_hash when it first executes, so it can exit if it notices
a different value. (This is becoming large enough for its own patch.)
> The other small thought I had here is that we should use the convenience
> function die_for_incompatible_opt3() here, since it uses an existing
> translation string for pairs of incompatible options.
> > (As an aside, though that function is actually implemented in the
> _opt4() variant, and it knows how to handle a pair, trio, and quartet of
> mutually incompatible options, there is no die_for_incompatible_opt2()
> function. It may be worth adding one here since I'm sure there are other
> spots which would benefit from such a function).
Interesting. I've not considered these functions before.
>> diff --git a/builtin/repack.c b/builtin/repack.c
>> index d6bb37e84ae..ab2a2e46b20 100644
>> --- a/builtin/repack.c
>> +++ b/builtin/repack.c
> > I'm surprised to see the new option plumbed into repack in this commit.
> I would have thought that it'd appear in the subsequent commit instead.
> The implementation below looks good, I just imagined it would be placed
> in the next commit instead of this one.
Yes, I should delay that to patch 2.
Thanks,
-Stole
| @@ -777,6 +777,13 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' ' | |||
| test_must_be_empty tmpfiles | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:02AM +0000, Derrick Stolee via GitGitGadget wrote:
> ---
> t/t7700-repack.sh | 7 +++++++
> t/test-lib-functions.sh | 26 ++++++++++++++++++++++++++
> 2 files changed, 33 insertions(+)
OK, I stand by my thinking in the previous patch that this one is where
the changes to builtin/repack.c belong.
> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh
> index c4c3d1a15d9..fc2cc9d37be 100755
> --- a/t/t7700-repack.sh
> +++ b/t/t7700-repack.sh
> @@ -777,6 +777,13 @@ test_expect_success 'repack -ad cleans up old .tmp-* packs' '
> test_must_be_empty tmpfiles
> '
>
> +test_expect_success '--full-name-hash option passes through to pack-objects' '
> + GIT_TRACE2_EVENT="$(pwd)/full-trace.txt" \
> + git repack -a --full-name-hash &&
> + test_subcommand_flex git pack-objects --full-name-hash <full-trace.txt
OK. To be honest, I am not sure I would have written the same test to
test trivially correct behavior, but I am not opposed to having such a
test either.
I do think that test_subcommand_flex may be unnecessary though, since
you could instead write this as:
test_subcommand "git pack-objects.*--full-name-hash" <full-trace.txt
and get the same behavior.
> +'
> +
> +
Nit: extra newline here.
Thanks,
TaylorThere was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 3:12 PM, Taylor Blau wrote:
> On Tue, Nov 05, 2024 at 03:05:02AM +0000, Derrick Stolee via GitGitGadget wrote:
>> ---
>> t/t7700-repack.sh | 7 +++++++
>> t/test-lib-functions.sh | 26 ++++++++++++++++++++++++++
>> 2 files changed, 33 insertions(+)
> > OK, I stand by my thinking in the previous patch that this one is where
> the changes to builtin/repack.c belong.
Yes. I should have done this already.
> I do think that test_subcommand_flex may be unnecessary though, since
> you could instead write this as:
> > test_subcommand "git pack-objects.*--full-name-hash" <full-trace.txt
> > and get the same behavior.
This does require knowing a bit about the internals of test_subcommand
that may be too much of a burden for future contributors.
Thanks,
-Stolee
| @@ -266,6 +266,14 @@ struct configured_exclusion { | |||
| static struct oidmap configured_exclusions; | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote:
> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
> index 2e28d02b20f..75b40f07bbd 100755
> --- a/ci/run-build-and-tests.sh
> +++ b/ci/run-build-and-tests.sh
> @@ -30,6 +30,7 @@ linux-TEST-vars)
> export GIT_TEST_NO_WRITE_REV_INDEX=1
> export GIT_TEST_CHECKOUT_WORKERS=2
> export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1
> + export GIT_TEST_FULL_NAME_HASH=1
> ;;
> linux-clang)
> export GIT_TEST_DEFAULT_HASH=sha1
Hmm. I appreciate what this new GIT_TEST_ variable is trying to do, but
I am somewhat saddened to see this list in linux-TEST-vars growing
rather than shrinking.
I'm most definitely part of the problem here, but I think too often we
add new entries to this list and let them languish without ever removing
them after they have served their intended purpose.
So I think the question is: what do we hope to get out of running the
test suite in a mode where we use the full-name hash all of the time? I
can't imagine any interesting breakage (other than individual tests'
sensitivity to specific delta/base pairs) that would be caught by merely
changing the hash function here.
I dunno. Maybe there is some exotic behavior that this shook out for you
during development which I'm not aware of. If that were the case, I
think that keeping this variable around makes sense, since the appearance
of that exotic behavior proves that the variable is useful at shaking
out bugs.
But assuming not, I think that I would just as soon avoid this test
variable entirely, which I think in this case amounts to dropping this
patch from the series.
Thanks,
TaylorThere was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 3:15 PM, Taylor Blau wrote:
> On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote:
>> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
>> index 2e28d02b20f..75b40f07bbd 100755
>> --- a/ci/run-build-and-tests.sh
>> +++ b/ci/run-build-and-tests.sh
>> @@ -30,6 +30,7 @@ linux-TEST-vars)
>> export GIT_TEST_NO_WRITE_REV_INDEX=1
>> export GIT_TEST_CHECKOUT_WORKERS=2
>> export GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL=1
>> + export GIT_TEST_FULL_NAME_HASH=1
>> ;;
>> linux-clang)
>> export GIT_TEST_DEFAULT_HASH=sha1
> > Hmm. I appreciate what this new GIT_TEST_ variable is trying to do, but
> I am somewhat saddened to see this list in linux-TEST-vars growing
> rather than shrinking.
You make good points that this does not need to be here.
It's enough that someone could manually check the test suite
with this test variable to make sure that enough of the other
options are tested with this feature.
Thanks,
-Stolee
| @@ -9,7 +9,9 @@ git-repack - Pack unpacked objects in a repository | |||
| SYNOPSIS | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:04AM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[email protected]>
>
> This also adds the '--full-name-hash' option introduced in the previous
> change and adds newlines to the synopsis.
I think "the previous change" is not quite accurate here, even if
you move the implementation to pass through '--full-name-hash' via
repack into the second patch.
It would be nice to have the option added in 'repack' in the same commit
as adjusts the documentation instead of splitting them apart.
Thanks,
TaylorThere was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 3:17 PM, Taylor Blau wrote:
> On Tue, Nov 05, 2024 at 03:05:04AM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <[email protected]>
>>
>> This also adds the '--full-name-hash' option introduced in the previous
>> change and adds newlines to the synopsis.
> > I think "the previous change" is not quite accurate here, even if
> you move the implementation to pass through '--full-name-hash' via
> repack into the second patch.
Ah, I should definitely rearrange the commits.
> It would be nice to have the option added in 'repack' in the same commit
> as adjusts the documentation instead of splitting them apart.
Part of the point of the split was that the synopsis in builtin/repack.c
needs more than just the addition of the --full-name-hash option in order
to make it match the Documentation synopsis.
But you're right, the code change is small enough that these things can
be combined.
Thanks,
-Stolee
| @@ -0,0 +1,94 @@ | |||
| #!/bin/sh | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[email protected]>
>
> As custom options are added to 'git pack-objects' and 'git repack' to
> adjust how compression is done, use this new performance test script to
> demonstrate their effectiveness in performance and size.
Nicely done, thank you for adding a perf test to allow readers to easily
verify these changes themselves.
> In the case of the Git repository, these numbers show some of the issues
> with this approach:
>
> [...]
>
> The thin pack that simulates a push is much worse with --full-name-hash
> in this case. The name hash values are doing a lot to assist with delta
> bases, it seems. The big pack and shallow clone cases are slightly worse
> with the --full-name-hash option. Only the full repack gains some
> benefits in size.
Not a problem with your patch, but just thinking aloud: do you think
there is an easy/straightforward way to suggest when to use
--full-name-hash or not?
> ---
> t/perf/p5313-pack-objects.sh | 94 ++++++++++++++++++++++++++++++++++++
> 1 file changed, 94 insertions(+)
> create mode 100755 t/perf/p5313-pack-objects.sh
>
> diff --git a/t/perf/p5313-pack-objects.sh b/t/perf/p5313-pack-objects.sh
> new file mode 100755
> index 00000000000..dfa29695315
> --- /dev/null
> +++ b/t/perf/p5313-pack-objects.sh
> @@ -0,0 +1,94 @@
> +#!/bin/sh
> +
> +test_description='Tests pack performance using bitmaps'
> +. ./perf-lib.sh
> +
> +GIT_TEST_PASSING_SANITIZE_LEAK=0
> +export GIT_TEST_PASSING_SANITIZE_LEAK
> +
> +test_perf_large_repo
> +
> +test_expect_success 'create rev input' '
> + cat >in-thin <<-EOF &&
> + $(git rev-parse HEAD)
> + ^$(git rev-parse HEAD~1)
> + EOF
> +
> + cat >in-big <<-EOF &&
> + $(git rev-parse HEAD)
> + ^$(git rev-parse HEAD~1000)
> + EOF
> +
> + cat >in-shallow <<-EOF
> + $(git rev-parse HEAD)
> + --shallow $(git rev-parse HEAD)
> + EOF
> +'
I was going to comment that these could probably be moved into the
individual perf test that cares about reading each of these inputs. But
having them shared here makes sense since we are naturally comparing
generating two packs with the same input (with and without
--full-name-hash). So the shared setup here makes sense to me.
> +
> +test_perf 'thin pack' '
> + git pack-objects --thin --stdout --revs --sparse <in-thin >out
> +'
> +
> +test_size 'thin pack size' '
> + test_file_size out
> +'
Nice. I always forget about this and end up writing 'wc -c <out'.
> +test_perf 'thin pack with --full-name-hash' '
> + git pack-objects --thin --stdout --revs --sparse --full-name-hash <in-thin >out
> +'
> +
> +test_size 'thin pack size with --full-name-hash' '
> + test_file_size out
> +'
> +
> +test_perf 'big pack' '
> + git pack-objects --stdout --revs --sparse <in-big >out
> +'
> +
> +test_size 'big pack size' '
> + test_file_size out
> +'
> +
> +test_perf 'big pack with --full-name-hash' '
> + git pack-objects --stdout --revs --sparse --full-name-hash <in-big >out
> +'
> +
> +test_size 'big pack size with --full-name-hash' '
> + test_file_size out
> +'
> +
> +test_perf 'shallow fetch pack' '
> + git pack-objects --stdout --revs --sparse --shallow <in-shallow >out
> +'
> +
> +test_size 'shallow pack size' '
> + test_file_size out
> +'
> +
> +test_perf 'shallow pack with --full-name-hash' '
> + git pack-objects --stdout --revs --sparse --shallow --full-name-hash <in-shallow >out
> +'
> +
> +test_size 'shallow pack size with --full-name-hash' '
> + test_file_size out
> +'
> +
> +test_perf 'repack' '
> + git repack -adf
> +'
> +
> +test_size 'repack size' '
> + pack=$(ls .git/objects/pack/pack-*.pack) &&
> + test_file_size "$pack"
Here and below, I think it's fine to inline this as in:
test_file_size "$(ls .git/objects/pack/pack-*.pack)"
...but I wonder: will using ".git" break this test in bare repositories?
Should we write instead:
pack="$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)" &&
test_file_size
?
Thanks,
TaylorThere was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 3:31 PM, Taylor Blau wrote:
> On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <[email protected]>
>> The thin pack that simulates a push is much worse with --full-name-hash
>> in this case. The name hash values are doing a lot to assist with delta
>> bases, it seems. The big pack and shallow clone cases are slightly worse
>> with the --full-name-hash option. Only the full repack gains some
>> benefits in size.
> > Not a problem with your patch, but just thinking aloud: do you think
> there is an easy/straightforward way to suggest when to use
> --full-name-hash or not?
The kinds of heuristics I would use are:
1. Are there enough commits that enough files have enough versions
across history that it's very important to keep deltas within a path?
2. Is the repository at least 500MB such that there is actually room for
a "meaningful" change in size?
3. Are there a lot of name-hash collisions? (The last patch in the series
helps do this through a test-helper, but isn't something we can expect
end users to check themselves.)
>> + cat >in-shallow <<-EOF
>> + $(git rev-parse HEAD)
>> + --shallow $(git rev-parse HEAD)
>> + EOF
>> +'
> > I was going to comment that these could probably be moved into the
> individual perf test that cares about reading each of these inputs. But
> having them shared here makes sense since we are naturally comparing
> generating two packs with the same input (with and without
> --full-name-hash). So the shared setup here makes sense to me.
I also wanted to avoid having these commands be part of the time
measurement, even if they are extremely small.
>> +
>> +test_perf 'thin pack' '
>> + git pack-objects --thin --stdout --revs --sparse <in-thin >out
>> +'
>> +
>> +test_size 'thin pack size' '
>> + test_file_size out
>> +'
> > Nice. I always forget about this and end up writing 'wc -c <out'.
I believe this is a Junio recommendation from an earlier version.
>> +test_size 'repack size' '
>> + pack=$(ls .git/objects/pack/pack-*.pack) &&
>> + test_file_size "$pack"
> > Here and below, I think it's fine to inline this as in:
> > test_file_size "$(ls .git/objects/pack/pack-*.pack)"
Generally I prefer to split things into stages so the verbose output
provides a clear definition of the value when calling the Git command.
> ...but I wonder: will using ".git" break this test in bare repositories?
> Should we write instead:
> > pack="$(ls $(git rev-parse --git-dir)/objects/pack/pack-*.pack)" &&
> test_file_size
> > ?
While this would break a bare repo, the perf lib makes a bare repo be
copied into a non-bare repo as follows:
test_perf_copy_repo_contents () {
for stuff in "$1"/*
do
case "$stuff" in
*/objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*)
;;
*)
cp -R "$stuff" "$repo/.git/" || exit 1
;;
esac
done
}
I'll still add the `git rev-parse` suggestion because it's safest.
Thanks,
-Stolee
| @@ -4576,6 +4586,20 @@ int cmd_pack_objects(int argc, | |||
| if (pack_to_stdout || !rev_list_all) | |||
| write_bitmap_index = 0; | |||
|
|
|||
| if (use_full_name_hash < 0) | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:06AM +0000, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <[email protected]>
>
> As demonstrated in the previous change, the --full-name-hash option of
> 'git pack-objects' is less effective in a trunctated history. Thus, even
> when the option is selected via a command-line option or config, disable
> this option when the '--shallow' option is specified. This will help
> performance in servers that choose to enable the --full-name-hash option
> by default for a repository while not regressing their ability to serve
> shallow clones.
>
> This will not present a compatibility issue in the future when the full
> name hash values are stored in the reachability bitmaps, since shallow
> clones disable bitmaps.
>
> Signed-off-by: Derrick Stolee <[email protected]>
> ---
> builtin/pack-objects.c | 6 ++++++
> t/perf/p5313-pack-objects.sh | 1 +
> 2 files changed, 7 insertions(+)
I appreciate demonstrating the value of declaring --shallow and
--full-name-hash incompatible by showing the performance numbers in the
previous patch.
But TBH I think that it would be equally fine or slightly better to say
up front "when combined with --shallow, this option produces larger
packs during testing, so the two are incompatible for now". You could
include some performance numbers there to illustrate that difference in
the commit log too if you wanted.
But I don't think it's worth introducing the pair as compatible only to
mark them incompatible later on in the same series.
Thanks,
TaylorThere was a problem hiding this comment.
On the Git mailing list, Derrick Stolee wrote (reply to this):
On 11/21/24 3:33 PM, Taylor Blau wrote:
> On Tue, Nov 05, 2024 at 03:05:06AM +0000, Derrick Stolee via GitGitGadget wrote:
>> From: Derrick Stolee <[email protected]>
>>
>> As demonstrated in the previous change, the --full-name-hash option of
>> 'git pack-objects' is less effective in a trunctated history. Thus, even
>> when the option is selected via a command-line option or config, disable
>> this option when the '--shallow' option is specified. This will help
>> performance in servers that choose to enable the --full-name-hash option
>> by default for a repository while not regressing their ability to serve
>> shallow clones.
>>
>> This will not present a compatibility issue in the future when the full
>> name hash values are stored in the reachability bitmaps, since shallow
>> clones disable bitmaps.
>>
>> Signed-off-by: Derrick Stolee <[email protected]>
>> ---
>> builtin/pack-objects.c | 6 ++++++
>> t/perf/p5313-pack-objects.sh | 1 +
>> 2 files changed, 7 insertions(+)
> > I appreciate demonstrating the value of declaring --shallow and
> --full-name-hash incompatible by showing the performance numbers in the
> previous patch.
> > But TBH I think that it would be equally fine or slightly better to say
> up front "when combined with --shallow, this option produces larger
> packs during testing, so the two are incompatible for now". You could
> include some performance numbers there to illustrate that difference in
> the commit log too if you wanted.
> > But I don't think it's worth introducing the pair as compatible only to
> mark them incompatible later on in the same series.
I disagree and here's why: they are not functionally incompatible. This
performance-focused change is worth justifying with performance test data
_and_ isolating from the initial implementation with its own reasoning
for future history spelunkers. Having these warning lines blame to this
patch instead of the initial implementation will make it much easier to
understand the justification of this change.
But maybe this patch can be removed if we use Jonathan's function. I'll
check the performance tests to see if this continues to be justified.
Thanks,
-Stolee
| @@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o | |||
| TEST_BUILTINS_OBJS += test-match-trees.o | |||
There was a problem hiding this comment.
On the Git mailing list, Taylor Blau wrote (reply to this):
On Tue, Nov 05, 2024 at 03:05:07AM +0000, Derrick Stolee via GitGitGadget wrote:
> Test this tree
> -----------------------------------------------------------------
> 5314.1: paths at head 4.5K
> 5314.2: number of distinct name-hashes 4.1K
> 5314.3: number of distinct full-name-hashes 4.5K
> 5314.4: maximum multiplicity of name-hashes 13
> 5314.5: maximum multiplicity of fullname-hashes 1
>
> Here, the maximum collision multiplicity is 13, but around 10% of paths
> have a collision with another path.
Neat.
> diff --git a/t/helper/test-name-hash.c b/t/helper/test-name-hash.c
> new file mode 100644
> index 00000000000..e4ecd159b76
> --- /dev/null
> +++ b/t/helper/test-name-hash.c
> @@ -0,0 +1,24 @@
> +/*
> + * test-name-hash.c: Read a list of paths over stdin and report on their
> + * name-hash and full name-hash.
> + */
> +
> +#include "test-tool.h"
> +#include "git-compat-util.h"
> +#include "pack-objects.h"
> +#include "strbuf.h"
> +
> +int cmd__name_hash(int argc UNUSED, const char **argv UNUSED)
> +{
> + struct strbuf line = STRBUF_INIT;
> +
> + while (!strbuf_getline(&line, stdin)) {
> + uint32_t name_hash = pack_name_hash(line.buf);
> + uint32_t full_hash = pack_full_name_hash(line.buf);
> +
> + printf("%10"PRIu32"\t%10"PRIu32"\t%s\n", name_hash, full_hash, line.buf);
I'm definitely nitpicking, but having a tab to separate these two 32-bit
values feels odd when we know already that they will be at most
10-characters wide.
I probably would have written:
printf("%10"PRIu32" %10"PRIu32"\t%s\n", name_hash, full_hash, line.buf);
instead, but this is obviously not a big deal either way ;-).
> diff --git a/t/perf/p5314-name-hash.sh b/t/perf/p5314-name-hash.sh
> new file mode 100755
> index 00000000000..9fe26612fac
> --- /dev/null
> +++ b/t/perf/p5314-name-hash.sh
> @@ -0,0 +1,41 @@
> +#!/bin/sh
> +
> +test_description='Tests pack performance using bitmaps'
> +. ./perf-lib.sh
> +
> +GIT_TEST_PASSING_SANITIZE_LEAK=0
> +export GIT_TEST_PASSING_SANITIZE_LEAK
Does this conflict with Patrick's series to remove these leak checking
annotations? I think it might, which is not unexpected given this series
was written before that one (and it's my fault for not reviewing it
earlier).
> +test_perf_large_repo
> +
> +test_size 'paths at head' '
> + git ls-tree -r --name-only HEAD >path-list &&
> + wc -l <path-list
> +'
> +
> +test_size 'number of distinct name-hashes' '
> + cat path-list | test-tool name-hash >name-hashes &&
> + cat name-hashes | awk "{ print \$1; }" | sort -n | uniq -c >name-hash-count &&
In these two (and a handful of others lower down in this same script)
the "cat ... |" is unnecessary. I think this one should be written as:
test-tool name-hash <path-list >name-hashes &&
awk "{ print \$1; }" <name-hashes | sort | uniq -c >name-hash-count &&
(sort -n is unnecessary, since we just care about getting the list in
sorted order so that "uniq -c" can count the number of unique values).
> + wc -l <name-hash-count
> +'
> +
> +test_size 'number of distinct full-name-hashes' '
> + cat name-hashes | awk "{ print \$2; }" | sort -n | uniq -c >full-name-hash-count &&
> + wc -l <full-name-hash-count
> +'
> +
> +test_size 'maximum multiplicity of name-hashes' '
> + cat name-hash-count | \
> + sort -nr | \
> + head -n 1 | \
> + awk "{ print \$1; }"
> +'
> +
> +test_size 'maximum multiplicity of fullname-hashes' '
> + cat full-name-hash-count | \
> + sort -nr | \
> + head -n 1 | \
> + awk "{ print \$1; }"
Nitpicking again, but you could extract the "sort | head | awk" pipeline
into a function.
Thanks,
Taylor
|
On the Git mailing list, Jonathan Tan wrote (reply to this): "Derrick Stolee via GitGitGadget" <[email protected]> writes:
> This series introduces a new name-hash algorithm, but does not replace the
> existing one. There are cases, such as packing a single snapshot of a
> repository, where the existing algorithm outperforms the new one.
I came up with a hash function that both uses information from a lot
more of the path (not the full name, though) and preserves the sortable
property (diff at the end of this email). It also contains fixes to the
existing algorithm: not wasting the most significant bits of the hash
if files in the repo mostly end in a lowercase alphabetic character, and
the cast from a possibly-signed-possibly-unsigned char to a uint32_t.
The results look quite good. In summary, the pack sizes are comparable
to Stolee's results in the case of fluentui, and better than Stolee's
results in the case of git.
Here's one run on the fluentui repo (git clone https://
github.com/microsoft/fluentui; cd fluentui; git checkout
a637a06df05360ce5ff21420803f64608226a875^ following the instructions
in [1]:
(before my change)
Test this tree
---------------------------------------------------------------------
5313.2: thin pack 0.03(0.01+0.01)
5313.3: thin pack size 1.1K
5313.4: thin pack with --full-name-hash 0.03(0.00+0.02)
5313.5: thin pack size with --full-name-hash 3.0K
5313.6: big pack 1.60(2.87+0.32)
5313.7: big pack size 57.9M
5313.8: big pack with --full-name-hash 1.41(1.94+0.37)
5313.9: big pack size with --full-name-hash 57.8M
5313.10: shallow fetch pack 1.69(2.70+0.22)
5313.11: shallow pack size 33.0M
5313.12: shallow pack with --full-name-hash 1.49(1.84+0.34)
5313.13: shallow pack size with --full-name-hash 33.6M
5313.14: repack 75.10(537.66+5.47)
5313.15: repack size 454.2M
5313.16: repack with --full-name-hash 18.10(92.50+5.14)
5313.17: repack size with --full-name-hash 174.8M
(after my change)
Test this tree
---------------------------------------------------------------------
5313.2: thin pack 0.03(0.01+0.02)
5313.3: thin pack size 1.1K
5313.4: thin pack with --full-name-hash 0.03(0.01+0.02)
5313.5: thin pack size with --full-name-hash 1.1K
5313.6: big pack 1.62(2.94+0.28)
5313.7: big pack size 57.9M
5313.8: big pack with --full-name-hash 1.35(2.07+0.37)
5313.9: big pack size with --full-name-hash 57.6M
5313.10: shallow fetch pack 1.63(2.52+0.29)
5313.11: shallow pack size 33.0M
5313.12: shallow pack with --full-name-hash 1.50(2.10+0.23)
5313.13: shallow pack size with --full-name-hash 33.1M
5313.14: repack 74.86(531.39+5.49)
5313.15: repack size 454.7M
5313.16: repack with --full-name-hash 19.71(111.39+5.12)
5313.17: repack size with --full-name-hash 165.6M
The tests were run by:
GENERATE_COMPILATION_DATABASE=yes make CC=clang && (cd t/perf && env GIT_PERF_LARGE_REPO=~/tmp/fluentui ./run -- p5313*.sh)
The similarity in sizes looked suspicious, so I replaced the contents
of the hash function with "return 0;" and indeed the sizes significantly
increased, so hopefully there is nothing wrong with my setup.
The git repo was called out in [1] as demonstrating "some of the issues
with this approach", but here are the results, run by:
GENERATE_COMPILATION_DATABASE=yes make CC=clang && (cd t/perf && ./run -- p5313*.sh)
Test this tree
--------------------------------------------------------------------
5313.2: thin pack 0.03(0.00+0.02)
5313.3: thin pack size 2.9K
5313.4: thin pack with --full-name-hash 0.03(0.00+0.02)
5313.5: thin pack size with --full-name-hash 2.9K
5313.6: big pack 1.69(2.80+0.28)
5313.7: big pack size 18.7M
5313.8: big pack with --full-name-hash 1.68(2.82+0.31)
5313.9: big pack size with --full-name-hash 18.8M
5313.10: shallow fetch pack 0.96(1.47+0.16)
5313.11: shallow pack size 12.1M
5313.12: shallow pack with --full-name-hash 1.01(1.51+0.14)
5313.13: shallow pack size with --full-name-hash 12.1M
5313.14: repack 17.05(69.99+4.33)
5313.15: repack size 116.5M
5313.16: repack with --full-name-hash 15.74(67.03+4.18)
5313.17: repack size with --full-name-hash 116.1M
[1] https://lore.kernel.org/git/c14ef6879e451401381ebbdb8f30d33c8f56c25b.1730775908.git.gitgitgadget@gmail.com/
> | Repo | Standard Repack | With --full-name-hash |
> |----------|-----------------|-----------------------|
> | fluentui | 438 MB | 168 MB |
> | Repo B | 6,255 MB | 829 MB |
> | Repo C | 37,737 MB | 7,125 MB |
> | Repo D | 130,049 MB | 6,190 MB |
> | Repo E | 100,957 MB | 22,979 MB |
> | Repo F | 8,308 MB | 746 MB |
> | Repo G | 4,329 MB | 3,643 MB |
If the results are similar for some of the above repos (I do not have
access to them), maybe it's worth considering using my hash function (or
a variation of it).
I'll also take a look at the rest of the patch set.
---
diff --git a/pack-objects.h b/pack-objects.h
index 88360aa3e8..c4f35eafa0 100644
--- a/pack-objects.h
+++ b/pack-objects.h
@@ -209,23 +209,24 @@ static inline uint32_t pack_name_hash(const char *name)
static inline uint32_t pack_full_name_hash(const char *name)
{
- const uint32_t bigp = 1234572167U;
- uint32_t c, hash = bigp;
+ uint32_t hash = 0, base = 0;
+ uint8_t c;
if (!name)
return 0;
- /*
- * Do the simplest thing that will resemble pseudo-randomness: add
- * random multiples of a large prime number with a binary shift.
- * The goal is not to be cryptographic, but to be generally
- * uniformly distributed.
- */
- while ((c = *name++) != 0) {
- hash += c * bigp;
- hash = (hash >> 5) | (hash << 27);
+ while ((c = (uint8_t) *name++) != 0) {
+ if (isspace(c))
+ continue;
+ if (c == '/') {
+ base = (base >> 6) ^ hash;
+ hash = 0;
+ } else {
+ uint8_t nybble_swapped = (c >> 4) + ((c & 15) << 4);
+ hash = (hash >> 2) + (nybble_swapped << 24);
+ }
}
- return hash;
+ return (base >> 6) ^ hash;
} |
|
User |
| @@ -266,6 +266,14 @@ struct configured_exclusion { | |||
| static struct oidmap configured_exclusions; | |||
There was a problem hiding this comment.
On the Git mailing list, Jonathan Tan wrote (reply to this):
"Derrick Stolee via GitGitGadget" <[email protected]> writes:
> Second, there are two tests in t5616-partial-clone.sh that I believe are
> actually broken scenarios.
I took a look...this is a tricky one.
> While the client is set up to clone the
> 'promisor-server' repo via a treeless partial clone filter (tree:0),
> that filter does not translate to the 'server' repo. Thus, fetching from
> these repos causes the server to think that the client has all reachable
> trees and blobs from the commits advertised as 'haves'. This leads the
> server to providing a thin pack assuming those objects as delta bases.
It is expected that the server sometimes sends deltas based on objects
that the client doesn't have. In fact, this test tests the ability of
Git to lazy-fetch delta bases.
> Changing the name-hash algorithm presents new delta bases and thus
> breaks the expectations of these tests.
To be precise, the change resulted in no deltas being sent (before this
change, one delta was sent). Here's what is meant to happen. The server has:
commitB - treeB - file1 ("make the tree big\nanother line\n"), file2...file100
|
commitA - treeA - file1...file100 ("make the tree big\n")
The client only has commitA. (The client does not have treeA or any
blob, since it was cloned with --filter=tree:0.)
When GIT_TEST_FULL_NAME_HASH=0 (matching the current behavior), the
server sends a non-delta commitB, a delta treeB (with base treeA), and
a non-delta blob "make the tree big\nanother line\n". This triggers a
lazy fetch of treeA, and thus treeB is inflated successfully. During
the subsequent connectivity check (with --exclude-promisor-objects,
see connected.c), it is noticed that the "make the tree big\n" blob is
missing, but since it is a promisor object (referenced by treeA, which
was fetched from the promisor remote), the connectivity check since
passes.
When GIT_TEST_FULL_NAME_HASH=1, the server sends a non-delta commitB,
a non-delta treeB, and a non-delta blob "make the tree big\nanother
line\n". No lazy fetch is triggered. During the subsequent connectivity
check, the "make the tree big\n" blob (referenced by treeB) is missing.
There is nothing that can vouch for it (the client does not have treeA,
remember) so the client does not consider it a promisor object, and thus
the connectivity check fails.
Investigating this was made a bit harder due to a missing "git -C
promisor-remote config --local uploadpack.allowfilter 1" in the test.
The above behavior is after this is included in the test.
I think the solution is to have an algorithm that preserves the property
that treeB is sent as a delta object - if not, we need to find another
way to test the lazy-fetch of delta bases. My proposal in [1] does do
that.
[1] https://lore.kernel.org/git/[email protected]/
There was a problem hiding this comment.
On the Git mailing list, Junio C Hamano wrote (reply to this):
Jonathan Tan <[email protected]> writes:
> ... During the subsequent connectivity
> check, the "make the tree big\n" blob (referenced by treeB) is missing.
> There is nothing that can vouch for it (the client does not have treeA,
> remember) so the client does not consider it a promisor object, and thus
> the connectivity check fails.
It is sad that it is a (probably unfixable) flaw in the "promisor
object" concept that the "promisor object"-ness of blobA depends on
the lazy-fetch status of treeA. This is not merely a test failure,
but it would cause blobA pruned if such a lazy fetch happens in the
wild and then "git gc" triggers, no? It may not manifest as a
repository corruption, since we would lazily fetch it again if the
user requests to fully fetch what commitA and treeA need, but it
does feel somewhat suboptimal.
Thanks for a detailed explanation on what is going on.There was a problem hiding this comment.
On the Git mailing list, Jonathan Tan wrote (reply to this):
Junio C Hamano <[email protected]> writes:
> It is sad that it is a (probably unfixable) flaw in the "promisor
> object" concept that the "promisor object"-ness of blobA depends on
> the lazy-fetch status of treeA. This is not merely a test failure,
> but it would cause blobA pruned if such a lazy fetch happens in the
> wild and then "git gc" triggers, no?
Right now, it won't be pruned since we never prune promisor objects
(we just concatenate all of them into one file). But in the future, we
might only keep reachable promisor objects, in which case, yes, blobA
will be pruned. In this case, though, I think blobA is like any other
unreachable object in git. If a user memorizes a commit hash but does
not point a ref to it (or point a ref to one of its descendants), that
commit is still subject to being lost by GC. I think it's the same case
here.There was a problem hiding this comment.
On the Git mailing list, Junio C Hamano wrote (reply to this):
Jonathan Tan <[email protected]> writes:
> Junio C Hamano <[email protected]> writes:
>> It is sad that it is a (probably unfixable) flaw in the "promisor
>> object" concept that the "promisor object"-ness of blobA depends on
>> the lazy-fetch status of treeA. This is not merely a test failure,
>> but it would cause blobA pruned if such a lazy fetch happens in the
>> wild and then "git gc" triggers, no?
>
> Right now, it won't be pruned since we never prune promisor objects
> (we just concatenate all of them into one file).
Sorry, but I am lost. In the scenario discussed, you have two
commits A and B with their trees and blobs. You initially only have
commit A because the partial clone is done with "tree:0". Then you
fetch commit B (A's child), tree B in non-delta form, and blob B
contained within tree B. Due to the tweak in the name hash
function, we do not know of tree A (we used to learn about it
because tree B was sent as a delta against it with the old name
hash). If blob B was sent as a delta against blob A, lazy fetch
would later materialize blob A even if you do not still have tree A,
no?
I thought the story was that we would not know who refers to blobA
when treeA hasn't been lazily fetched, hence we cannot tell if blobA
is a "promisor object" to begin with, no?
The blob in such a scenario may be reclaimed by GC and we may still
be able to refetch from the promisor, so it may not be the end of
the world, but feels suboptimal.
There was a problem hiding this comment.
On the Git mailing list, Jonathan Tan wrote (reply to this):
Junio C Hamano <[email protected]> writes:
> Jonathan Tan <[email protected]> writes:
>
> > Junio C Hamano <[email protected]> writes:
> >> It is sad that it is a (probably unfixable) flaw in the "promisor
> >> object" concept that the "promisor object"-ness of blobA depends on
> >> the lazy-fetch status of treeA. This is not merely a test failure,
> >> but it would cause blobA pruned if such a lazy fetch happens in the
> >> wild and then "git gc" triggers, no?
> >
> > Right now, it won't be pruned since we never prune promisor objects
> > (we just concatenate all of them into one file).
>
> Sorry, but I am lost. In the scenario discussed, you have two
> commits A and B with their trees and blobs. You initially only have
> commit A because the partial clone is done with "tree:0". Then you
> fetch commit B (A's child), tree B in non-delta form, and blob B
> contained within tree B. Due to the tweak in the name hash
> function, we do not know of tree A (we used to learn about it
> because tree B was sent as a delta against it with the old name
> hash).
Yes, that's correct.
> If blob B was sent as a delta against blob A, lazy fetch
> would later materialize blob A even if you do not still have tree A,
> no?
Just to be clear, this is not happening right now (blob B is sent whole,
not as a delta). But let's suppose that blob B was sent as a delta, then
yes, the lazy fetch would materialize blob A...
> I thought the story was that we would not know who refers to blobA
> when treeA hasn't been lazily fetched, hence we cannot tell if blobA
> is a "promisor object" to begin with, no?
...ah, in this case, blob A vouches for itself. Whenever we lazy fetch,
all objects that are fetched go into promisor packs (packfiles with an
associated .promisor file), so we know that they are promisor objects.There was a problem hiding this comment.
On the Git mailing list, Junio C Hamano wrote (reply to this):
Jonathan Tan <[email protected]> writes:
> ...ah, in this case, blob A vouches for itself. Whenever we lazy fetch,
> all objects that are fetched go into promisor packs (packfiles with an
> associated .promisor file), so we know that they are promisor objects.
Thanks.
| @@ -816,6 +816,7 @@ TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o | |||
| TEST_BUILTINS_OBJS += test-match-trees.o | |||
There was a problem hiding this comment.
On the Git mailing list, Jonathan Tan wrote (reply to this):
"Derrick Stolee via GitGitGadget" <[email protected]> writes:
> From: Derrick Stolee <[email protected]>
>
> Add a new test-tool helper, name-hash, to output the value of the
> name-hash algorithms for the input list of strings, one per line.
I've looked at all 7 patches.
I didn't really understand the concern with shallow in patch 6 (in
particular, the documentation of "git pack-objects --shallow" seems
to imply that it's for use by a server to a shallow client, but at
the point that the server would need such a feature, it probably would
already have bitmaps packed with the new hash algorithm). I didn't look
at it further, though, since I had an algorithm that seemed to also do
OK in the shallow test. So we might be able to drop patch 6.
Other than that, and other than all my comments and Taylor's comments,
this series looks good.
|
This patch series was integrated into seen via git@9d8e429. |
|
On the Git mailing list, Junio C Hamano wrote (reply to this): Jonathan Tan <[email protected]> writes:
> + while ((c = (uint8_t) *name++) != 0) {
> + if (isspace(c))
> + continue;
> + if (c == '/') {
> + base = (base >> 6) ^ hash;
> + hash = 0;
> + } else {
> + uint8_t nybble_swapped = (c >> 4) + ((c & 15) << 4);
> + hash = (hash >> 2) + (nybble_swapped << 24);
> + }
> }
> + return (base >> 6) ^ hash;
> }
Nice. The diff relative to the --full-name-hash version is a bit
hard to grok, but compared to the current hash function, there are
two and a half changes that matter:
(0) it is more careful with bytes with the MSB set (i.e. non-ASCII
pathnames).
(1) it hashes each path component separetely and rotates the whole
thing only at a directory boundary. I'd imagine that this
would make a big difference for languages that force overly
long filenames at each level.
(2) it gives more weight to lower bits by swapping nybbles of each
byte.
I wonder if we do even better if we reverse all 8 bits instead of
swapping nybbles (if we were to do so, it might be more efficient to
shift in from the right instead of left end of the base and hash
accumulators in the loop and then swap the whole resulting word at
the end).
Thanks for a fun read. |
|
On the Git mailing list, Junio C Hamano wrote (reply to this): Junio C Hamano <[email protected]> writes:
> ... (if we were to do so, it might be more efficient to
> shift in from the right instead of left end of the base and hash
> accumulators in the loop and then swap the whole resulting word at
> the end).
That is garbage. We could do the "shift in from the right and then
reverse the result" for the hash accumulator, but not the "base"
one. Sorry for the noise.
|
|
This patch series was integrated into seen via git@c586020. |
The pack_name_hash() method has not been materially changed since it was introduced in ce0bd64 (pack-objects: improve path grouping heuristics., 2006-06-05). The intention here is to group objects by path name, but also attempt to group similar file types together by making the most-significant digits of the hash be focused on the final characters. Here's the crux of the implementation: /* * This effectively just creates a sortable number from the * last sixteen non-whitespace characters. Last characters * count "most", so things that end in ".c" sort together. */ while ((c = *name++) != 0) { if (isspace(c)) continue; hash = (hash >> 2) + (c << 24); } As the comment mentions, this only cares about the last sixteen non-whitespace characters. This cause some filenames to collide more than others. This collision is somewhat by design in order to promote hash locality for files that have similar types (.c, .h, .json) or could be the same file across a directory rename (a/foo.txt to b/foo.txt). This leads to decent cross-path deltas in cases like shallow clones or packing a repository with very few historical versions of files that share common data with other similarly-named files. However, when the name-hash instead leads to a large number of name-hash collisions for otherwise unrelated files, this can lead to confusing the delta calculation to prefer cross-path deltas over previous versions of the same file. Here are some examples that I've seen while investigating repositories that are growing more than they should be: * "/CHANGELOG.json" is 15 characters, and is created by the beachball [1] tool. Only the final character of the parent directory can differentiate different versions of this file, but also only the two most-significant digits. If that character is a letter, then this is always a collision. Similar issues occur with the similar "/CHANGELOG.md" path, though there is more opportunity for differences in the parent directory. * Localization files frequently have common filenames but differentiates via parent directories. In C#, the name "/strings.resx.lcl" is used for these localization files and they will all collide in name-hash. [1] https://github.com/microsoft/beachball I've come across many other examples where some internal tool uses a common name across multiple directories and is causing Git to repack poorly due to name-hash collisions. It is clear that the existing name-hash algorithm is optimized for repositories with short path names, but also is optimized for packing a single snapshot of a repository, not a repository with many versions of the same file. In my testing, this has proven out where the name-hash algorithm does a good job of finding peer files as delta bases when unable to use a historical version of that exact file. However, for repositories that have many versions of most files and directories, it is more important that the objects that appear at the same path are grouped together. Create a new pack_full_name_hash() method and a new --full-name-hash option for 'git pack-objects' to call that method instead. Add a simple pass-through for 'git repack --full-name-hash' for additional testing in the context of a full repack, where I expect this will be most effective. The hash algorithm is as simple as possible to be reasonably effective: for each character of the path string, add a multiple of that character and a large prime number (chosen arbitrarily, but intended to be large relative to the size of a uint32_t). Then, shift the current hash value to the right by 5, with overlap. The addition and shift parameters are standard mechanisms for creating hard-to-predict behaviors in the bits of the resulting hash. This is not meant to be cryptographic at all, but uniformly distributed across the possible hash values. This creates a hash that appears pseudorandom. There is no ability to consider similar file types as being close to each other. In a later change, a test-tool will be added so the effectiveness of this hash can be demonstrated directly. For now, let's consider how effective this mechanism is when repacking a repository with and without the --full-name-hash option. Specifically, let's use 'git repack -adf [--full-name-hash]' as our test. On the Git repository, we do not expect much difference. All path names are short. This is backed by our results: | Stage | Pack Size | Repack Time | |-----------------------|-----------|-------------| | After clone | 260 MB | N/A | | Standard Repack | 127MB | 106s | | With --full-name-hash | 126 MB | 99s | This example demonstrates how there is some natural overhead coming from the cloned copy because the server is hosting many forks and has not optimized for exactly this set of reachable objects. But the full repack has similar characteristics with and without --full-name-hash. However, we can test this in a repository that uses one of the problematic naming conventions above. The fluentui [2] repo uses beachball to generate CHANGELOG.json and CHANGELOG.md files, and these files have very poor delta characteristics when comparing against versions across parent directories. | Stage | Pack Size | Repack Time | |-----------------------|-----------|-------------| | After clone | 694 MB | N/A | | Standard Repack | 438 MB | 728s | | With --full-name-hash | 168 MB | 142s | [2] https://github.com/microsoft/fluentui In this example, we see significant gains in the compressed packfile size as well as the time taken to compute the packfile. Using a collection of repositories that use the beachball tool, I was able to make similar comparisions with dramatic results. While the fluentui repo is public, the others are private so cannot be shared for reproduction. The results are so significant that I find it important to share here: | Repo | Standard Repack | With --full-name-hash | |----------|-----------------|-----------------------| | fluentui | 438 MB | 168 MB | | Repo B | 6,255 MB | 829 MB | | Repo C | 37,737 MB | 7,125 MB | | Repo D | 130,049 MB | 6,190 MB | Future changes could include making --full-name-hash implied by a config value or even implied by default during a full repack. It is important to point out that the name hash value is stored in the .bitmap file format, so we must disable the --full-name-hash option when bitmaps are being read or written. Later, the bitmap format could be updated to be aware of the name hash version so deltas can be quickly computed across the bitmapped/not-bitmapped boundary. Signed-off-by: Derrick Stolee <[email protected]>
derrickstolee
force-pushed
the
full-name
branch
from
ab341dd
to
01939a1
Compare
The new '--full-name-hash' option for 'git repack' is a simple pass-through to the underlying 'git pack-objects' subcommand. However, this subcommand may have other options and a temporary filename as part of the subcommand execution that may not be predictable or could change over time. The existing test_subcommand method requires an exact list of arguments for the subcommand. This is too rigid for our needs here, so create a new method, test_subcommand_flex. Use it to check that the --full-name-hash option is passing through. Signed-off-by: Derrick Stolee <[email protected]>
Add a new environment variable to opt-in to the --full-name-hash option in 'git pack-objects'. This allows for extra testing of the feature without repeating all of the test scenarios. Unlike many GIT_TEST_* variables, we are choosing to not add this to the linux-TEST-vars CI build as that test run is already overloaded. The behavior exposed by this test variable is of low risk and should be sufficient to allow manual testing when an issue arises. But this option isn't free. There are a few tests that change behavior with the variable enabled. First, there are a few tests that are very sensitive to certain delta bases being picked. These are both involving the generation of thin bundles and then counting their objects via 'git index-pack --fix-thin' which pulls the delta base into the new packfile. For these tests, disable the option as a decent long-term option. Second, there are two tests in t5616-partial-clone.sh that I believe are actually broken scenarios. While the client is set up to clone the 'promisor-server' repo via a treeless partial clone filter (tree:0), that filter does not translate to the 'server' repo. Thus, fetching from these repos causes the server to think that the client has all reachable trees and blobs from the commits advertised as 'haves'. This leads the server to providing a thin pack assuming those objects as delta bases. Changing the name-hash algorithm presents new delta bases and thus breaks the expectations of these tests. An alternative could be to set up 'server' as a promisor server with the correct filter enabled. This may also point out more issues with partial clone being set up as a remote-based filtering mechanism and not a repository-wide setting. For now, do the minimal change to make the test work by disabling the test variable. Third, there are some tests that compare the exact output of a 'git pack-objects' process when using bitmaps. The warning that disables the --full-name-hash option causes these tests to fail. Disable the environment variable to get around this issue. Signed-off-by: Derrick Stolee <[email protected]>
As custom options are added to 'git pack-objects' and 'git repack' to adjust how compression is done, use this new performance test script to demonstrate their effectiveness in performance and size. The recently-added --full-name-hash option swaps the default name-hash algorithm with one that attempts to uniformly distribute the hashes based on the full path name instead of the last 16 characters. This has a dramatic effect on full repacks for repositories with many versions of most paths. It can have a negative impact on cases such as pushing a single change. This can be seen by running pt5313 on the open source fluentui repository [1]. Most commits will have this kind of output for the thin and big pack cases, though certain commits (such as [2]) will have problematic thin pack size for other reasons. [1] https://github.com/microsoft/fluentui [2] a637a06df05360ce5ff21420803f64608226a875 Checked out at the parent of [2], I see the following statistics: Test HEAD --------------------------------------------------------------------- 5313.2: thin pack 0.37(0.43+0.02) 5313.3: thin pack size 1.2M 5313.4: thin pack with --full-name-hash 0.06(0.09+0.02) 5313.5: thin pack size with --full-name-hash 20.4K 5313.6: big pack 2.01(7.73+0.23) 5313.7: big pack size 20.3M 5313.8: big pack with --full-name-hash 1.32(2.77+0.27) 5313.9: big pack size with --full-name-hash 19.9M 5313.10: shallow fetch pack 1.40(3.01+0.08) 5313.11: shallow pack size 34.4M 5313.12: shallow pack with --full-name-hash 1.08(1.25+0.14) 5313.13: shallow pack size with --full-name-hash 35.4M 5313.14: repack 90.70(672.88+2.46) 5313.15: repack size 439.6M 5313.16: repack with --full-name-hash 18.53(123.41+2.53) 5313.17: repack size with --full-name-hash 169.7M In this case, we see positive behaviors such as a significant shrink in the size of the thin pack and full repack. The big pack is slightly smaller with --full-name-hash than without. The shallow pack is slightly larger with --full-name-hash. In the case of the Git repository, these numbers show some of the issues with this approach: Test HEAD -------------------------------------------------------------------- 5313.2: thin pack 0.00(0.00+0.00) 5313.3: thin pack size 589 5313.4: thin pack with --full-name-hash 0.00(0.00+0.00) 5313.5: thin pack size with --full-name-hash 14.9K 5313.6: big pack 2.07(3.57+0.17) 5313.7: big pack size 17.6M 5313.8: big pack with --full-name-hash 2.00(3.07+0.19) 5313.9: big pack size with --full-name-hash 17.9M 5313.10: shallow fetch pack 1.41(2.23+0.06) 5313.11: shallow pack size 12.1M 5313.12: shallow pack with --full-name-hash 1.22(1.66+0.04) 5313.13: shallow pack size with --full-name-hash 12.4M 5313.14: repack 15.75(89.29+1.54) 5313.15: repack size 126.4M 5313.16: repack with --full-name-hash 15.56(89.78+1.32) 5313.17: repack size with --full-name-hash 126.0M The thin pack that simulates a push is much worse with --full-name-hash in this case. The name hash values are doing a lot to assist with delta bases, it seems. The big pack and shallow clone cases are slightly worse with the --full-name-hash option. Only the full repack gains some benefits in size. The results are similar with the nodejs/node repo: Test HEAD --------------------------------------------------------------------- 5313.2: thin pack 0.01(0.01+0.00) 5313.3: thin pack size 1.6K 5313.4: thin pack with --full-name-hash 0.01(0.00+0.00) 5313.5: thin pack size with --full-name-hash 3.1K 5313.6: big pack 4.26(8.03+0.24) 5313.7: big pack size 56.0M 5313.8: big pack with --full-name-hash 4.16(6.55+0.22) 5313.9: big pack size with --full-name-hash 56.2M 5313.10: shallow fetch pack 7.67(11.80+0.29) 5313.11: shallow pack size 104.6M 5313.12: shallow pack with --full-name-hash 7.52(9.65+0.23) 5313.13: shallow pack size with --full-name-hash 105.9M 5313.14: repack 71.22(317.61+3.95) 5313.15: repack size 739.9M 5313.16: repack with --full-name-hash 48.85(267.02+3.72) 5313.17: repack size with --full-name-hash 793.5M The Linux kernel repository was the initial target of the default name hash value, and its naming conventions are practically build to take the most advantage of the default name hash values: Test HEAD ------------------------------------------------------------------------- 5313.2: thin pack 0.15(0.01+0.03) 5313.3: thin pack size 4.6K 5313.4: thin pack with --full-name-hash 0.03(0.02+0.01) 5313.5: thin pack size with --full-name-hash 6.8K 5313.6: big pack 18.51(33.74+0.95) 5313.7: big pack size 201.1M 5313.8: big pack with --full-name-hash 16.01(29.81+0.88) 5313.9: big pack size with --full-name-hash 202.1M 5313.10: shallow fetch pack 11.49(17.61+0.54) 5313.11: shallow pack size 269.2M 5313.12: shallow pack with --full-name-hash 11.24(15.25+0.56) 5313.13: shallow pack size with --full-name-hash 269.8M 5313.14: repack 1001.25(2271.06+38.86) 5313.15: repack size 2.5G 5313.16: repack with --full-name-hash 625.75(1941.96+36.09) 5313.17: repack size with --full-name-hash 2.6G Finally, an internal Javascript repo of moderate size shows significant gains when repacking with --full-name-hash due to it having many name hash collisions. However, it's worth noting that only the full repack case has enough improvement to be worth it. But the improvements are significant: 6.4 GB to 862 MB. Test HEAD -------------------------------------------------------------------------- 5313.2: thin pack 0.03(0.02+0.00) 5313.3: thin pack size 1.2K 5313.4: thin pack with --full-name-hash 0.03(0.03+0.00) 5313.5: thin pack size with --full-name-hash 2.6K 5313.6: big pack 2.20(3.23+0.30) 5313.7: big pack size 130.7M 5313.8: big pack with --full-name-hash 2.33(3.17+0.34) 5313.9: big pack size with --full-name-hash 131.0M 5313.10: shallow fetch pack 3.56(6.02+0.32) 5313.11: shallow pack size 44.5M 5313.12: shallow pack with --full-name-hash 2.94(3.94+0.32) 5313.13: shallow pack size with --full-name-hash 45.3M 5313.14: repack 2435.22(12523.11+23.53) 5313.15: repack size 6.4G 5313.16: repack with --full-name-hash 473.25(1805.11+17.22) 5313.17: repack size with --full-name-hash 861.9M These tests demonstrate that it is important to be careful about which cases are best for using the --full-name-hash option. Signed-off-by: Derrick Stolee <[email protected]>
As demonstrated in the previous change, the --full-name-hash option of 'git pack-objects' is less effective in a trunctated history. Thus, even when the option is selected via a command-line option or config, disable this option when the '--shallow' option is specified. This will help performance in servers that choose to enable the --full-name-hash option by default for a repository while not regressing their ability to serve shallow clones. This will not present a compatibility issue in the future when the full name hash values are stored in the reachability bitmaps, since shallow clones disable bitmaps. Signed-off-by: Derrick Stolee <[email protected]>
Add a new test-tool helper, name-hash, to output the value of the name-hash algorithms for the input list of strings, one per line. Since the name-hash values can be stored in the .bitmap files, it is important that these hash functions do not change across Git versions. Add a simple test to t5310-pack-bitmaps.sh to provide some testing of the current values. Due to how these functions are implemented, it would be difficult to change them without disturbing these values. Create a performance test that uses test_size to demonstrate how collisions occur for these hash algorithms. This test helps inform someone as to the behavior of the name-hash algorithms for their repo based on the paths at HEAD. My copy of the Git repository shows modest statistics around the collisions of the default name-hash algorithm: Test this tree ----------------------------------------------------------------- 5314.1: paths at head 4.5K 5314.2: number of distinct name-hashes 4.1K 5314.3: number of distinct full-name-hashes 4.5K 5314.4: maximum multiplicity of name-hashes 13 5314.5: maximum multiplicity of fullname-hashes 1 Here, the maximum collision multiplicity is 13, but around 10% of paths have a collision with another path. In a more interesting example, the microsoft/fluentui [1] repo had these statistics at time of committing: Test this tree ----------------------------------------------------------------- 5314.1: paths at head 19.6K 5314.2: number of distinct name-hashes 8.2K 5314.3: number of distinct full-name-hashes 19.6K 5314.4: maximum multiplicity of name-hashes 279 5314.5: maximum multiplicity of fullname-hashes 1 [1] https://github.com/microsoft/fluentui That demonstrates that of the nearly twenty thousand path names, they are assigned around eight thousand distinct values. 279 paths are assigned to a single value, leading the packing algorithm to sort objects from those paths together, by size. In this repository, no collisions occur for the full-name-hash algorithm. In a more extreme example, an internal monorepo had a much worse collision rate: Test this tree ----------------------------------------------------------------- 5314.1: paths at head 221.6K 5314.2: number of distinct name-hashes 72.0K 5314.3: number of distinct full-name-hashes 221.6K 5314.4: maximum multiplicity of name-hashes 14.4K 5314.5: maximum multiplicity of fullname-hashes 2 Even in this repository with many more paths at HEAD, the collision rate was low and the maximum number of paths being grouped into a single bucket by the full-path-name algorithm was two. Signed-off-by: Derrick Stolee <[email protected]>
derrickstolee
force-pushed
the
full-name
branch
from
01939a1
to
c6cb866
Compare
|
On the Git mailing list, Derrick Stolee wrote (reply to this): On 11/21/24 10:01 PM, Junio C Hamano wrote:
> Jonathan Tan <[email protected]> writes:
> >> + while ((c = (uint8_t) *name++) != 0) {
>> + if (isspace(c))
>> + continue;
>> + if (c == '/') {
>> + base = (base >> 6) ^ hash;
>> + hash = 0;
>> + } else {
>> + uint8_t nybble_swapped = (c >> 4) + ((c & 15) << 4);
>> + hash = (hash >> 2) + (nybble_swapped << 24);
>> + }
>> }
>> + return (base >> 6) ^ hash;
>> }
> > Nice. The diff relative to the --full-name-hash version is a bit
> hard to grok, but compared to the current hash function, there are
> two and a half changes that matter:
> > (0) it is more careful with bytes with the MSB set (i.e. non-ASCII
> pathnames).
> > (1) it hashes each path component separetely and rotates the whole
> thing only at a directory boundary. I'd imagine that this
> would make a big difference for languages that force overly
> long filenames at each level.
I was confused by the "rotates the whole thing only at a directory
boundary" statement. I think one way to say what you mean is
Each path component is hashed similarly to the standard name-hash,
and parent path component hashes are contributed via XOR after a
down-shift of 6 bits per level.
So we are getting something like
[ name-hash for level 0 ]
......[ name-hash for level 1 ](truncated by 6)
............[name-hash for level 2](truncated by 12)
..................[...for level 3 ](truncated by 18)
........................[ level 4 ](truncated by 24)
..............................[ 5 ](truncated by 30)
and at each layer we get the "last 16 bytes matter" issue, though it
is balanced quite well. Also, the name-hash in each layer is adjusted
for nybble swaps.
(I don't think my explanation is _better_ but just that it matches my
personal mental model slightly better.)
> (2) it gives more weight to lower bits by swapping nybbles of each
> byte.
> > I wonder if we do even better if we reverse all 8 bits instead of
> swapping nybbles (if we were to do so, it might be more efficient to
> shift in from the right instead of left end of the base and hash
> accumulators in the loop and then swap the whole resulting word at
> the end).
I will give this a try in my private repos as well as with the name-hash
collision perf test from patch 7.
Thanks,
-Stolee
|
|
On the Git mailing list, Jonathan Tan wrote (reply to this): Junio C Hamano <[email protected]> writes:
> I wonder if we do even better if we reverse all 8 bits instead of
> swapping nybbles (if we were to do so, it might be more efficient to
> shift in from the right instead of left end of the base and hash
> accumulators in the loop and then swap the whole resulting word at
> the end).
>
> Thanks for a fun read.
Ah, yes, reversing is better than swapping nybbles (the least
significant 2 bits have more entropy than the next-least significant
2 bits). When writing this, I didn't think of shifting in from the
right (if I had thought of that, I would have indeed reversed the bits
instead). |
|
On the Git mailing list, Junio C Hamano wrote (reply to this): Derrick Stolee <[email protected]> writes:
>> (1) it hashes each path component separetely and rotates the whole
>> thing only at a directory boundary. I'd imagine that this
>> would make a big difference for languages that force overly
>> long filenames at each level.
>
> I was confused by the "rotates the whole thing only at a directory
> boundary" statement.
Yeah, I guess it was confusing. What I meant was that the entire
result is shifted down with new material from left to right, but
unlike the original, the outer thing (i.e. what is given to the
caller as the result) is shifted only at the directory boundary, so
we are not as aggressive to lose early bits by shifting them down to
the right, as we are not shifting as fast as before.
> I think one way to say what you mean is
>
> Each path component is hashed similarly to the standard name-hash,
> and parent path component hashes are contributed via XOR after a
> down-shift of 6 bits per level.
Yes. |
|
This patch series was integrated into seen via git@d7c3f0a. |
|
This patch series was integrated into seen via git@e52a408. |
|
This patch series was integrated into seen via git@d138150. |
|
On the Git mailing list, Patrick Steinhardt wrote (reply to this), regarding 812257e (outdated): On Tue, Nov 05, 2024 at 03:05:01AM +0000, Derrick Stolee via GitGitGadget wrote:
> It is important to point out that the name hash value is stored in the
> .bitmap file format, so we must disable the --full-name-hash option when
> bitmaps are being read or written. Later, the bitmap format could be
> updated to be aware of the name hash version so deltas can be quickly
> computed across the bitmapped/not-bitmapped boundary.
I was wondering a bit about this: is there any reason why we cannot have
both, that is reap the benefits of "--full-name-hash" but end up writing
a bitmap with the old name hash so that we can continue to generate
bitmaps?
Forgive me if this question is naive, I'm more at home in the refs
subsystem :)
> diff --git a/pack-objects.h b/pack-objects.h
> index b9898a4e64b..88360aa3e8e 100644
> --- a/pack-objects.h
> +++ b/pack-objects.h
> @@ -207,6 +207,27 @@ static inline uint32_t pack_name_hash(const char *name)
> return hash;
> }
>
> +static inline uint32_t pack_full_name_hash(const char *name)
> +{
> + const uint32_t bigp = 1234572167U;
> + uint32_t c, hash = bigp;
It would be nice to have a comment here detailing how you came up with
that number, and what its requirements are. You briefly mention it in
the comment further down, but I think this could be expanded a bit.
Patrick |
|
On the Git mailing list, Patrick Steinhardt wrote (reply to this), regarding 259734e (outdated): On Tue, Nov 05, 2024 at 03:05:03AM +0000, Derrick Stolee via GitGitGadget wrote:
> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
> index c53e93be2f7..425aa8d8789 100755
> --- a/t/t5616-partial-clone.sh
> +++ b/t/t5616-partial-clone.sh
> @@ -516,7 +516,18 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas' '
> # Exercise to make sure it works. Git will not fetch anything from the
> # promisor remote other than for the big tree (because it needs to
> # resolve the delta).
> - GIT_TRACE_PACKET="$(pwd)/trace" git -C client \
> + #
> + # TODO: the --full-name-hash option is disabled here, since this test
> + # is fundamentally broken! When GIT_TEST_FULL_NAME_HASH=1, the server
> + # recognizes delta bases in a different way and then sends a _blob_ to
> + # the client with a delta base that the client does not have! This is
> + # because the client is cloned from "promisor-server" with tree:0 but
> + # is now fetching from "server" withot any filter. This is violating the
s/withot/without/
Also present in copies of this comment.
> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
> index 0f0c86f9cb2..03f8c976720 100755
> --- a/t/t7406-submodule-update.sh
> +++ b/t/t7406-submodule-update.sh
> @@ -1094,6 +1094,8 @@ test_expect_success 'submodule update --quiet passes quietness to fetch with a s
> ) &&
> git clone super4 super5 &&
> (cd super5 &&
> + # This test var can mess with the stderr output checked in this test.
> + GIT_TEST_FULL_NAME_HASH=0 \
> git submodule update --quiet --init --depth=1 submodule3 >out 2>err &&
Nit: This line should now be indented.
> test_must_be_empty out &&
> test_must_be_empty err
> diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh
> index fc2cc9d37be..e3787bacdad 100755
> --- a/t/t7700-repack.sh
> +++ b/t/t7700-repack.sh
> @@ -309,6 +309,9 @@ test_expect_success 'no bitmaps created if .keep files present' '
> keep=${pack%.pack}.keep &&
> test_when_finished "rm -f \"\$keep\"" &&
> >"$keep" &&
> +
> + # Disable --full-name-hash test due to stderr comparison.
> + GIT_TEST_FULL_NAME_HASH=0 \
> git -C bare.git repack -ad 2>stderr &&
Same here.
> test_must_be_empty stderr &&
> find bare.git/objects/pack/ -type f -name "*.bitmap" >actual &&
> @@ -320,6 +323,9 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '
> blob=$(test-tool genrandom big $((1024*1024)) |
> git -C bare.git hash-object -w --stdin) &&
> git -C bare.git update-ref refs/tags/big $blob &&
> +
> + # Disable --full-name-hash test due to stderr comparison.
> + GIT_TEST_FULL_NAME_HASH=0 \
> git -C bare.git repack -ad 2>stderr &&
And here.
Patrick |
|
On the Git mailing list, Patrick Steinhardt wrote (reply to this), regarding c14ef68 (outdated): On Tue, Nov 05, 2024 at 03:05:05AM +0000, Derrick Stolee via GitGitGadget wrote:
> These tests demonstrate that it is important to be careful about which
> cases are best for using the --full-name-hash option.
Is it possible to give general guidelines in our documentation that
guides the end user for when to use the option and when not to use it?
And if the answer is yes, is it possible for us to figure out at runtime
what the current scenario is via some heuristics and enable the option
automatically in a subset of cases?
Patrick |
|
This patch series was integrated into seen via git@b40cf2a. |
|
This patch series was integrated into seen via git@1f9eff1. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a recreation of the topic in [1] that was closed. (I force-pushed my branch and GitHub won't let me reopen the PR for GitGitGadget to create this as v3.)
[1] https://lore.kernel.org/git/[email protected]/
I've been focused recently on understanding and mitigating the growth of a few internal repositories. Some of these are growing much larger than expected for the number of contributors, and there are multiple aspects to why this growth is so large.
This is part of the RFC I submitted [2] involving the path-walk API, though this doesn't use the path-walk API directly. In full repack cases, it seems that the --full-name-hash option gets nearly as good compression as the --path-walk option introduced in that series. I continue to work on that feature as well, so we can review it after this series is complete.
[2] https://lore.kernel.org/git/[email protected]/
The main issue plaguing these repositories is that deltas are not being computed against objects that appear at the same path. While the size of these files at tip is one aspect of growth that would prevent this issue, the changes to these files are reasonable and should result in good delta compression. However, Git is not discovering the connections across different versions of the same file.
One way to find some improvement in these repositories is to increase the window size, which was an initial indicator that the delta compression could be improved, but was not a clear indicator. After some digging (and prototyping some analysis tools) the main discovery was that the current name-hash algorithm only considers the last 16 characters in the path name and has some naturally-occurring collisions within that scope.
This series introduces a new name-hash algorithm, but does not replace the existing one. There are cases, such as packing a single snapshot of a repository, where the existing algorithm outperforms the new one.
However, my findings show that when a repository has many versions of files at the same path (and especially when there are many name-hash collisions) then there are significant gains to be made using the new algorithm.
(This table is updated in v2 with even more private examples that were found while communicating findings internally.)
Since there has been some interest in these repacking results since [3] was published, I'll say that "Repo D" is the one mentioned in that post. The --path-walk repack further improves the results to under 4GB.
[3] https://www.jonathancreamer.com/how-we-shrunk-our-git-repo-size-by-94-percent/
I include Repo G here as an example where the improvement is less drastic, since this repo does not demonstrate a very high rate of name-hash collisions; the collisions that exist seem to be in paths that are not changed very often. Thus, the standard name-hash algorithm is nearly as effective in these full repacks.
The main change in this series is in patch 1, which adds the algorithm and the option to 'git pack-objects' and 'git repack'. The remaining patches are focused on creating more evidence around the value of the new name-hash algorithm and its effects on the packfiles created with it.
I will also try to make clear that I've been focused on client-side performance and size concerns. Based on discussions in v1, it appears that the following is true:
This feature is completely orthogonal to delta islands.
Changing the name-hash function can lead to compatibility issues with
.bitmapfiles, as they store a name-hash value. Without augmenting the data structure to indicate which name-hash value was used at write time, the full-name-hash values should not be stored in the.bitmapfiles or used when reading.bitmapfiles and other objects. Thus, the full-name-hash is marked as incompatible with bitmaps for now. (In this version, the 'git pack-objects' process will not fail but will prefer the standard name hash algorithm.)The --path-walk option is likely incompatible with the delta-islands feature without significant work, so this suggests that the --full-name-hash is a better long-term solution for servers. This would still require the .bitmap modifications to make it work, but it's a smaller leap to get there.
Thanks, -Stolee
UPDATES SINCE PREVIOUS VERSION
This is recreated after pursuing the path-walk API with 'git pack-objects --path-walk' [4] and also cutting the series into fewer patches and only including the path-walk API in [5].
[4] https://lore.kernel.org/git/[email protected]/
[5] https://lore.kernel.org/git/[email protected]/
This re-roll has these changes:
The performance script is updated to include a simulation of a depth-1 shallow clone.
The performance numbers in the commit messages are updated with the latest results and include more example repositories.
Small style nits, especially around whitespace.
To prevent the --full-name-hash option from having compatibility issues with .bitmap files, the previous version would cause the 'git pack-objects' process to fail. Now, the process succeeds, but the --full-name-hash option is disabled with a warning.
These patches are rebased on a recent copy of 'master' and thus there are some new tests that need adjustment to work with GIT_TEST_FULL_NAME_HASH=1. They all care about an exact match with stderr and the warning to disable --full-name-hash causes a mismatch.
Here is the range-diff since the previous version:
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: [email protected]
cc: Jonathan Tan [email protected]