Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deflate decompression #1927

Merged
merged 5 commits into from
Aug 8, 2023
Merged

Conversation

cipherbrain
Copy link
Contributor

This solve issue 1462 by using 'deflate' algo instead of 'zlib' algo to decompress responses with the 'deflate' Content-Encoding.

The full series of tests (cargo test) still works after this.

My code (see isseu #1462) now works, where I was getting " deflate decompression error" before my patch.

Copy link
Owner

@seanmonstar seanmonstar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me, thanks!

@cipherbrain
Copy link
Contributor Author

You're welcome.

But the CI doesn't seem to like the patch, and rejected it. I'm not sure if it's my patch, some unrelated error, or how I could reproduce the errors on my machine to have a chance to solve them.

Doing "cargo test --all-features" doesn't work even without my patch, and "cargo test" works with my patch.

If you have some idea on what I could do to have it pass the tests, I'm willing to try that.

Anyways, thanks for your work on this great library!

@seanmonstar
Copy link
Owner

Does running cargo test --feature deflate --test deflate work, with your patch? It looks like this is the failure:

deflate_single_byte_chunks' panicked at 'text: reqwest::Error { kind: Decode, source: Custom { kind: Other, error: DecompressError(General { msg: None }) } }', tests/deflate.rs:149:33

@T-Sujeeban
Copy link
Contributor

The deflate test fails because it also uses Zlib instead of Deflate
The line 95 in tests/deflate.rs should be changed from
let mut encoder = libflate::zlib::Encoder::new(Vec::new()).unwrap();
to this
let mut encoder = libflate::deflate::Encoder::new(Vec::new());

@seanmonstar seanmonstar changed the title Solve issue #1462 Fix deflate decompression Aug 8, 2023
@seanmonstar seanmonstar merged commit 8396233 into seanmonstar:master Aug 8, 2023
31 checks passed
@seanmonstar
Copy link
Owner

Thanks for finding that!

kodiakhq bot pushed a commit to pdylanross/fatigue that referenced this pull request Aug 22, 2023
Bumps reqwest from 0.11.18 to 0.11.19.

Release notes
Sourced from reqwest's releases.

v0.11.19
What's Changed

Add ClientBuilder::http1_ignore_invalid_headers_in_responses() option.
Add ClientBuilder::http1_allow_spaces_after_header_name_in_responses() option.
Add support for ALL_PROXY environment variable.
Add support for use_preconfigured_tls when combined with HTTP/3.
Fix deflate decompression from using the zlib decoder.
Fix Response::{text, text_with_charset}() to strip BOM characters.
Fix a panic when HTTP/3 is used if UDP isn't able to connect.
Fix some dependencies for HTTP/3.
Increase MSRV to 1.63.

New Contributors

@​nyurik made their first contribution in seanmonstar/reqwest#1849
@​smndtrl made their first contribution in seanmonstar/reqwest#1856
@​attila-lin made their first contribution in seanmonstar/reqwest#1869
@​ollyswanson made their first contribution in seanmonstar/reqwest#1898
@​VivekPanyam made their first contribution in seanmonstar/reqwest#1903
@​bouzuya made their first contribution in seanmonstar/reqwest#1922
@​cipherbrain made their first contribution in seanmonstar/reqwest#1927
@​T-Sujeeban made their first contribution in seanmonstar/reqwest#1926
@​eric-seppanen made their first contribution in seanmonstar/reqwest#1852




Changelog
Sourced from reqwest's changelog.

v0.11.19

Add ClientBuilder::http1_ignore_invalid_headers_in_responses() option.
Add ClientBuilder::http1_allow_spaces_after_header_name_in_responses() option.
Add support for ALL_PROXY environment variable.
Add support for use_preconfigured_tls when combined with HTTP/3.
Fix deflate decompression from using the zlib decoder.
Fix Response::{text, text_with_charset}() to strip BOM characters.
Fix a panic when HTTP/3 is used if UDP isn't able to connect.
Fix some dependencies for HTTP/3.
Increase MSRV to 1.63.




Commits

8b49fc9 v0.11.19
87ff5d9 improve error message if incompabitle Identity with selected backend (#1852)
42f57b4 Fix panic in building h3 client when udp is forbidden (#1945)
4aa8516 msrv: bump to 1.63 (#1947)
1f6c2cf Add ClientBuilder::http1_allow_spaces_after_header_name_in_responses() (#1932)
6f07b9f Add ClientBuilder::http1_ignore_invalid_headers_in_responses() (#1926)
8396233 Fix deflate decompression (#1927)
b0c07a2 Bump rustls to v0.21.6 (#1928)
99bbae6 Bump webpki-roots to v0.25 (#1922)
61b1b2b Bump wasm-streams dependency to 0.3 (#1903)
Additional commits viewable in compare view




Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR
@dependabot recreate will recreate this PR, overwriting any edits that have been made to it
@dependabot merge will merge this PR after your CI passes on it
@dependabot squash and merge will squash and merge this PR after your CI passes on it
@dependabot cancel merge will cancel a previously requested merge and block automerging
@dependabot reopen will reopen this PR if it is closed
@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
@jongiddy
Copy link

Is this correct behavior? RFC9110 says deflate coding is a deflate stream inside a zlib container, with a note that some non-conformant implementations use plain deflate.

@seanmonstar
Copy link
Owner

Ah, I hadn't checked that out, seems the proper fix is more involved. Would it be to use zlib, and if that fails then try deflate?

@inikulin
Copy link

Would it be to use zlib, and if that fails then try deflate?

Yes, this is usually how browsers handle it

seanmonstar added a commit that referenced this pull request Aug 23, 2023
seanmonstar added a commit that referenced this pull request Aug 23, 2023
Nutomic pushed a commit to Nutomic/reqwest that referenced this pull request Nov 7, 2024
Nutomic pushed a commit to Nutomic/reqwest that referenced this pull request Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants