Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multipart Upload Fails at final step. #431

Open
USMortality opened this issue Apr 18, 2023 · 0 comments
Open

Multipart Upload Fails at final step. #431

USMortality opened this issue Apr 18, 2023 · 0 comments

Comments

@USMortality
Copy link

Multipart upload fails, see error below, when setting multi_part to false, it works, but gives a warning due to large file size...

I'm using MinIO behind Cloudflare. Upload via mc copy command works fine, so its prob. a problem with this plugin. Thx

## load package
library("aws.s3")

## code goes here
    put_object(
      file = local_file_name,
      object = file_name,
      bucket = data_bucket,
      multipart = TRUE,
      show_progress = TRUE
    )

## session info for your system
> sessionInfo()
R version 4.2.3 (2023-03-15)
Platform: x86_64-apple-darwin22.3.0 (64-bit)
Running under: macOS Ventura 13.3.1

Matrix products: default
BLAS:   /usr/local/Cellar/openblas/0.3.23/lib/libopenblasp-r0.3.23.dylib
LAPACK: /usr/local/Cellar/r/4.2.3/lib/R/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] imputeTS_3.3     rvest_1.0.3      rtweet_1.1.0     ggpubr_0.6.0    
 [5] feasts_0.3.1     fable_0.3.3      fabletools_0.3.3 clock_0.6.1     
 [9] scales_1.2.1     quantmod_0.4.22  TTR_0.24.3       xts_0.12.2      
[13] zoo_1.8-11       aws.s3_0.3.21    tsibble_1.1.3    httr_1.4.5      
[17] jsonlite_1.8.4   readxl_1.4.2     lubridate_1.9.2  forcats_1.0.0   
[21] stringr_1.5.0    dplyr_1.1.1      purrr_1.0.1      readr_2.1.4     
[25] tidyr_1.3.0      tibble_3.2.1     ggplot2_3.4.2    tidyverse_2.0.0 

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.10               lattice_0.20-45          
 [3] digest_0.6.31             lmtest_0.9-40            
 [5] utf8_1.2.3                mime_0.12                
 [7] aws.signature_0.6.0       R6_2.5.1                 
 [9] cellranger_1.1.0          backports_1.4.1          
[11] pillar_1.8.1              rlang_1.1.0              
[13] curl_4.3.3                car_3.1-1                
[15] fracdiff_1.5-2            gridtext_0.1.5           
[17] munsell_0.5.0             broom_1.0.4              
[19] anytime_0.3.9             compiler_4.2.3           
[21] pkgconfig_2.0.3           base64enc_0.1-3          
[23] forecast_8.20             urca_1.3-3               
[25] ggtext_0.1.2              nnet_7.3-18              
[27] tidyselect_1.2.0          quadprog_1.5-8           
[29] fansi_1.0.4               tzdb_0.3.0               
[31] withr_2.5.0               grid_4.2.3               
[33] distributional_0.3.1.9000 nlme_3.1-162             
[35] gtable_0.3.1              lifecycle_1.0.3          
[37] magrittr_2.0.3            cli_3.6.0                
[39] stringi_1.7.12            carData_3.0-5            
[41] farver_2.1.1              ggsignif_0.6.4           
[43] tseries_0.10-52           timeDate_4022.108        
[45] xml2_1.3.3                ellipsis_0.3.2           
[47] generics_0.1.3            vctrs_0.6.1              
[49] tools_4.2.3               glue_1.6.2               
[51] hms_1.1.2                 abind_1.4-5              
[53] parallel_4.2.3            timechange_0.2.0         
[55] colorspace_2.1-0          stinepack_1.4            
[57] rstatix_0.7.2            

Log:

> put_object(
+       file = local_file_name,
+       object = file_name,
+       bucket = data_bucket,
+       multipart = TRUE,
+       show_progress = TRUE
+     )
Uploading part 1 of 2-part upload
  |======================================================================| 100%
Uploading part 2 of 2-part upload
  |======================================================================| 100%
Completing multi-part upload
List of 10
 $ url        : chr "https://s3.mortality.watch/data/mortality/world_weekly_104w_sma.csv"
 $ status_code: int 400
 $ headers    :List of 11
  ..$ date           : chr "Tue, 18 Apr 2023 03:43:32 GMT"
  ..$ content-type   : chr "text/plain"
  ..$ content-length : chr "341"
  ..$ accept-ranges  : chr "bytes"
  ..$ vary           : chr "Origin"
  ..$ cf-cache-status: chr "DYNAMIC"
  ..$ report-to      : chr "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=hlWyUJpFuQAk0kDwNS%2BurQ0Qm3Iwe1Msj"| __truncated__
  ..$ nel            : chr "{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}"
  ..$ server         : chr "cloudflare"
  ..$ cf-ray         : chr "7b99e7763e0d102d-LAX"
  ..$ alt-svc        : chr "h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"
  ..- attr(*, "class")= chr [1:2] "insensitive" "list"
 $ all_headers:List of 1
  ..$ :List of 3
  .. ..$ status : int 400
  .. ..$ version: chr "HTTP/2"
  .. ..$ headers:List of 11
  .. .. ..$ date           : chr "Tue, 18 Apr 2023 03:43:32 GMT"
  .. .. ..$ content-type   : chr "text/plain"
  .. .. ..$ content-length : chr "341"
  .. .. ..$ accept-ranges  : chr "bytes"
  .. .. ..$ vary           : chr "Origin"
  .. .. ..$ cf-cache-status: chr "DYNAMIC"
  .. .. ..$ report-to      : chr "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=hlWyUJpFuQAk0kDwNS%2BurQ0Qm3Iwe1Msj"| __truncated__
  .. .. ..$ nel            : chr "{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}"
  .. .. ..$ server         : chr "cloudflare"
  .. .. ..$ cf-ray         : chr "7b99e7763e0d102d-LAX"
  .. .. ..$ alt-svc        : chr "h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"
  .. .. ..- attr(*, "class")= chr [1:2] "insensitive" "list"
 $ cookies    :'data.frame':    0 obs. of  7 variables:
  ..$ domain    : logi(0) 
  ..$ flag      : logi(0) 
  ..$ path      : logi(0) 
  ..$ secure    : logi(0) 
  ..$ expiration: 'POSIXct' num(0) 
  ..$ name      : logi(0) 
  ..$ value     : logi(0) 
 $ content    : raw [1:341] 3c 3f 78 6d ...
 $ date       : POSIXct[1:1], format: "2023-04-18 03:43:32"
 $ times      : Named num [1:6] 0 0.000056 0 0.000218 0.000222 ...
  ..- attr(*, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ...
 $ request    :List of 7
  ..$ method    : chr "POST"
  ..$ url       : chr "https://s3.mortality.watch/data/mortality/world_weekly_104w_sma.csv"
  ..$ headers   : Named chr [1:6] "application/json, text/xml, application/xml, */*" "text/plain" "224" "20230418T034332Z" ...
  .. ..- attr(*, "names")= chr [1:6] "Accept" "Content-Type" "Content-Length" "x-amz-date" ...
  ..$ fields    : NULL
  ..$ options   :List of 4
  .. ..$ useragent          : chr "libcurl/7.87.0 r-curl/4.3.3 httr/1.4.5"
  .. ..$ post               : logi TRUE
  .. ..$ readfunction       :function (nbytes, ...)  
  .. ..$ postfieldsize_large: num 224
  ..$ auth_token: NULL
  ..$ output    : list()
  .. ..- attr(*, "class")= chr [1:2] "write_memory" "write_function"
  ..- attr(*, "class")= chr "request"
 $ handle     :Class 'curl_handle' <externalptr> 
 - attr(*, "class")= chr "aws_error"
 - attr(*, "headers")=List of 11
  ..$ date           : chr "Tue, 18 Apr 2023 03:43:32 GMT"
  ..$ content-type   : chr "text/plain"
  ..$ content-length : chr "341"
  ..$ accept-ranges  : chr "bytes"
  ..$ vary           : chr "Origin"
  ..$ cf-cache-status: chr "DYNAMIC"
  ..$ report-to      : chr "{\"endpoints\":[{\"url\":\"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=hlWyUJpFuQAk0kDwNS%2BurQ0Qm3Iwe1Msj"| __truncated__
  ..$ nel            : chr "{\"success_fraction\":0,\"report_to\":\"cf-nel\",\"max_age\":604800}"
  ..$ server         : chr "cloudflare"
  ..$ cf-ray         : chr "7b99e7763e0d102d-LAX"
  ..$ alt-svc        : chr "h3=\":443\"; ma=86400, h3-29=\":443\"; ma=86400"
  ..- attr(*, "class")= chr [1:2] "insensitive" "list"
 - attr(*, "request_canonical")= chr "POST\n/data/mortality/world_weekly_104w_sma.csv\n\ncontent-length:224\nhost:s3.mortality.watch\nx-amz-date:2023"| __truncated__
 - attr(*, "request_string_to_sign")= chr "AWS4-HMAC-SHA256\n20230418T034332Z\n20230418/s3/s3/aws4_request\n73802dd93b74ca21116b9115649e353d454512adddd4f6"| __truncated__
 - attr(*, "request_signature")= chr "AWS4-HMAC-SHA256 Credential=minio/20230418/s3/s3/aws4_request,SignedHeaders=content-length;host;x-amz-date,Sign"| __truncated__
NULL
Error in parse_aws_s3_response(r, Sig, verbose = verbose) : 
  Bad Request (HTTP 400).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant