curl / Mailing Lists / curl-library / Single Mail

curl-library

Re: Ranged PUTs, Content-Range, and Content-Length

From: Ray Satiro via curl-library <curl-library_at_cool.haxx.se>
Date: Sun, 5 May 2019 03:48:39 -0400

On 5/4/2019 4:04 AM, Ray Satiro wrote:
>
> Also as you noted CURLOPT_RANGE does not work like
> CURLOPT_RESUME_FROM, in that the former does not have any effect on
> Content-Length. Content-Length is determined by CURLOPT_INFILESIZE
> minus CURLOPT_RESUME_FROM. Yet if you specify CURLOPT_RESUME_FROM then
> CURLOPT_RANGE is ignored, and only CURLOPT_RESUME_FROM changes the
> starting offset of the file.
>
> If you use CURLOPT_RANGE and CURLOPT_INFILESIZE without
> CURLOPT_RESUME_FROM then libcurl ends up in this block [1]:
>
>       else {
>         /* Range was selected and then we just pass the incoming range and
>            append total size */
>         conn->allocptr.rangeline =
>           aprintf("Content-Range: bytes %s/%" CURL_FORMAT_CURL_OFF_T
> "\r\n",
>                   data->state.range, data->state.infilesize);
>       }
>
> But I think that will not work correctly since Content-Length is also
> CURLOPT_INFILESIZE. IIRC it is not good to set the Content-Length
> header on your own because then internally libcurl doesn't change the
> length to 0 when it does authentication before the PUT. Therefore I
> think the best thing to do here is do not use CURLOPT_RANGE (at least
> until its use is clarified) and set the range header yourself, as I
> suggested. I will try experimenting with it some later.
>

I played around with it and the idea I proposed initially to add in your
offset to READFUNCTION and SEEKFUNCTION is not needed since you can use
libcurl to resume from an arbitrary position. You should still implement
a regular seek function and (if on windows) a regular read function.

Instead of setting INFILESIZE to the file's actual size you would set it
to the resume + part size (or the file's actual size if it goes over).
Here it is boiled down:

  curl_off_t end = (filesize < (resume + partsize) ? filesize : (resume
+ partsize));
  curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, end);
  curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, resume);

  char range_header[256];
  sprintf(range_header, "Content-Range: bytes %" CURL_FORMAT_CURL_OFF_T "-"
                        "%" CURL_FORMAT_CURL_OFF_T "/*",
                        resume, end - 1);
  struct curl_slist *header_list = NULL;
  header_list = curl_slist_append(NULL, range_header);
  curl_easy_setopt(curl, CURLOPT_HTTPHEADER, header_list);
 
  res = curl_easy_perform(curl);

  if(end == filesize)
    done;

For example say your input file size is 6 and your part size is 4 then
there would be two transfers.

Content-Range: bytes 0-3/*
Content-Length: 4

Content-Range: bytes 4-5/*
Content-Length: 2

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2019-05-05