Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Feature-Request: sendfile(2)
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Eric Wong via curl-library <curl-library_at_cool.haxx.se>
Date: Thu, 25 Feb 2021 23:52:16 +0000
Emil Engler via curl-library <curl-library_at_cool.haxx.se> wrote:
> Hello curl-library,
> I recently thought about implementing the syscall sendfile(2) into
> libcurl to speed up downloads to a file. For those of you who don't
> know what it is:
"downloads to a file"? Meaning socket => regular file?
I hope I understand you correctly, since...
> sendfile() copies data between one file descriptor and another.
> Because this copying is done within the kernel, sendfile() is more
> efficient than the combination of read(2) and write(2),
> which would require transferring data to and from user space.
>
> From https://linux.die.net/man/2/sendfile
Which says:
The in_fd argument must correspond to a file which supports
mmap(2)-like operations (i.e., it cannot be a socket).
So curl, as an HTTP/FTP/etc. client would only benefit using
sendfile to upload (e.g. curl -T $FILE), not download.
> However implementing this syscall also raises some problems, including
> limited support. AFAIK only Linux and FreeBSD support this syscall
> (maybe even macOS as it inherited a lot from FreeBSD).
> The lack of standardization also leads to the problem that the syscall
> might differ on the platforms where it is supported.
>
> So my idea would be to implement that feature and disable it for now
> by using a configure option. Has someone already pitched the idea
> in the past? If so, I would be happy for a link to this thread :-)
Based on my past experiences on the server side, sendfile was
often a source of bugs, especially on BSDs. It's gotten better
over the years, but it's also less relevant now that TLS is
popular.
Maybe it still makes sense with --proxy talking to localhost
(e.g. Tor) or on private networks (VPNs, LANs), but only for
uploads...
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-02-26
Date: Thu, 25 Feb 2021 23:52:16 +0000
Emil Engler via curl-library <curl-library_at_cool.haxx.se> wrote:
> Hello curl-library,
> I recently thought about implementing the syscall sendfile(2) into
> libcurl to speed up downloads to a file. For those of you who don't
> know what it is:
"downloads to a file"? Meaning socket => regular file?
I hope I understand you correctly, since...
> sendfile() copies data between one file descriptor and another.
> Because this copying is done within the kernel, sendfile() is more
> efficient than the combination of read(2) and write(2),
> which would require transferring data to and from user space.
>
> From https://linux.die.net/man/2/sendfile
Which says:
The in_fd argument must correspond to a file which supports
mmap(2)-like operations (i.e., it cannot be a socket).
So curl, as an HTTP/FTP/etc. client would only benefit using
sendfile to upload (e.g. curl -T $FILE), not download.
> However implementing this syscall also raises some problems, including
> limited support. AFAIK only Linux and FreeBSD support this syscall
> (maybe even macOS as it inherited a lot from FreeBSD).
> The lack of standardization also leads to the problem that the syscall
> might differ on the platforms where it is supported.
>
> So my idea would be to implement that feature and disable it for now
> by using a configure option. Has someone already pitched the idea
> in the past? If so, I would be happy for a link to this thread :-)
Based on my past experiences on the server side, sendfile was
often a source of bugs, especially on BSDs. It's gotten better
over the years, but it's also less relevant now that TLS is
popular.
Maybe it still makes sense with --proxy talking to localhost
(e.g. Tor) or on private networks (VPNs, LANs), but only for
uploads...
-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.se/mail/etiquette.html
Received on 2021-02-26