cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: curl with sendfile

From: Brian McQueen <mcqueenorama_at_gmail.com>
Date: Mon, 10 Mar 2008 13:30:14 -0700

I'm doing many huge downloads via libcurl and sendfile would be useful
in that instance. Its within a daemon and its doing many massive
downloads all the time. I'm seeing way too much disk IO, which I
believe is due to the comparatively small write buffer, we typically
use a 16mb buffer before trying to write. The max buffer size
defaults to 16k in libcurl I could recompile with a larger write
buffer, or make my own buffer, but if I could use sendfile it would be
best. I can see it would be hard to fit it in with the rest of
libcurl though.

On Mon, Mar 10, 2008 at 12:12 PM, Jamie Lokier <jamie_at_shareable.org> wrote:
>
> Brian McQueen wrote:
> > Does libcurl use linux's high performance sendfile routine when
> > appropriate? I don't see it as fitting into the write callback
> > scheme. The write callback scheme seems to use memory buffers in user
> > space for handling the writes, but sendfile skips user space and sends
> > directly from one fd to another fd.
>
> Sendfile is most useful for servers, as they typically handle a much
> higher sending load than clients. It's quite unusual to have a
> machine with hundreds of simultaneous client uploads, whereas
> thousands simultanous downloads from a server is not uncommon.
> Sendfile saves memory as well as copying time in this scenario,
> because all the simultaneous downloads share page cache memory and TCP
> socket buffers.
>
> Since curl isn't a server, you'd only use sendfile when uploading
> files to one. Would it really make a noticable difference to any
> client application to do this?
>
> -- Jamie
>
Received on 2008-03-10