Re: Curl's network performance is slower than IDM.

On Thu, 18 Jun 2020, Hongyi Zhao wrote:

(I've removed curl-users as a receiver, let's stick to the curl-library list.)

> No, wget and curl give similar results. OTOH, wget has fewer features than
> curl, say, socks5 proxy supporting, so, I rarely use it.

When we compare transfer speeds, I don't see how its too relevant with all
those other features.

When a widely used tool with a completely separate HTTP stack implementation
gets similar transfer speeds as curl, it suggests to me that at least curl is
not doing something completely bonkers.

The question is then of course still what IDM does that makes it that much
faster or perhaps what curl should do to improve. But without more details and
knowledge of IDM's set of tricks, I'm not sure we will achieve that right now.

>> can use with the cmdline tool if you do -Z transfers.
>
> When I try with this option, it doesn't give any boosting for speed,

-Z does multiple transfers in parallel, if that is faster or not will of
course vary a lot depending on many factors. I was just highlighting that the
command line tool can do parallel transfers (using the multi interface).

> But, the PyIDM's developer told me the following point of view on the
> *TCP connections* discussed here:
> https://github.com/pyIDM/PyIDM/issues/107#issuecomment-645273695.

I don't think you're helping this dicusssion by relaying a separate
discussion from another forum. At least it just makes me go even more confused
about what we're actually talking about.

To me, the interesting discussion here is if curl does something that makes it
do its single-connection transfers slower than it otherwise could. Can we
stick to that?

> I don’t think you can open multiple tcp connections asking for the
> same chunk range of data

Yes you can certainly do that with libcurl.

> For example curl-multi can run multiple easy handles from one thread but
> still every handle will request different chunk range from server then your
> application will concatenate these chunks into one file

libcurl's multi API is an *API* - it leaves the program logic to the
application. If the application wants to act as described above, then do that.
If you want it to do something else, then do that instead.

> I believe he meant IDM might use multi-connections silently ignoring
> your settings

Right, which is what I asked about in my initial response because if it does,
that could indeed explain the transfer speed differences and there's no
mystery anymore.

> The only way to make sure it can’t do this is to download a file from a
> server which doesn’t support “range requests” which the server doesn’t have
> ‘Accept-Ranges: bytes’ in response headers

You could also just inspect the network traffic with wireshark.

-- 
  / daniel.haxx.se | Commercial curl support up to 24x7 is available!
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2020-06-18