curl / Mailing Lists / curl-library / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

Re: How to do optimal FTP upload for multiple files?

From: Taras Kushnir via curl-library <curl-library_at_cool.haxx.se>
Date: Fri, 19 Jul 2019 19:09:08 +0000

Hi Daniel

From what I saw, timeout messages were always together with "control connection looks dead". Regarding upload time, it's hard to answer this question, since all uploads failed. I attached a "typical" (from that user) log where you can see time to first failure and following failures on your own. Log contains only upload thread logs with curl tracing.

I didn't investigate curl commandline tool upload time on my machine since for me priority #1 is to fix upload errors of users of my app (since they will just stop using it for upload). I will do so right after finding some solution to this problem.

I understand about NAT/firewall/etc. timeout, but the thing is that libcurl 7.57 works for them right at the same time (of course, it could be a very big coincidence). Currently my only option to be "on the safe side" is to downgrade back, however, I'm still hesitating.

So far I will try to disable UPLOAD_BUFFERSIZE setting and increase CURLOPT_FTP_RESPONSE_TIMEOUT and see if it will help.

Best regards,
Taras Kushnir

________________________________
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Friday, July 19, 2019 10:33 AM
To: Taras Kushnir
Cc: curl-library_at_cool.haxx.se; Daniel JeliŮski
Subject: Re: How to do optimal FTP upload for multiple files?

On Fri, 19 Jul 2019, Taras Kushnir wrote:

> On the same machine at the same time, app with libcurl 7.57 uploads without
> timeout and app with libcurl 7.65 and UPLOAD_BUFFERSIZE change uploads with
> timeout.

And how long did the uploads take in both these cases? Was the timeout always
the control connection one Daniel showed?

> Since Iím not a big expert in libcurl (thatís why Iím seeking your help in
> this mailing list) of course I thought that UPLOAD_BUFFERSIZE is the problem
> (because thatís all that I personally changed).

I understand that, but it is important to keep an eye on *all* the details and
facts so that we don't accidentally focus on the wrong things.

If changing UPLOAD_BUFFERSIZE actually makes the upload slower, it can of
course explain why the control connection would time out if that slowness
pushes the transfer time over the "magic threshold" (which to us happens to be
unknown). It of course doesn't at all explain *why* changing that size makes
the upload slower, because as I've said already, it really shouldn't.

If you do FTP upload with the curl command line tool, does it also show the
same kind of "slowness" ?

> Of course, I can just try to increase the timeout in my app and see if it
> helps, but at least itís important (and interesting) to understand ďwhy?Ē.

The timeout in question here is most likely in a router, NAT or firewall
between the client and the server so there's no timeout in the app that you
can change for this!

--
  / daniel.haxx.se | Get the best commercial curl support there is - from me
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/


-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html

Received on 2019-07-19