Re: How to do optimal FTP upload for multiple files?

Hi Daniel,

Thanks for getting back to me.

I use latest curl a of today: 7.65.1, built on Windows and macOS from sources.

Slowness was reported by few users on Windows 10 x64. Slowness (and, in other case, timeouts) were reported against couple of totally different FTP servers around the globe. At least 5 different (from my user's logs) servers run by different companies, so unfortunately there's no way to blame them here. Only my use of libcurl.

Cannot say about RTT to servers from user's perspective, I don't have this information ("it works on my machine"). I only have my users' bugreports.

There're 2 sample logs (with libcurl tracing) attached, one app uses libcurl 7.57 and does not have UPLOAD_BUFFERSIZE tuning (the one which is successfull). Another log from libcurl 7.65.1 has UPLOAD_BUFFERSIZE set to 2MB (also tried with 1MB and 512KB ) and decreased by half each time user gets timeout. Logs contain lots of noise (from my app), but they have full libcurl debug trace.

In majority of reported cases after lines "We are fully uploaded and fine" there's a timeout (after 10sec). Also for other FTP server we have timeout even before during upload. With exactly same configs first version does not fail.

"Almost satisfactory" means reported user experience when my app uploaded user's file at speeds previously experienced with FileZilla (reported by user) when I only tweaked UPLOAD_BUFFERSIZE and nothing else.

On my own machine UPLOAD_BUFFESIZE tuning does not make any difference like for those few users who reported issues. However, FTP upload with libcurl is significantly slower than by using FileZilla, for example. Not exactly sure if this is related to the user's timeouts problem, but I assume it somehow is.

Code that was used for uploading looks almost like this https://github.com/ribtoks/xpiks/blob/master/src/libxpks_stub/Connectivity/curlftpuploader.cpp#L105 Function uploadFile() however, is neglibibly different in reality (some refactoring, few easy_setopt() tunings etc.).

Thank you,
Taras

________________________________
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Tuesday, July 16, 2019 3:50 PM
To: Taras Kushnir via curl-library
Cc: Taras Kushnir
Subject: Re: How to do optimal FTP upload for multiple files?

On Tue, 16 Jul 2019, Taras Kushnir via curl-library wrote:

> The reason why I’m asking this is the following: I got reports from users
> that my app “uploads very slowly” so I started this investigation. Default
> upload code heavily (like x3-x4 on my machine) underperforms FileZilla
> upload and commandline “ftp" utility even for 1 file to 1 server (that
> eliminates my theoretical threading implementation bugs etc.).

I've not seen/heard this before.

What libcurl version on what platform? Have you seen this "slowness" on more
than one version/platform?

Is this against any or particular servers?

When you do these "slow" uploads, what's the RTT to server, roughly?

> If, when doing FTP upload for 1 file for 1 server, I’m setting
> CURLOPT_UPLOAD_BUFFERSIZE, upload speed increases for some users of my app
> to “almost satisfactory”.

When you set it to what value? What does "almost satisfactory" mean in terms
how it compares to other tools then?

> However, same setting cause FTP response timeouts on _every_ transfer for
> other users (who, if I understand correctly, have much lower bandwidth).

That sounds really weird and would rather imply a (separate) bug.

> On my machine (macbook pro 2015) and bandwidth I feel almost no difference
> when playing with UPLOAD_BUFFERSIZE.

In my own tests, UPLOAD_BUFFERSIZE mostly make a difference when you have near
infinite bandwidth, like uploading on localhost. (And when when using SFTP,
but I assume we're not talking about that here.)

> Therefore comes a question: is there any idiomatic way of using libcurl for
> such cases that will solve upload speed issues for users both with big and
> small bandwidth?

Increasing buffer size shouldn't hurt transfer speeds for any users!

--
  / daniel.haxx.se | Get the best commercial curl support there is - from me
                   | Private help, bug fixes, support, ports, new features
                   | https://www.wolfssl.com/contact/

-------------------------------------------------------------------
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html