curl / Mailing Lists / curl-library / Single Mail
Buy commercial curl support from WolfSSL. We help you work out your issues, debug your libcurl applications, use the API, port to new platforms, add new features and more. With a team lead by the curl founder himself.

How to do optimal FTP upload for multiple files?

From: Taras Kushnir via curl-library <>
Date: Tue, 16 Jul 2019 17:41:01 +0000

Hello everybody

I’m developing a cross-platform app that among other things uploads N files to M FTP servers (same batch of files to every FTP server). So far I have K threads where each of them in a “for" loop uploads a file from the batch (each thread creates an “easy" curl handle, configures it and does curl_easy_perform()). libcurl code in my for-loop is mostly “stolen” from official FTP upload examples (example with ftp retry).

The reason why I’m asking this is the following: I got reports from users that my app “uploads very slowly” so I started this investigation. Default upload code heavily (like x3-x4 on my machine) underperforms FileZilla upload and commandline “ftp" utility even for 1 file to 1 server (that eliminates my theoretical threading implementation bugs etc.).

If, when doing FTP upload for 1 file for 1 server, I’m setting CURLOPT_UPLOAD_BUFFERSIZE, upload speed increases for some users of my app to “almost satisfactory”. However, same setting cause FTP response timeouts on _every_ transfer for other users (who, if I understand correctly, have much lower bandwidth). On my machine (macbook pro 2015) and bandwidth I feel almost no difference when playing with UPLOAD_BUFFERSIZE.

Therefore comes a question: is there any idiomatic way of using libcurl for such cases that will solve upload speed issues for users both with big and small bandwidth?

Thank you,
Taras Kushnir

Received on 2019-07-16