Menu

#1188 slow upload on Windows

closed-fixed
performance (4)
5
2015-04-15
2013-02-04
No

my colleague experienced low ftp upload speed with curl(7.26, 7.28.1). ~500kb/s
(win client to linux ftp server, both located in different countries so connection was not so ideal).
but at the same time upload speed through ordinary windows client(FileZilla) was much higher ~6mb/s.
finally we had tested ftp upload with farmanager which has ability to tune IO buffer size. by using 16k buffer we achieve same 500kb/s limit and than we specify 256kb IO buffer then speed goes up to ~6mb/s

curl upload speed is very sensitive to CURL_MAX_WRITE_SIZE. default 16kb is not enough. i have made custom curl build with increased buffer size upto 256kb and finally problem was resolved.

may be the root of problem is not in this value and only some internal upload pipeline should use increased value.

Discussion

  • Daniel Stenberg

    Daniel Stenberg - 2013-02-04

    I'd ask you to please read up on the recent discussion on the curl-library list http://curl.haxx.se/mail/lib-2013-02/0009.html and contribute there.

     
  • Daniel Stenberg

    Daniel Stenberg - 2013-02-04
    • labels: --> performance
    • summary: slow upload --> slow upload on Windows
    • milestone: -->
     
  • Andrew Kurushin

    Andrew Kurushin - 2013-02-05

    i have made following changes according to the thread discussion:
    [---]
    Index: connect.c
    ===================================================================
    --- connect.c (revision)
    +++ connect.c (working copy)
    @@ -846,12 +846,32 @@
    Buffer Size

    */
    +#define DETECT_OS_NONE 0
    +#define DETECT_OS_PREVISTA 1
    +#define DETECT_OS_VISTA 2
    +
    void Curl_sndbufset(curl_socket_t sockfd)
    {
    int val = CURL_MAX_WRITE_SIZE + 32;
    int curval = 0;
    int curlen = sizeof(curval);

    • OSVERSIONINFO osver;
    • static int detectOsState = DETECT_OS_NONE;
      +
    • if (detectOsState == DETECT_OS_NONE) {
    • memset(&osver, 0, sizeof(osver));
    • osver.dwOSVersionInfoSize = sizeof(osver);
    • detectOsState = DETECT_OS_PREVISTA;
    • if(GetVersionEx(&osver)) {
    • if (osver.dwMajorVersion >= 6) {
    • detectOsState = DETECT_OS_VISTA;
    • }
    • }
    • }
    • if (detectOsState == DETECT_OS_VISTA) {
    • return;
    • }
      if(getsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (char *)&curval, &curlen) == 0)
      if(curval > val)
      return;
      [---]

    and made several test with 3 custom builds of
    curl 7.28.1 (i386-pc-win32) libcurl/7.28.1 OpenSSL/1.0.1c zlib/1.2.7 libssh2/1.4.3
    Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
    Features: AsynchDNS Largefile NTLM SPNEGO SSL libz

    by uploading 70mb file from windows2k8r2 client to linux ftp and sftp server

    ordinary(without sndbufset patch):
    sftp=~6239Kbytes/s
    ftp=~17.90Mbytes/s

    with sndbufset patch:
    sftp=~5629Kbytes/s
    ftp=~19.1Mbytes/s

    with CURL_MAX_WRITE_SIZE 491520:
    sftp=21.4Mbytes/s
    ftp=40.7Mbytes/s

     

    Last edit: Andrew Kurushin 2013-02-05
  • Andrew Kurushin

    Andrew Kurushin - 2013-02-05

    so from my point of view: sndbufset skipping does not fix upload speed problem

     

    Last edit: Andrew Kurushin 2013-02-05
  • Daniel Stenberg

    Daniel Stenberg - 2013-03-04
    • status: open --> open-confirmed
     
  • Daniel Stenberg

    Daniel Stenberg - 2013-03-04

    I concur that there's an issue here. I have however not seen a satisfactory solution being suggested so this issue isn't going anywhere...

     
  • Daniel Stenberg

    Daniel Stenberg - 2013-03-04
    • assigned_to: Daniel Stenberg
     
  • Christian Hägele

    I think we need some more individuals who are doing tests in real work scenarios. Testing in the 'lab' doesn't help much here.

    For the tests I made (FTP upload on a 1GBit-Link with a latency of 19ms) the provided patch made a significant difference. (Both sides where Windows 2008 R2. FTP-Server was MS-IIS)

    For some reason it didn't help much with other testers. I don't know why.

     
  • Daniel Stenberg

    Daniel Stenberg - 2013-03-05

    Thanks Christian for the input. I think we should adjust the patch to be suitable (not break non-windows builds) and apply it then as we don't know of any downsides but we know it might help performance.

     
    • Andrew Kurushin

      Andrew Kurushin - 2013-04-02

      This patch make changes inside USE_WINSOCK section. I guess it will not break non windows builds.

       
  • Daniel Stenberg

    Daniel Stenberg - 2013-04-01

    I can't adjust the patch myself. Unless I get help from someone on windows who can make sure it does the right thing on windows I will be forced to close this issue without treatment to be handled "later".

     
    • Andrew Kurushin

      Andrew Kurushin - 2013-04-02

      At least this aproach solve performance issue for Christian and not made problem worse for others.

       
  • Daniel Stenberg

    Daniel Stenberg - 2013-04-02

    which patch is "this" ?

     
    • Andrew Kurushin

      Andrew Kurushin - 2013-04-02

      please look at attachment from this comment
      https://sourceforge.net/p/curl/bugs/1188/#ca3a

       
  • Daniel Stenberg

    Daniel Stenberg - 2013-04-02

    Ah yes it does, thanks for pointing that out. But with the patch applied, the comment above the function is no longer entirely accurate. I would appreciate a clarification/explanation that matches the new code.

     
    • Andrew Kurushin

      Andrew Kurushin - 2013-04-02

      Please look at following attachment:
      - added comment about os detection
      - patch againts 7.29

       

      Last edit: Andrew Kurushin 2013-04-02
  • Daniel Stenberg

    Daniel Stenberg - 2013-04-02

    Thanks a lot, applied and pushed now in commit 43e045fc3e8430b2d2f3a1. I consider this issue closed now.

     
  • Daniel Stenberg

    Daniel Stenberg - 2013-04-02
    • status: open-confirmed --> closed-fixed
    • milestone: --> not_used
     
  • zenden2k

    zenden2k - 2015-04-13

    Actually, this solution doesn't work for me.
    I have a 100 mbit connection and I still have to use CURLOPT_SOCKOPTFUNCTION
    and setsockopt(SO_SNDBUF...) with buffer size = 1MB in my code. Without this workaround or using 16 kb buffer gives me just 10% of the maximum upload speed.
    Tested on Windows XP, 7, 8.1.
    More interesting that this actually depends on ISP.

     

    Last edit: zenden2k 2015-04-13
  • Daniel Stenberg

    Daniel Stenberg - 2015-04-15

    You're commenting on an old and dead report. Nobody will care about those comments.

    File a new bug report or post on the mailing list.