Bugs item #2817378, was opened at 2009-07-06 12:48
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2817378&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SCP/SFTP
Group: bad behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Sergey Prokopenko (s-prokop)
Assigned to: Daniel Stenberg (bagder)
Summary: high cpu usage during SFTP copying

Initial Comment:
We use curl-library to support FTP and SFTP protocols in our projects (Linux RHEL).
We were completing migration from FTP to SFTP and unexpectedly faced with two issues using curl library (used latest versions).

1. The CPU usage is very high (80-90%) during SFTP copying.
Any process or thread, which is using curl lib, consumes about 90% cpu usage during SFTP copying.
It is a too much for us, since sometimes we transfer very big files and our systems becomes overloaded during this period.

SFTP cli command , for example, consumes less CPU (about 20).

I took a look more closely and observed that main bottleneck is libssh2 lib but curl code has some "slow" points too.
Please see oprofile report below.

CPU: AMD64 processors, speed 2311.61 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
Counted DATA_CACHE_REFILLS_FROM_SYSTEM events (Data cache refills from system) with a unit mask of 0x1f (All cache states) count 1155806
Counted RETIRED_BRANCH_INSTRUCTIONS events (Retired branches (conditional, unconditional, exceptions, interrupts)) with a unit mask of 0x00 (No unit mask) count 1155806
samples % samples % samples % app name symbol name
333703 41.1265 4 57.1429 2760 27.8226 no-vmlinux (no symbols)
224337 27.6479 0 0 4524 45.6048 libc-2.9.so (no symbols)
161857 19.9477 2 28.5714 1901 19.1633 oprofiled (no symbols)
13144 1.6199 0 0 30 0.3024 librt-2.9.so (no symbols)
11590 1.4284 0 0 66 0.6653 libcrypto.so.0.9.8 (no symbols)
6033 0.7435 0 0 49 0.4940 libcurl.so.4.1.1 Curl_readwrite
4993 0.6154 0 0 42 0.4234 anon (tgid:8637 range:0xb80b7000-0xb80b8000) (no symbols)
3872 0.4772 0 0 40 0.4032 libssh2.so.1 libssh2_channel_read_ex
3416 0.4210 0 0 21 0.2117 nvidia_drv.so (no symbols)
3085 0.3802 0 0 46 0.4637 libcurl.so.4.1.1 Curl_socket_ready
2918 0.3596 0 0 9 0.0907 libcurl.so.4.1.1 Curl_pgrsUpdate
2828 0.3485 0 0 22 0.2218 libxul.so (no symbols)
2676 0.3298 1 14.2857 20 0.2016 Xorg (no symbols)
2478 0.3054 0 0 25 0.2520 libglib-2.0.so.0.2000.1 (no symbols)
2198 0.2709 0 0 20 0.2016 libssh2.so.1 libssh2_sftp_write
1973 0.2432 0 0 14 0.1411 libssh2.so.1 _libssh2_transport_read
1848 0.2278 0 0 10 0.1008 libmozjs.so (no symbols)
1587 0.1956 0 0 6 0.0605 libcurl.so.4.1.1 Curl_debug
1572 0.1937 0 0 10 0.1008 libcurl.so.4.1.1 showit
1548 0.1908 0 0 26 0.2621 libcurl.so.4.1.1 Curl_perform
1520 0.1873 0 0 19 0.1915 libgobject-2.0.so.0.2000.1 (no symbols)
1398 0.1723 0 0 17 0.1714 libcurl.so.4.1.1 curlx_tvnow
1242 0.1531 0 0 30 0.3024 libssh2.so.1 sftp_packet_read
1041 0.1283 0 0 9 0.0907 libcairo.so.2.10800.6 (no symbols)
981 0.1209 0 0 6 0.0605 curl my_trace
822 0.1013 0 0 14 0.1411 libcurl.so.4.1.1 __i686.get_pc_thunk.bx
808 0.0996 0 0 20 0.2016 libpixman-1.so.0.13.2 (no symbols)
774 0.0954 0 0 9 0.0907 libssh2.so.1 __i686.get_pc_thunk.bx
757 0.0933 0 0 10 0.1008 libcurl.so.4.1.1 Curl_write
746 0.0919 0 0 10 0.1008 libcurl.so.4.1.1 .plt
714 0.0880 0 0 7 0.0706 libcurl.so.4.1.1 Curl_sftp_send
708 0.0873 0 0 4 0.0403 libgdk-x11-2.0.so.0.1600.1 (no symbols)
708 0.0873 0 0 7 0.0706 libgtk-x11-2.0.so.0.1600.1 (no symbols)
620 0.0764 0 0 7 0.0706 libpthread-2.9.so pthread_mutex_lock
614 0.0757 0 0 1 0.0101 libssh2.so.1 sftp_packet_require
557 0.0686 0 0 5 0.0504 libcurl.so.4.1.1 Curl_pgrsSetUploadCounter
546 0.0673 0 0 4 0.0403 libcurl.so.4.1.1 ssh_block2waitfor
522 0.0643 0 0 7 0.0706 libssh2.so.1 crypt_encrypt
510 0.0629 0 0 8 0.0806 libcurl.so.4.1.1 Curl_speedcheck
503 0.0620 0 0 0 0 libssh2.so.1 _libssh2_cipher_crypt
488 0.0601 0 0 11 0.1109 curl cutil_tvnow
462 0.0569 0 0 7 0.0706 libssh2.so.1 .plt
439 0.0541 0 0 5 0.0504 libssh2.so.1 libssh2_channel_eof

Can it be considered as a bug or it is curl/libssh2 "feature" ?
Possible, you are already aware about this issue and have a patch.

Appretiate for your help

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2009-07-07 11:50

Message:
This report is still very vague and hard to deal with without more specific
findings or recipes on what we can do to see this problem.

What are you doing exactly and how do you determine that it is a too high
load? Comparing against the openssh sftp tool is fun but it doesn't really
help us much. Can you try the libssh2 sftp example(s) and see if they show
any problems? (and if so take this issue to the libssh2 project) oprofile
really is a very rough tool to get anything detailed out from.

----------------------------------------------------------------------

Comment By: Sergey Prokopenko (s-prokop)
Date: 2009-07-07 08:01

Message:
Sorry for the prio.

Yes, I tried latest curl and libssh2. The same effect.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2009-07-06 22:51

Message:
Why are you saying the libcurl version doesn't matter? Did you try with
7.19.5 and found the same effect?

Please don't change the prio, that field is for the project members to
change.

----------------------------------------------------------------------

Comment By: Sergey Prokopenko (s-prokop)
Date: 2009-07-06 13:04

Message:
libssh2-1.1
curl-7.18.2
Red Hat Enterprise Linux Server release 5.1 (Tikanga). But this bug is
valid for any Linux with any version of curl or libssh2

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2817378&group_id=976
Received on 2009-07-07