Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slow loading of directory content via SFTP when timeout is set #3506

Closed
DiceMaster opened this issue Jan 30, 2019 · 5 comments
Closed

Slow loading of directory content via SFTP when timeout is set #3506

DiceMaster opened this issue Jan 30, 2019 · 5 comments
Labels
SCP/SFTP

Comments

@DiceMaster
Copy link

Directory listing is wery slow when connecting to a SFTP server with CURLOPT_TIMEOUT or -m cmd option. Without a timeout everything works fast.

Tested on my Mac (macos 10.14.2) when connecting to other Mac with curl v7.64.0 (or with daily snapshot 20190129) with following command:
curl sftp://<host>/ -m 20 -u <user>

There is some delay for loading of each subdirectory/file.
In curl v7.61.0 the same command performs almost instantly.

After some investigation I found that the bug was introduced with changes to lib/ssh.c in commit 1a89099
@koskokos2
Copy link

koskokos2 commented Jan 31, 2019 •
edited
Loading

Noticed the very same problem and for me it works very slow even without any additional arguments just:
curl sftp://<auth>@<host>/

In version 7.61.0 that command takes (for a folder with 4 entries) 0.7s and in 7.61.1 (first affected version) it takes 6.7s for a localhost sftp server.

@manfredsc
Copy link

Same for me, 7.61.1 is way slower than version 7.61.0 when doing a simple sftp listing.
The trace-ascii output is exactly the same for both versions.

It seems curl-7.61.1 does exactly one line of print-out per second as one can see with --trace-time:

14:45:01.186936 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    3 0        root         4096 Jun 12  2014 dev.
14:45:02.187983 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Jun 12  2014 lib.
14:45:03.189041 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Oct  7  2014 bin.
14:45:04.190077 => Send data, 60 bytes (0x3c)
0000: drwxr-xr-x    2 0        root         4096 Jun 12  2014 tmp.

So there is somewhere an artificial 1 second delay per printed line! Hope this rings some bells...
Please tell how to further narrow this annoying bug. Thanks.

@bagder
Copy link
Member

bagder commented Mar 4, 2019

I can reproduce. Fix pending.

bagder added a commit that referenced this issue Mar 4, 2019
@bagder
ssh: loop the state machine if not done and not blocking

Verified

This commit was signed with the committer’s verified signature.
bagder Daniel Stenberg
GPG key ID: 5CC908FDB71E12C2
Verified
Learn about vigilant mode
d9af8f2 
If the state machine isn't complete, didn't fail and it didn't return
due to blocking it can just as well loop again.

This addresses the problem with SFTP directory listings where we would
otherwise return back to the parent and as the multi state machine
doesn't have any code for using CURLM_CALL_MULTI_PERFORM for as long the
doing phase isn't complete, it would return out when in reality there
was more data to deal with.

Fixes #3506
@bagder bagder mentioned this issue Mar 4, 2019
Closed
@manfredsc
Copy link

Works for me, version 7.64.0 plus this patch.
Thanks!

@bagder
Copy link
Member

bagder commented Mar 4, 2019

Thanks for confirming!

@bagder bagder closed this as completed in d6f73af Mar 5, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
SCP/SFTP
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ssh: loop the state machine if not done and not blocking curl/curl
4 participants
@bagder @DiceMaster @koskokos2 @manfredsc