Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

curl_multi: resumed transfer is not completed although all data was received #3994

Closed
sstruchtrup opened this issue Jun 7, 2019 · 5 comments
Closed

curl_multi: resumed transfer is not completed although all data was received #3994

sstruchtrup opened this issue Jun 7, 2019 · 5 comments

Comments

@sstruchtrup
Copy link

I did this

I am using libcurl with the multi interface (and libev) to receive data via http. Received data will be written to a relatively slow flash memory. curl_easy_pause() is used for flow control.

I noticed stuck transfers when the transfer is paused on the last invocation on the write callback, i.e. when the last chunk (in a non-http sense) of data is received. Once the transfer is resumed, it does not complete, although all data was received.

I observed the problem both when using curl_easy_pause(easy, CURLPAUSE_ALL) (consuming the data) from within the callback and as well when returning CURL_WRITEFUNC_PAUSE from the callback (without consuming the data).

Restarting the transfer will deliver the remaining data, in case it was not consumed (by returning CURL_WRITEFUNC_PAUSE). But in any case the transfer is not completed, that is CURLMSG_DONE is not received. It looks it will eventually be delivered, likely once the server closes the connection.

The application is single-threaded.

I will submit a example to demonstrate the problem, based on the evhiperfifo.c example, modified to pause the transfer on the first invocation and restart it after one second.

Fetching a resource with a short body (single callback invocation) will demonstrate the problem, e.g. http://google.com.

I expected the following

The transfer shall complete once all data was received. CURLMSG_DONE should be delivered by curl_multi_info_read().

curl/libcurl version

[curl -V output]

curl 7.64.0 (x86_64-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.8.6/openssl/zlib nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

I tested current curl master as well as curl 7.40 and 7.29/7.30, all seem to show this problem.

operating system

Linux / Ubuntu 18.04
sstruchtrup pushed a commit to sstruchtrup/curl that referenced this issue Jun 7, 2019
Modify evhiperfifo.c to demonstrate curl#3994
e62b9f6 
Fetch an URL with a short body, e.g. http://google.com/
@sstruchtrup
Copy link
Author

Modified example in sstruchtrup/curl@e62b9f6

Output for http://google.com:

...
REMAINING: 1
event_cb  w 0x557d9dae8a80 revents 1
* Mark bundle as not supporting multiuse
* HTTP 1.1 or later with persistent connection
< HTTP/1.1 301 Moved Permanently
< Location: http://www.google.com/
< Content-Type: text/html; charset=UTF-8
< Date: Fri, 07 Jun 2019 14:41:44 GMT
< Expires: Sun, 07 Jul 2019 14:41:44 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 219
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
<
Received 219 bytes
*** pausing transfer ***
* Paused 219 bytes in buffer for type 01
Progress: http://google.com/ (219/219)
Progress: http://google.com/ (219/219)
sock_cb e 0x557d9daf1fc8 s 6 what 4 cbp 0x7ffd6efa9590 sockp 0x557d9dae8a60
socket callback: s=6 e=0x557d9daf1fc8 what=REMOVE
remsock
REMAINING: 1
*** timeout, restarting transfer ***
Received 219 bytes

The transfer is complete, all 219 bytes were consumed, but no CURLMSG_DONE is received.

@sstruchtrup
Copy link
Author

Using the curl multi interface without an external event loop, e.g. via curl_multi_wait() does not show this behaviour.

@sstruchtrup sstruchtrup changed the title curl_multi: unpaused transfer is not completed although all data was received curl_multi: resumed transfer is not completed although all data was received Jun 7, 2019
bagder added a commit that referenced this issue Jun 8, 2019
@bagder
unpause: trigger a timeout for event-based transfers
893250f 
... so that timeouts or other state machine actions get going again
after a changing pause state. For example, if the last delivery was
paused there's no pending socket activity.

Reported-by: sstruchtrup on github
Fixes #3994
@bagder bagder mentioned this issue Jun 8, 2019
Closed
@bagder
Copy link
Member

bagder commented Jun 9, 2019

I could reproduce the problem and the fix in #4001 made it work for me. It'd be great if you could see if it does for you as well?

@sstruchtrup
Copy link
Author

I can confirm that it solves it for me as well, both for the sample code as well as for the actual application.
Thanks for the quick fix!

@bagder
Copy link
Member

bagder commented Jun 9, 2019

Lovely, thanks for confirming!

@bagder bagder closed this as completed in 680f141 Jun 9, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Sep 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

unpause: trigger a timeout for event-based transfers curl/curl
2 participants
@bagder @sstruchtrup