Bugs item #1779054, was opened at 2007-08-22 00:38
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1779054&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: ftp
Group: bad behaviour
Status: Open
>Resolution: None
Priority: 5
Private: No
Submitted By: Kris (tinker105)
Assigned to: Daniel Stenberg (bagder)
Summary: very long ftp replies confuse curl

Initial Comment:
On an upload using either the curl command line or a client using the "easy" API, if the FTP reply to the STOR command exceeds about 16K (e.g. due to a custom server including debug info), curl will stop reading it and will subsequently falsely report that the connection is dead. The rest of the long reply will be emitted (if verbose set) after subsequent commands and curl will create a new connection (to perform a 2nd transfer).

I tried to fix the problem by doubling the size of UrlState.buffer and the (BUFSIZE) limit checks in ftp_readresp and Curl_GetFTPResponse but encountered a different "reckoned dead" problem, later, after the 2nd STOR, this one emitted from Curl_retry_request.

> curl --version
curl 7.16.4 (sparc-sun-solaris2.6) libcurl/7.16.4 OpenSSL/0.9.8a zlib/1.1.3
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: Largefile NTLM SSL libz

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2007-08-28 09:50

Message:
Logged In: YES
user_id=1110
Originator: NO

ok, any chance you can help us by trying to craft up a test case that
repeats your problem? I can't see how your case is different than the ones
we've fixed recently...

See test case 1003, 1005 and 1006 and use as basis for an attempt. They
are all doing different tests with very long ftp server responses.

Your server isn't responding this in a very slow manner, is it?

The "left intact" message means that libcurl aborted the transfer and left
the connection "alive". And yes, leaving such a connection alive assuming
it is okay will mess up the subsequent reconnection quite a bit so that is
a plain wrong action.

----------------------------------------------------------------------

Comment By: Kris (tinker105)
Date: 2007-08-27 16:59

Message:
Logged In: YES
user_id=1437892
Originator: YES

Still no joy here (using the 8/24 snapshot). I'm including some snippets
of the verbose output to show what's happening. To elaborate, after about
16K of reply, curl stops reading the reply, emits the "left intact" msg,
there's a short delay, then it emits the "dead" msg, the "QUIT", and (most
of) the remainder of the reply. It then establishes a new connection to
handle the 2nd put in the script.

[...]
> EPSV
* Connect data stream passively
< 229 Entering Extended Passive Mode (|||57707|)
* Trying 10.1.4.7... * connected
* Connecting to 10.1.4.7 (10.1.4.7) port 57707
> TYPE I
< 200 Type set to I.
> STOR myfile
< 150 Opening BINARY mode data connection.
< 226- %L-I-XLDINFO, \Xfer started Mon Aug 27 10:16:45 2007\
[... about 16K bytes of reply, then]
< 226- %L-I-XLDINFO, \ C000604 (*:*+19) INTEXT(20) TERM(:)\
< 226- %L-I-XLDINFO, \ C000605 (*:*+19) INTEXT(20) TERM(:)\
* Connection #0 to host sthdev2 left intact
* Connection #0 seems to be dead!
> QUIT
< FO, \ C000606 (*:*+19) INTEXT(20) TERM(:)\
< 226- %L-I-XLDINFO, \ C000607 (*:*+19) INTEXT(20) TERM(:)\
< 226- %L-I-XLDINFO, \ C000608 (*:*+19) INTEXT(20) TERM(:)\
[... about 100 more lines]
< 226- %WORLD-S-XWOK, A request was successfully completed.
< 226 ***** Xfer finished ***** code=<0>
* Closing connection #0
* About to connect() to sthdev2 port 1985 (#0)
* Trying 10.1.4.7... * connected
* Connected to sthdev2 (10.1.4.7) port 1985 (#0)
< 220 sthdev2 FTP server (Version 3.3) ready.
> USER xxxx
[... 2nd PUT in script succeeds OK]

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-08-24 16:01

Message:
Logged In: YES
user_id=1110
Originator: NO

Committed now. tinker, can you please try again?

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-08-24 15:38

Message:
Logged In: YES
user_id=1110
Originator: NO

Great Dan! I've used this now to nail the problem and I have a working
version now, I'll just polish it up a bit and commit soon...

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-08-24 02:01

Message:
Logged In: YES
user_id=236775
Originator: NO

File Added: test1006

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-08-24 01:45

Message:
Logged In: YES
user_id=236775
Originator: NO

I've added test case 1005 that tests many short lines from the server, and
it passes, too. However, if you shorten the reply in test case 1005 by 583
characters, then the test hangs. curl hits a boundary condition when it
truncates the long line, and ends up parsing a line with a code of
"250250", which is an unexpected code.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-08-23 22:33

Message:
Logged In: YES
user_id=1110
Originator: NO

Oh well, I found it obvious how it was a problem with a single line that
long, with multiple lines I don't exactly see how it is a problem in the
code...

Any chance you can offer me access to such a server setup to debug
against?

----------------------------------------------------------------------

Comment By: Kris (tinker105)
Date: 2007-08-23 18:22

Message:
Logged In: YES
user_id=1437892
Originator: YES

No, it isn't all one line - it's a standard "long reply" with a large
number of normal-length lines. I downloaded and tried the latest CVS
(7/23) and the problem is still there.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-08-22 16:18

Message:
Logged In: YES
user_id=1110
Originator: NO

Thanks, this is now fixed in CVS and test case 1003 was added to verify
it.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2007-08-22 09:37

Message:
Logged In: YES
user_id=1110
Originator: NO

Is that >16K as in one single line with only CRLF at the end of the huge
chunk?

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1779054&group_id=976
Received on 2007-08-28