Bugs item #3048197, was opened at 2010-08-19 00:59
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3048197&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: ftp
Group: bad behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: catalin (catalinr)
Assigned to: Daniel Stenberg (bagder)
Summary: Incorrect data uploaded in case of CURLE_SEND_ERROR

Initial Comment:
These are my [hopefully right] conclusions:
- When doing an FTP upload, in the progressCallback function the uploadedTillNow argument (last one) holds the correct size for the uploaded data (well, rather "sent data");
- the uploaded data is sent in chunks of at most CURL_MAX_WRITE_SIZE bytes (btw, CURL_MAX_WRITE_SIZE is not documented anywhere and it'd be useful to know that that is the maximum size attempted to be uploaded at one time);
- in case the link is broken/disconnected, the remote destination file gets appended with a chunk of the size reported by progressCallback but the last KB are composed of NULL bits and not the real data. So a subsequent APPE[-nd] to that file will render a file of the same size with the original one, but somewhere in the middle it'll have the wrong NULL bytes.

The workaround for this was to add the size reported by the progressCallback in a totalTillNow variable, and after any network error that would require a resume subtract CURL_MAX_WRITE_SIZE*8 from totalTillNow, and setup a REST upload with the calculated starting point. (See #3048174 for issues with CURLOPT_RESUME_FROM).

I'm not sure if my workaround is the best solution, but it works. OTOH, that behavior when uploading and getting disconnected leads to corrupted files at destination and that is IMO very wrong...

libcurl 7.21.1
msw Vista
mingw-gcc 3.4.5

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2010-08-21 21:20

Message:
When the connection breaks, libcurl CANNOT send anything further as the
connection is no more, nor can it check anything else on the remote site as
the connection... broke! Having libcurl try to reconnect just to check the
end of the file in case it got disconnected just previously is completely
out of the question.

Alas, the problem you see at disconnect depends on what the server does on
a disconnect. libcurl cannot guarantee what the server does, nor can it
assume anything. Some servers are likely to act differently than others on
disconnect. Appending zero-bytes to the file does sound like a case of bad
behaviour ON THE SERVER END.

1. I can't see any error in libcurl's side
2. It sounds like bad behavior on the server side
3. you have not presented any way to repeat this problem

I can't see what libcurl can do about this.

Anyone who decides to append data to an existing file because it got
aborted in a previous upload attempt may of course consider to check the
end of the file to see that the end looks OK before blindly appending more
data to it. libcurl will not do that automatically though but does provide
the powers to get the data etc.

----------------------------------------------------------------------

Comment By: catalin (catalinr)
Date: 2010-08-21 07:02

Message:
I'm sure I fail to see a lot more than you do, but I'm just signaling what
looks like bad behavior to me. Maybe you can find a way to try and
reproduce this as I don't think I can make a sample program that will make
a server disconnect (or an ISP to interrupt it etc), can I?
Of course that ftp server (comes with a BusyBox linux on a NAS device) may
be broken but I have some doubts about that and IMO it's worth
investigated...

The error received at my end was CURLE_SEND_ERROR and IIRC once I also got
CURLE_RECV_ERROR (although only uploads were being done, but maybe it was
about receiving some response from the server).
A long-shot interpretation would be that curl sends the size of the packet
being uploaded, but only part of the actual data gets to destination.
Again, I may be far away with my guess...
I don't think those zeroes are exactly random neither... Comparing the
source and destination files, the difference is made by the zeroes in the
destination file, not some _random_ bits being there.

Maybe a shorter way would be to get an ftp upload to break with
CURLE_SEND_ERROR and then check the end of the uploaded file part?

HTH

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2010-08-19 15:15

Message:
I don't see how libcurl sends any data as zeroes. Also, I fail to see how
it could send that block of zeroes if the connection disconnected?

To me it sounds like your server behaves oddly and add random data to the
file being written at the time of the disconnect. I don't think libcurl can
do anything about it.

If this is not the case, can you please clarify your point for us?

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3048197&group_id=976
Received on 2010-08-21