Bugs item #3215314, was opened at 2011-03-16 06:44
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3215314&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: SCP/SFTP
Group: bad behaviour
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Rajesh Naganathan (mydreamzz)
Assigned to: Daniel Stenberg (bagder)
Summary: Post quote operation to rename fails in Windows

Initial Comment:
There is already posting done for this issue(http://curl.haxx.se/mail/lib-2011-02/0322.html) and below is the abstract.

We are using CURLOPT_POSTQUOTE option to rename a file after transfer of the file using sftp protocol.But the POSTQUOTE option for rename is failing when the sftp server is runing in Windows O/S.The same code works perfectly when the sftp server is Linux or Sun OS. Following is the Log from the server side.
  
Complete FTP:
2011-02-23 14:46:31,203 INFO SFTPSubsystemServer [48:temp] Opened /C-drive/.A20110223.0826+0000-XYZ.TBD.0000075BCD15.1 for writing
2011-02-23 14:46:31,844 ERROR SFTPSubsystemServer [48:temp] Failed to move file .A20110223.0826+0000-XYZ.TBD.0000075BCD15.1 => A20110223.0826+0000-XYZ.TBD.0000075BCD15.1: The process cannot access the file because it is being used by another process.
  
Cerebres SFTP :
2011/02/23 15:05:33 [5] File Open command received
2011/02/23 15:05:33 [5] File handle to '//.A20110223.0931+0000-XYZ.TBD.0000075BCD15.1' opened
2011/02/23 15:05:33 [5] Upload started for file 'C:\.A20110223.0931+0000-XYZ.TBD.0000075BCD15.1'
2011/02/23 15:05:34 [5] Unable to rename file 'C:\.A20110223.0931+0000-XYZ.TBD.0000075BCD15.1'
2011/02/23 15:05:34 [5] Upload complete for file 'C:\.A20110223.0931+0000-XYZ.TBD.0000075BCD15.1'
2011/02/23 15:05:34 [5] Handle '//.A20110223.0931+0000-XYZ.TBD.0000075BCD15.1' closed
  
It was suggested that libcurl uses libssh2 to do the SFTP operations and the thing libcurl can do here is to make sure that libssh2_sftp_close() is done on the file properly before it does the POST_QUOTE operations. Since this can be supported by libcurl,it would be better for libcurl to do the libssh2_sftp_close first followed by the execution of the rename operation instead of any server specific implementation.

Currently this works fine in Linux because linux doesnt care about the open files..Suppose Linux also makes sure that the file should be closed to perform the rename operation,the entire post quote operation supported by libcurl will fail on Linux and Windows.

So its better if libcurl could support the behaviour of doing the libssh2_close before doing the postquote operations.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-20 23:40

Message:
Thanks for the report, this problem is now fixed in the git repository.

To try it out, you either checkout/update your git clone:
http://curl.haxx.se/source.html

or you try tomorrow's daily snapshot: http://curl.haxx.se/snapshots/

----------------------------------------------------------------------

Comment By: Rajesh Naganathan (mydreamzz)
Date: 2011-04-20 07:35

Message:
I have attached the change that is proposed by you.The SSH_SFTP_CLOSE state
first looks at if a nextstate is set.If the nextstate is set,it will set
that state as the current state and also it will set the nextstate as
SSH_SFTP_CLOSE.This is done to ensure,that the final error code or result
is passed out to the caller using the SSH_SFTP_CLOSE state .
Old statemovement for rename was
sftp_done->POSTQUOTE_INIT->SSH_SFTP_QUOTE->QUOTE_RENAME->NEXT_QUOTE->SSH_SFTP_CLOSE->SSH_STOP
This state machine movement is changed to happen as :
sftp_done
->SSH_SFTP_CLOSE->POSTQUOTE_INIT->SSH_SFTP_QUOTE->QUOTE_RENAME->NEXT_QUOTE->SSH_SFTP_CLOSE->SSH_STOP
        

If suppose during the statemovement like sftp_done
->SSH_SFTP_CLOSE->POSTQUOTE_INIT->QUOTE->QUOTE_RENAME,the rename command
failed in the libssh2 call in QUOTE_RENAME state,it will transition to the
SSH_SFTP_CLOSE setting the nextstate to SSH_NO_STATE.This will ensure that
there is no infinite loop in the SSH_SFTP_CLOSE state as it always looks
for nextstate variable now.Since the QUOTE_RENAME has set the nextstate to
SSH_NO_STATE,it will go to the state SSH_STOP where the result is passed to
the caller. The new statemachine is also attached.

----------------------------------------------------------------------

Comment By: Rajesh Naganathan (mydreamzz)
Date: 2011-04-18 19:30

Message:
Perfect.I agree with your solution.I tested it in few Windows SFTP server
and it worked .

But a few more things has to be taken care to ensure the error code is
passed to the caller properly.I would make a final fix around this solution
and will post the diff file.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-17 14:23

Message:
If you read the state machine code again, you'll see that it deals with
LIBSSH2_ERROR_EAGAIN return codes all over, and that's what we need to
continue to do as basically all libssh2 functions MIGHT return that and
then we need to call that function again when the correct action on the
socket has occurred.

I think the correct fix to this problem (and I do acknowledge that there
is a problem) is to change the flow for post-quote operations, and let the
SSH_SFTP_CLOSE code run first and then have the state machine code there
check for "nextstate" and then switch to the the post-quote state instead
of immediately use SSH_STOP once sftp close is done, so that the post quote
code can run properly.

What do you think about that?

----------------------------------------------------------------------

Comment By: Rajesh Naganathan (mydreamzz)
Date: 2011-04-13 12:16

Message:
In sftp_done function,the sshc->nextstate is set to SSH_SFTP_CLOSE and
then the state is moved to SSH_SFTP_POSTQUOTE_INIT.But there is no place
before POSTQUOTE_INIT the sftp handle is closed.The fix that is proposed is
to close the handle via libssh2_sftp_close before even entering the
POSTQUOTE_INIT state.We tried to enable debug logs and saw that before
POSTQUOTE_INIT,the sftp handle was never closed using libssh2_sftp_close.If
you feel that its really closed,can u please point to us ?

And regarding the second point ignoring closing of handle during error:
This is the case even in the state machine transition for SSH_SFTP_CLOSE
.Do you suggest if a retry attempt or so should be made?

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-11 22:23

Message:
Thanks for reporting this issue and helping us improve curl and libcurl.

We're awaiting feedback in this issue. Due to this, I have set the state
of this issue to pending and it will automatically get closed later on
unless we get further info.

Please consider answering the outstanding questions or providing the
missing info so that we can proceed to resolve this issue!

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-06 23:26

Message:
Well, first we better be sure why the current code doesn't work before we
add more code to do the same thing existing code already attempts to do.

Then, as I already mentioned, the fix you provided doesn't work properly
as it doesn't wait until the function has completed so it very well may not
at all close the handle. It actually could even suggest that what solves
the issue is not the closing at all, but just the little extra time this
operation added between the existing close to the post quote operation.

----------------------------------------------------------------------

Comment By: Rajesh Naganathan (mydreamzz)
Date: 2011-04-06 12:21

Message:
I dont know for sure why the SSH_SFTP_CLOSE didnt cose the handle before
executing postquote.Since the windows server was rejecting the rename
command,we tried to close the handle which will close all file handles in
the server side so that we can start with the POSTQUOTE operation freshly
without any file handles that are in server side.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-02 09:32

Message:
Thanks for reporting this issue and helping us improve curl and libcurl.

We're awaiting feedback in this issue. Due to this, I have set the state
of this issue to pending and it will automatically get closed later on
unless we get further info.

Please consider answering the outstanding questions or providing the
missing info so that we can proceed to resolve this issue!

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-03-21 19:46

Message:
Thanks, but that approach is not working properly.

libssh2_sftp_close() is an operating that very well may return
LIBSSH2_ERROR_EAGAIN due to its non-blocking nature and then the closure
hasn't completed and just discarding the handle at that point seems like a
very weird solution.

Also, I would think that the SSH_SFTP_CLOSE should be run properly before
post-quote operations and that state closes the handle already. Have you
figured out why it doesn't or what we can do to make it do so?

----------------------------------------------------------------------

Comment By: Rajesh Naganathan (mydreamzz)
Date: 2011-03-16 10:41

Message:
I have attached the diff file "mycurlchange.diff" here to show that the
handle can be closed before executing the post quote commands.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-03-16 09:18

Message:
Thanks, can you please make a full "proper" diff/patch and attach here?

Here are some hints on how to do it:

http://curl.haxx.se/dev/contribute.html#How_To_Make_a_Patch_without_git

----------------------------------------------------------------------

Comment By: Rajesh Naganathan (mydreamzz)
Date: 2011-03-16 06:49

Message:
Doing the below change in libcurl makes the post quote work in both windows
and linux environment .
 

static CURLcode sftp_done(struct connectdata *conn, CURLcode status,
                               bool premature)
{
  struct ssh_conn *sshc = &conn->proto.sshc;
  int rc;
  struct SessionHandle *data = conn->data;

  if(status == CURLE_OK) {
    /* Before we shut down, see if there are any post-quote commands to
send: */
    if(!status && !premature && conn->data->set.postquote) {

        /* added following part */

        if(sshc->sftp_handle) {
            rc = libssh2_sftp_close(sshc->sftp_handle);

            if(rc == LIBSSH2_ERROR_EAGAIN || rc == 0) {
              sshc->sftp_handle = NULL;
            }
            else if (rc < 0) {
              infof(data, "Failed to close libssh2 file %d\n", rc);
            }
        }
        sshc->nextstate = SSH_SFTP_CLOSE;
        state(conn, SSH_SFTP_POSTQUOTE_INIT);
    }
    else
      state(conn, SSH_SFTP_CLOSE);
  }
  return ssh_done(conn, status);

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3215314&group_id=976
Received on 2011-04-20