Bugs item #3285747, was opened at 2011-04-13 13:49
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3285747&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: ftp
Group: hang
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: michal lev (michalev1)
Assigned to: Daniel Stenberg (bagder)
Summary: 100% CPU when FTP server is heavy load

Initial Comment:
Hi,

I'm having a multi threaded application written in c++ that uses curl lib version 7.19.4.
When the FTP server is on heavy load, some of the threads of my application are waiting to the FTP response for the timeout period I defined in CURLOPT_FTP_RESPONSE_TIMEOUT. But during this time of waiting these threads consume lots of CPU and as a result my application consumes most of the machine's CPU.

While digging in the CURL code I can see that these threads are in the "ftp_easy_statemach" function inside the while loop, and they are stuck in this loop until the response timeout is over.
The state of the ftp connection is "FTP_WAIT220"

During the while loop the rc is always 1 and the result is CURLE_OK .
 
on ftp.c:
static CURLcode ftp_easy_statemach(struct connectdata *conn)
{
  curl_socket_t sock = conn->sock[FIRSTSOCKET];
  int rc;
  struct SessionHandle *data=conn->data;
  struct ftp_conn *ftpc = &conn->proto.ftpc;
  CURLcode result = CURLE_OK;

  while(ftpc->state != FTP_STOP) {
    long timeout_ms = ftp_state_timeout(conn);

    if(timeout_ms <=0 ) {
      failf(data, "FTP response timeout");
      return CURLE_OPERATION_TIMEDOUT; /* already too little time */
    }

    rc = Curl_socket_ready(ftpc->sendleft?CURL_SOCKET_BAD:sock, /* reading */
                           ftpc->sendleft?sock:CURL_SOCKET_BAD, /* writing */
                           (int)timeout_ms);

    if(rc == -1) {
      failf(data, "select/poll error");
      return CURLE_OUT_OF_MEMORY;
    }
    else if(rc == 0) {
      result = CURLE_OPERATION_TIMEDOUT;
      break;
    }
    else {
      result = ftp_statemach_act(conn);
      if(result)
        break;
    }
  }

  return result;
}

Is it a known issue?
Is there a workaround ?

I saw this open bug , but it didn't talk about consuming 100% CPU until we get to the timeout.
73. if a connection is made to a FTP server but the server then just never
  sends the 220 response or otherwise is dead slow, libcurl will not
  acknowledge the connection timeout during that phase but only the "real"
  timeout - which may surprise users as it is probably considered to be the
  connect phase to most people. Brought up (and is being misunderstood) in:
  http://curl.haxx.se/bug/view.cgi?id=2844077

Please advise.

Thanks

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2011-04-22 22:32

Message:
As I can't repeat this problem I need you to try to figure out what's
happening.

Curl_socket_ready() is documented to return a CURL_CSELECT_* bitmask, so 1
means first argument's socket was readable.

So what does "heavy load" mean? You say it occurs with FileZilla then, but
what about under less loaded situations? Where's the threshold?

----------------------------------------------------------------------

Comment By: michal lev (michalev1)
Date: 2011-04-21 16:10

Message:
What that i had see is that the loop we get r=1 on this line in
in Curl_socket_ready()
r = select((int)maxfd + 1, &fds_read, &fds_write, &fds_err, ptimeout);

Is this is the "the socket as it was signaled to be readable"?

For sure the loop is a busy-loop , and we get to ftp_statemach_act() many
times with ftpcode 0, I saw it on debug.

Something that I wanted to mention is that we had this problem only when
we used FileZilla as FTP server, when we used Microsoft FTP server the
problem not occured.

can you think about differences between these 2 server on heavy load
situation?

can you focus me on the details you need in ftp_readresp()? which line
interest you?
 

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-17 20:14

Message:
ok, but within ftp_statemach_act(), the function ftp_readresp() is called
and that should read from the socket as it was signaled to be readable. It
shouldn't do that very many times though until it _does_ return an ftpcode
and then it should continue.

While ftpcode equals zero, it should be reading from the socket and it
should not busy-loop. Can you see why it would do that?

----------------------------------------------------------------------

Comment By: michal lev (michalev1)
Date: 2011-04-17 18:06

Message:
Thank you for the response.

result = ftp_statemach_act(conn);
The return value for this line is (CURLE_OK)

The ftocode inside ftp_statemach_act is 0 - as you can see we don't enter
to the ftpcode if - so we don't act like the FTP_WAIT220 state should act.
the result stay - CURLE_OK, and then begin one more cycle of the while
loop is starting - until we got the timeout, and meanwhile we got 100 % cpu
consuming.

static CURLcode ftp_statemach_act(struct connectdata *conn)
{
.
.
.
.

  if(ftpcode) {
    /* we have now received a full FTP server response */
    switch(ftpc->state) {
    case FTP_WAIT220:
      if(ftpcode != 220) {
        failf(data, "Got a %03d ftp-server response when 220 was
expected",
              ftpcode);
        return CURLE_FTP_WEIRD_SERVER_REPLY;
      }
.
.
.
.} /* if(ftpcode) */

  return result;
}

Is this the information you needed?

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2011-04-17 14:13

Message:
No, it's not a known issue and it is not the known bug #73.

I read the code now and I'm trying to understan how ftp_statemach_act()
can be called many times while being in the FTP_WAIT220 state. Can you?

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=3285747&group_id=976
Received on 2011-04-22