RE: CURLE_OPERATION_TIMEDOUT using curl_multi_socket_action
Date: Mon, 11 Mar 2013 16:53:31 +0530
It looks like there is some confusion regarding the thread pool.
1. There is a queue of jobs (structure having a pointer to a function and values for calling the function)
2. A few worker threads are running, each of which remove one job from the queue and call the job's function.
3. One thread which periodically calls epoll_wait and processes the epoll_event by just pushing the job to the job queue. There is no call to curl_multi_socket_action here. Call to curl_multi_socket_action is made when the job is picked up by one of the worker threads.
> > 4. I add this job to the IO queue.
> > 5. When the job is executed again, call curl_multi_socket_action with the
> > proper socket descriptor and ev_bitmask.
> I don't understand this statement. You already added the handle, what you mean
> with "when it is executed again"?
By "executing a job", I mean when the job is removed from the job queue and its function is called.
> curl_multi_socket_action() is of course called by you whenever there is
> action to do on a specific socket that you monitor or when you deem that the
> timeout has triggered. Right?
Yes, but the call does not take place until the worker thread executes the job.
> > 6. If the HTTP transfer is not completed, add the job back to the IO queue.
> I don't get this either. You already added the handle, surely you'll just keep
> transferring until curl_multi_info_read() tells you that the particular
> transfer is complete?
Yes, I will. But the entire process (of calling curl_multi_socket_action and listening for required events) is delayed because of jobs being queued up.
> > 1. I am using HTTPS to connect to the server. So every time I add the easy
> > handle, steps 5 and 6 are executed multiple times. This is fine if the
> > number of "jobs" is less, but as they increase, I get a
> > CURLE_OPERATION_TIMEDOUT.
> I don't understand this description either. I think you should consider
> providing us with a full example source code showing what you do and ideally
> one that repeats the problem you have.
I think the description I have provided should make it clearer. I will work on writing a sample code and will mail it by tomorrow.
> > I am guessing that 30s is not enough for the SSL certificate verification.
> If that is really the case, it sounds like exceptionally slow machines
> involved. Modern machines do SSL handshakes counted in milliseconds.
Again, there is a confusion here. The 30s is inclusive of the delay in queuing the job, waiting for an IO event (which IS a few ms), pushing to job queue and waiting for a free worker thread to execute the job (which then calls curl_multi_socket_action)
> > Is there anyway for me to force libcurl to finish the send operation?
> What exactly would that mean? libcurl will always do as much as it can without
> blocking, then wait until it "gets permission" to try again and then continue
> from there. Would would a "force" operation even be able to do?
The operation I am interested in requires to wait for the response, which could take a couple of minutes before it arrives. Hence, I am looking for an interface where I can tell curl to do send() in blocking manner, and recv() in a non-blocking manner. I am not sure if this is even technically feasible.
> > 2. In step 6, if I use curl_easy_perform to do some HTTP transfers on the
> > same server, before going to step 1 again. A new connection is established
> > to the server. Is that expected?
> Not if it can re-use an idle existing connection, no.
Please find attached sample to reproduce this.
- text/plain attachment: curl_reuse_issue.cc