On 3/8/2016 9:39 AM, Daniel Stenberg wrote:

> On Tue, 8 Mar 2016, Boutin Maël wrote:
>
> Currently i call multi_perform when there are data to be sent. How can i
>> obtain the send status ? By performing a select on the handles ?
>>
>
> Yes, you would call select on the file descriptors with the actions
> libcurl told you, and if one of those signal readiness you call
> curl_multi_perform().
>

Do you mean you plan on leaving open ~1000 connections concurrently?
> Wouldn't curl_multi_wait [1] be a safer choice instead of curl_multi_fdset
> here?

Thank you for your help,

Yes i may have ~1000 concurrent connections. But i'm still stucked at how i
should handle errors. Currently i have a working code like this:

Curl initialisation :

poCurrentHandle->m_pstHandle = curl_easy_init();
curl_easy_setopt(poCurrentHandle->m_pstHandle, CURLOPT_POST, 1L);
curl_easy_setopt(poCurrentHandle->m_pstHandle, CURLOPT_READFUNCTION,
pushBuffer);
curl_easy_setopt(poCurrentHandle->m_pstHandle, CURLOPT_UNIX_SOCKET_PATH,
"./test");
curl_easy_setopt(poCurrentHandle->m_pstHandle, CURLOPT_HTTPHEADER,
m_pstCurlHeaders);

Thread loop :

while(1)
{
  bSomethingToDo = false;

  for(int i = 0; i < MAX_CONNECTIONS; i++)
  {
    // ... some stuff
    if (poCurrentConnection->getNbRemainingBuffers() != 0 &&
poCurrentConnection->getConnectionState() != eHSSConnectionStateENDED)
    {
      bSomethingToDo = true;
      if (poCurrentConnection->getConnectionState() ==
eHSSConnectionStateWAIT)
      {
        //If the connection was in WAIT state then it means we have to
unpause the CURL connection to send the new buffers
        curl_easy_pause(poCurrentConnection->getCurlHandle()->m_pstHandle,
CURLPAUSE_CONT);
      }
    }
  }
  if (bSomethingToDo)
  {
    // Call multi_perform
    curl_multi_perform(m_pstCurlMultiHandle, &iRunningFDs);
  }
  else
  {
    usleep(10000);
  }
}

And here is the callback which is called by multi_perform:

size_t CPushingThread::pushBuffer(void * pi_pData, size_t pi_iBuffSize,
size_t pi_iBuffNbElement, void * pi_pConnection)
{
  CHSSConnection * poConnection = (CHSSConnection *) pi_pConnection;
  CHSSBuffer * poBuffer = NULL;
  size_t uiBytesToSend = 0;

  poBuffer = poConnection->getBufferToSend();
  if (poBuffer)
  {
    // ... Copy buffer for libcurl
  }
  else {
    // No data to be sent, PAUSE the connection, we don't want to close it
while we wait for new data except if the connection is in ENDED state
    if (poConnection->getConnectionState() == eHSSConnectionStateENDED)
    {
      uiBytesToSend = 0;
    }
    else
    {
      uiBytesToSend = CURL_READFUNC_PAUSE;
    }
  }

  return uiBytesToSend;

}

This code works quite well, i can stream data up to 60Gbps. However i still
lack the error handling, the recipients may for any reason close the
connection either by sending a HTTP response (although it should not, we
are sending a long running POST that never ends) or by terminating the TCP
connection. My question is where should i check the response in this
context ? I bet this is the piece of code i need to change:

  if (bSomethingToDo)
  {
    // Call multi_perform
    curl_multi_perform(m_pstCurlMultiHandle, &iRunningFDs);
  }
  else
  {
    usleep(10000);
  }

This might seem trivial for you but i'm still learning libcurl and i find
the documentation quite misleading for starters like me..

Regards

2016-03-08 22:55 GMT+01:00 Daniel Stenberg <daniel_at_haxx.se>:

> On Tue, 8 Mar 2016, Ray Satiro via curl-library wrote:
>
> Do you mean you plan on leaving open ~1000 connections concurrently?
>> Wouldn't curl_multi_wait [1] be a safer choice instead of curl_multi_fdset
>> here?
>>
>
> Oh right, for any solution that intends to use perhaps more than a hundred
> simultaneous transfers I strongly recommend going
> curl_multi_socket_action() and event-based. Anything based on select/poll
> gets slow at some point with a growing number of descriptors.
>
> --
>
> / daniel.haxx.se
>
> -------------------------------------------------------------------
> List admin: https://cool.haxx.se/list/listinfo/curl-library
> Etiquette: https://curl.haxx.se/mail/etiquette.html
>

-- 
Maël BOUTIN

-------------------------------------------------------------------
List admin: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Received on 2016-03-09