Bugs item #1634515, was opened at 2007-01-13 03:32
Message generated for change (Comment added) made by ntopilski
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1634515&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: bad behaviour
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: nickt (ntopilski)
Assigned to: Daniel Stenberg (bagder)
Summary: Large Virtual Memory size

Initial Comment:
Hello,

I am a first time user of the libcurl library.
I am compiling and running under Fedora Core 5 on Intel machine.

I run into an issue that when launching libcurl from threads, if the number of threads is above 9 the virtual memory consumption increases disproportionatly.

In the code sample below:

#include <fstream>
#include <iomanip>
#include <time.h>
#include <cstdlib>
#include <cstdio>
#include <pthread.h>
#include <iostream>
#include <curl/curl.h>

using namespace std;

pthread_mutex_t COUNTER_MUTEX = PTHREAD_MUTEX_INITIALIZER;
unsigned int THREAD_COUNTER = 0;

void* executeStuff (void* arg )
{
  char* ip = (char*) arg;

  CURL* easyhandle = curl_easy_init();

  char execStr[128];
  sprintf(execStr, "http://%s/get_stuff.php?name=file_i_need.xml", ip);

  curl_easy_setopt(easyhandle, CURLOPT_URL, execStr);
  curl_easy_setopt(easyhandle, CURLOPT_NOSIGNAL, 1);
  curl_easy_setopt(easyhandle, CURLOPT_TIMEOUT, 10);
  int success =curl_easy_perform(easyhandle);

  cout << "Executing : " << execStr << " Reslut ( 0 = OK): " << success << endl;

  curl_easy_cleanup(easyhandle);

  pthread_mutex_lock(&COUNTER_MUTEX);
  --THREAD_COUNTER;
  pthread_mutex_unlock(&COUNTER_MUTEX);
}

int main ()
{
  char* ips[] = {
                        "10.1.1.1",
                        "10.1.1.2",
                        "10.1.1.3",
                        "10.1.1.4",
                        "10.1.1.5",
                        "10.1.1.6",
                        "10.1.1.7",
                        "10.1.1.8",
                        "10.1.1.9",
                        "10.1.1.10",
                        "10.1.1.11"};

  pthread_t thread_id;

  for ( int i=0; i<9; ++i)
  {
    pthread_create(&thread_id, NULL, &executeStuff, (void*) ips[i] );

    pthread_mutex_lock(&COUNTER_MUTEX);
    ++THREAD_COUNTER;
    pthread_mutex_unlock(&COUNTER_MUTEX);
  }

  while ( THREAD_COUNTER > 0 )
    sleep(20);

  cout << "Done!" << endl;

  return 0;
}

the amount of virtual memory consumed will be 90 MB, on the other hand if there are only 8 threads launched, only 80K of virtual memory be consumed.

Please advise,

Is this a normal behaviour or do I have problem with my code.

Thank you very much.

----------------------------------------------------------------------

>Comment By: nickt (ntopilski)
Date: 2007-01-13 18:52

Message:
Logged In: YES
user_id=1690842
Originator: YES

Dan,

I've followed your advice and added

curl_global_init(CURL_GLOBAL_ALL);

right at the top of main and
curl_global_cleanup();
before return. It did not address the issue.
Nevertheless, thank you for pointing out that I am missing an essential
piece of initialization and clean up
code.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-01-13 18:35

Message:
Logged In: YES
user_id=236775
Originator: NO

You need to call curl_global_init in main() before starting any threads.
You're probably hitting a race condition at 9 threads due to this improper
initialization.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1634515&group_id=976
Received on 2007-01-13