Bugs item #1634515, was opened at 2007-01-13 04:32
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1634515&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: libcurl
Group: bad behaviour
Status: Open
>Resolution: Invalid
Priority: 5
Private: No
Submitted By: nickt (ntopilski)
Assigned to: Daniel Stenberg (bagder)
Summary: Large Virtual Memory size

Initial Comment:
Hello,

I am a first time user of the libcurl library.
I am compiling and running under Fedora Core 5 on Intel machine.

I run into an issue that when launching libcurl from threads, if the number of threads is above 9 the virtual memory consumption increases disproportionatly.

In the code sample below:

#include <fstream>
#include <iomanip>
#include <time.h>
#include <cstdlib>
#include <cstdio>
#include <pthread.h>
#include <iostream>
#include <curl/curl.h>

using namespace std;

pthread_mutex_t COUNTER_MUTEX = PTHREAD_MUTEX_INITIALIZER;
unsigned int THREAD_COUNTER = 0;

void* executeStuff (void* arg )
{
  char* ip = (char*) arg;

  CURL* easyhandle = curl_easy_init();

  char execStr[128];
  sprintf(execStr, "http://%s/get_stuff.php?name=file_i_need.xml", ip);

  curl_easy_setopt(easyhandle, CURLOPT_URL, execStr);
  curl_easy_setopt(easyhandle, CURLOPT_NOSIGNAL, 1);
  curl_easy_setopt(easyhandle, CURLOPT_TIMEOUT, 10);
  int success =curl_easy_perform(easyhandle);

  cout << "Executing : " << execStr << " Reslut ( 0 = OK): " << success << endl;

  curl_easy_cleanup(easyhandle);

  pthread_mutex_lock(&COUNTER_MUTEX);
  --THREAD_COUNTER;
  pthread_mutex_unlock(&COUNTER_MUTEX);
}

int main ()
{
  char* ips[] = {
                        "10.1.1.1",
                        "10.1.1.2",
                        "10.1.1.3",
                        "10.1.1.4",
                        "10.1.1.5",
                        "10.1.1.6",
                        "10.1.1.7",
                        "10.1.1.8",
                        "10.1.1.9",
                        "10.1.1.10",
                        "10.1.1.11"};

  pthread_t thread_id;

  for ( int i=0; i<9; ++i)
  {
    pthread_create(&thread_id, NULL, &executeStuff, (void*) ips[i] );

    pthread_mutex_lock(&COUNTER_MUTEX);
    ++THREAD_COUNTER;
    pthread_mutex_unlock(&COUNTER_MUTEX);
  }

  while ( THREAD_COUNTER > 0 )
    sleep(20);

  cout << "Done!" << endl;

  return 0;
}

the amount of virtual memory consumed will be 90 MB, on the other hand if there are only 8 threads launched, only 80K of virtual memory be consumed.

Please advise,

Is this a normal behaviour or do I have problem with my code.

Thank you very much.

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2007-01-17 11:11

Message:
Logged In: YES
user_id=1110
Originator: NO

Given that the posted examples have been pretty bad libcurl-using
citizens, I can't say that I acknowledge this report.

Can you please provide an updated example that repeats this claimed
problem? Also, plese tell us what libcurl version you're using.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-01-14 01:45

Message:
Logged In: YES
user_id=236775
Originator: NO

The pointer in write_data should definitely NOT be freed in the callback.
Also, there are still writes to stdout occuring, via cout. If those are
not thread safe, then they need to be protected as well.

Beyond that, I don't know what else could be causing the problem. Can you
use libcurl's memory debugging facility or valgrind to track down the
source of the memory use?

----------------------------------------------------------------------

Comment By: nickt (ntopilski)
Date: 2007-01-14 00:21

Message:
Logged In: YES
user_id=1690842
Originator: YES

I've modified the code to handle write callback:

size_t write_data ( void* ptr, size_t size, size_t nmemb, void *stream )
{
  pthread_mutex_lock(&WRITE_MUTEX);
  curl_free(ptr);
  pthread_mutex_unlock(&WRITE_MUTEX);

  return size*nmemb;
}

void* executeStuff (void* arg )
{
  char* ip = (char*) arg;

  CURL* easyhandle = curl_easy_init();

  char execStr[128];
  sprintf(execStr, "http://%s/get_stuff.php?name=file_i_need.xml", ip);

  curl_easy_setopt(easyhandle, CURLOPT_URL, execStr);
  curl_easy_setopt(easyhandle, CURLOPT_NOSIGNAL, 1);
  curl_easy_setopt(easyhandle, CURLOPT_TIMEOUT, 10);
  curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, write_data);
  int success =curl_easy_perform(easyhandle);

  cout << "Executing : " << execStr << " Reslut ( 0 = OK): " << success <<
endl;

  curl_easy_cleanup(easyhandle);

  pthread_mutex_lock(&COUNTER_MUTEX);
  --THREAD_COUNTER;
  pthread_mutex_unlock(&COUNTER_MUTEX);
}

But the issue still persists. Since the IPs that I have
are really fake the issue seems to be in their initialization.
But good point to keep in mind.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-01-13 22:32

Message:
Logged In: YES
user_id=236775
Originator: NO

You're also not setting your own write callback function, so the default
libcurl one is going to run which writes the output to stdout using
fwrite. I doubt that's thread safe by default. Try setting your own write
callback function and put a mutex around the fwrite.

----------------------------------------------------------------------

Comment By: nickt (ntopilski)
Date: 2007-01-13 19:52

Message:
Logged In: YES
user_id=1690842
Originator: YES

Dan,

I've followed your advice and added

curl_global_init(CURL_GLOBAL_ALL);

right at the top of main and
curl_global_cleanup();
before return. It did not address the issue.
Nevertheless, thank you for pointing out that I am missing an essential
piece of initialization and clean up
code.

----------------------------------------------------------------------

Comment By: Dan Fandrich (dfandrich)
Date: 2007-01-13 19:35

Message:
Logged In: YES
user_id=236775
Originator: NO

You need to call curl_global_init in main() before starting any threads.
You're probably hitting a race condition at 9 threads due to this improper
initialization.

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=1634515&group_id=976
Received on 2007-01-17