We have implemented a limiting function in the callback made by
cURL when it crawls a site. Our code can be checked out via CVS
on SourceForge, or you can download it from our site at grub.org.

Following is a snippet of the code that we added to the callback,
the rest of the code is also located in the Crawler.cpp file in
the crawler directory, notably the ::adjustbandwidth() routine that
takes care of calculating the current bandwidth usage and saving
that into the struct that you see the callback accessing here.

// located in write memory callback function
// begin our bandwidth limiting routine
while (true && mem->bandwidth_pointer)
  {
    // get semaphore
    pthread_mutex_lock(&band_mutex);

    int usage = mem->bandwidth_pointer->bandwidth_usage;
    int limit = mem->bandwidth_pointer->bandwidth_limit;

    if (usage > limit)
    {
       // fetch our sleep time
       int sleep_time = mem->bandwidth_pointer->bandwidth_time
                        - time(NULL);

       if (sleep_time < 1) sleep_time = 1;

       // release the semaphore before we start waiting
       pthread_mutex_unlock(&band_mutex);

       // hang out until we have bandwidth
       sleep(sleep_time);
    }
    else
    {
       // increment the current bandwidth usage
       mem->bandwidth_pointer->bandwidth_usage += realsize;

       // release our semaphore
       pthread_mutex_unlock(&band_mutex);

       // break out of loop and proceed
       break;
    }
}

Hope this helps!

Kord

On Thu, 28 Mar 2002, Guido Neitzer wrote:

> Hi.
>
> Is it possible to limit the bandwidth libcurl uses for its connections?
>
> Background: I'm working on a synching tool e.g. for websites (ftp-
> based directory sync) and don't want to kill our whole bandwidth with
> one or more concurrent sync-jobs. Our webservers should be accessible
> and fast enough.
>
>
> cug
>
>
>
Received on 2002-03-28