Re: Reducing memory consumption.
Date: Sun, 14 Dec 2008 19:57:56 +0100 (CET)
On Sun, 14 Dec 2008, Tsachi wrote:
> (I am using "libcurl 7.19.1" with c-ares 1.5.3 (over linux)). I am writing a
> program which fetches many http links simultaneously. I have noticed that
> every time I call curl_easy_init() around 70K of memory is allocated. Now
> taking into consideration that I want to use around 1000 handles, this might
> be an issue (~70 MB in memory).
> Is there a way to reduce that ?
Not any particular that don't require code changes, no. But I think in
A) some of the allocations that are made could possibly be re-used better
between easy handles, like the buffer or uploadbuffer areas that
probably are only used one at a time anyway
B) more buffers can be made allocated on-demand (like uploadbuffer for
downloads and buffer for uploads) to reduce the initial requirement
It's been a while since we worked on reducing the memory consumption but I do
think it could be worth investigating where the biggest allocations are made
and how much of them that are truly needed for each handle.
-- / daniel.haxx.seReceived on 2008-12-14