On Thu, 7 Sep 2006, Christian Grade wrote:

> My requirement is/was to have a libcurl module which consumes data (remote
> paths/urls) from [two] producing threads. At the same time, the libcurl
> module is/was supposed to produce data (mainly files) for [three] consuming
> threads. Modelling the logics for the *data- structure-wise separation* of
> the *transfer statús*, I found I'd better adapt "multi.c" by building in
> some locking mechanisms and by replacing the "multi.c" list with a
> *lock-free* alternative: one thread transferring, one thread retrieving data
> about transfers. I didn't get far with this.

I think it sounds totally crazy. To me, it sounds like you've had made your
mind up how this would be done already before you ever saw libcurl or read its
documentation. And since then you've tried to squeeze libcurl into working
with this design.

When using the multi interface to transfer multiple files, it doesn't make any
sense to split it up into multiple threads. If using many threads is your
game, then I suggest you instead simply use indivual easy transfers in each
thread.

> When the necessity arose to even adapt "url.c" (monitored strings) and as I
> didn't find out how to retrieve a stack of redirection urls and when I heard
> the multi-interface will undergo a major overhaul soon anyway, I took a
> break from fiddling with it, pondering about alternatives.

Well, the multi interface API is not about to change, but the internals are
gonna be somewhat changed within a few days when I commit the HTTP pipelining
support. Further, "a stack of redirection urls" is not a problem to the multi
interface. Not now and not tomorrow.

You will get a slight worry if the stack is more than 1000 simultanoues
transfers using the goold old *perform() approach but you can then switch to
the *socket() way and enjoy unlimited number of transfers supported at a speed
that I don't think any other HTTP library in the wild can match.

> I must have overlooked toUpper( "curlopt_private" ). Keyword 'private'
> (suggesting "better keep hands off")

Yes, libcurl keeps its hands off it. Private for you, the application.

> Now the existence of this option fulfills the *should-at-least-have* part.

Thanks for your conforting words. You sure know how to take people.

> I see three transfer scenarios (downstreaming):
> [1] Module supplied with url, file name, optional offset (resumption)
> [2] Module supplied with url, continuous buffer, salvation function
> [3] Module supplied with url, chunked buffer, what-if/process function

libcurl can of course support them all fine.

> This introduces new data members which one has to care for: book-keeping.

Yeah, sure. You have the data and the URLs in some kind of entities.

> So, one could associate these with 'easy handles' but it would be more
> performant, less tedious to implement if these were in the multi-interface
> already.

What would be in the multi interface already?

> One would have two lists for the same concept in parallel otherwise.

Two lists of what?

> So, why 'unique' IDs for connections?

There are "unique IDs", not for connections but for transfers. They are called
easy handles.

> When the libcurl module is running, and this practically an infinite mode

In your case, it seems to be yes.

> there's little need to cope with 'easy handles' themselves outside a
> universal module.

The easy handle is the handle to a specific transfer. This is a very common
approach used all over. How else would you control options and specifics for a
single transfer?

> The number of maximum simultaneous transfers is 'pretty' constant; naturally
> subject to hardware limitations.

Constant? Yes if you fill up all your RAM and when all your other apps have
been swapped to disk, then there's likely a fixed limit on the amount of
simultaneous transfers you can do on the single particular host you use. But
in reality that won't appear as constant.

> Why would the outside have to care which handle is used for a connection?

Because you want to set specific options for each single transfer. Because you
want to know for which particular transfer data comes from or goes to. Because
the multi interface is a sister-API to the easy interface and makes life
simple to users who wants to move from asynchronous easy transfers to
asynchronous multi transfers.

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html