On Son 10.06.2007 11:43, Robert Iakobashvili wrote:
>On 6/6/07, Daniel Stenberg <daniel_at_haxx.se> wrote:
>
>>I'm not following. libcurl will only and always use one single
>>connection to do a HTTP request.
>>
>>
>Let's say, somebody wants to emulate behavior of Firefox or MSIE, using
>libcurl.

That's a very interesting question for a commandline tool, imho.

>To the best of my knowledge, the browsers are fetching a page, looking
>into the page body and are either re-using existing connection by
>pipelining or opening new connections to fetch the found images.
>
>Any your ideas regarding the "best mode" of such implementation and
>which current features or new libcurl features to be added to
>facilitate such emulation would be very much appreciated. Thank you.

It is a similar behavior as a crawler isn't it?!

I think there are more then one way ;-)

I would go like this:

---
set max_connections_per_site <NUM>
set deep_or_breadth          (DEEP|BREADTH)
set deep_or_breadth_count    <NUM>
set wait_human_read_time     <NUM>
repeat as long as deep_or_breadth_count is not reached {
get site (e.g.: index.html)
parse site and ( count links and get the needed elements from remote
                 server
              || if element is to get make new connection but not more
                 the max_connections_per_site
               )
wait_human_read_time()
if breadth && deep_or_breadth_count not reached {
    get the next link from same directory-level (e.g.: /, /news/, ...)
  }elsif deep && deep_or_breadth_count not reached {
    get the next link from the next directory-level
  }
}
---
I'am not sure if this is a behaviour which libcurl should have or the
curl-tool or some other programs which use libcurl.
Do you plan to add this into curl-loader ;-)?
thoughts
Aleks