On Fri, 9 Sep 2005, Dave wrote:

> I think the existing code is pretty close to working the way I need it to.
> There seems to be two problems. Sometimes it just exits without doing any
> retry's, even though I specify "--retry 32" in my command line.

Retry (currently) only retries for transient errors (basically that means for
HTTP: response code >= 500) and timeouts, not for closed connections during a
successful transfer.

But --retry is a fairly new addition to curl, it most probably needs some
tweaking still to become really good.

> Sometimes it does a retry and continues appending data to the file but when
> it resets the elapsed time to the beginning. So if I specify "--max-time
> 100" and it runs for 99 seconds and then does a retry I get 199 seconds
> worth of data in my file. When it does multiple retry's the file just keeps
> getting bigger and bigger.

This should probably be considered a bug. I can't see how the current
behaviour can be useful, and it is indeed suprising to the user. I'll work on
a fix for this.

> So maybe these are bugs? It seems like it should keep doing retry's until it
> has tried 32 times.

Only if we change the current meaning for --retry. Also, when you download a
stream we don't know how big it'll be so how can curl tell if the closure is
premature or not?

> It also seems like it should keep track of the total time, even after doing
> a number of retry's.

Yes it should.

> Do you think it would be possible to add a "--Stream <bytes>" option to
> cURL? Its purpose would be to keep appending data to the file. It would
> automatically retry or reconnect whenever there is a problem. It would only
> quit after the file size had reached the specified number of bytes or if the
> retry parameters have been exceeded.

If no other ways exist to achieve this, then I don't think I'd mind. But I
wouldn't call it --stream since it would actually mean "continue to retry
downloading data from the URL at least until N bytes have been downloaded".
Stream or not.

> This option would also be useful in cases where the server allows the client
> to pre-buffer data (the server sends the stream faster then real-time). In
> this case the --max-time option doesn't work as well.

I still believe the range option is what you want and unless it works already,
we could investigate why it doesn't.

> The script idea is interesting but I am not sure how I can get it to work.
>
> I can use a script get it to run again but I need to have it append the new
> stream data to the file that it created when it ran (and failed) the first
> time. Is there an option to append to an existing file? I tried "-C -" and it
> doesnt work.

What happens then? -C - is the way to do resumed transfers. And that's the way
--retry does internally when it retries transfers.

> I also need to have it quit when the file reaches the correct size (or the
> proper amount of time has expired). Any suggestions as to how I can get that
> to work? I cant figure out how to do it with a Win-ME command line script.

In a sane script environment you could start the transfer in the background,
sleep for N seconds and then kill the transfer if it still is transferring.

For the maximum size downloaded, the range option seems to be what should
work.

Windows doesn't offer any proper scripting abilities at all by default, so I
can only recommend that you download and install something before you try to
script anything (like perl, python, tcl or whatever your personal preference
is).

curl the command line tool will never be able to provide every imaginable
option or combination of options. It does however provide enough options to
allow users to do almost everything, only sometimes some of the things are
better done with a little extra scripting.

Also, we offer libcurl bindings to almost every imaginable script or
development environment, should you consider the curl tool options too
restricted.

-- 
  Commercial curl and libcurl Technical Support: http://haxx.se/curl.html