cURL / Mailing Lists / curl-library / Single Mail


Re: How to disconnect when unexpectable data coming ?

From: Daniel Stenberg <>
Date: Sun, 13 Oct 2002 12:39:20 +0200 (MET DST)

On Sun, 13 Oct 2002, [gb2312] wrote:

> I am trying to write a auto-downloading web crawler . But I do not want to
> download some file which are bigger then expected, for example bigger than
> 10M . So I decide to calculate the Content-Length , if Content-Length > 10
> M, disconnet it . Can I still use the persistent connection capability?

If you use libcurl for this task, it'll attempt and use peristent connections
as much as possible. So, yes, it will.

But if you want to stop a download in the middle, like in this deescribed
scenario, there is no way for a client to tell the server to stop the
download while still keeping the connection alive. The only way to stop
receiving this once it started, is to disconnect the TCP stream. This of
course breaks the persistency.

> Anyone please give me some hints how to perform operation like this with
> efficient performance ?

The only really safe way to do this (and retain persistency), would be to
HEAD the document first and avoid GETting it if it is too large.

Of course, I might be wrong.

 Daniel Stenberg -- curl, cURL, Curl, CURL. Groks URLs.
This email is sponsored by:ThinkGeek
Welcome to geek heaven.
Received on 2002-10-13