cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: ftp enhancement - FTP wildcard download

From: Pavel Raiskup <xraisk00_at_gmail.com>
Date: Thu, 28 Jan 2010 09:09:59 +0100

> It would
> be great to make it working with the easier variant first, though keeping
> the future enhancement in mind. If time permits, you can move to the
> more
> complex variant later.

Yep, I'm going to try do it so. But it could be a little uncomfortable
do it opened for future improving and also stay effective.

>> int wildcard_callback(char *matchedstring, FILE *f,
>> int filetype)
>> {
>> // keep "f" or close it and open new .. by default keep on "f ==
>> stdout"
>> return 0; // discard downloading this file .. or
>> return 1;
>> }
>

> The incoming data may be simply processed on the fly. I
> suggest to pass a general (void *) pointer to the callback, previously
> set by
> the application. Perhaps shared with CURLOPT_WRITEDATA or something?

I think CURLOPT_WRITEDATA expects FILE *ptr (but yes, in prototype should
stay void pointer) -- and it should not broke the possibility of
processing incoming data on the fly. It is possible this way: Not set the
"CURLOPT_WILDCARDFUNCTION" and set CURLOPT_WRITEFUNCTION.. All the matched
files are processed in this callback .. possibly you can change only
CURLOPT_WRITEDATA to let the lib save all incomming data to the FILE
pointer without any callbacks. I thought the CURLOPT_WILDCARDFUNCTION
should be used e.g. for changing FILE* pointer to other file in local
filesystem for each matched file. It could be useful for copy remote files
in matched directory to "local" dependently to matched filename. (for
example ftp://site/dir/to/1.txt => ~/dir/to/1.txt; ftp://site/dir/to/2.txt
=> ~/dir/to/2.txt; etc ..)

>> And I have forgotten to idea about deepness of recoursive match ...
>
> I also don't think it would be that easy. For inspiration you can look
> at the gnulib FTS module works to estimate what walk-flags etc. are
> usually necessary for such a traversal:
>
> http://www.kernel.org/doc/man-pages/online/pages/man3/fts.3.html

I agree, this module is too complex. But I dont't think it is necessary to
resolve pre-order, post-order, link cycles, and other problems. I was
hoping to implement simply pre-order -- it allows user to react e.g. to
dynamically creating directories or creating files in directories. If I
solve it only over FTP, I can neglect symlinks too. Only resolving the
stop value which sais how deep can I go.. I don't know if this principle
makes itself unusable for other protocols and it could be the problem?
Note that discarding concrete file(s) is not impossible, it is not
necessary to resolve it in tree-traversal algorithm -- it could be done in
callback.

Pavel
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2010-01-28