curl-library

From: Serge van den Broek <SVandenbroek_at_isabel.be> Date: Mon, 12 Sep 2005 15:30:14 +0200 · > 2) I tried, to solve the above problem, to rebuild my application with

--
Michael - < mike(@)php.net >
------------------------------
Message: 2
Date: Fri, 09 Sep 2005 17:32:32 +0200
From: Gisle Vanem <giva_at_bgnett.no>
Subject: AF_UNSPEC and bind()
To: curl-library <curl-library_at_cool.haxx.se>
Message-ID: <170701c5b553$b24e8310$0600000a_at_broadpark.no>
Content-Type: text/plain; format=flowed; charset=iso-8859-1;
	reply-type=original
I commited this change in tftp.c:
@@ -533,12 +535,18 @@
   state->sockfd = state->conn->sock[FIRSTSOCKET];
   state->state = TFTP_STATE_START;
 
+#ifdef WIN32
+  /* AF_UNSPEC == 0 (from above calloc) doesn't work on Winsock */
+  state->local_addr.sa_family = conn->ip_addr->ai_family;
+#endif
+
and wonder if not this should be done for all. I'm not sure why
AF_UNSPEC
magically works on Linux (that's where this was tested, no?) and what it
really means.
w/o the above change, I got "Address family not supported" under Winsock
(no surprice).
--gv
------------------------------
Message: 3
Date: Fri, 09 Sep 2005 08:45:58 -0700
From: Jonathan Cobb <jonathan_at_kyuss.org>
Subject: segfault in Curl_hostcache_prune
To: curl-library_at_cool.haxx.se
Message-ID: <4321AE36.5010701_at_kyuss.org>
Content-Type: text/plain; charset=us-ascii; format=flowed
Hi all,
I'm writing an apache module that uses libcurl internally.  We're using
the latest libcurl (1.14.1)
We allocate a curl handle using memory from a pool associated with the
http request.  When we allocate the handle, we register a cleanup
function with the pool.  So, when the request completes the pool cleanup
function is called, which then calls the curl_cleanup function.
Sometimes (infrequently and difficult to reproduce), a segfault occurs
in the cleanup.  The backtrace is below, at the end of this message.
The segfault is caused by dereferencing a null pointer, in the
Curl_hostcache_prune function (in hostip.c):
void Curl_hostcache_prune(struct SessionHandle *data)
{
   time_t now;
   if((data->set.dns_cache_timeout == -1) || !data->hostcache)
     /* cache forever means never prune, and NULL hostcache means
        we can't do it */
     return;
...
}
Somehow, the "data" pointer is null.  I added
   if (!data || ...)
to the first "if" check and I'm thinking this will make it go away.
But I'm wondering if this is the right thing to do.  Under what
conditions would this pointer be null?  Also, I have a suspicion that
maybe our cleanup is getting called twice - if that were the case, could
that cause the data pointer to be null?
Any/all help is much appreciated.  If there's more info I can provide to
help just let me know.
Thanks,
    - jonathan
---- backtrace --------------------------------------------------
(gdb) bt
#0  Curl_hostcache_prune (data=0x0) at hostip.c:260
#1  0x00422e27 in Curl_disconnect (conn=0x92400f0) at url.c:1422
#2  0x004232be in ConnectionKillOne (data=0x9235598) at url.c:1678
#3  0x00421b95 in Curl_close (data=0x9235598) at url.c:204
#4  0x0041a83f in curl_cleanup (ctx=0x91d8740) at mod_px.c:363
#5  0x003fee6d in apr_pool_cleanup_run () from /usr/lib/libapr-0.so.0
#6  0x003fe5bd in apr_pool_destroy () from /usr/lib/libapr-0.so.0
#7  0x003fe578 in apr_pool_clear () from /usr/lib/libapr-0.so.0
#8  0x08066995 in child_main (child_num_arg=152930112) at
/usr/src/debug/httpd-2.0.46/server/mpm/prefork/prefork.c:613
#9  0x08066c34 in make_child (s=0x5, slot=49) at
/usr/src/debug/httpd-2.0.46/server/mpm/prefork/prefork.c:788
#10 0x08066ed9 in perform_idle_server_maintenance (p=0x90740a8) at
/usr/src/debug/httpd-2.0.46/server/mpm/prefork/prefork.c:923
#11 0x08067570 in ap_mpm_run (_pconf=0x1c, plog=0x909e150, s=0x0) at
/usr/src/debug/httpd-2.0.46/server/mpm/prefork/prefork.c:1118
#12 0x0806da4f in main (argc=1, argv=0xbfff9dc4) at
/usr/src/debug/httpd-2.0.46/server/main.c:661
------------------------------
Message: 4
Date: 09 Sep 2005 10:29:32 -0600
From: "John F. Kelly" <johnfkelly_at_earthlink.net>
Subject: Re: AF_UNSPEC and bind()
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <1126283373.5388.12.camel_at_nikita>
Content-Type: text/plain
On Fri, 2005-09-09 at 09:32, Gisle Vanem wrote:
> 
> and wonder if not this should be done for all. I'm not sure why
AF_UNSPEC
> magically works on Linux (that's where this was tested, no?) and what
it
> really means.
The OS probably selects a rational value from what it knows about the
socket.
> 
> w/o the above change, I got "Address family not supported" under
Winsock
> (no surprice).
Hmm....it worked for me on Linux 2.4, 2.6, Solaris 8 and Windows XP
(MinGW).  Where are you seeing the problem?
> 
> --gv
------------------------------
Message: 5
Date: Fri, 09 Sep 2005 18:36:56 +0200
From: Gisle Vanem <giva_at_bgnett.no>
Subject: Re: AF_UNSPEC and bind()
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <000901c5b55c$b16d5bc0$0600000a_at_broadpark.no>
Content-Type: text/plain; format=flowed; charset=iso-8859-1;
	reply-type=original
> Hmm....it worked for me on Linux 2.4, 2.6, Solaris 8 and Windows XP
> (MinGW).  Where are you seeing the problem?
In bind(). Your original tftp.c ignored the ret-val from bind(), so no
wonder it worked for you :-)
--gv
------------------------------
Message: 6
Date: 09 Sep 2005 11:24:39 -0600
From: "John F. Kelly" <johnfkelly_at_earthlink.net>
Subject: Re: AF_UNSPEC and bind()
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <1126286679.5390.21.camel_at_nikita>
Content-Type: text/plain
On Fri, 2005-09-09 at 10:36, Gisle Vanem wrote:
> 
> In bind(). Your original tftp.c ignored the ret-val from bind(), so no
> wonder it worked for you :-)
In bind() on which platform?  Again, it works --bind() and all---on
linux 2.4, 2.6, Solaris and XP.
14194 bind(3, {sa_family=AF_UNSPEC,
sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
14194 recvfrom(3, "\0\3\0\24 -f \"$(DESTDIR)$(bindir)/$$f"..., 516, 0,
{sa_family=AF_INET, sin_port=htons(33620),
sin_addr=inet_addr("135.9.39.88")}, [16]) = 516
> 
> --gv
------------------------------
Message: 7
Date: Fri, 09 Sep 2005 13:49:13 -0400
From: Brian Akins <bakins_at_web.turner.com>
Subject: Re: multithreads or multi interface
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <4321CB19.3090104_at_web.turner.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Martin Vogt wrote:
> No, this would overcomplicate it. And you have context switches.
> I hope to get 16000 requests per second, but I may be dreaming.
>
I know that ab can do 30k + requests per second... it uses poll.
-- 
Brian Akins
Lead Systems Engineer
CNN Internet Technologies
------------------------------
Message: 8
Date: Fri, 9 Sep 2005 23:06:49 +0200 (CEST)
From: Daniel Stenberg <daniel-curl_at_haxx.se>
Subject: Re: curl_multi_fdset alternatives
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <Pine.LNX.4.63.0509091316130.31314_at_yvahk3.pbagnpgbe.fr>
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
On Fri, 9 Sep 2005, Alexander Lazic wrote:
> My main point was libcurl have sockets and poll()/select on these, the
app 
> have sockets an $POLL on these, therefor when i use
curl_multi_socket() the 
> poll()/select() of the libcurl i can't influence, i hope i have say it
> clearer.
Right. The application can't influence that, no, but it has no need to.
If the 
app just selects/polls/your-favourite-poll-replacement on the sockets
and tell 
libcurl what socket that had action with curl_multi_socket() then the
lib can 
do it any way it likes internally, without it affecting the application.
Possibly it should just trust the application and don't do any
select/poll at 
all!
> The app must know which are his own sockets an which are the libcurl 
> sockets, imho.
Oh yes it really should, since among other things it should tell libcurl
when 
action has been dected on libcurl's sockets and not otherwise.
>>> I think this is the major problem because some functions make their
own 
>>> socket()/connect() calls.
>> 
>> The application's sockets, connects etc are not a concern of libcurl.
The 
>> application deals with its own sockets.
>
> Yes but when i call the *easy* or *multi* functions they make some 
> handshakes, such as ssl or ftp or whatever, how can i see if the
protocol 
> handshakes was ok or nok?
First, this curl_multi_socket() thing of course implies using the multi 
interface.
But then, why would the application care or even want to know about the 
connect/handshake status of each connection? It makes no difference to
it.
> I think we need some more states then:
...
> 5 CURL_CONNECT_OK      - the protocol handshake was ok
> 6 CURL_CONNECT_NOK     - the protocol handshake was nok
What for? The application waits for "action" on libcurl's sockets and
then it 
tells libcurl there is action. Nothing more.
>> This fix, together with added HTTP pipeling support, ought to make
libcurl 
>> outperform most of the competition.
>
> What is higher in your priority list, if you have one ;-)?
Since I might get funded for this task, I'm waiting for that to get
decided 
until I proceed with this topic. I expect to get further info no later
than 
October.
But in general my TODO is flexible and if I'm not totally occupied by
people's 
problems on the lists or keeping the web site in shape, I usually pick
an 
issue in docs/TODO or docs/KNOWN_BUGS that I feel is challanging.
-- 
  Commercial curl and libcurl Technical Support:
http://haxx.se/curl.html
------------------------------
Message: 9
Date: Fri, 9 Sep 2005 23:33:22 +0200 (CEST)
From: Daniel Stenberg <daniel-curl_at_haxx.se>
Subject: Re: segfault in Curl_hostcache_prune
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <Pine.LNX.4.63.0509092325351.2177_at_yvahk3.pbagnpgbe.fr>
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
On Fri, 9 Sep 2005, Jonathan Cobb wrote:
> I'm writing an apache module that uses libcurl internally.  We're
using the 
> latest libcurl (1.14.1)
I assume that's a typo and you mean 7.14.1.
> Sometimes (infrequently and difficult to reproduce), a segfault occurs
in 
> the cleanup.  The backtrace is below, at the end of this message.
>
> The segfault is caused by dereferencing a null pointer, in the 
> Curl_hostcache_prune function (in hostip.c):
...
> Somehow, the "data" pointer is null.  I added
>  if (!data || ...)
> to the first "if" check and I'm thinking this will make it go away.
It might hide the problem for you, but I would guess you'll get a crash 
elsewhere instead. See below.
> But I'm wondering if this is the right thing to do.  Under what
conditions 
> would this pointer be null?  Also, I have a suspicion that maybe our
cleanup 
> is getting called twice - if that were the case, could that cause the
data 
> pointer to be null?
Let me first quickly describe how the two main structs within libcurl
are 
related.
There's the SessionHandle struct. That is the main one, which is
returned as a 
handle in the curl_easy_init() function. This struct will always be
around as 
long as there is an easy handle (CURL *). This pointer is always called
'data' 
in the code.
Within the 'data' struct, there is a list of 'struct connectdata'. One
such 
for each connection this SessionHandle keeps alive (usually the max
amount is 
5).
Within the connectdata struct, there's a pointer called 'data' that
points 
back to the "parent" SessionHandle struct. Thus, this pointer can never 
legally be NULL and will always point to a valid struct.
In the backstrace you showed, the 'data' member within the connectdata
struct 
obviously was NULL when used within Curl_disconnect(). That simply
shouldn't 
happen. It shows a major error has occurred already then.
Calling the cleanup twice might cause something like that, or possibly
if 
memory is overwritten either by your app or by libcurl itself.
I have not seen any similar bug getting reported nor have this area of
the 
code been modified in a long while.
-- 
  Commercial curl and libcurl Technical Support:
http://haxx.se/curl.html
------------------------------
Message: 10
Date: Fri, 09 Sep 2005 15:32:06 -0700
From: Jonathan Cobb <jonathan_at_kyuss.org>
Subject: Re: segfault in Curl_hostcache_prune
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <43220D66.5000508_at_kyuss.org>
Content-Type: text/plain; charset=us-ascii; format=flowed
Daniel Stenberg wrote:
> On Fri, 9 Sep 2005, Jonathan Cobb wrote:
> 
>> I'm writing an apache module that uses libcurl internally.  We're 
>> using the latest libcurl (1.14.1)
> 
> 
> I assume that's a typo and you mean 7.14.1.
Yes, that was a typo.
>> Somehow, the "data" pointer is null.  I added
>>  if (!data || ...)
>> to the first "if" check and I'm thinking this will make it go away.
> 
> 
> It might hide the problem for you, but I would guess you'll get a
crash 
> elsewhere instead. See below.
Yup, that "fix" still resulted in a segfault, somewhere else.
> Calling the cleanup twice might cause something like that, or possibly
> if memory is overwritten either by your app or by libcurl itself.
> 
> I have not seen any similar bug getting reported nor have this area of
> the code been modified in a long while.
A thought -- could this be caused by opening a CURL handle with 
curl_easy_init, then calling curl_easy_reset (possibly multiple times)? 
  I'm guessing that curl_easy_reset is safe to call multiple times and 
would not reset the data pointer, but I thought I'd ask.
I've switched the allocation of the CURL handle from the request pool to
the connection pool.  The idea is that Apache will sometimes reuse a 
single TCP connection across multiple requests (to support http 
keepalives).
I'm thinking maybe the request cleanup is getting called multiple times 
(even though there should be separate request pools for each request), 
and the connection cleanup will only be called once (though now there 
will be multiple CURL handles in the pool to cleanup).  It might be a 
long shot, but I'll see if that fixes things.
   - jonathan.
------------------------------
Message: 11
Date: Sat, 10 Sep 2005 10:18:50 +0200
From: Alexander Lazic <al-curllibrary_at_none.at>
Subject: Re: curl_multi_fdset alternatives
To: libcurl development <curl-library_at_cool.haxx.se>
Message-ID: <20050910081850.GA24446_at_none.at>
Content-Type: text/plain; charset=us-ascii; format=flowed
On Fre 09.09.2005 23:06, Daniel Stenberg wrote:
>On Fri, 9 Sep 2005, Alexander Lazic wrote:
>
>Possibly it should just trust the application and don't do any
select/poll 
>at all!
Ok.
>>The app must know which are his own sockets an which are the libcurl
>>sockets, imho.
>
>Oh yes it really should, since among other things it should tell
>libcurl when action has been dected on libcurl's sockets and not
>otherwise.
Hm do you mean that libcurl also call the (*curl_socket_callback) for
they sockets?
>First, this curl_multi_socket() thing of course implies using the multi
>interface.
Sure, sorry.
>But then, why would the application care or even want to know about the
>connect/handshake status of each connection? It makes no difference to
>it.
Hm if the callback get CURL_POLL_(IN|OUT|INOUT) then the app can
read()/write() to the socket, right?
I think i can't help you for this, because i'am not sure that i have
understand the hole idea behind the curl_multi_socket() :-(
Greetings
al ;-)
------------------------------
_______________________________________________
curl-library mailing list
curl-library_at_cool.haxx.se
http://cool.haxx.se/mailman/listinfo/curl-library
End of curl-library Digest, Vol 20, Issue 21
********************************************
 
Serge Vandenbroek
Analyst Programmer
E-mail : SVandenbroek_at_isabel.be
Tel   : +32 (0)2 545.17.67
Fax  : +32 (0)2 545.17.19
 
Isabel NV/S.A. 
Keizerinlaan 13/15 Bld de l'Imperatrice 
1000 Bruxelles /Brussel
RPR Bruxelles /RPM Brussel: BE 0455 530 509
--------------------------------------------------------
Need assistance? 
Get help from Isabel Web Support 
www.isabel.be/support
--------------------------------------------------------
DISCLAIMER : Confidential information may be contained in this message and it may be legally privileged. If you are not the addressee indicated in this message (or responsible for delivery of the message to such person), you may not copy or deliver this message to anyone. In such case, you should destroy this message and kindly notify the sender by reply email. Please advise immediately if you or your employer does not consent to Internet email for messages of this kind. Opinions, conclusions and other information in this message that do not relate to the official business of Isabel shall be understood as neither given nor endorsed by it. Messages and attachments are swept by a virusscanner. If this message contains password-protected attachments, the files have not been scanned for viruses by the Isabel mail domain.


curl-library

Error recompiling curl with more recent version