Bugs item #2041827, was opened at 2008-08-07 17:40
Message generated for change (Comment added) made by bagder
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2041827&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: http
Group: crash
Status: Open
Resolution: Works For Me
Priority: 7
Private: No
Submitted By: Felix (felix-sf191)
Assigned to: Daniel Stenberg (bagder)
Summary: Segfault in http_output_auth w/ FORBID_REUSE (7.18.2)

Initial Comment:
I use the PHP API of libcurl, but the segfaults happen in libcurl itself. The situation is a bit complicated, but I try to include all information I have.

In my PHP application I do not receive a segmentation fault but a sig abort sent from glibc:

*** glibc detected *** double free or corruption (fasttop): 0x0879d1a0 ***

For my application is quite big I have tried to isolate the problem with a simpler script. Now I get segfaults instead of the sigabort, but that might be caused by the same problem.

I use the multi api to perform two requests to the same server at once. One of the requests receives a 200 response directly, the other has to follow one redirect to get the 200. Everything works fine until I set CURLOPT_FORBID_REUSE (= true). Then every now and then (more than 1 out of 10) the program segfaults with a backtrace ending in http_output_auth:

-------------------------------
#0 0xb7d57cea in http_output_auth () from libcurl.so.4
#1 0xb7d590bb in Curl_http () from libcurl.so.4
#2 0xb7d6587d in Curl_do () from lib/libcurl.so.4
#3 0xb7d74d8e in multi_runsingle () from libcurl.so.4
#4 0xb7d75839 in curl_multi_perform () from libcurl.so.4
#5 0x080b3e12 in zif_curl_multi_exec ()
#6 0x0838885f in zend_do_fcall_common_helper_SPEC ()
#7 0x083875d8 in execute ()
#8 0x08363170 in zend_execute_scripts ()
#9 0x08318ac2 in php_execute_script ()
#10 0x083f0494 in main ()
-------------------------------

As the client PHP script I have used the curl_multi_exec_example from http://de.php.net/manual/en/function.curl-multi-exec.php. Besides the URLs the only significant change is that I have CURLOPT_FORBID_REUSE set.
-------------------------------
$url = "http://example.com/redir.php";
$url2 = $url ."?uri=". urlencode($url);

// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();

// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, $url);
curl_setopt($ch2, CURLOPT_URL, $url2);

curl_setopt($ch1, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch2, CURLOPT_FORBID_REUSE, true);

curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, true);

/* from here on no changes to the example from http://de.php.net/manual/en/function.curl-multi-exec.php */

//create the multiple cURL handle
$mh = curl_multi_init();

//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);

$active = null;
//execute the handles
do {
    $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    if (curl_multi_select($mh) != -1) {
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
echo "\n";
-------------------------------

As the server side script you can use s/th like this:

-------------------------------
if (!empty($_GET['uri'])) {
    header("Location: ". $_GET['uri']);
} else {
    echo "done";
}
-------------------------------

I don't think it is http_output_auth's fault. That is probably just the first place where the error occurs.

The segfaults disapper when either
 * both requests are redirects
 * the two requests connect to different servers
 * CURLOPT_FORBID_REUSE is set to FALSE

I use PHP-5.2.6, libcurl-7.18.2 on Debian etch.

----------------------------------------------------------------------

>Comment By: Daniel Stenberg (bagder)
Date: 2008-08-19 23:35

Message:
Logged In: YES
user_id=1110
Originator: NO

Now I've tried it again with your URLs MANY times and none failed.

I don't seem to get re-used connections at all.

----------------------------------------------------------------------

Comment By: Felix (felix-sf191)
Date: 2008-08-12 09:55

Message:
Logged In: YES
user_id=2099045
Originator: YES

I also tried a slow script. But I think it must not be actually slow. The
200 has just to return before the 302. When the answer of the 200 comes
really late (3 seconds) then libcurl will not try to reuse it to follow the
redirect. So the responses have to be more or less at the same time.

I couldn't get the segfault with your new URLs.

But have you got a single run (with your URLs) that said "Re-using
existing connection!"?

How often have you tried my URLs? I had to run the program about 10 - 20
times to get the bad timing.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-12 00:18

Message:
Logged In: YES
user_id=1110
Originator: NO

Unfortunately that (using your URLs) didn't help either.

It still doesn't crash for me. I also tried creating a URL of mine
(http://daniel.haxx.se/test/slow.cgi) that waits 3 seconds before it
returns 200 so that the 302 one is the first to respond, but I still can't
make it fail... Does it fail for you if you use those URLs of mine?

----------------------------------------------------------------------

Comment By: Felix (felix-sf191)
Date: 2008-08-11 12:36

Message:
Logged In: YES
user_id=2099045
Originator: YES

I built curl using "./configure --prefix=... --enable-debug"
-V says:

curl 7.18.2 (i686-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8h zlib/1.2.3
libidn/0.6.5
Protocols: ftp http file https ftps
Features: IDN Largefile NTLM SSL libz

I have used your test program with my URLs. My server is local, so
responses are fast. That my be the reason why sometimes the 302 comes
first.

I do not have segfaults with your test program with your URLs. (Sorry for
the many requests in the last minutes :)). But I realized, that your URLs
have a different setup. I needed to have one direct 200 and one 302 to make
the segfault "work". So I have changed your first URL to receive a 200 from
your server. But still no segfault. I always get the 200 first from your
server.

Right now I have moved my redir-script to a slow, remote server. I still
see the same behaviour (segfaulting now and then). Maybe you can try with
my URLs. Change the first request to "http://213.221.113.41/test/redir.php"
and the second to
"http://213.221.113.41/test/redir.php?uri=http%3A%2F%2F213.221.113.41%2Ftest%2Fredir.php".

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-11 11:57

Message:
Logged In: YES
user_id=1110
Originator: NO

What's the configure line you use then? Or perhaps, what does curl -V show
with this built libcurl?

Also, is this with my test program? It does show different URLs in the
output...

I rebuilt libcurl 7.18.2 with debug, but still I can't make this fail and
valgrind reports no errors. I guess this is either depending on some
specific version of something underlying, or there's a timing issue or
something. Is your testing server local or remote? Ie how fast is the
response time when you connect and send requests to it?

----------------------------------------------------------------------

Comment By: Felix (felix-sf191)
Date: 2008-08-11 10:34

Message:
Logged In: YES
user_id=2099045
Originator: YES

I have rebuild libcurl with --enable-debug and I still get the segfault.
The short story:

#0 0xb7f9ae24 in Curl_do (connp=0x806e780, done=0xbf9f5910) at
url.c:4576
#1 0xb7fab2aa in multi_runsingle (multi=0x8050184, easy=0x806e774) at
multi.c:1102
#2 0xb7fabd7f in curl_multi_perform (multi_handle=0x8050184,
running_handles=0xbf9f5b44)
    at multi.c:1460
#3 0x080487ba in runner ()
#4 0x080488ed in main ()

The long version with complete output is attached.

But still the segfault occurs every time when a connection is to be reused
(which happens when the redirect returns first).
File Added: curl_segfault.txt

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-09 00:08

Message:
Logged In: YES
user_id=1110
Originator: NO

Very odd.

I tried again on a second machine with CVS HEAD (Debian Testing) and it
runs just as fine. I then tried to use the debian installed libcurl 7.18.2
version instead of CVS, and that too works just as fine.

I also tried running both versions with valgrind but it reported no
problems. Can you build libcurl from source with --enable-debug and get a
stack trace from the crash?

----------------------------------------------------------------------

Comment By: Felix (felix-sf191)
Date: 2008-08-08 11:03

Message:
Logged In: YES
user_id=2099045
Originator: YES

I have tried your c program on my machine and I still get the segmentation
fault every few runs:

-------------------------------------------
[New Thread -1211963712 (LWP 29199)]
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* About to connect() to fh.devel port 7825 (#1)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#1)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

> GET /tests/redir.php?uri=redir.php HTTP/1.1
Host: fh.devel7825
Accept: */*

< HTTP/1.1 302 Found
< Date: Fri, 08 Aug 2008 07:56:01 GMT
< Server: Apache
< Location: http://fh.devel:7825/tests/redir.php
< Content-Length: 0
< Content-Type: text/html
<
* Expire cleared
* Closing connection #1
* Issue another request to this URL:
'http://fh.devel:7825/tests/redir.php'
< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 07:56:01 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Re-using existing connection! (#0) with host fh.devel
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* Expire cleared
* Closing connection #0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1211963712 (LWP 29199)]
0xb7f1aa8a in http_output_auth () from libcurl.so.4
(gdb) bt
#0 0xb7f1aa8a in http_output_auth () from libcurl.so.4
#1 0xb7f1be5b in Curl_http () from libcurl.so.4
#2 0xb7f2890d in Curl_do () from libcurl.so.4
#3 0xb7f3786e in multi_runsingle () from libcurl.so.4
#4 0xb7f38319 in curl_multi_perform () from libcurl.so.4
#5 0x080487ba in runner ()
#6 0x080488ed in main ()
-------------------------------------------

I have build your program in a Debian Vserver with gcc (GCC) 4.1.2
20061115 (prerelease) (Debian 4.1.1-21)
using the command line:
gcc `curl-config --cflags` `curl-config --libs` test-2041827.c

On a CentOS 5.2 with gcc version 4.1.2 20071124 (Red Hat 4.1.2-42) I get
the sigabort that I originally got:
----------------------------------------------
Program received signal SIGABRT, Aborted.
[Switching to Thread -1212470720 (LWP 24705)]
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7dcad20 in raise () from /lib/libc.so.6
#2 0xb7dcc631 in abort () from /lib/libc.so.6
#3 0xb7e02e6b in __libc_message () from /lib/libc.so.6
#4 0xb7e0ab16 in _int_free () from /lib/libc.so.6
#5 0xb7e0e070 in free () from /lib/libc.so.6
#6 0xb7f048b2 in Curl_safefree (ptr=0x806d9d8) at url.c:283
#7 0xb7efbe51 in Curl_http (conn=0x806cd00, done=0xbf8fa068) at
http.c:2086
#8 0xb7f0885c in Curl_do (connp=0x806c9ec, done=0xbf8fa068) at
url.c:4578
#9 0xb7f19eaf in multi_runsingle (multi=0x8050170, easy=0x806c9e0) at
multi.c:1102
#10 0xb7f1a907 in curl_multi_perform (multi_handle=0x8050170,
running_handles=0xbf8fa274) at multi.c:1460
#11 0x0804878a in runner (multi_handle=0x8050170) at test-2041827.c:52
#12 0x080488bd in main () at test-2041827.c:96
---------------------------------------------------------

It is interesting how the verbose output differs when the program works
and when it fails.

Working version:
------------------------------------------
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* About to connect() to fh.devel port 7825 (#1)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#1)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

> GET /tests/redir.php?uri=http://fh.devel:7825/tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 08:37:09 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Closing connection #0
< HTTP/1.1 302 Found
< Date: Fri, 08 Aug 2008 08:37:09 GMT
< Server: Apache
< Location: http://fh.devel:7825/tests/redir.php
< Content-Length: 0
< Content-Type: text/html
<
* Closing connection #1
* Issue another request to this URL:
'http://fh.devel:7825/tests/redir.php'
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 08:37:09 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Closing connection #0
------------------------------------------
The direct 200 is returned first. Then the 302 returns. _Both_ connections
are closed. To follow the redirect another connection has to be opened.

Now the segfaulting/sigaborting version:
------------------------------------------
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* About to connect() to fh.devel port 7825 (#1)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#1)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

> GET /tests/redir.php?uri=http://fh.devel:7825/tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

< HTTP/1.1 302 Found
< Date: Fri, 08 Aug 2008 08:41:19 GMT
< Server: Apache
< Location: http://fh.devel:7825/tests/redir.php
< Content-Length: 0
< Content-Type: text/html
<
* Expire cleared
* Closing connection #1
* Issue another request to this URL:
'http://fh.devel:7825/tests/redir.php'
< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 08:41:19 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Re-using existing connection! (#0) with host fh.devel
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* Expire cleared
* Closing connection #0
segmentation fault
------------------------------------------

In this case the 302 answer comes first. That seems to lead to a different
behaviour. The connection of the redirected request is closed. When the
direct 200 arrives, that sane connection is reused to follow the redirect:
"Re-using existing connection! (#0)". Every time this message occurs, the
program fails.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-08 00:14

Message:
Logged In: YES
user_id=1110
Originator: NO

I wrote up a version of that program in plain C using libcurl directly,
and I cannot seem to repeat any crash out of it. See the attached file for
the exact code I used.

File Added: test-2041827.c

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2041827&group_id=976
Received on 2008-08-19