Bugs item #2041827, was opened at 2008-08-07 17:40
Message generated for change (Comment added) made by felix-sf191
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2041827&group_id=976

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: http
Group: crash
Status: Open
Resolution: Works For Me
Priority: 5
Private: No
Submitted By: Felix (felix-sf191)
Assigned to: Daniel Stenberg (bagder)
Summary: Segfault in http_output_auth w/ FORBID_REUSE (7.18.2)

Initial Comment:
I use the PHP API of libcurl, but the segfaults happen in libcurl itself. The situation is a bit complicated, but I try to include all information I have.

In my PHP application I do not receive a segmentation fault but a sig abort sent from glibc:

*** glibc detected *** double free or corruption (fasttop): 0x0879d1a0 ***

For my application is quite big I have tried to isolate the problem with a simpler script. Now I get segfaults instead of the sigabort, but that might be caused by the same problem.

I use the multi api to perform two requests to the same server at once. One of the requests receives a 200 response directly, the other has to follow one redirect to get the 200. Everything works fine until I set CURLOPT_FORBID_REUSE (= true). Then every now and then (more than 1 out of 10) the program segfaults with a backtrace ending in http_output_auth:

-------------------------------
#0 0xb7d57cea in http_output_auth () from libcurl.so.4
#1 0xb7d590bb in Curl_http () from libcurl.so.4
#2 0xb7d6587d in Curl_do () from lib/libcurl.so.4
#3 0xb7d74d8e in multi_runsingle () from libcurl.so.4
#4 0xb7d75839 in curl_multi_perform () from libcurl.so.4
#5 0x080b3e12 in zif_curl_multi_exec ()
#6 0x0838885f in zend_do_fcall_common_helper_SPEC ()
#7 0x083875d8 in execute ()
#8 0x08363170 in zend_execute_scripts ()
#9 0x08318ac2 in php_execute_script ()
#10 0x083f0494 in main ()
-------------------------------

As the client PHP script I have used the curl_multi_exec_example from http://de.php.net/manual/en/function.curl-multi-exec.php. Besides the URLs the only significant change is that I have CURLOPT_FORBID_REUSE set.
-------------------------------
$url = "http://example.com/redir.php";
$url2 = $url ."?uri=". urlencode($url);

// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();

// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, $url);
curl_setopt($ch2, CURLOPT_URL, $url2);

curl_setopt($ch1, CURLOPT_FORBID_REUSE, true);
curl_setopt($ch2, CURLOPT_FORBID_REUSE, true);

curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, true);

/* from here on no changes to the example from http://de.php.net/manual/en/function.curl-multi-exec.php */

//create the multiple cURL handle
$mh = curl_multi_init();

//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);

$active = null;
//execute the handles
do {
    $mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);

while ($active && $mrc == CURLM_OK) {
    if (curl_multi_select($mh) != -1) {
        do {
            $mrc = curl_multi_exec($mh, $active);
        } while ($mrc == CURLM_CALL_MULTI_PERFORM);
    }
}

//close the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
echo "\n";
-------------------------------

As the server side script you can use s/th like this:

-------------------------------
if (!empty($_GET['uri'])) {
    header("Location: ". $_GET['uri']);
} else {
    echo "done";
}
-------------------------------

I don't think it is http_output_auth's fault. That is probably just the first place where the error occurs.

The segfaults disapper when either
 * both requests are redirects
 * the two requests connect to different servers
 * CURLOPT_FORBID_REUSE is set to FALSE

I use PHP-5.2.6, libcurl-7.18.2 on Debian etch.

----------------------------------------------------------------------

>Comment By: Felix (felix-sf191)
Date: 2008-08-11 12:36

Message:
Logged In: YES
user_id=2099045
Originator: YES

I built curl using "./configure --prefix=... --enable-debug"
-V says:

curl 7.18.2 (i686-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8h zlib/1.2.3
libidn/0.6.5
Protocols: ftp http file https ftps
Features: IDN Largefile NTLM SSL libz

I have used your test program with my URLs. My server is local, so
responses are fast. That my be the reason why sometimes the 302 comes
first.

I do not have segfaults with your test program with your URLs. (Sorry for
the many requests in the last minutes :)). But I realized, that your URLs
have a different setup. I needed to have one direct 200 and one 302 to make
the segfault "work". So I have changed your first URL to receive a 200 from
your server. But still no segfault. I always get the 200 first from your
server.

Right now I have moved my redir-script to a slow, remote server. I still
see the same behaviour (segfaulting now and then). Maybe you can try with
my URLs. Change the first request to "http://213.221.113.41/test/redir.php"
and the second to
"http://213.221.113.41/test/redir.php?uri=http%3A%2F%2F213.221.113.41%2Ftest%2Fredir.php".

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-11 11:57

Message:
Logged In: YES
user_id=1110
Originator: NO

What's the configure line you use then? Or perhaps, what does curl -V show
with this built libcurl?

Also, is this with my test program? It does show different URLs in the
output...

I rebuilt libcurl 7.18.2 with debug, but still I can't make this fail and
valgrind reports no errors. I guess this is either depending on some
specific version of something underlying, or there's a timing issue or
something. Is your testing server local or remote? Ie how fast is the
response time when you connect and send requests to it?

----------------------------------------------------------------------

Comment By: Felix (felix-sf191)
Date: 2008-08-11 10:34

Message:
Logged In: YES
user_id=2099045
Originator: YES

I have rebuild libcurl with --enable-debug and I still get the segfault.
The short story:

#0 0xb7f9ae24 in Curl_do (connp=0x806e780, done=0xbf9f5910) at
url.c:4576
#1 0xb7fab2aa in multi_runsingle (multi=0x8050184, easy=0x806e774) at
multi.c:1102
#2 0xb7fabd7f in curl_multi_perform (multi_handle=0x8050184,
running_handles=0xbf9f5b44)
    at multi.c:1460
#3 0x080487ba in runner ()
#4 0x080488ed in main ()

The long version with complete output is attached.

But still the segfault occurs every time when a connection is to be reused
(which happens when the redirect returns first).
File Added: curl_segfault.txt

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-09 00:08

Message:
Logged In: YES
user_id=1110
Originator: NO

Very odd.

I tried again on a second machine with CVS HEAD (Debian Testing) and it
runs just as fine. I then tried to use the debian installed libcurl 7.18.2
version instead of CVS, and that too works just as fine.

I also tried running both versions with valgrind but it reported no
problems. Can you build libcurl from source with --enable-debug and get a
stack trace from the crash?

----------------------------------------------------------------------

Comment By: Felix (felix-sf191)
Date: 2008-08-08 11:03

Message:
Logged In: YES
user_id=2099045
Originator: YES

I have tried your c program on my machine and I still get the segmentation
fault every few runs:

-------------------------------------------
[New Thread -1211963712 (LWP 29199)]
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* About to connect() to fh.devel port 7825 (#1)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#1)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

> GET /tests/redir.php?uri=redir.php HTTP/1.1
Host: fh.devel7825
Accept: */*

< HTTP/1.1 302 Found
< Date: Fri, 08 Aug 2008 07:56:01 GMT
< Server: Apache
< Location: http://fh.devel:7825/tests/redir.php
< Content-Length: 0
< Content-Type: text/html
<
* Expire cleared
* Closing connection #1
* Issue another request to this URL:
'http://fh.devel:7825/tests/redir.php'
< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 07:56:01 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Re-using existing connection! (#0) with host fh.devel
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* Expire cleared
* Closing connection #0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1211963712 (LWP 29199)]
0xb7f1aa8a in http_output_auth () from libcurl.so.4
(gdb) bt
#0 0xb7f1aa8a in http_output_auth () from libcurl.so.4
#1 0xb7f1be5b in Curl_http () from libcurl.so.4
#2 0xb7f2890d in Curl_do () from libcurl.so.4
#3 0xb7f3786e in multi_runsingle () from libcurl.so.4
#4 0xb7f38319 in curl_multi_perform () from libcurl.so.4
#5 0x080487ba in runner ()
#6 0x080488ed in main ()
-------------------------------------------

I have build your program in a Debian Vserver with gcc (GCC) 4.1.2
20061115 (prerelease) (Debian 4.1.1-21)
using the command line:
gcc `curl-config --cflags` `curl-config --libs` test-2041827.c

On a CentOS 5.2 with gcc version 4.1.2 20071124 (Red Hat 4.1.2-42) I get
the sigabort that I originally got:
----------------------------------------------
Program received signal SIGABRT, Aborted.
[Switching to Thread -1212470720 (LWP 24705)]
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7dcad20 in raise () from /lib/libc.so.6
#2 0xb7dcc631 in abort () from /lib/libc.so.6
#3 0xb7e02e6b in __libc_message () from /lib/libc.so.6
#4 0xb7e0ab16 in _int_free () from /lib/libc.so.6
#5 0xb7e0e070 in free () from /lib/libc.so.6
#6 0xb7f048b2 in Curl_safefree (ptr=0x806d9d8) at url.c:283
#7 0xb7efbe51 in Curl_http (conn=0x806cd00, done=0xbf8fa068) at
http.c:2086
#8 0xb7f0885c in Curl_do (connp=0x806c9ec, done=0xbf8fa068) at
url.c:4578
#9 0xb7f19eaf in multi_runsingle (multi=0x8050170, easy=0x806c9e0) at
multi.c:1102
#10 0xb7f1a907 in curl_multi_perform (multi_handle=0x8050170,
running_handles=0xbf8fa274) at multi.c:1460
#11 0x0804878a in runner (multi_handle=0x8050170) at test-2041827.c:52
#12 0x080488bd in main () at test-2041827.c:96
---------------------------------------------------------

It is interesting how the verbose output differs when the program works
and when it fails.

Working version:
------------------------------------------
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* About to connect() to fh.devel port 7825 (#1)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#1)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

> GET /tests/redir.php?uri=http://fh.devel:7825/tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 08:37:09 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Closing connection #0
< HTTP/1.1 302 Found
< Date: Fri, 08 Aug 2008 08:37:09 GMT
< Server: Apache
< Location: http://fh.devel:7825/tests/redir.php
< Content-Length: 0
< Content-Type: text/html
<
* Closing connection #1
* Issue another request to this URL:
'http://fh.devel:7825/tests/redir.php'
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 08:37:09 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Closing connection #0
------------------------------------------
The direct 200 is returned first. Then the 302 returns. _Both_ connections
are closed. To follow the redirect another connection has to be opened.

Now the segfaulting/sigaborting version:
------------------------------------------
* About to connect() to fh.devel port 7825 (#0)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* About to connect() to fh.devel port 7825 (#1)
* Trying 172.31.51.105... * connected
* Connected to fh.devel (172.31.51.105) port 7825 (#1)
> GET /tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

> GET /tests/redir.php?uri=http://fh.devel:7825/tests/redir.php HTTP/1.1
Host: fh.devel:7825
Accept: */*

< HTTP/1.1 302 Found
< Date: Fri, 08 Aug 2008 08:41:19 GMT
< Server: Apache
< Location: http://fh.devel:7825/tests/redir.php
< Content-Length: 0
< Content-Type: text/html
<
* Expire cleared
* Closing connection #1
* Issue another request to this URL:
'http://fh.devel:7825/tests/redir.php'
< HTTP/1.1 200 OK
< Date: Fri, 08 Aug 2008 08:41:19 GMT
< Server: Apache
< Content-Length: 4
< Content-Type: text/html
<
* Re-using existing connection! (#0) with host fh.devel
* Connected to fh.devel (172.31.51.105) port 7825 (#0)
* Expire cleared
* Closing connection #0
segmentation fault
------------------------------------------

In this case the 302 answer comes first. That seems to lead to a different
behaviour. The connection of the redirected request is closed. When the
direct 200 arrives, that sane connection is reused to follow the redirect:
"Re-using existing connection! (#0)". Every time this message occurs, the
program fails.

----------------------------------------------------------------------

Comment By: Daniel Stenberg (bagder)
Date: 2008-08-08 00:14

Message:
Logged In: YES
user_id=1110
Originator: NO

I wrote up a version of that program in plain C using libcurl directly,
and I cannot seem to repeat any crash out of it. See the attached file for
the exact code I used.

File Added: test-2041827.c

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=100976&aid=2041827&group_id=976
Received on 2008-08-11