cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: multithread curl. unexpected SIGSEGV

From: Azat Khuzhin <dohardgopro_at_gmail.com>
Date: Mon, 17 Sep 2012 17:40:53 +0400

URL with https protocol comes after redirect.
But because of some specific stuff that this application must do, I
must handle redirect myself.
Maybe I forgot something.

Here is sample of code, that I use for this:
https://gist.github.com/3737325

On Mon, Sep 17, 2012 at 5:20 PM, Azat Khuzhin <dohardgopro_at_gmail.com> wrote:
> Here is link on pastebin with normal formating.
> http://pastebin.com/raw.php?i=8wmaB6MG
>
> On Mon, Sep 17, 2012 at 5:18 PM, Azat Khuzhin <dohardgopro_at_gmail.com> wrote:
>> On Mon, Sep 17, 2012 at 5:12 PM, Azat Khuzhin <dohardgopro_at_gmail.com> wrote:
>>> On Mon, Sep 17, 2012 at 4:58 PM, Daniel Stenberg <daniel_at_haxx.se> wrote:
>>>> On Mon, 17 Sep 2012, Azat Khuzhin wrote:
>>>>
>>>>> I have SIGSEGV when I run curl in 400 threads. valgrind doesn't detect any
>>>>> memory errors.
>>>>>
>>>>> I can't post here all program sources, but here is some info:
>>>>
>>>>
>>>> You shouldn't post "all program sources" - because we don't want them. You
>>>> should write a new stand-alone example that repeats your problem and then
>>>> post _that_ source code.
>>>>
>>>> If you can reproduce the problem in a separate application, chances are the
>>>> problem is in your code.
>>>
>>> I don't think that this justified, but I try when I have more time.
>>> Because this application crawl URL's and source code is not enough to
>>> reproduce this error, also you need URL's (I think that ~5 million
>>> rows should be enough.)
>>>>
>>>>
>>>>> At all time program runs, only one valgrind message (valgrind
>>>>> --show-reachable=yes --leak-check=full) :
>>>>> ==30908== Thread 3:
>>>>> ==30908== Conditional jump or move depends on uninitialised value(s)
>>>>> ==30908== at 0x4C2B837: __GI___rawmemchr (mc_replace_strmem.c:1110)
>>>>> ==30908== by 0x532A2AF: _IO_str_init_static_internal (strops.c:45)
>>>>> ==30908== by 0x531E9D4: vsscanf (iovsscanf.c:44)
>>>>> ==30908== by 0x530AB97: sscanf (sscanf.c:34)
>>>>> ==30908== by 0x4074E9: parseUrl (crawler.c:28)
>>>>> ==30908== by 0x407961: startRoutine (crawler.c:125)
>>>>> ==30908== by 0x4E35B4F: start_thread (pthread_create.c:304)
>>>>> ==30908== by 0x538D70C: clone (clone.S:112)
>>>>
>>>>
>>>> ... this has nothing to do with libcurl but rather shows a problem in your
>>>> application.
>>>
>>> I know, but it seems that this is not important error
>>>>
>>>>
>>>>> And it caught SIGSEGV:
>>>>> ==30908== Process terminating with default action of signal 11 (SIGSEGV)
>>>>> ==30908== General Protection Fault
>>>>> ==30908== at 0x50926C8: curl_slist_free_all (in
>>>>> /usr/lib/x86_64-linux-gnu/libcurl.so.4.2.0)
>>>>> ==30908== by 0x4031AF: curlSetOptionPC (curl.c:138)
>>>>> ==30908== by 0x407CB2: startRoutine (crawler.c:185)
>>>>> ==30908== by 0x4E35B4F: start_thread (pthread_create.c:304)
>>>>> ==30908== by 0x538D70C: clone (clone.S:112)
>>>>
>>>>
>>>> This could just as well be a thread/synch problem in your application. If
>>>> you look at the curl_slist_free_all() implementation you'll see that it
>>>> doesn't have a lot of logic that can go wrong.
>>>
>>> Every thread have it own curl handler.
>>> At this moment I run my application under "valgrind --tool=drd" but
>>> pthread_create is very slow under this tool,
>>> so if I found some thing important/interesting I post here.
>>>>
>>>>
>>>>> Also valgrind show that Curl_ssl_connect is called (how ? If I allow only
>>>>> HTTP protocol)
>>>>
>>>>
>>>> That seems strange yes and I can't explain it. Can't you just set a
>>>> break-point and see how it gets there?
>>>
>>> It's a good idea. I give it a try and post here results when find something.
>>
>> Breakpoint 1, Curl_ssl_connect (conn=conn_at_entry=0x62f550,
>> sockindex=sockindex_at_entry=0) at sslgen.c:199
>> 199 sslgen.c: No such file or directory.
>> (gdb) bt
>> #0 Curl_ssl_connect (conn=conn_at_entry=0x62f550,
>> sockindex=sockindex_at_entry=0) at sslgen.c:199
>> #1 0x00007ffff7969895 in Curl_http_connect (conn=0x62f550,
>> done=0x7ffff3986abf) at http.c:1333
>> #2 0x00007ffff797bb14 in Curl_protocol_connect
>> (conn=conn_at_entry=0x62f550,
>> protocol_done=protocol_done_at_entry=0x7ffff3986abf) at url.c:3437
>> #3 0x00007ffff797bdea in Curl_setup_conn (conn=conn_at_entry=0x62f550,
>> protocol_done=0x7ffff3986abf) at url.c:5243
>> #4 0x00007ffff7994074 in Curl_async_resolved (conn=0x62f550,
>> protocol_done=protocol_done_at_entry=0x7ffff3986abf) at hostasyn.c:139
>> #5 0x00007ffff7987ba1 in connect_host (conn=0x7ffff3986ac0,
>> data=0x611770) at transfer.c:1952
>> #6 Curl_do_perform (data=0x611770) at transfer.c:2087
>> #7 0x0000000000403a34 in curlRequestGet (ptCurl=0x7ffff398ce00,
>> pccUrl=0x7ffff398be00
>> "https://www.google.com/bookmarks/mark?bkmk=http%3A%2F%2Fcmjornal.xl.pt%2Fdetalhe%2Fo-dia-em-imagens%2Fa-holandesa-laura-dekker-16-anos-tornou-se-ontem-a-mais-jovem-navegadora-solitaria-a-dar-a-volta-a"...)
>> at gen/gen_curl.c:272
>> #8 0x0000000000407a64 in startRoutine (arg=0x60e920) at
>> crawler/src/crawler.c:143
>> #9 0x00007ffff7bc7b50 in start_thread (arg=<optimized out>) at
>> pthread_create.c:304
>> #10 0x00007ffff76a870d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>> #11 0x0000000000000000 in ?? ()
>> (gdb) frame 8
>> #8 0x0000000000407a64 in startRoutine (arg=0x60e920) at
>> crawler/src/crawler.c:143
>> 143 gotBytes = curlRequestGet(&curlPage, url);
>> (gdb) p url
>> $1 = "https://www.google.com/bookmarks/mark?bkmk=http%3A%2F%2Fcmjornal.xl.pt%2Fdetalhe%2Fo-dia-em-imagens%2Fa-holandesa-laura-dekker-16-anos-tornou-se-ontem-a-mais-jovem-navegadora-solitaria-a-dar-a-volta-a"
>>>
>>> Thanks for reply.
>>>>
>>>> --
>>>>
>>>> / daniel.haxx.se
>>>> -------------------------------------------------------------------
>>>> List admin: http://cool.haxx.se/list/listinfo/curl-library
>>>> Etiquette: http://curl.haxx.se/mail/etiquette.html
>>>
>>>
>>>
>>> --
>>> Azat Khuzhin
>>
>>
>>
>> --
>> Azat Khuzhin
>
>
>
> --
> Azat Khuzhin

-- 
Azat Khuzhin
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette:  http://curl.haxx.se/mail/etiquette.html
Received on 2012-09-17