cURL / Mailing Lists / curl-library / Single Mail

curl-library

Re: A tale of two SPARC servers wherein one gets wedged on the testsuite

From: dev <dev_at_cor0.com>
Date: Fri, 16 Oct 2015 03:44:28 -0400 (EDT)

> >
> > test 0574...OK (404 out of 984, remaining: 12:46)
> > test 0575...
>
> Does it work if you run only that single test case?
>
> Like this:
>
> $ cd tests
> $ ./runtests.pl 575

I can certainly try however I just posted a reply to Rainer wherein I
show that I did a full rebuild and everything went smooth with the
exception of two tests that have been historically weird.

> I can't see what makes 575 particularly sensitive for this platform
> though.

Me neither. Really, I am fairly familiar with the platform and SPARC in
general and I just can not figure what happened here.

> > So I hit CTRL-C over and over. The things ends like so :
> >
> > test 0576...FAILED
>
> That just shows we're lousy at handling ctrl-c to end the tests
> properly. I've
> sometimes seen it work if ctrl-c is hit a second time.

ha .. yes .. I tried a single CTRL-C and then watched. Then decided a
bunch of them was needed before I resort to kill -9 foo.

> > Also all the defunct processes vanish neatly in a puff of smoke.
>
> ... at least we clean those up nicely then!

Let me share with you what I saw on the server :

mimas # ps -ef
     UID PID PPID C STIME TTY TIME CMD
    root 0 0 0 Sep 20 ? 0:00 sched
    root 5 0 0 Sep 20 ? 0:00 zpool-mimas_rpool
    root 6 0 0 Sep 20 ? 0:00 kmem_task
    root 1 0 0 Sep 20 ? 0:00 /sbin/init
    root 2 0 0 Sep 20 ? 0:00 pageout
    root 3 0 1 Sep 20 ? 0:00 fsflush
    root 7 0 0 Sep 20 ? 0:00 vmtasks
 dclarke 1369 29865 0 - ? 0:00 <defunct>
    root 11 1 0 Sep 20 ? 0:00
/lib/svc/bin/svc.startd
    root 13 1 0 Sep 20 ? 0:00
/lib/svc/bin/svc.configd
    root 179 1 0 Sep 20 ? 0:00 /usr/sbin/nscd
 dclarke 12943 12933 0 Sep 24 pts/1 0:00 -sh
    root 451 1 0 Sep 20 ? 0:00 /usr/lib/pool/poold
    root 104 1 0 Sep 20 ? 0:00 devfsadmd
 dclarke 1548 29865 0 - ? 0:00 <defunct>
    root 258 1 0 Sep 20 ? 0:00 /usr/lib/inet/ntpd -p
/var/run/ntp.pid -g
    root 186 1 0 Sep 20 ? 0:00 /usr/sbin/ipmon -Ds
    root 653 645 0 Sep 20 pts/2 0:00 -sh
    root 236 1 0 Sep 20 ? 0:00
/lib/svc/method/iscsi-initiator
    root 169 1 0 Sep 20 ? 0:00
/usr/lib/sysevent/syseventd
    root 189 1 0 Sep 20 ? 0:00 /usr/lib/picl/picld
 dclarke 27045 27044 0 01:33:17 pts/4 0:00 server/sws --pidfile
.http_server.pid --logfile log/http_server.log --ipv4 --po
   named 337 1 0 Sep 20 ? 0:00 /usr/local/sbin/named
-u named -c /etc/opt/iscbind/named/named.conf -d 1
  daemon 161 1 0 Sep 20 ? 0:00 /usr/lib/crypto/kcfd
    root 393 1 0 Sep 20 ? 0:00 /usr/lib/fm/fmd/fmd
    root 470 1 0 Sep 20 ? 0:00 /usr/lib/sendmail -bl
-q15m
   smmsp 468 1 0 Sep 20 ? 0:00 /usr/lib/sendmail -Ac
-q15m
    root 405 11 0 Sep 20 ? 0:00 /usr/lib/saf/sac -t
300
    root 323 1 0 Sep 20 ? 0:00 /usr/sbin/cron
    root 466 1 0 Sep 20 ? 0:00 /usr/lib/inet/in.ndpd
    root 431 1 0 Sep 20 ? 0:00 /usr/sbin/syslogd
    root 494 1 0 Sep 20 ? 0:00 /usr/lib/inet/inetd
start
  daemon 372 1 0 Sep 20 ? 0:00 /usr/sbin/rpcbind
    root 423 11 0 Sep 20 console 0:00 /usr/lib/saf/ttymon
-g -d /dev/console -l console -m ldterm,ttcompat -h -p mima
 dclarke 635 634 0 Sep 20 ? 0:00 /usr/lib/ssh/sshd
    root 409 405 0 Sep 20 ? 0:00 /usr/lib/saf/ttymon
    root 634 441 0 Sep 20 ? 0:00 /usr/lib/ssh/sshd
    root 413 1 0 Sep 20 ? 0:00 /usr/lib/utmpd
    root 12932 441 0 Sep 24 ? 0:00 /usr/lib/ssh/sshd
 dclarke 2105 29865 0 - ? 0:00 <defunct>
    root 441 1 0 Sep 20 ? 0:00 /usr/lib/ssh/sshd
 dclarke 645 635 0 Sep 20 pts/2 0:00 -sh
    root 457 1 0 Sep 20 ? 0:00 /sbin/dhcpagent
 dclarke 12933 12932 0 Sep 24 ? 0:00 /usr/lib/ssh/sshd
 dclarke 28208 28207 0 01:34:46 pts/4 0:00 server/sws --pidfile
.http2_server.pid --logfile log/http2_server.log --connect
 dclarke 29865 27018 0 01:36:55 pts/4 0:00 perl -I.
./ftpserver.pl --pidfile .ftp_server.pid --logfile log/ftp_server.log
 dclarke 1965 29865 0 - ? 0:00 <defunct>
 dclarke 28207 27018 0 01:34:46 pts/4 0:00 sh -c server/sws
--pidfile ".http2_server.pid" --logfile "log/http2_server.log"
 dclarke 1217 29865 0 - ? 0:00 <defunct>
 dclarke 1433 29865 0 - ? 0:00 <defunct>
 dclarke 13397 901 0 01:09:05 pts/4 0:00 /usr/bin/time -p
/usr/local/bin/gmake check
 dclarke 27044 27018 0 01:33:17 pts/4 0:00 sh -c server/sws
--pidfile ".http_server.pid" --logfile "log/http_server.log" -
 dclarke 13425 13424 0 01:09:08 pts/4 0:00 /usr/local/bin/bash
-c (cd tests; /usr/local/bin/gmake all quiet-test)
 dclarke 1482 29865 0 - ? 0:00 <defunct>
 dclarke 13426 13425 0 01:09:08 pts/4 0:00 /usr/local/bin/gmake
all quiet-test
 dclarke 13424 13398 0 01:09:08 pts/4 0:00 /usr/local/bin/bash
-c (cd tests; /usr/local/bin/gmake all quiet-test)
 dclarke 901 890 0 Oct 03 pts/4 0:00 /usr/local/bin/bash
 dclarke 2027 29865 0 - ? 0:00 <defunct>
 dclarke 1050 27018 0 01:38:41 pts/4 0:00 sh -c server/tftpd
--pidfile ".tftp_server.pid" --logfile "log/tftp_server.log"
 dclarke 1465 29865 0 - ? 0:00 <defunct>
 dclarke 228 29865 0 - ? 0:00 <defunct>
 dclarke 1499 29865 0 - ? 0:00 <defunct>
 dclarke 1051 1050 0 01:38:41 pts/4 0:00 server/tftpd
--pidfile .tftp_server.pid --logfile log/tftp_server.log --ipv4 --
 dclarke 27018 13426 0 01:33:12 pts/4 0:00 /usr/local/bin/perl
-I. ./runtests.pl -a -s
 dclarke 1531 29865 0 - ? 0:00 <defunct>
 dclarke 811 759 0 - ? 0:00 <defunct>
 dclarke 828 759 0 - ? 0:00 <defunct>
 dclarke 777 759 0 - ? 0:00 <defunct>
 dclarke 13398 13397 0 01:09:05 pts/4 0:00 /usr/local/bin/gmake
check
 dclarke 21387 12943 0 Oct 02 pts/1 0:00 /usr/local/bin/bash
    root 29054 21387 0 Oct 11 pts/1 0:00 -sh
 dclarke 1565 29865 0 - ? 0:00 <defunct>
 dclarke 794 759 0 - ? 0:00 <defunct>
 dclarke 880 879 0 Oct 03 ? 0:00 /usr/lib/ssh/sshd
 dclarke 1048 29865 0 - ? 0:00 <defunct>
    root 879 441 0 Oct 03 ? 0:00 /usr/lib/ssh/sshd
 dclarke 845 759 0 - ? 0:00 <defunct>
 dclarke 890 880 0 Oct 03 pts/4 0:00 -sh
 dclarke 1386 29865 0 - ? 0:00 <defunct>
 dclarke 941 29865 0 - ? 0:00 <defunct>
 dclarke 226 29865 0 - ? 0:00 <defunct>
 dclarke 761 760 0 01:38:20 pts/4 0:00 ./server/sockfilt
--ipv6 --port 8996 --pidfile ./.ftp_ipv6_sockctrl.pid --logfi
 dclarke 759 27018 0 01:38:18 pts/4 0:00 perl -I.
./ftpserver.pl --pidfile .ftp_ipv6_server.pid --logfile log/ftp_ipv6_s
 dclarke 757 29865 0 - ? 0:00 <defunct>
 dclarke 760 759 0 01:38:20 pts/4 0:00 sh -c
./server/sockfilt --ipv6 --port 8996 --pidfile
"./.ftp_ipv6_sockctrl.pid"
 dclarke 708 29865 0 - ? 0:00 <defunct>
 dclarke 740 29865 0 - ? 0:00 <defunct>
 dclarke 563 29865 0 - ? 0:00 <defunct>
 dclarke 596 595 0 01:37:45 pts/4 0:00 server/sws --pidfile
.http_ipv6_server.pid --logfile log/http_ipv6_server.log -
 dclarke 546 29865 0 - ? 0:00 <defunct>
 dclarke 595 27018 0 01:37:45 pts/4 0:00 sh -c server/sws
--pidfile ".http_ipv6_server.pid" --logfile "log/http_ipv6_ser
 dclarke 283 29865 0 - ? 0:00 <defunct>
 dclarke 264 29865 0 - ? 0:00 <defunct>
 dclarke 465 29865 0 - ? 0:00 <defunct>
 dclarke 445 29865 0 - ? 0:00 <defunct>
 dclarke 209 29865 0 - ? 0:00 <defunct>
 dclarke 1599 29865 0 - ? 0:00 <defunct>
 dclarke 1582 29865 0 - ? 0:00 <defunct>
 dclarke 29883 29865 0 - ? 0:00 <defunct>
 dclarke 190 29865 0 - ? 0:00 <defunct>
 dclarke 281 29865 0 - ? 0:00 <defunct>
 dclarke 187 29865 0 - ? 0:00 <defunct>
 dclarke 29900 29865 0 - ? 0:00 <defunct>
 dclarke 29867 29866 0 01:36:56 pts/4 0:00 ./server/sockfilt
--ipv4 --port 8992 --pidfile ./.ftp_sockctrl.pid --logfile ./
 dclarke 207 29865 0 - ? 0:00 <defunct>
 dclarke 262 29865 0 - ? 0:00 <defunct>
 dclarke 1948 29865 0 - ? 0:00 <defunct>
 dclarke 29866 29865 0 01:36:56 pts/4 0:00 sh -c
./server/sockfilt --ipv4 --port 8992 --pidfile "./.ftp_sockctrl.pid"
--lo
 dclarke 1721 29865 0 - ? 0:00 <defunct>
 dclarke 2067 29865 0 - ? 0:00 <defunct>
 dclarke 2044 29865 0 - ? 0:00 <defunct>
 dclarke 2046 29865 0 - ? 0:00 <defunct>
 dclarke 2048 29865 0 - ? 0:00 <defunct>
 dclarke 2050 29865 0 - ? 0:00 <defunct>
 dclarke 2137 29865 0 - ? 0:00 <defunct>
 dclarke 2071 29865 0 - ? 0:00 <defunct>
 dclarke 2069 29865 0 - ? 0:00 <defunct>
 dclarke 2073 29865 0 - ? 0:00 <defunct>
 dclarke 2177 29865 0 - ? 0:00 <defunct>
 dclarke 2196 29865 0 - ? 0:00 <defunct>
 dclarke 2154 29865 0 - ? 0:00 <defunct>
 dclarke 2156 29865 0 - ? 0:00 <defunct>
 dclarke 2158 29865 0 - ? 0:00 <defunct>
 dclarke 2160 29865 0 - ? 0:00 <defunct>
 dclarke 2288 29865 0 - ? 0:00 <defunct>
 dclarke 2179 29865 0 - ? 0:00 <defunct>
 dclarke 2322 29865 0 - ? 0:00 <defunct>
 dclarke 2555 29865 0 - ? 0:00 <defunct>
 dclarke 2401 29865 0 - ? 0:00 <defunct>
 dclarke 2290 29865 0 - ? 0:00 <defunct>
 dclarke 2852 2851 0 01:42:11 pts/4 0:00
/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/tests/libtest/.libs/lib575
f
 dclarke 2324 29865 0 - ? 0:00 <defunct>
 dclarke 2403 29865 0 - ? 0:00 <defunct>
 dclarke 2687 2686 0 01:41:58 pts/4 0:00 server/rtspd
--pidfile .rtsp_server.pid --logfile log/rtsp_server.log --ipv4 --
 dclarke 2866 29865 0 - ? 0:00 <defunct>
 dclarke 2686 27018 0 01:41:58 pts/4 0:00 sh -c server/rtspd
--pidfile ".rtsp_server.pid" --logfile "log/rtsp_server.log"
 dclarke 2825 29865 0 - ? 0:00 <defunct>
 dclarke 2823 29865 0 - ? 0:00 <defunct>
 dclarke 2827 29865 0 - ? 0:00 <defunct>
 dclarke 2835 29865 0 - ? 0:00 <defunct>
 dclarke 2829 29865 0 - ? 0:00 <defunct>
 dclarke 2831 29865 0 - ? 0:00 <defunct>
 dclarke 2841 29865 0 - ? 0:00 <defunct>
 dclarke 2833 29865 0 - ? 0:00 <defunct>
 dclarke 2837 29865 0 - ? 0:00 <defunct>
 dclarke 2839 29865 0 - ? 0:00 <defunct>
 dclarke 2847 29865 0 - ? 0:00 <defunct>
 dclarke 2843 29865 0 - ? 0:00 <defunct>
 dclarke 2851 27018 0 01:42:11 pts/4 0:00 sh -c
./libtest/lib575 ftp://127.0.0.1:8992/fully_simulated/UNIX/*
>log/stdout5
 dclarke 2845 29865 0 - ? 0:00 <defunct>
 dclarke 2849 29865 0 - ? 0:00 <defunct>
 dclarke 2882 29865 0 - ? 0:00 <defunct>
 dclarke 2868 29865 0 - ? 0:00 <defunct>
 dclarke 2870 29865 0 - ? 0:00 <defunct>
 dclarke 2872 29865 0 - ? 0:00 <defunct>
 dclarke 2874 29865 0 - ? 0:00 <defunct>
 dclarke 2876 29865 0 - ? 0:00 <defunct>
 dclarke 2878 29865 0 - ? 0:00 <defunct>
 dclarke 2880 29865 0 - ? 0:00 <defunct>
 dclarke 2884 29865 0 - ? 0:00 <defunct>
 dclarke 2886 29865 0 - ? 0:00 <defunct>
 dclarke 2888 29865 0 01:42:13 pts/4 0:00 sh -c
./server/sockfilt --ipv4 --port 0 --pidfile "./.ftp_sockdata.pid"
--logf
 dclarke 2889 2888 0 01:42:13 pts/4 0:00 ./server/sockfilt
--ipv4 --port 0 --pidfile ./.ftp_sockdata.pid --logfile ./log
    root 3063 29054 1 17:57:24 pts/1 0:00 ps -ef
mimas #

That is some truely strange events going on there.

> > So I guess the question is, on a slower older system do I need to
> > grant
> > special considerations or perhaps indicate the the timeouts on a
> > test should
> > be extended or any thoughts ?
>
> Maybe. We need some more details from the failed tests to be able to
> tell for
> sure.

I was able to run a build and a full testsuite no problem and this is
the same server, no reboot of course and no processes or services
modified.

  * baffled *

> Also, if you want to try it out, you can add '575' to
> tests/data/DISABLED
> before running all the tests as then it'll skip that single test. In
> case you
> want to make sure everything else still runs fine.

What I may do is go back and re-run test 1060 and 1061 as they are all
that I see as issues now. However only on that system. On another much
faster and resource rich system I see :

.
.
.
test 2036...OK (975 out of 984, remaining: 00:04)
test 2039...OK (978 out of 984, remaining: 00:03)
test 2040...OK (979 out of 984, remaining: 00:02)
test 2044...OK (983 out of 984, remaining: 00:00)
test 2045...Killed
Killed
Killed
Killed
Killed
Terminated
Terminated
Terminated
Terminated
Terminated
OK (984 out of 984, remaining: 00:00)
TESTDONE: 754 tests out of 754 reported OK: 100%
TESTDONE: 994 tests were considered during 519 seconds.

That is then followed by a pile of activity and then this output :

libtool: link: /opt/solarisstudio12.4/bin/cc -errfmt=error -erroff=%none
-errshort=full -xstrconst -xildoff -m64 -xmemalign=8s -xnolibmil -Xa
-xcode=pic32 -xregs=no%appl -xlibmieee -mc -xs -ftrap=%none -Qy
-xbuiltin=%none -xdebugformat=dwarf -xunroll=1 -xtarget=ultraT2
-xcache=8/16/4:4096/64/16 -s -xO2 -o .libs/http2-serverpush
http2-serverpush.o -L/usr/local/lib -L/usr/local/ssl/lib
../../lib/.libs/libcurl.so /usr/local/lib/libidn.so
/usr/local/lib/libintl.so -lc /usr/local/lib/libiconv.so
/usr/local/lib/libssh2.so -lssl -lcrypto -lldap -lz -lrt -lsocket -lnsl
-R/usr/local/lib
gmake[2]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/examples'
gmake[1]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/examples'
gmake[1]: Entering directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
Making check in opts
gmake[2]: Entering directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl/opts'
gmake[2]: Nothing to be done for `check'.
gmake[2]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl/opts'
gmake[2]: Entering directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
/usr/local/bin/gmake check-TESTS
gmake[3]: Entering directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
gmake[4]: Entering directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
OPTS="$(ls ./opts/CURLOPT*.3 | /usr/local/bin/gsed -e 's,^.*/,,' -e
's,\.3$,,')" && \
for opt in $OPTS; do grep "^\.IP $opt$" ./curl_easy_setopt.3 >/dev/null
|| echo Missing $opt; done > check-easy
PASS: check-easy
OPTS="$(ls ./opts/CURLMOPT*.3 | /usr/local/bin/gsed -e 's,^.*/,,' -e
's,\.3$,,')" && \
for opt in $OPTS; do grep "^\.IP $opt$" ./curl_multi_setopt.3 >/dev/null
|| echo Missing $opt; done > check-multi
PASS: check-multi
============================================================================
Testsuite summary for curl -
============================================================================
# TOTAL: 2
# PASS: 2
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
gmake[4]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
gmake[3]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
gmake[2]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'
gmake[1]: Leaving directory
`/usr/local/build/curl-7.45.0_SunOS5.10_sparcv9.001/docs/libcurl'

real 1565.01
user 1054.68
sys 427.42

Here we see a real chunk of time used and a strange set of results.

Really ... I am glad I ran this on two different servers both with
Solaris 10 and both with very very similar tool chains. I am getting to
see different results between them. Not the least of which is that
mimas ( small little Netra class SPARC server ) seems to stop its
testsuite early.

Strange strange strange.

Dennis Clarke
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Received on 2015-10-16