Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Re-running curl tests
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Ray Satiro via curl-library <curl-library_at_lists.haxx.se>
Date: Tue, 24 Oct 2023 15:32:42 -0400
On 10/24/2023 1:56 PM, Jacob Hoffman-Andrews via curl-library wrote:
> I often have trouble efficiently debugging curl tests, and I wanted
> to ask if I'm doing it wrong, and/or if there are improvements possible.
>
> My workflow is like this: If I notice a set of tests are failing, I
> will usually run, e.g.:
>
> make; cd tests; ./runtests.pl <http://runtests.pl/> 719
>
> One issue I run into is that `make` must be run in the curl repository
> root, while `runtests.pl <http://runtests.pl/>` must be run in the
> `tests` directory. Is it possible to make `runtests.pl
> <http://runtests.pl/>` agnostic as to the current directory from which
> it is run? This would save some overhead changing directory back and
> forth.
>
> Another issue is that if tests are running long (for instance, if I
> ran `make test` and then changed my mind), hitting Ctrl-C takes a very
> long time to shut down the test runner. At least 10 seconds, and often
> much more. Sometimes the Ctrl-C doesn't register with the test runner
> at all.
>
> Lastly, when tests fail in CI, it can be hard to find the cause. For
> instance, in this test run
> (https://github.com/curl/curl/actions/runs/6628686909/job/18006346659),
> I saw that test 719 failed, so I did Ctrl-F 719 to find it, with no
> results. Later I realized that GitHub cuts off super long log output,
> so my browser did not have a copy of the lines related to test case
> 719. Using the "Search logs" text box in GitHub's log viewer did turn
> up the relevant log lines. A couple possible fixes here: When tests
> fail we could emit a final log line that says to use the search box
> rather than the browser's built-in search. Also, one of the major
> culprits was the (failing) test1474, which emits ~6000 lines of mostly
> "xxxxxx". Perhaps we could suppress or summarize output for certain
> tests that have super long output?
You can run individual tests from the top level directory by passing the
arguments to runtests via TFLAGS.
make test TFLAGS=719
make test-full TFLAGS=719
By default 'test' configuration in the top level Makefile [1] will
invoke the tests/Makefile [2] with configuration 'quiet-test'
(./runtests.pl -a -s [tflags]). 'test-full' is invoked as 'full-test'
configuration (./runtests.pl -a -p -r [tflags]).
It is only necessary to run make again from the top level directory if
you have modified curl because a test failed. In that case curl needs to
be rebuilt and the lib/unit tests may need to be rebuilt as well. If
curl does not need to be rebuilt and only tests need to be rebuilt (eg
you modified a libtest but not curl) then you can run make from the
tests directory. If only the XML of a test (ie tests/data dir files) was
changed then, in most cases, you do not need to run make at all to
rebuild anything because the XML is not used by the make system. After
that you run runtests.pl.
Regarding the GitHub Actions CI output, sometimes I can easily find what
I'm looking for and sometimes I can't. I'm not sure if it's my browser
or GitHub. Sometimes I will go to the failed section (like in the URL
you gave) and the expanded output of the failed section (eg run tests)
will be blank, so I have to reload the page. Then, even if I get the
output and I search for the failed test in Chrome (CTRL+F) it often
can't find the test. For example your failed CI job output says:
TESTFAIL: These test cases failed: 371 719 728 1148 1268 1278 1471 1472
2053
So I search for "test 0371" and nothing is found by Chrome. Instead I
have to scroll up until I see test 0371. The text is there but there's
some issue with the asynchronous loading that the text does not always
parse or show properly, or is delayed so it can't be found. A workaround
for that is to view the raw log which will download the whole thing at
once. Click the sprocket icon in that panel choose "View raw logs" then
search for "test 0371" and Chrome finds it instantly.
Regarding the long output, we set it that way for the CI purposely so
that there would be a postmortem of every failed test. Refer to
configuration test-ci/ci-test in the makefiles to see the exact options
used.
[1]: https://github.com/curl/curl/blob/curl-8_4_0/Makefile.am#L197-L216
[2]: https://github.com/curl/curl/blob/curl-8_4_0/tests/Makefile.am#L67-L108
Date: Tue, 24 Oct 2023 15:32:42 -0400
On 10/24/2023 1:56 PM, Jacob Hoffman-Andrews via curl-library wrote:
> I often have trouble efficiently debugging curl tests, and I wanted
> to ask if I'm doing it wrong, and/or if there are improvements possible.
>
> My workflow is like this: If I notice a set of tests are failing, I
> will usually run, e.g.:
>
> make; cd tests; ./runtests.pl <http://runtests.pl/> 719
>
> One issue I run into is that `make` must be run in the curl repository
> root, while `runtests.pl <http://runtests.pl/>` must be run in the
> `tests` directory. Is it possible to make `runtests.pl
> <http://runtests.pl/>` agnostic as to the current directory from which
> it is run? This would save some overhead changing directory back and
> forth.
>
> Another issue is that if tests are running long (for instance, if I
> ran `make test` and then changed my mind), hitting Ctrl-C takes a very
> long time to shut down the test runner. At least 10 seconds, and often
> much more. Sometimes the Ctrl-C doesn't register with the test runner
> at all.
>
> Lastly, when tests fail in CI, it can be hard to find the cause. For
> instance, in this test run
> (https://github.com/curl/curl/actions/runs/6628686909/job/18006346659),
> I saw that test 719 failed, so I did Ctrl-F 719 to find it, with no
> results. Later I realized that GitHub cuts off super long log output,
> so my browser did not have a copy of the lines related to test case
> 719. Using the "Search logs" text box in GitHub's log viewer did turn
> up the relevant log lines. A couple possible fixes here: When tests
> fail we could emit a final log line that says to use the search box
> rather than the browser's built-in search. Also, one of the major
> culprits was the (failing) test1474, which emits ~6000 lines of mostly
> "xxxxxx". Perhaps we could suppress or summarize output for certain
> tests that have super long output?
You can run individual tests from the top level directory by passing the
arguments to runtests via TFLAGS.
make test TFLAGS=719
make test-full TFLAGS=719
By default 'test' configuration in the top level Makefile [1] will
invoke the tests/Makefile [2] with configuration 'quiet-test'
(./runtests.pl -a -s [tflags]). 'test-full' is invoked as 'full-test'
configuration (./runtests.pl -a -p -r [tflags]).
It is only necessary to run make again from the top level directory if
you have modified curl because a test failed. In that case curl needs to
be rebuilt and the lib/unit tests may need to be rebuilt as well. If
curl does not need to be rebuilt and only tests need to be rebuilt (eg
you modified a libtest but not curl) then you can run make from the
tests directory. If only the XML of a test (ie tests/data dir files) was
changed then, in most cases, you do not need to run make at all to
rebuild anything because the XML is not used by the make system. After
that you run runtests.pl.
Regarding the GitHub Actions CI output, sometimes I can easily find what
I'm looking for and sometimes I can't. I'm not sure if it's my browser
or GitHub. Sometimes I will go to the failed section (like in the URL
you gave) and the expanded output of the failed section (eg run tests)
will be blank, so I have to reload the page. Then, even if I get the
output and I search for the failed test in Chrome (CTRL+F) it often
can't find the test. For example your failed CI job output says:
TESTFAIL: These test cases failed: 371 719 728 1148 1268 1278 1471 1472
2053
So I search for "test 0371" and nothing is found by Chrome. Instead I
have to scroll up until I see test 0371. The text is there but there's
some issue with the asynchronous loading that the text does not always
parse or show properly, or is delayed so it can't be found. A workaround
for that is to view the raw log which will download the whole thing at
once. Click the sprocket icon in that panel choose "View raw logs" then
search for "test 0371" and Chrome finds it instantly.
Regarding the long output, we set it that way for the CI purposely so
that there would be a postmortem of every failed test. Refer to
configuration test-ci/ci-test in the makefiles to see the exact options
used.
[1]: https://github.com/curl/curl/blob/curl-8_4_0/Makefile.am#L197-L216
[2]: https://github.com/curl/curl/blob/curl-8_4_0/tests/Makefile.am#L67-L108
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-10-24