Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: Re-running curl tests
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Jeffrey Walton via curl-library <curl-library_at_lists.haxx.se>
Date: Tue, 24 Oct 2023 14:14:42 -0400
On Tue, Oct 24, 2023 at 1:56 PM Jacob Hoffman-Andrews via curl-library
<curl-library_at_lists.haxx.se> wrote:
>
> I often have trouble efficiently debugging curl tests, and I wanted to ask if I'm doing it wrong, and/or if there are improvements possible.
>
> My workflow is like this: If I notice a set of tests are failing, I will usually run, e.g.:
>
> make; cd tests; ./runtests.pl 719
>
> One issue I run into is that `make` must be run in the curl repository root, while `runtests.pl` must be run in the `tests` directory. Is it possible to make `runtests.pl` agnostic as to the current directory from which it is run? This would save some overhead changing directory back and forth.
>
> Another issue is that if tests are running long (for instance, if I ran `make test` and then changed my mind), hitting Ctrl-C takes a very long time to shut down the test runner. At least 10 seconds, and often much more. Sometimes the Ctrl-C doesn't register with the test runner at all.
>
> Lastly, when tests fail in CI, it can be hard to find the cause. For instance, in this test run (https://github.com/curl/curl/actions/runs/6628686909/job/18006346659), I saw that test 719 failed, so I did Ctrl-F 719 to find it, with no results. Later I realized that GitHub cuts off super long log output, so my browser did not have a copy of the lines related to test case 719. Using the "Search logs" text box in GitHub's log viewer did turn up the relevant log lines. A couple possible fixes here: When tests fail we could emit a final log line that says to use the search box rather than the browser's built-in search. Also, one of the major culprits was the (failing) test1474, which emits ~6000 lines of mostly "xxxxxx". Perhaps we could suppress or summarize output for certain tests that have super long output?
>
For completeness, this is what the project recommended (to me) several
years ago:
cd tests && ./runtests.pl -a -p 165 1448 2046 2047 >logfile 2>&1
Since you mentioned debugging, I'm not sure if you are trying to run a
particular test under gdb or similar. I find that is significantly
trickier since you have to follow the child process once the
executable is started. And it is complicated by the script, which is
something gdb cannot debug. I seem to recall gdb fails to load the
script, and you have to resort to other trickery to load the actual
executable called by the test script.
Maybe someone can comment on using gdb to debug a particular codepath
in the library from the test suite.
Jeff
Date: Tue, 24 Oct 2023 14:14:42 -0400
On Tue, Oct 24, 2023 at 1:56 PM Jacob Hoffman-Andrews via curl-library
<curl-library_at_lists.haxx.se> wrote:
>
> I often have trouble efficiently debugging curl tests, and I wanted to ask if I'm doing it wrong, and/or if there are improvements possible.
>
> My workflow is like this: If I notice a set of tests are failing, I will usually run, e.g.:
>
> make; cd tests; ./runtests.pl 719
>
> One issue I run into is that `make` must be run in the curl repository root, while `runtests.pl` must be run in the `tests` directory. Is it possible to make `runtests.pl` agnostic as to the current directory from which it is run? This would save some overhead changing directory back and forth.
>
> Another issue is that if tests are running long (for instance, if I ran `make test` and then changed my mind), hitting Ctrl-C takes a very long time to shut down the test runner. At least 10 seconds, and often much more. Sometimes the Ctrl-C doesn't register with the test runner at all.
>
> Lastly, when tests fail in CI, it can be hard to find the cause. For instance, in this test run (https://github.com/curl/curl/actions/runs/6628686909/job/18006346659), I saw that test 719 failed, so I did Ctrl-F 719 to find it, with no results. Later I realized that GitHub cuts off super long log output, so my browser did not have a copy of the lines related to test case 719. Using the "Search logs" text box in GitHub's log viewer did turn up the relevant log lines. A couple possible fixes here: When tests fail we could emit a final log line that says to use the search box rather than the browser's built-in search. Also, one of the major culprits was the (failing) test1474, which emits ~6000 lines of mostly "xxxxxx". Perhaps we could suppress or summarize output for certain tests that have super long output?
>
For completeness, this is what the project recommended (to me) several
years ago:
cd tests && ./runtests.pl -a -p 165 1448 2046 2047 >logfile 2>&1
Since you mentioned debugging, I'm not sure if you are trying to run a
particular test under gdb or similar. I find that is significantly
trickier since you have to follow the child process once the
executable is started. And it is complicated by the script, which is
something gdb cannot debug. I seem to recall gdb fails to load the
script, and you have to resort to other trickery to load the actual
executable called by the test script.
Maybe someone can comment on using gdb to debug a particular codepath
in the library from the test suite.
Jeff
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-10-24