Buy commercial curl support from
WolfSSL. We help you work out your issues, debug your libcurl
applications, use the API, port to new platforms, add new features and more.
With a team lead by the curl founder himself.
Re: Subtle H2 performance degradation in 8.7.1 vs 8.6.0
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Mon, 1 Apr 2024 15:52:55 +0200
Hi Dmitry,
we were discussing this in the project when I introduced the change. The advantage of writing
HTTP/2 data frames directly is that curl does not buffering and less memory (way less in some
cases).
The disadvantage, as you noticed, is that applications are called much more often. We thought
about adding optional buffering in curl, but decided against it - for now. Many applications
need to see response data as soon as possible, so it would need to be opt-in.
I think it would be more beneficial to provide an alternate callback to CURLOPT_WRITEFUNCTION
that also gets an "End-of-Stream" flag and a "Flush" behaviour, so applications can implelement
some easy buffering according to their needs.
Kind Regards,
Stefan
> Am 29.03.2024 um 23:47 schrieb Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>:
>
> Hi All,
>
> Trying 8.7.1 and TOT, I discovered a subtle performance degradation for H2 transfers vs 8.6.0, which I would like to report.
>
> It is subtle because it is mostly visible on low-end hardware platforms with weak CPUs in cases when a client application needs to perform heavy processing of incoming data (i.e. JSON parsing and processing).
> I stepped on small performance degradation with 8.7.1 when I analyzed overall H2 transfer time metrics in my application which included total transfer process time (download + processing).
>
> As it turned out, 8.7.1 (and TOT) processes H2 transfers with a larger number of write callbacks than 8.6.0, and with smaller chunk sizes.
>
> Here is the illustration for two H2 multiplexed 1MB transfers downloaded over LAN:
>
> 8.6.0, 2 H2, LAN: **** chunks=134, min_size=36, max_size=16384, avg_size=15650, time=32ms,
> <size:count map> [36:1, 117:1, 149:1, 230:1, 567:2, 16100:2, 16375:122, 16384:4]
>
> 8.7.1, 2 H2, LAN: **** chunks =146, min_size=45, max_size=16375, avg_size=14364, time=36ms,
> <size:count map> [45:8, 158:8, 576:2, 16217:8, 16330:8, 16375:112]
>
> As we can see from the example data, 8.7.1 had smaller data chunks on average (14364 vs 15650),
> the total number of chunks was greater (146 vs 134), and the full CURL_MAX_WRITE_SIZE (16384) was never used.
>
> (A similar picture if I use just one H2 download and slower speeds).
>
> While it is not an issue for fast CPUs, a bigger number of write callbacks with smaller sizes might affect overall transfer performance on slow CPUs.
> So, I thought it could be helpful to provide feedback from this perspective.
>
> Thanks,
> Dmitry Karpov
>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
Date: Mon, 1 Apr 2024 15:52:55 +0200
Hi Dmitry,
we were discussing this in the project when I introduced the change. The advantage of writing
HTTP/2 data frames directly is that curl does not buffering and less memory (way less in some
cases).
The disadvantage, as you noticed, is that applications are called much more often. We thought
about adding optional buffering in curl, but decided against it - for now. Many applications
need to see response data as soon as possible, so it would need to be opt-in.
I think it would be more beneficial to provide an alternate callback to CURLOPT_WRITEFUNCTION
that also gets an "End-of-Stream" flag and a "Flush" behaviour, so applications can implelement
some easy buffering according to their needs.
Kind Regards,
Stefan
> Am 29.03.2024 um 23:47 schrieb Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>:
>
> Hi All,
>
> Trying 8.7.1 and TOT, I discovered a subtle performance degradation for H2 transfers vs 8.6.0, which I would like to report.
>
> It is subtle because it is mostly visible on low-end hardware platforms with weak CPUs in cases when a client application needs to perform heavy processing of incoming data (i.e. JSON parsing and processing).
> I stepped on small performance degradation with 8.7.1 when I analyzed overall H2 transfer time metrics in my application which included total transfer process time (download + processing).
>
> As it turned out, 8.7.1 (and TOT) processes H2 transfers with a larger number of write callbacks than 8.6.0, and with smaller chunk sizes.
>
> Here is the illustration for two H2 multiplexed 1MB transfers downloaded over LAN:
>
> 8.6.0, 2 H2, LAN: **** chunks=134, min_size=36, max_size=16384, avg_size=15650, time=32ms,
> <size:count map> [36:1, 117:1, 149:1, 230:1, 567:2, 16100:2, 16375:122, 16384:4]
>
> 8.7.1, 2 H2, LAN: **** chunks =146, min_size=45, max_size=16375, avg_size=14364, time=36ms,
> <size:count map> [45:8, 158:8, 576:2, 16217:8, 16330:8, 16375:112]
>
> As we can see from the example data, 8.7.1 had smaller data chunks on average (14364 vs 15650),
> the total number of chunks was greater (146 vs 134), and the full CURL_MAX_WRITE_SIZE (16384) was never used.
>
> (A similar picture if I use just one H2 download and slower speeds).
>
> While it is not an issue for fast CPUs, a bigger number of write callbacks with smaller sizes might affect overall transfer performance on slow CPUs.
> So, I thought it could be helpful to provide feedback from this perspective.
>
> Thanks,
> Dmitry Karpov
>
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-04-01