Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0

From: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Date: Tue, 5 Dec 2023 02:59:47 +0000

I ran more tests today both on Linux and Windows, and I think I understand now where the problem is.
First, I should tell that the main factor which affects speed deviations in the 8.4/main branches is the receive socket buffer.

Only if I set it to 64KB, I can get speed deviations like they were in 7.84.

And in 7.84, the socket receive buffer didn't affect speed deviations that much - I get deviations under 1% both for 64KB and 1MB socket buffers.
(Only if I set CURLOPT_BUFFERSIZE in 7.84, I get big speed deviations).

I applied your patch to verify the delay math, and I think it is correct - in both 7.84 and the main branch the logic is essentially the same.
But I saw that with 1MB socket buffer in the main branch, the number of Curl_pgrsLimitWaitTime() calls was multiple time less
and the "wait" times were much longer.

These much longer "wait" times on larger sockets buffers seem to be the main reason for the speed deviations in the main branch
because they made the total throttling time less precise.

So, I looked deeper and found out why this problem occurs in the 8.4/main but didn't occur in 7.84.

It looks like the readwrite_data() function in transfer.c has changed from 7.84 in a way that contributes to our problem.

Now, it looks like:
static CURLcode readwrite_data(struct Curl_easy *data,...)
{
   curl_off_t max_recv = data->set.max_recv_speed?
                      data->set.max_recv_speed : CURL_OFF_T_MAX;

    /* This is where we loop until we have read everything there is to
   read or we get a CURLE_AGAIN */
   do {
        ... read from socket
        max_recv -= blen;
        ...
   } while((max_recv > 0) && data_pending(data) && maxloops--);
   ...
}

In the do-while loop, we keep reading from socket until we can, so with relatively big socket buffers we
can read a lot of data before we can have a chance to check for throttling again, and thus we can have big speed deviations due to less precise "wait" time calculations.

If I replace the "max_recv -= blen" by " max_recv = 0" to force breaking the do-while loop,
then I get the same quite precise speed throttling as we had in 7.84 (~0.1% on average in my tests).

I guess this do-while loop was added as some kind of optimization allowing to continuously read from the socket,
but I think that for the speed throttling case, it is better to break this loop, because if someone sets a speed limit then he/she is probably more concerned about that limit
rather than some optimizations that break that limit.

And yes, I agree that "Accurate rate limiting is... hard.", but we can definitely achieve very good rate limit in the main because we had it in 7.84.

Thanks,
Dmitry Karpov


-----Original Message-----
From: Daniel Stenberg <daniel_at_haxx.se>
Sent: Saturday, December 2, 2023 8:58 AM
To: Dmitry Karpov via curl-library <curl-library_at_lists.haxx.se>
Cc: Dmitry Karpov <dkarpov_at_roku.com>
Subject: [EXTERNAL] Re: Large download speed deviations for CURLOPT_MAX_RECV_SPEED_LARGE in 8.4.0

On Sat, 2 Dec 2023, Dmitry Karpov via curl-library wrote:

> No, buffering is the same. I explicitly set 1MB socket receive buffer
> to avoid any default buffering differences between 8.4. and 7.84.
>
> The deviations are not sensitive to file sizes. They slightly vary
> between file sizes, but still are quite large for all file sizes.

I'm sure there are lots of factors around, as the rate limiting logic in curl is very simple. To help the rate limiter somewhat, you can adjust the buffer size accordingly and make sure it is not made larger than number of bytes/second you want to transfer.

I don't think it helps that you post more numbers of your results. I want to know if the math is right or wrong for you. The patch I posted would show that.

Currently the rate limit stores a time stamp and the amount of data every three seconds and then base the calculations on that. It (the three second
interval) was picked fairly arbitrarily out of the air, so I bet we can consider alternatives of solving this and I'm certainly open for suggestions.

Note that any transfer that is completed within say six seconds or so, is not likely to every get very accurate rate limiting because of this.

Accurate rate limiting is... hard.

-- 
  / daniel.haxx.se
  | Commercial curl support up to 24x7 is available!
  | Private help, bug fixes, support, ports, new features
  | https://curl.se/support.html
-- 
Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
Etiquette:   https://curl.se/mail/etiquette.html

Received on 2023-12-05