Buy commercial curl support from WolfSSL. We help you work
out your issues, debug your libcurl applications, use the API, port to new
platforms, add new features and more. With a team lead by the curl founder
himself.
Re: WebSocket feature request: is it possible to call write function when full frame is loaded only?
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Vitalii B. Avramenko via curl-library <curl-library_at_lists.haxx.se>
Date: Fri, 3 Feb 2023 11:39:54 +0200
Hello,
I unfortunately don't know `libcurl` sources well, and I program primarily for devices with huge memory available, so in my context it is not a big issue to increase the limit; if we have 128 GB of memory, why not to make 1 megabyte buffer? Of course, there is possibility that we'll have something unusually huge, but in 99,99% of situation just increasing the buffer is enough, and if data size will be even more than huge buffer, it is almost for sure error situation, so `libcurl` may just fail with some error code. As a summary:
1. We add `CULOPT_BUFFER_SIZE` where we can overwrite the buffer size, or we may even supply own buffer pointer to `libcurl` as a pointer to structure that contains size and data fields.
2. If frame is even bigger, `libcurl` generates buffer overflow error, or calls any sort of callback that allows custom program to decide what to do: increase the buffer or cancel the operation, something like this:
bool OnBufferOverflow(size_t new_size);
and if we return `true`, `libcurl` has to increase the buffer, and it has to cancel the operation if we return `false`.
3. We make this mode optional and disabled by default to not affect existing applications or applications that expect huge data amount and don't want to load it in memory.
`curl_ws_meta` is also good solution, even it is not fully automated. May be just update documentation is good idea. For me the entry point to websocket support in curl was this page:
https://curl.se/docs/websocket.html
Google gives it when you search "libcurl websocket", so it is logical entry point for new protocol users. I see here you mention `curl_ws_meta`, but I simply missed it. My logic was like that: "Well, meta is for low level programmers, and I just need the stream of messages, so I can skip this function.". May be link to any working example of write function with `curl_ws_meta` call, or any highlight that this function is very important and that without it you will most probably not receive the frames stream like you expect them will help other `libcurl` users to investigate more effectively.
Actually idea of simple "write function" that just gives you infinite stream of data is very good for applications that just receive streams of data, including trading, social media monitoring, etc..., and updating API to increase `libcurl` buffer size, or to supply custom buffer will simplify life for many users that don't need anything complex and just need stream of data. There are possibility that we'll get unusually huge payload, but it is for sure not mainstream situation; unlike HTTP that we sometimes use to download huge files, websocket is realtime protocol for fast exchange of small messages, so simplification of API is very good idea.
Date: Fri, 3 Feb 2023 11:39:54 +0200
Hello,
I unfortunately don't know `libcurl` sources well, and I program primarily for devices with huge memory available, so in my context it is not a big issue to increase the limit; if we have 128 GB of memory, why not to make 1 megabyte buffer? Of course, there is possibility that we'll have something unusually huge, but in 99,99% of situation just increasing the buffer is enough, and if data size will be even more than huge buffer, it is almost for sure error situation, so `libcurl` may just fail with some error code. As a summary:
1. We add `CULOPT_BUFFER_SIZE` where we can overwrite the buffer size, or we may even supply own buffer pointer to `libcurl` as a pointer to structure that contains size and data fields.
2. If frame is even bigger, `libcurl` generates buffer overflow error, or calls any sort of callback that allows custom program to decide what to do: increase the buffer or cancel the operation, something like this:
bool OnBufferOverflow(size_t new_size);
and if we return `true`, `libcurl` has to increase the buffer, and it has to cancel the operation if we return `false`.
3. We make this mode optional and disabled by default to not affect existing applications or applications that expect huge data amount and don't want to load it in memory.
`curl_ws_meta` is also good solution, even it is not fully automated. May be just update documentation is good idea. For me the entry point to websocket support in curl was this page:
https://curl.se/docs/websocket.html
Google gives it when you search "libcurl websocket", so it is logical entry point for new protocol users. I see here you mention `curl_ws_meta`, but I simply missed it. My logic was like that: "Well, meta is for low level programmers, and I just need the stream of messages, so I can skip this function.". May be link to any working example of write function with `curl_ws_meta` call, or any highlight that this function is very important and that without it you will most probably not receive the frames stream like you expect them will help other `libcurl` users to investigate more effectively.
Actually idea of simple "write function" that just gives you infinite stream of data is very good for applications that just receive streams of data, including trading, social media monitoring, etc..., and updating API to increase `libcurl` buffer size, or to supply custom buffer will simplify life for many users that don't need anything complex and just need stream of data. There are possibility that we'll get unusually huge payload, but it is for sure not mainstream situation; unlike HTTP that we sometimes use to download huge files, websocket is realtime protocol for fast exchange of small messages, so simplification of API is very good idea.
-- Best regards, Vitalii B. Avramenko. -- Unsubscribe: https://lists.haxx.se/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2023-02-03