Ref: https://api.libssh.org/master/group__libssh__session.html#ga7a7b16a4bed6d8d58f10bdb269172ff7

It seems like that doc could be clearer. Can you try #4974 please?

/cc @nmav

Thank you, very speedy!
Rebuilt and verified, this can work as documented now, and be made to succeed.

In the interim I was using a quick copy-paste job from lib/vssh/libssh2.c, because that had the exact same task with the hex conversion and case-insensitive comparison afterwards. That used a msnprintf loop for the hex conversion, this does feel neater. One nicety the libssh2 version does have though is that it sets a failf() message on failure, letting the developers looking at the curl error buffer know exactly what was compared to what.

It's probably not worth any further effort. While CURLOPT_SSH_HOST_PUBLIC_KEY_MD5 technically functions now, it's still practically pointless. Most ssh servers have multiple host keys, and the libcurl developer has no influence over which one will be picked for verification here. For my same test server, libssh verifies against the fingerprint of the ssh-ed25519 key, libssh2/1.9.0 against the ecdsa-sha2-nistp256 key, and libssh2/1.8.2 against the ssh-rsa. Each of those possibilities are a valid choice, and can't be faulted for not matching the documentation. I suppose people using the libcurl API could brute force it by looping through trying to make the connection using each possible MD5 hash.
Maybe the libssh developers could chime in, and tell us if it would be easy enough to extend this check to match either of the host keys?

Then again I only ever ended up experimenting with this because of #4972, when for an extended time I couldn't figure out why I cannot make the verification fail on one of my test systems through the file I set as CURLOPT_SSH_KNOWNHOSTS, and was panicking it wasn't doing the verification at all. Turned out it was actually doing the verification, only ignoring my KNOWNHOSTS setting, and picking up the OpenSSH configuration instead… But that's the other report.

I don't know the etiquette on if I should choose the "Close" button myself now, so I'm leaving it as is, but I do consider this bug report fixed & closed. Thank you!

Thanks for that additional information. I agree it's rather complicated. I do not have a libssh build and I'm not familiar enough with the SSH code to have a strong opinion. We can leave this open for feedback. I've since updated the libssh check to more closely match the one in libssh2 and pushed those changes to the PR, if you could try that.