Currently OpenSSL in client mode stops handshake only if the keylength of server selected DH parameters is less than 768 bit (hardcoded in source).
In my client I want to stop handshake if the keylength of Server selected DH parameters is less than 2048-bit. The preferred way would be to set via API, e.g. option setting exposed by OpenSSL.
Is there any way to set the minimum key length using public APIs?
Is there any way to set the minimum key length using public APIs?
Yes (or maybe I should say, "I believe so"). Use your Diffie-Hellman callback. The callback is set with SSL_CTX_set_tmp_dh_callback and SSL_set_tmp_dh_callback.
Usually the Diffie-Hellman callback is used on the server to generate its keys. But according to OpenSSL's SSL_CTX_set_tmp_dh_callback(3)
man page, its "... to be used when a DH parameters are required for tmp_dh_callback
...".
For an example of using the callback in the context of a server (which should be similar to using it in a client), see 'No Shared Cipher' Error with EDH-RSA-DES-CBC3-SHA. It performs key length checks.