[FFmpeg-devel] [PATCH v2 2/4] avformat/tls_openssl: fix dtls_handshake return code

[Jack Lau]

> On Jul 11, 2025, at 23:05, Steven Liu <lingjiujianke-at-gmail.com at ffmpeg.org> wrote:
> 
> Jack Lau <jacklau1222gm-at-gmail.com at ffmpeg.org <mailto:jacklau1222gm-at-gmail.com at ffmpeg.org>> 于2025年7月11日周五 21:22写道：
>> 
>> If the handshake is still in progress, dtls_handshake should
>> return a positive status code.
>> 
>> Signed-off-by: Jack Lau <jacklau1222 at qq.com>
>> ---
>> libavformat/tls_openssl.c | 7 +++----
>> 1 file changed, 3 insertions(+), 4 deletions(-)
>> 
>> diff --git a/libavformat/tls_openssl.c b/libavformat/tls_openssl.c
>> index 8639ac9758..ffd9cd51d2 100644
>> --- a/libavformat/tls_openssl.c
>> +++ b/libavformat/tls_openssl.c
>> @@ -716,15 +716,14 @@ static int openssl_dtls_verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
>> 
>> static int dtls_handshake(URLContext *h)
>> {
>> -    int ret = 0, r0, r1;
>> +    int ret = EINPROGRESS, r0, r1;
> AVERROR(EINPROGRESS) ?
Why I use the positive return code because avio_handshake specify:
0 on a complete and successful handshake 
> 0 if the handshake progressed, but is not complete 
< 0 for an AVERROR code

So when the dtls_handshake is still in processing, I return a positive code EINPROGRESS to match this description

>>     TLSContext *p = h->priv_data;
>> 
>>     r0 = SSL_do_handshake(p->ssl);
>>     r1 = SSL_get_error(p->ssl, r0);
>>     if (r0 <= 0) {
>>         if (r1 != SSL_ERROR_WANT_READ && r1 != SSL_ERROR_WANT_WRITE && r1 != SSL_ERROR_ZERO_RETURN) {
>> -            av_log(p, AV_LOG_ERROR, "TLS: Read failed, r0=%d, r1=%d %s\n", r0, r1, openssl_get_error(p));
>> -            ret = AVERROR(EIO);
>> +            ret = print_ssl_error(h, r1);
>>             goto end;
>>         }
>>     } else {
>> @@ -734,7 +733,7 @@ static int dtls_handshake(URLContext *h)
>>     /* Check whether the DTLS is completed. */
>>     if (SSL_is_init_finished(p->ssl) != 1)
>>         goto end;
>> -
>> +    ret = 0;
>>     p->tls_shared.state = DTLS_STATE_FINISHED;
>> end:
>>     return ret;
>> --
>> 2.49.0
>> 
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".