Fix logical error in rtp engine check in dtls flow

[SouradeepBera]

For an inbound call leg requiring SRTP over DTLS, switch_channel_answer does not return. It gets stuck in an infinite do-while loop in switch_core_media_check_dtls()
do { if (engine->rtp_session) checking = check_engine(engine); } while (switch_channel_ready(session->channel) && checking);
The check_engine() function should return 0 on success to break the loop. However. in the second half of the function it flips the logic and returns 0 if it is not acceptable and 1 otherwise, resulting in the infinite loop

[signalwire-ci]

Unit-tests failed: https://public-artifacts.signalwire.cloud/drone/signalwire/freeswitch/1722/artifacts.html

[ChrisZhangJin]

Hi, we experienced the same issue now!! Are you sure the change you made can solve this issue?? I saw your pr was not accepted. @SouradeepBera

For an inbound call leg requiring SRTP over DTLS, switch_channel_answer does not return. It gets stuck in an infinite do-while loop in switch_core_media_check_dtls() do { if (engine->rtp_session) checking = check_engine(engine); } while (switch_channel_ready(session->channel) && checking); The check_engine() function should return 0 on success to break the loop. However. in the second half of the function it flips the logic and returns 0 if it is not acceptable and 1 otherwise, resulting in the infinite loop

[SouradeepBera]

Seems to work for us @ChrisZhangJin

[SouradeepBera]

@andywolk can you please review this fix?

[ChrisZhangJin]

HI team, in my case, I found the issue was there, as you can see the image i captured from the source code. our sip trunk (over dtls) remote side didn't send the session ticket to complete the handshake, so that the code was looping this dtls_state_handshake all the time.
after adding some simple logs to print the result of SSL_get_error, I found all the ret code from that was SSL_ERROR_WAIT_READ, which matched what i thought that the remote server didn't reponse the session ticket to me.
Accordingly, the result of SSL_is_init_finished always is 0

here is the captured pcap file