Hello,

Consider such situation:
Client and server both passed DTLS handshake. Then server for some
application-level reason frees SSL object, without sending close alert.
For example, there are no application-level data from client for some
timeout.
Then client also frees connection, but with close alert. When server
receives this alert he thinks that it is probaly handshake message from
new client and creates new SSL object. SSL_do_handshake() then
returns SSL_ERROR_WANT_READ. Because there is no data
from this client, server frees SSL object. Here occurs memory leak.

The problem may be with following code in function "void dtls1_free(SSL
*s)":

while( (item = pqueue_pop(s->d1->unprocessed_rcds.q)) != NULL)
{
OPENSSL_free(item->data);
pitem_free(item);
}
pqueue_free(s->d1->unprocessed_rcds.q);

In described situation ((DTLS1_RECORD_DATA *)item->data)->rbuf.buf
allocates 18 Kbytes of memory, which should be freed. Following code frees
this memory:

while( (item = pqueue_pop(s->d1->unprocessed_rcds.q)) != NULL)
{
DTLS1_RECORD_DATA *rdata;
rdata = (DTLS1_RECORD_DATA *)item->data;
if (rdata->rbuf.buf != NULL)
{
OPENSSL_free(rdata->rbuf.buf);
}
OPENSSL_free(item->data);
pitem_free(item);
}
pqueue_free(s->d1->unprocessed_rcds.q);

There are also same code for s->d1->processed_rcds in dtls1_free(...).

Pavel

__________________________________________________ ____________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majordomo@openssl.org