FWIW, I think we mostly agree but are attacking the general issue from
different angles.

On Wed, 2008-03-05 at 10:44 -0500, Thor Lancelot Simon wrote:
> Well, I did not suggest that one would ever use a file descriptor per
> operation. But, let me try again.
> Note first that that OpenSSL does not abstract I/O completion waiting
> (the API provides SSL_read(), SSL_write(), SSL_connect(), SSL_shutdown(),
> but *not* SSL_poll() or SSL_select). When I proposed correcting this,
> last year, I got a pretty strongly negative reception.

I was not involved in the discussion so am not familiar with the
arguments. However I agree that SSL_[poll|select]() would be unwise
because it presumes to make the SSL/TLS stack transport-aware, whereas
the BIO scheme is an honest (if sometimes inelegant) attempt to leave
the stack transport-agnostic. SSL/TLS can be fed data on all sides
purely from memory, and the real "transport" could come from lots of
non-fd-based sources. Eg. if you somehow didn't trust multi-user windows
(though we all know windows is 100% trustworthy so this is obviously an
outlandish proposition ...) then you could theoretically use SSL/TLS to
encapsulate message passing - ie. the ability to watch IPC would not
reveal the contents of that IPC. Hmm, I can just about smell the
presence of a numbskull who'll try to build a DRM scheme around
this ...

Just as the SSL user needs to "know" transport specifics to create BIOs
in the first place for hooking up to the SSL object, they also need to
"know" how (and on what) to wait when they see WANT_READ/WANT_WRITE
codes and so-forth. I think the same would essentially be true for the
use of asynchronous crypto. The point is to keep openssl agnostic, not
the application. (Now I've probably offended a vast array of library
fascists who will exclaim that libraries are supposed to "hide the
problem" rather than "hiding themselves from the problem" - to which I
can only say, (1) this is security, not ODBC-for-javascript, and (2)
there is a world beyond POSIX, hard to imagine as that may be.)

> Note next that there are already a nontrivial number of nontrivial
> applications using the existing non-blocking select()/poll() driven API
> to OpenSSL, despite its warts. This API *already* requires, quite
> explicitly, extracting the file descriptor from each SSL session and
> adding those descriptors to a select() or poll() set. Applications are
> already written to this API and it does not abstract completion waiting
> as you suggest it should (I happen to agree with you, but neither you
> nor I can adjust all of everyone else's code out there in the world!).

Actually I don't claim that it should abstract completion waiting, only
that the SSL/TLS API shouldn't. For one thing, async crypto will be a
libcrypto interface - not an SSL/TLS-specific interface. Just as BIOs
are a libcrypto interface.

> This API also *already* requires any operations that would block to be
> reissued at the SSL layer by retrying exactly the same operation again
> after the file descriptor for that session comes up ready with select
> or poll.

Sure, but that's because you are *not* suspended miles down a code path
that you need to find your way back to. It means the state-machine has
advanced as far as it can go and to advance further, you will need the
presence of either input on your read BIO or some drainage on your write
BIO. Those events are on the exterior of the SSL/TLS state-machine and
represent a stable state, the code-paths exit cleanly in that state and
when progress is possible, we will go down new code paths to do new
work. This is not the same as trying to pick up where you left off way
down a code-path that has not been designed with suspend-and-retry
semantics in mind. Eg. if you were calling SSL_read() when you got a
WANT_WRITE, but you then called SSL_write() - it should *not* crash.
What you're describing, about being able to replay your way down a prior
code-path, would be very hard to make robust in an equivalent way.

NB, this would be different if the SSL/TLS code had, since the
beginning, the idea of a third BIO-like dependency - ie. if the
state-machine was designed to interact with a read BIO, a write BIO, and
a crypto-interconnect. If that were the case, WANT_CRYPTO would already
exist, just like WANT_READ and WANT_WRITE. But the code is not like that
and making it like that would amount (IMHO) to a rewrite.

> I do not love this API, but it is what all the existing, non-threaded
> non-blocking OpenSSL API consumers are written to, and I think it would
> be a considerable mistake to change it now.

I quite agree.

> So, given my "note first" and "note second" I believe I have established
> that, to do non-blocking I/O in a single-threaded application with OpenSSL,
> one must already:
> 1) Dip beneath the SSL abstraction layer to get the file
> descriptor for each session.

No, this is one form it must be able to take, but this is not *the*
general method for OpenSSL to support.

> 2) Use select() or poll() to wait on all those file descriptors.

During which time the SSL/TLS object (and "token" handling) is in *what*
state exactly?

> 3) Retry operations, exactly as issued prior to completion waiting,
> when their sessions' descriptors come up ready in select or
> poll (whether to select for read-ready or write-ready is known
> by the WANT_READ or WANT_WRITE SSL errors returned when the
> operation would block).

I've been down this route before and the "retry" is where I lose
interest in this approach. The tree of possible code-paths that *could*
go asynchronous is frightening, and adapting them to return back up the
call-stack cleanly in a "WANT_CRYPTO" manner, as well as going back down
the call-stack to "retry" is at best going to be a usually-works hack.
And if there's one place we don't want cut-corner hacks, it's in an
SSL/TLS implementation. In other words, I've never seen CERT issue an
security advisory on the basis of design insanity, but I don't want us
pioneering in this field.

> I propose, then, to extend the existing interface to accomodate non-blocking
> operation of ENGINEs themselves, by:

I see where you're heading, and some pieces of the idea are (IMHO) going
to be part of any meaningful solution. But from a big-picture
perspective, I think you're heading down a rat-hole with this approach.
If we take openssl as it stands, and the considerable inertia associated
with its existing user-base (a valid point you made earlier), I think
the SSL/TLS implementation will just not let us go the way you're

And to your probably significant frustration, I'm not going to try and
describe what I think we should do instead. At least, not in this email.
I'm going to try and mock-up an overview of how I think openssl should
address this and I will certainly run it by you for feedback.

> Given the already existing API for non-blocking operation in OpenSSL, I
> must confess I am pretty much baffled how else it _could_ work.

I'll give you a clue, create a stack object, extend it with state about
your current context, swapcontext(3) to the new stack (or whatever
compiled-in equivalent your platform or mood provides for), call any SSL
API you like, presume that leads down to crypto code that "wants" to go
asynchronous, the async crypto code detects that async suspension has
been made *available* (the stack object stuff), set appropriate
suspension flags and swapcontext() back to the "user's" context, use the
suspension flags to glue the "token" logic so the user can know when to
go back to that SSL object, and when it's time, the retry is a
swapcontext(3) (or sigsetjmp() - read Ralf's GNU-Pth docs and/or usenix
paper if you want to know how these things work) straight back to the
crypto code.

Damn, there I go describing what I said I wouldn't describe yet. And
trying to do so in one sentence has no doubt rendered it completely
obscure. Anyway, the way to do this robustly with the existing openssl
code, and do so without processing waste, is to leave the call-stack
as-is when you are way down in the crypto code - you context-switch back
out to suspend and context-switch back in to resume.

The rest is details and having an interface that can port to unusual
environments (I reiterate: not just file-descriptors). The "waiting
mechanism" is really the detail of how you represent and convey that
suspension between the caller/user and the the crypto code. Moreover,
SSL/TLS is just one use-case of this mechanism, it's not
SSL/TLS-specific, it's libcrypto-specific.

More news when I have some.


__________________________________________________ ____________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majordomo@openssl.org