Andy Polyakov wrote, responding to my earlier post:
> Paul,
>
> > With the wide variety of platforms, it is not unusual for
> > problems to show up on only one platform.

>
> Yes, but when it does, it's more and more likely to mean that it's
> something wrong with end-developer environment, e.g. a
> compiler bug...
> Naturally provided that platform is correctly recognized by
> ./config...


Config knows about VOS and selects "no-threads no-shared no-asm no-dso".

grep vos-gcc Configure
"vos-gcc","gcc:-b hppa1.1-stratus-vos -O3 -Wall -Wuninitialized
-D_POSIX_C_SOURCE=200112L
-D_BSD:unknown):VOS:-Wl,-map:BN_LLONG:::::::::::::.so:",
"debug-vos-gcc","gcc:-b hppa1.1-stratus-vos -O0 -g -Wall
-D_POSIX_C_SOURCE=200112L -D_BSD -DBN_DEBUG -DREF_CHECK -DCONF_DEBUG
-DBN_CTX_DEBUG
-DCRYPTO_MDEBUG:unknown):VOS:-Wl,-map:BN_LLONG:::::::::::::.so:",

Now, this is the configure line for our PA-RISC platform, and we pass all
self-tests there. Except for the "-b hppa1.1-stratus-vos" option, it is
identical to the configure line for our new Intel platform. (The shipping
source code for OpenSSL 0.9.7e has the patches to run unmodified on our
PA-RISC platforms. I haven't yet submitted the patches for our Intel
platform).

I see that I didn't specify an explicit endianness here...but that wouldn't
have mattered, as you note below.

> > I had a similar problem with MD4, MD5, RIPEMD and SHA on my system
> > because some of the OpenSSL source code assumes that certain
> > platforms have certain endian characteristics. In my case, the code
> > tested the _i386 macro and decided the platform must be little
> > endian.

>
> This is wastly misleading. Endianess assumptions are based upon
> -D[LB]_ENDIAN macro in corresponding ./Configure line, not upon CPU
> macros set up by compiler. Well, assembler code does assume endianess
> [naturally!], so that if you get rid of -D[LB]_ENDIAN [or change it],
> you better configure with no-asm. Also note that [LB]_ENDIAN is
> essentially a performance option and the code works correctly
> without it.


It wasn't an assembler problem, as we were not using the assembly code.

My bold reference is to the following code fragment, which appears in
md4_locl.h, md5_locl.h, sha_locl.h, and rmd_locl.h. The macros do not
reference the endian macros, but test the platform-specific (and
compiler-based) macros to determine whether to use the aligned or unaligned
data copy code. But they also have a built-in assumption about host
endianness.

#if defined(__i386) || defined(__i386__) || defined(_M_IX86) ||
defined(__INTEL__)
/*
* *_block_host_order is expected to handle aligned data while
* *_block_data_order - unaligned. As algorithm and host (x86)
* are in this case of the same "endianness" these two are
* otherwise indistinguishable.
....
*/
#define md4_block_data_order md4_block_host_order
#endif

#define DATA_ORDER_IS_LITTLE_ENDIAN

Maybe I don't understand this code as well as I thought, but the way I read
this code, the algorithm is expecting data in LE order, and is presuming
that Intel platforms also store data in LE order. That assumption is
incorrect for our (admittedly rather unusual) platform. As coded, md4 fails
on our BE Intel platform. When I recoded this #if statement to not be true,
md4 worked. What am I missing?

> > As it happens, I have a
> > big-endian environment on an Intel box,

>
> Is it really possible? Is there an x86 big-endian? Is it possible to
> switch Intel x86 to big-endian? What OS does it run? Which
> config line does OpenSSL picks then?


So many questions!

It is not only possible, we've done it, and we're shipping products. We use
a P4 Xeon and of course it only processes data in little-endian format.
There is no hardware option to operate in big-endian mode (sadly). However,
the compilers that we wrote or ported produce the necessary byte-swap
instructions. All user-visible data in main memory is stored in big-endian
format, and the compilers byte-swap when necessary. It runs the Stratus VOS
operating system. The config line is for "vos", as mentioned above.

> [snip]


> > Take a close look at some of the #if macro tests in, for example,
> > crypto/sha/sha_locl.h. You could be hitting a similar problem.

>
> I bet not:-) As mentioned, it smells a compiler bug...


OK, you're on. I'll take a Sam Adams, please, if I'm right. What's your
pleasure? This problem happened with two completely different compilers, by
the way.

Thanks
PG
--
Paul Green, Senior Technical Consultant,
Stratus Technologies, Maynard, MA USA
Voice: +1 978-461-7557; FAX: +1 978-461-3610

__________________________________________________ ____________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majordomo@openssl.org