Hi Users

I am using openssl in my Application. It works on 32 bit Linux
environment. Now I am porting my application to
64 Bit on HP-UX. I compiled the library with aCC -c +O2 -g +DD64 +Z
-D_HPUX_SOURCE -DRW_MULTI_THREAD -D_REENTRANT
flags. The shared library got successfully created and the
libsecuritymodule.sl is archive of all 64-Bit objects (ELF-64 relocatable
object files - IA64).

The problem that I am facing is

(Nested Function call flow)

(SecurityWrap1.cpp) (decrypt.c) (smime.c) (pk7_smime1.c)
(pk7_doit1.c) (p_dec1.c)
VerifyAndDecrypt ( ) -> DecryptMessage( ) -> openssl( ) ->
PKCS7_decrypt( ) -> PKCS7_dataDecode( ) -> EVP_PKEY_decrypt( ) ->

(rsa_lib1.c) (rsa_eay1.c)
(rsa_pk1_1.c)
RSA_private_decrypt( ) Line:233 -> RSA_eay_private_decrypt( ) ->
RSA_padding_check_PKCS1_type_2( )

Padding Check logic returns -1 ( Which says Padding Check failed / Block
Type is not O2). This causes openssl to return 4 as error code, and
Decryption fails.
I think the issue is related to computation of BIGNUM by function
BN_bn2bin( ) in bn_lib.c.

This is piece of code from rsa_pk1_1.c (forgive me for pasting the code
here):

int RSA_padding_check_PKCS1_type_2(unsigned char *to, int tlen,
unsigned char *from, int flen, int num)
{
int i,j;
unsigned char *p;

p=from;
if ((num != (flen+1)) || (*(p++) != 02))
{
RSAerr(RSA_F_RSA_PADDING_CHECK_PKCS1_TYPE_2,RSA_R_ BLOCK_TYPE_IS_NOT_02);
return(-1);
}

The num variable is 128 while flen is 178. So the if condition gets
validated and this returns Error.
Please help me what is wrong here?


Prateek Shrivastava
Tata Consultancy Services
Mailto: prateek.shrivastava@tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Outsourcing
____________________________________________
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you