Hi everybody,

I'm currently in some kind of a dead-end and hope that someone here
can help me.
I want to use the crypto library of OpenSSL to sign and verify
messages using RSASSA-PSS as specified in RSA PKCS#1.

The problem is, that whenever I sign a message and try to verfiy it,
the verification always fails.
I have a test function which does nothing more than sign a message and
verify it imediatly afterwards.
The actual input data of the test function is an unsigned char array
of size 302 and a 2048 bit RSA key in DER format. The output of the
sign function should be 256 bytes long.

Can anybody spot some obvious mistakes? The PSS functions are
unfortunately very poorly documented

Regards
Morris

The implementations of the sign, verify and test functions look like
this:

#include
#include
#include

void sign(unsigned char* output, unsigned char* input, int
inputLength, const unsigned char* key, int keyLength)
{
// temp buffer
unsigned char buffer[256];

// apply SHA-1 hash function
unsigned char hashed[EVP_sha1()->md_size];
SHA_CTX sha_ctx;
SHA1_Init(&sha_ctx);
SHA1_Update(&sha_ctx, input, inputLength);
SHA1_Final(hashed, &sha_ctx);

// apply PSS padding using SHA-1 and salt length 20
RSA *rsa = d2i_RSAPrivateKey(NULL, &key, keyLength);
RSA_padding_add_PKCS1_PSS(rsa, buffer, hashed, EVP_sha1(), 20);

// encrypt hashed and padded signature
RSA_private_encrypt(256, buffer, output, rsa, RSA_NO_PADDING);
}

int verify(unsigned char* input, const unsigned char* key, int
keyLength)
{
// temp buffer
unsigned char buffer[256];

// apply SHA-1 hash function
unsigned char hashed[EVP_sha1()->md_size];
SHA_CTX sha_ctx;
SHA1_Init(&sha_ctx);
SHA1_Update(&sha_ctx, input, 256);
SHA1_Final(hashed, &sha_ctx);

// decrypt signature
RSA* rsa = d2i_RSAPrivateKey(NULL, &key, keyLength);
RSA_public_decrypt(256, input, buffer, rsa, RSA_NO_PADDING);

// verify signature using SHA-1 and salt length 20
return RSA_verify_PKCS1_PSS(rsa, hashed, EVP_sha1(), buffer, 20);
}

int test(unsigned char* input, int inputLength, const unsigned char*
key, int keyLength)
{
unsigned char signature[256];

sign(signature, input, inputLength, key, keyLength);

if(verify(signature, key, keyLength))
{
printf("verified!");
}
else
{
printf("not verified!");
}
}

Hi again,

> I'm currently in some kind of a dead-end and hope that someone here
> can help me.
> I want to use the crypto library of OpenSSL to sign and verify
> messages using RSASSA-PSS as specified in RSA PKCS#1.

I've just found my misconception in the verify function. Everything is
working as intended now.
In the case that somebody is interested, see the new code below.

Regards
Morris

#include
#include
#include

void sign(unsigned char* output, unsigned char* input, int
inputLength, const unsigned char* key, int keyLength)
{
// temp buffer
unsigned char buffer[256];

// apply SHA-1 hash function
unsigned char hashed[EVP_sha1()->md_size];
SHA_CTX sha_ctx;
SHA1_Init(&sha_ctx);
SHA1_Update(&sha_ctx, input, inputLength);
SHA1_Final(hashed, &sha_ctx);

// apply PSS padding using SHA-1 and salt length 20
RSA *rsa = d2i_RSAPrivateKey(NULL, &key, keyLength);
RSA_padding_add_PKCS1_PSS(rsa, buffer, hashed, EVP_sha1(), 20);

// encrypt hashed and padded signature
RSA_private_encrypt(256, buffer, output, rsa, RSA_NO_PADDING);
}

int verify(unsigned char* signature, unsigned char* input, int
inputLength, const unsigned char* key, int keyLength)
{
// temp buffer
unsigned char buffer[256];

// apply SHA-1 hash function
unsigned char hashed[EVP_sha1()->md_size];
SHA_CTX sha_ctx;
SHA1_Init(&sha_ctx);
SHA1_Update(&sha_ctx, input, inputLength);
SHA1_Final(hashed, &sha_ctx);

// decrypt signature
RSA* rsa = d2i_RSAPrivateKey(NULL, &key, keyLength);
RSA_public_decrypt(256, signature, buffer, rsa, RSA_NO_PADDING);

// verify signature using SHA-1 and salt length 20
return RSA_verify_PKCS1_PSS(rsa, hashed, EVP_sha1(), buffer, 20);
}

int test(unsigned char* input, int inputLength, const unsigned char*
key, int keyLength)
{
unsigned char signature[256];

sign(signature, input, inputLength, key, keyLength);

if(verify(signature, input, inputLength, key, keyLength))
{
printf("verified!");
}
else
{
printf("not verified!");
}
}

You said that you want to sing and verify messages, but, in fact, you're encrypting the hash when you call this function:
// encrypt hashed and padded signature
RSA_private_encrypt(256, buffer, output, rsa, RSA_NO_PADDING);

Why don't you use RSA_sign() instead?

another question: in which way did u decide to pass the key? const unsigned char? why?
Can you post all the code, plz?

thanks in advance