Program crashes in EVP_CipherUpdate while doing T-DES Decryption - Openssl

This is a discussion on Program crashes in EVP_CipherUpdate while doing T-DES Decryption - Openssl ; Hi all, Platform: Redhat Linux Advanced Server 4.0 Compiler: g++ 3.3.2 (C++ compiler used) Linkage: openssl libraries statically linked with the application program. Version: OpenSSL 0.9.7a Feb 19 2003 (I tried with a recent version of openssl, but same result). ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: Program crashes in EVP_CipherUpdate while doing T-DES Decryption

  1. Program crashes in EVP_CipherUpdate while doing T-DES Decryption

    Hi all,

    Platform: Redhat Linux Advanced Server 4.0
    Compiler: g++ 3.3.2 (C++ compiler used)
    Linkage: openssl libraries statically linked with the application program.
    Version: OpenSSL 0.9.7a Feb 19 2003 (I tried with a recent version of
    openssl, but same result).

    I am trying out the EVP API functions to encrypt and decrypt a string using
    Triple DES in CBC mode. The program is actually a shared object, which is
    loaded by another EXE. The program gets a base64 string, which I first do a
    base64 decode and then apply the T-des decryption EVP.

    The encryption works fine, whereas the decryption program crashes at
    EVP_CipherUpdate with the following stack trace:

    #0 0x00000000 in ?? ()
    #1 0xb4f9df57 in EVP_EncryptUpdate () from /home/ambarish/appssl.so
    #2 0xb4f9e047 in EVP_EncryptUpdate () from /home/ambarish/appssl.so
    #3 0xb4f9e18d in EVP_DecryptUpdate () from
    /home/ambarish/PFG/impersonation/authzplugin/appssl.so
    #4 0xb4f5d4ec in tdesDecrypt (in=0x8fac420
    "ۥYKԶ#oBUKEA֮\016\223\217\026\223\ 226<\036=", len=40,
    output=0x8fac450 "\nf", outlen=0xb6650ecc) at tdes.cpp:141
    #5 0xb4f57b0e in DecrFn (pContext=0x8f7b8c8, pFnBlock=0x8fac408,
    pInfo=0x8fac3f0) at appssl.cpp:259

    It is not multi-threaded at this point (I have to provide MT support later),
    so I have not used the Crypto locking functions in the code.


    The code snippet is given below. The encryption program is exactly the same
    except:
    EVP_CipherInit_ex(&ctx, EVP_des_ede3_cbc(), NULL, key, iv, ENCRYPT); //
    This works fine.


    ....

    unsigned char *binText; // The binary text after
    base64 decode
    unsigned int binTextLen = 0; // Spaces allocated for
    cipher text
    unsigned int cipherTextLen = 0; // Cipher Text Length after
    base64 decoding
    unsigned char *plainText; // The plantext after
    decryption
    unsigned int plainTextLen = 0; // The plaintext length

    // Base64 Decode the cookie, and then Decrpyt the Cookie

    binTextLen = (((strlen(input) + 3)/4) * 3);
    binText = new unsigned char [binTextLen];

    b64Decode(binText, dnipcookie, &cipherTextLen); // We base64 decode
    input. o/p will be held in binText

    plainText = new unsigned char [cipherTextLen];
    tdesDecrypt(binText, cipherTextLen, plainText, &plainTextLen);
    // Tdes Decrypt. The output will be held in plainText.

    plainText[plainTextLen] = '\0';

    delete[] plainText;
    delete[] binText;
    ....

    void b64Decode(unsigned char *ret, const char *buf, unsigned int *len)
    {
    int tlen;
    int buflen = strlen(buf);

    tlen = EVP_DecodeBlock(ret, (unsigned char *)buf, buflen);

    *len = tlen;
    }


    int tdesDecrypt(unsigned char *in, unsigned int len, unsigned char *output,
    unsigned int *outlen)
    {
    unsigned char initVector[8] =
    {0xfe,0xdc,0xba,0x98,0x76,0x54,0x32,0x10}; //The initialization
    vector will be derived from this
    unsigned char iv[EVP_MAX_IV_LENGTH], key[EVP_MAX_KEY_LENGTH];

    int outbuflen=0;
    int outbuflen2=0;

    EVP_CIPHER_CTX ctx;

    unsigned char salt[8] = {0x53,0x41,0x4C,0x54,0x73,0x61,0x6C,0x74};
    unsigned char keyInput[24] =
    {0x3F,0x6F,0x6B,0x69,0x20,0x5E,0x5F,0x45,0x65,0x54 ,0x5D,0x56,0x63,0x68,0x6E,
    0x6F,0x14,0x32,0x2C,0x41,0x3F,0xD3,0x9B,0xA3};

    memcpy(iv,initVector,sizeof(iv));

    int keysize = EVP_BytesToKey(EVP_des_ede3_cbc(), EVP_md5(), salt,
    keyInput, sizeof(keyInput), 1, key, iv);

    EVP_CIPHER_CTX_init(&ctx);
    EVP_CipherInit_ex(&ctx, EVP_des_ede3_cbc(), NULL, key, iv, DECRYPT);

    EVP_CipherUpdate(&ctx, output, &outbuflen, in,len); ////////////
    Crashes here. /////////////

    EVP_CipherFinal_ex(&ctx, &output[outbuflen], &outbuflen2);

    EVP_CIPHER_CTX_cleanup(&ctx);

    output[outbuflen + outbuflen2] = '\0';
    *outlen = outbuflen + outbuflen2;

    return 0;
    }


    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
    __________________________________________________ ____________________
    OpenSSL Project http://www.openssl.org
    User Support Mailing List openssl-users@openssl.org
    Automated List Manager majordomo@openssl.org


  2. Re: Program crashes in EVP_CipherUpdate while doing T-DES Decryption

    Given the coredump stack trace you provide, it looks like
    ctx->cipher->do_cipher == NULL, while it should be a valid pointer to
    a cypher-specific en/decrypt function (see evp_cipher_st typedef in
    evp.h).
    Please check with assert() or debugger if this member != NULL before
    entering the decrypt function. If, like I assume, it *is* NULL, that
    means the INIT went wrong. Since you do not check error codes here (or
    cut away that code to shorten the example), I don't know what error
    was returned by the ...init_ex() call there.
    By the way, what is the value of the 'DECRYPT' define? (last parameter
    to your Init_ex()) Should be 0 or 1.

    Ger


    On Mon, Aug 4, 2008 at 3:06 PM, Ambarish Mitra
    wrote:
    > Hi all,
    >
    > Platform: Redhat Linux Advanced Server 4.0
    > Compiler: g++ 3.3.2 (C++ compiler used)
    > Linkage: openssl libraries statically linked with the application program___________________________________________ ___________________________

    OpenSSL Project http://www.openssl.org
    User Support Mailing List openssl-users@openssl.org
    Automated List Manager majordomo@openssl.org


  3. Re: Program crashes in EVP_CipherUpdate while doing T-DES Decryption

    Shoot. Forgot to mention: I checked this on latest 0.9.9 CVS, so
    slight differences may appear)

    --
    Met vriendelijke groeten / Best regards,

    Ger Hobbelt

    --------------------------------------------------
    web: http://www.hobbelt.com/
    http://www.hebbut.net/
    mail: ger@hobbelt.com
    mobile: +31-6-11 120 978
    --------------------------------------------------
    __________________________________________________ ____________________
    OpenSSL Project http://www.openssl.org
    User Support Mailing List openssl-users@openssl.org
    Automated List Manager majordomo@openssl.org


  4. RE: Program crashes in EVP_CipherUpdate while doing T-DES Decryption




    Given the coredump stack trace you provide, it looks like
    ctx->cipher->do_cipher == NULL, while it should be a valid pointer to
    a cypher-specific en/decrypt function (see evp_cipher_st typedef in
    evp.h).
    Please check with assert() or debugger if this member != NULL before
    entering the decrypt function.

    ----------
    AM: Indeed it is NULL. Thanks for pointing out the cause. I checked the code
    of EVP_CipherUpdate in crypto/evp/evp_enc.c and got an idea how it works.


    If, like I assume, it *is* NULL, that
    means the INIT went wrong.

    --------
    AM: Now, I will look into the init routines closely. All along I was looking
    at a wrong place.


    Since you do not check error codes here (or
    cut away that code to shorten the example), I don't know what error
    was returned by the ...init_ex() call there.
    By the way, what is the value of the 'DECRYPT' define? (last parameter
    to your Init_ex()) Should be 0 or 1.

    ----------
    Yes, #define DECRYPT 0
    is present in the code.


    DISCLAIMER
    ==========
    This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
    __________________________________________________ ____________________
    OpenSSL Project http://www.openssl.org
    User Support Mailing List openssl-users@openssl.org
    Automated List Manager majordomo@openssl.org


+ Reply to Thread