[BUGFIX] BN_GF2m_mod_arr() infinite loop

The following code will make BN_GF2m_mod_arr() into infinite loop.

int main(int argc, char *argv[])
{
BIGNUM *bn = NULL, *res = NULL, *p = NULL;

BN_hex2bn(&bn3, "448692853686179295b477565726f6e5d");
BN_hex2bn(&p, "100000000000000000000000000000087");
res = BN_new();
BN_GF2m_mod(res, bn3, p);
}

Because in final round of reduction d0 == 0 and z[dN] != 0, which
makes z[dN] can not be changed for ever. This is fixed by set
z[dn] = 0 if d0 == 0.

This patch is based on openssl SNAPSHOT 20080519, and has been tested
on x86_64 with openssl/test/bntest.c and above program.

Signed-off-by: Huang Ying <ying.huang@intel.com>

---
crypto/bn/bn_gf2m.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

--- a/crypto/bn/bn_gf2m.c
+++ b/crypto/bn/bn_gf2m.c
@@ -322,7 +322,11 @@ int BN_GF2m_mod_arr(BIGNUM *r, const BIG
if (zz == 0) break;
d1 = BN_BITS2 - d0;

- if (d0) z[dN] = (z[dN] << d1) >> d1; /* clear up the top d1 bits */
+ /* clear up the top d1 bits */
+ if (d0)
+ z[dN] = (z[dN] << d1) >> d1;
+ else
+ z[dN] = 0;
z[0] ^= zz; /* reduction t^0 component */

for (k = 1; p[k] != 0; k++)

______________________________________________________________________
OpenSSL Project [url]http://www.openssl.org[/url]
Development Mailing List [email]openssl-dev@openssl.org[/email]
Automated List Manager [email]majordomo@openssl.org[/email]