The count we passed to memcmp() in mpi_eq() and mpi_eq_abs() was actually the number of significant words in the MPI, rather than the number of bytes we wanted to compare. Multiply by 4 to get the correct value.
To make the intent of the code more apparent, introduce a private MPI_MSW() macro which evaluates to the number of significant words (or 1-based index of the most significant word). This also comes in handy in mpi_{add,sub,mul}_abs().
Add a couple of test cases which not only demonstrate the bug we fixed here but also demonstrate why we must compare whole words: on a big-endian machine, we would be comparing the unused upper bytes of the first and only word instead of the lower bytes which actually hold a value...
We failed to clear the negative flag when handling trivial cases, so if one of the terms was 0 and the other was negative, the result would be an exact copy of the non-zero term instead of its absolute value.
- Use the new vector byte-order conversion functions where appropriate.
- Use memset_s() instead of memset() where appropriate.
- Use consistent names and types for function arguments.
- Reindent, rename and reorganize to conform to Cryb style and idiom.
SHA224 and SHA256 were left mostly unchanged. MD2 and MD4 were completely rewritten as the previous versions (taken from XySSL) seem to have been copied from RSAREF.
This breaks the ABI as some context structures have grown or shrunk and some function arguments have been changed from int to size_t.
We now have separate encryption and decryption methods, and can process an arbitrary amount of plaintext or ciphertext per call, rounded down to the block size (if applicable). For stream ciphers, we also have a keystream method which fills the provided buffer with an arbitrary amount of keystream (once again, rounded down if applicable).
Not only is this a slightly more logical name, but it allows us to expose the verbose flag, previously private to cryb_t_main.c, as the equally logically named t_verbose.
These tests reveal that both functions assume that the target is initially positive zero if it is not identical with one of the operands, and fail in various ways if it is not, or if it is uninitialized.
Unlike assert(3), which uses abort(3), this has no other side effects (before raising SIGABRT) than an fprintf() call. The test framework will catch the SIGABRT, report that the test case failed, and proceed with the next case.
All further instances of asprintf() or vasprintf() in our codebase are either in libcryb-test or in individual unit tests, and in all cases, the only consequence of a failed call is that the result will say "no description" instead of either a description of the test or an explanation of how it failed. Therefore, we can simply ignore the problem and cast the call to void to satisfy gcc.
POSIX requires <stdio.h> and <unistd.h> to define off_t and ssize_t like <sys/types.h> does, but not all platforms respect that. Play it safe by always including <sys/types.h>.
