md5_append nbytes should be size_t

There will never be a time when the length of a string can be negative.
This makes the function arguments semantically correct but also solves
some of the casting and bit shifting errors inside the function.