Truncated output from CFB mode when calling from C

Question

I have pretty annoying issue which I'm unable to solve for 2 days. I have an encrypt() method which makes use of Crypto++ library written in C++. The method is implemented as follows:

string CRijndaelHelper::Encrypt(string text)
{
    CFB_Mode< AES >::Encryption e;
    e.SetKeyWithIV(_key, sizeof(_key), _iv);

    string cipher, encoded;

    // CFB mode must not use padding. Specifying
    //  a scheme will result in an exception
    StringSource ss(text, true,
        new StreamTransformationFilter(e,
            new StringSink(cipher)
        ));     

    return cipher;
};

When I call this method within the native environment it works perfectly, encrypting the whole 26Kb xml string without any problem.

Eventually I've had to implement the encryption in C# code, for that purpose I wrote the following wrapper code in native dll for later use with PInvoke:

extern "C"
API_EXPORT char* _cdecl encrypt(char* contents)
{   
    string cont(contents);
    CRijndaelHelper rij;
    string transformedText = rij.Encrypt(cont);     
    return marshalReturn(transformedText);  
}

While the PInvoke part looks as follows:

[DllImport("EncryptDecrypt.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.Cdecl)]
[return: MarshalAs(UnmanagedType.LPStr)]
public static extern string encrypt([MarshalAs(UnmanagedType.LPStr)] string contents);

And everything looks working perfect except that I get in transformedText variable only first 540 bytes encrypted and that's all.

Please advise.

Answer 1

... returns a string which is correctly encrypted when invoked from native code

The problem is not you C++ encrypt, which uses a std::string. The problem is with marshalling it back to managed code as a char*.

Change CRijndaelHelper::Encrypt to the following to remove the embedded NULL that will be sprinkled liberally with a probability of 1/255:

#include <cryptopp/base64.h>
using CryptoPP::Base64Encoder;
...

string CRijndaelHelper::Encrypt(string text)
{
    CFB_Mode< AES >::Encryption e;
    e.SetKeyWithIV(_key, sizeof(_key), _iv);

    string cipher, encoded;

    // CFB mode must not use padding. Specifying
    //  a scheme will result in an exception
    StringSource ss(text, true,
        new StreamTransformationFilter(e,
            new Base64Encoder(
                new StringSink(cipher)
        )));     

    return cipher;
};

With the Base64 encoding, the marshaling as a char* will not truncate the encryption results on the first NULL byte encountered.

Then, for decryption:

StringSource ss(cipher, true,
    new Base64Decoder(
        new StreamTransformationFilter(d,
            new StringSink(text)
    )));   

Answer 2

Your fundamental problem is that you are working on the wrong types. Encryption works with byte arrays and not with text. You need to stop using string, char* and so on. You need to use unsigned char* in the unmanaged code, and byte[] in the managed code.

It might be hard for you to accept this advice, but I proffer it all the same. On top of the problems with zero bytes being treated as null terminators (the reason for your truncation), your current approach simply neglects the issue of text encoding. You cannot do that.

You presumably have good reasons why you wish to transform a string to another string. That's fine, but the encryption part of the transformation needs to operate on byte arrays. The way to handle encryption of text to text is to use a chain of transformations. The transformation when encrypting runs like this:

1. Convert string to byte array using well-defined encoding. For instance, UTF-8.
2. Encrypt byte array, which results in another byte array being output.
3. Encode this encrypted byte array as text using, for example, base64.

This respects the fact that encryption operates on byte arrays, as well as being explicit about the text encoding.

In the opposite direction you reverse the steps:

1. Decode the base64 to an encrypted byte array.
2. Decrypt this byte array.
3. Decode the decrypted byte array using UTF-8 to obtain the original string.