I've spent two days on this so far and combed through every source at my disposal, so this is the last resort.
I have an X509 certificate whose public key I have stored in the iPhone's keychain (simulator only at this point). On the ASP.NET side, I've got the certificate in the cert store with a private key. When I encrypt a string on the iPhone and decrypt it on the server, I get a CryptographicException "Bad data." I tried the Array.Reverse suggested in the RSACryptoServiceProvider page on a longshot, but it did not help.
I have compared the base-64 strings on both sides and they're equal. I've compared the raw byte arrays after decoding and they too are equal. If I encrypt on the server using the public key, the byte array is different from the iPhone's version and readily decrypts using the private key. The raw plaintext string is 115 characters so it's within the 256-byte limitation of my 2048-bit key.
Here's the iPhone encryption method (pretty much verbatim from the CryptoExercise sample app's wrapSymmetricKey method):
+ (NSData *)encrypt:(NSString *)plainText usingKey:(SecKeyRef)key error:(NSError **)err
{
size_t cipherBufferSize = SecKeyGetBlockSize(key);
uint8_t *cipherBuffer = NULL;
cipherBuffer = malloc(cipherBufferSize * sizeof(uint8_t));
memset((void *)cipherBuffer, 0x0, cipherBufferSize);
NSData *plainTextBytes = [plainText dataUsingEncoding:NSUTF8StringEncoding];
OSStatus status = SecKeyEncrypt(key, kSecPaddingNone,
(const uint8_t *)[plainTextBytes bytes],
[plainTextBytes length], cipherBuffer,
&cipherBufferSize);
if (status == noErr)
{
NSData *encryptedBytes = [[[NSData alloc]
initWithBytes:(const void *)cipherBuffer
length:cipherBufferSize] autorelease];
if (cipherBuffer)
{
free(cipherBuffer);
}
NSLog(@"Encrypted text (%d bytes): %@",
[encryptedBytes length], [encryptedBytes description]);
return encryptedBytes;
}
else
{
*err = [NSError errorWithDomain:@"errorDomain" code:status userInfo:nil];
NSLog(@"encrypt:usingKey: Error: %d", status);
return nil;
}
}
And here's the server-side C# decryption method:
private string Decrypt(string cipherText)
{
if (clientCert == null)
{
// Get certificate
var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
store.Open(OpenFlags.ReadOnly);
foreach (var certificate in store.Certificates)
{
if (certificate.GetNameInfo(X509NameType.SimpleName, false) == CERT)
{
clientCert = certificate;
break;
}
}
}
using (var rsa = (RSACryptoServiceProvider)clientCert.PrivateKey)
{
try
{
var encryptedBytes = Convert.FromBase64String(cipherText);
var decryptedBytes = rsa.Decrypt(encryptedBytes, false);
var plaintext = Encoding.UTF8.GetString(decryptedBytes);
return plaintext;
}
catch (CryptographicException e)
{
throw(new ApplicationException("Unable to decrypt payload.", e));
}
}
}
My suspicion was that there was some encoding problems between the platforms. I know that one is big-endian and the other is little-endian but I don't know enough to say which is which or how to overcome the difference. Mac OS X, Windows, and the iPhone are all little-endian so that's not the problem.
New theory: if you set the OAEP padding Boolean to false, it defaults to PKCS#1 1.5 padding. Apparently, SecKey only has SecPadding definitions of PKCS1, PKCS1MD2, PKCS1MD5, and PKCS1SHA1. Perhaps Microsoft's PKCS#1 1.5 != Apple's PKCS1 and so the padding is affecting the binary output of the encryption. I tried using kSecPaddingPKCS1 with the fOAEP set to falseand it still didn't work.kSecPaddingPKCS1 is equivalent to PKCS#1 1.5. Back to the drawing board on theories…
Other newly-tried theories:
- Certificate on iPhone (.cer file) is not exactly the same as the PKCS#12 bundle on the server (.pfx file) and so it could never work. Installed .cer file in different cert store and server-encrypted string roundtripped just fine;
- Conversion to base-64 and act of POSTing to server resulted in oddness that wasn't present in same class roundtrip so I first tried some URLEncoding/Decoding and then posted raw binary from iPhone, verified that it was equal, and got same bad data;
- My original string was 125 bytes so I thought it might be truncating in UTF-8 (long shot) so I cropped it down to a 44-byte string with no result;
- Looked back over the System.Cryptography library to make sure I was using an appropriate class and discovered `RSAPKCS1KeyExchangeDeformatter`, became elated at new prospects, and dejected when it behaved exactly the same.
Success!
It turned out that I had some cruft in my Keychain on the iPhone Simulator that was muddying the waters, so to speak. I deleted the Keychain DB at ~/Library/Application Support/iPhone Simulator/User/Library/Keychains/keychain-2-debug.db to cause it to be re-created and it worked fine. Thank you for all of your help. Figures it would have been something simple but non-obvious. (Two things I learned: 1) uninstalling the app from the simulator does not clear its Keychain entries and 2) start absolutely fresh periodically.)
NOTE: The generic path for the keychain file is dependent on the iOS version: ~/Library/Application Support/iPhone Simulator/[version]/Library/Keychains/keychain-2-debug.db e.g., ~/Library/Application Support/iPhone Simulator/4.3/Library/Keychains/keychain-2-debug.db
