How to get a SecIdentityRef from a SecCertificateR

Question

Here's what I've already tried:

First idea was, just put both into an array, use that array with kSecUseItemList, so keychain calls will operate only on the items in this array, not on real keychains and then get the identity like this:

NSDictionary * searchQuery = @{
    (__bridge id)kSecClass:(__bridge id)kSecClassIdentity,
    (__bridge id)kSecUseItemList:@[(__bridge id)key, (__bridge id)cert],
    (__bridge id)kSecReturnRef:@YES
};
CFTypeRef foundItem = NULL;
OSStatus copyStatus = SecItemCopyMatching(
    (__bridge CFDictionaryRef)searchQuery, &foundItem
);

Turned out that this doesn't work. To quote from the docs:

@constant kSecUseItemList Specifies a dictionary key whose value is a CFArray of items. If provided, this array is treated as the set of all possible items to search, or add if the API being called is SecItemAdd. The items in this array may be of type SecKeyRef, SecCertificateRef, SecIdentityRef, or CFDataRef (for a persistent item reference.) The items in the array must all be of the same type. When this attribute is provided, no keychains are searched.

Well, they are not of the same type, so this cannot work.

My second attempt was to add both items to keychain (using SecItemAdd()), which works as expected, then find the certificate (using SecItemCopyMatching()) which also succeeds and finally getting my identity using:

SecIdentityRef identity = NULL;
OSStatus copyStatus = SecIdentityCreateWithCertificate(NULL, cert, &identity);

But that fails with errKCItemNotFound.

Looking at the items in Keychain Access app, the certificate and the private key are both there, they are both correct but they are not displayed as forming an Identity (they are not listed under "My Certificates", the cert is only listed under "Certificates" and the key under "Keys").

Okay, what am I doing wrong or what important step am I missing?

If I export the key to PKCS#8 and the cert to DER notation, then use openssl on command line to combine both into a PKCS#12 file and import that file with Keychain Access, then they are displayed as an Identity in keychain access and this identity also works correctly (so the private key is really the correct key for the public key in the cert). But this is not really an option, as my code must not rely on OpenSSL and would ideally be portable to iOS.

As far as I understood the documentation, the identity matching is done by matching public key hashes, so that could be related to my problem. How would the system know the hash of the public key for me SecKeyRef, which is only a raw private RSA key?

The docs also say that I can add a SecIdentityRef directly with SecAddItem(), in which case I guess everything would probably work as expected (the identity itself cannot be added, the cert and the private key will be added, but I assume the identity binding will be okay when adding them that way), but that sounds like a chicken-egg-problem, since how would I get that identity reference in the first place?

I cannot understand why there is no SecCreateIdentity(...) function that simply takes a SecCertificateRef and SecKeyRef on input and returns a SecIdentityRef on output.

Update

Here's some interesting info I found in SecKey.h:

@constant kSecKeyLabel type blob, for private and public keys this contains the hash of the public key. This is used to associate certificates and keys. Its value matches the value of the kSecPublicKeyHashItemAttr of a certificate and it's used to construct an identity from a certificate and a key. For symmetric keys this is whatever the creator of the key passed in during the generate key call.

This value is not correctly set. The public key in the certs hashes to 0x966C57... but my private key contains 0x097EAD... and this looks like the hash of the private key itself. I will try if I can somehow set this value to the correct one.

Update 2

That seems to be another dead end. When I try to set kSecAttrApplicationLabel with SecKeyUpdate() on a key prior to adding it to keychain, I get errKCItemNotFound, which is expected, as the documentation says:

A SecKeyRef instance that represents a key that is stored in a keychain can be safely cast to a SecKeychainItemRef for manipulation as a keychain item. On the other hand, if the key is not stored in a keychain, casting the object to a SecKeychainItemRef and passing it to Keychain Services functions returns errors.

Fair enough. So I first add the key, then retrieve it back from keychain and finally try to update kSecAttrApplicationLabel, but this fails as well, this the error is errKCNoSuchAttr.

Oh, in case anyone wonders why I'm updating kSecAttrApplicationLabel when I said the attribute is named kSecKeyLabel in my first update: The kSecKeyLabel is an enumeration value of the old attribute enumerations that Apple used with various API calls that have all been deprecated. The new API calls (like SecItemUpdate()) work with dictionaries and as using enum values as dictionary keys is a bit ugly, Apple defined a new set of dictionary keys that are CFStringRef.

@constant kSecAttrApplicationLabel Specifies a dictionary key whose value is the key's application label attribute. This is different from the kSecAttrLabel (which is intended to be human-readable). This attribute is used to look up a key programmatically; in particular, for keys of class kSecAttrKeyClassPublic and kSecAttrKeyClassPrivate, the value of this attribute is the hash of the public key. This item is a type of CFDataRef. Legacy keys may contain a UUID in this field as a CFStringRef.

So this seems to be the correct attribute to update, doesn't it? Except that the error implies no such attribute exists for the item. Even though the same header file explicitly lists this attribute as a possible attribute for SecKeyRef items:

kSecClassKey item attributes:
    kSecAttrAccess (OS X only)
    kSecAttrAccessControl
    kSecAttrAccessGroup (iOS; also OS X if kSecAttrSynchronizable specified)
    kSecAttrAccessible (iOS; also OS X if kSecAttrSynchronizable specified)
    kSecAttrKeyClass
    kSecAttrLabel
    kSecAttrApplicationLabel
     [... and so on ...]

Update 3

The first answer I got suggested to use SecItemCopyMatching() instead, however, please understand that this code:

CFTypeRef result = NULL;
OSStatus status = SecItemCopyMatching(
    (__bridge CFDictionaryRef)@{
        (__bridge id)kSecClass:(__bridge id)kSecClassIdentity,
        (__bridge id)kSecMatchItemList:@[(__bridge id)cert],
        (__bridge id)kSecReturnRef:@YES
     }, &result
);

is really functional identical to his code:

SecIdentityRef result = NULL;
OSStatus status = SecIdentityCreateWithCertificate(
    NULL, cert &result
);

The later one is just the older API call (older, but not deprecated) from the time where keychain access was limited to working with Sec...Ref CoreFoundation "objects" (Apple's attempt to mimic a bit of OO in pure C), whereas the first one is the newer API where you usually only work with dictionary representation of keychain items (as that casts toll-free to Obj-C, you just need some bridging casts when you use ARC) but where you also have the choice to fall back to CoreFoundation "objects" (when using attributes like e.g. kSecMatchItemList, kSecUseItemList, or kSecReturnRef). I'm' actually rather sure that in fact there's just one real API and the other one is just implemented on top of the other one (depending which one on top of which one, the newer one may just exist for convenience or the older one was just kept for backward compatibility).

Answer 1

The trick is to export the in-memory key first and then re-import it directly to the keychain instead of just adding it there. See the code below (careful, it's C++):

static OSStatus AddKeyToKeychain(SecKeyRef privateKey, SecKeychainRef targetKeychain)
{
    // This is quite similar to pal_seckey's ExportImportKey, but
    // a) is used to put something INTO a keychain, instead of to take it out.
    // b) Doesn't assume that the input should be CFRelease()d and overwritten.
    // c) Doesn't return/emit the imported key reference.
    // d) Works on private keys.
    SecExternalFormat dataFormat = kSecFormatWrappedPKCS8;
    CFDataRef exportData = nullptr;

    SecItemImportExportKeyParameters keyParams = {};
    keyParams.version = SEC_KEY_IMPORT_EXPORT_PARAMS_VERSION;
    keyParams.passphrase = CFSTR("ExportImportPassphrase");

    OSStatus status = SecItemExport(privateKey, dataFormat, 0, &keyParams, &exportData);

    SecExternalFormat actualFormat = dataFormat;
    SecExternalItemType actualType = kSecItemTypePrivateKey;
    CFArrayRef outItems = nullptr;

    if (status == noErr)
    {
        status =
            SecItemImport(exportData, nullptr, &actualFormat, &actualType, 0, &keyParams, targetKeychain, &outItems);
    }

    if (exportData != nullptr)
        CFRelease(exportData);

    CFRelease(keyParams.passphrase);
    keyParams.passphrase = nullptr;

    if (outItems != nullptr)
        CFRelease(outItems);

    return status;
}

The code has been taken from here.

Answer 2

The answer from @bartonjs really solves my problem, I just want to provide some extra helpful information here, as it is not very obvious why it solves the problem:

First of all, please note that security objects can either exist only in memory or they can exist in memory backed up by keychain storage. Every security object has data (the actual data that defines the object) and metadata (additional information that describes the data). But only objects backed up by keychain storage can have metadata as the metadata attributes are defined as part of the keychain API and not as part of the object API. If objects are not stored in a keychain, they simply cannot have metadata. See Update 2 of my question.

The public key hash, which is required to match private keys to their certificates, is stored within the metadata. So a SecKeyRef not backed up by keychain storage cannot have such a hash. Generating an identity (SecIdentityRef) with a key not stored in a keychain is thus simply not possible. The certificate doesn't have to be stored in a keychain but the key has.

So far I was just adding my key using SecItemAdd(), which seems to do exactly what the name implies, it just adds the item to keychain, and it only does what the name implies, so it won't do anything but adding the item to keychain as is. If the item already has a public key hash, it will also have a public key hash when being added to a (new/different) keychain, but only items already within keychains can have this attribute. The result was an item not having a correct public key hash set and this is the cause of all my problems.

Now my code uses the SecItemImport() function, which is a lot more powerful, as an "import" may require a lot more steps than just adding something keychain. Apparently this import function will also make sure that the public key hash in the meta data is correctly populated when importing the item. To make this import possible, my new code first needs to export the existing key and so it can then re-import the key directly into the desired keychain.

Update

Maybe interesting to know, there exists a function with the following syntax:

SecIdentityRef SecIdentityCreate(
    CFAllocatorRef allocator, 
    SecCertificateRef certificate, 
    SecKeyRef privateKey
);

that works exactly the way I was desiring it. ffmpeg is actually using it. But this is private API, you must not use it if you plan to submit your software to any app store (using private API will have your software rejected).