Mac Launch Daemon unable to retrieve password from

We have a Launch Daemon which (necessarily, for various reasons) runs as root, and which communicates with a server component via the network. It needs to authenticate with the service, so when it first obtains the password, we save it to the system keychain. On subsequent launches, the idea is to retrieve the password from the keychain and use it to authenticate with the network service.

This has been working fine, but on macOS 10.12 the existing code stopped working, and we've been entirely stumped on how to fix this. It boils down to this:

Regardless of whether we're saving a new password or retrieving an old one, we obtain a reference to the system keychain using this:

SecKeychainCopyDomainDefault(kSecPreferencesDomainSystem, &system_keychain);

We also disable user interaction for good measure, although we'd expect it to already be off in the context of a daemon.

SecKeychainSetUserInteractionAllowed(false);

When saving a new password to the keychain, we use

OSStatus status = SecKeychainAddInternetPassword(
    system_keychain,
    urlLength, server_base_url,
    0, NULL,
    usernameLength, username,
    0, NULL,
    0,
    kSecProtocolTypeAny, kSecAuthenticationTypeAny,
    passwordLength, password,
    NULL);

This much works. Success is reported, and I can see the item in the "system" keychain in Keychain Access.app.

Retrieving it on subsequent runs of our daemon is done with this line:

status = SecKeychainFindInternetPassword(
    system_keychain,
    urlLength, url,
    0, NULL,
    usernameLength, username,
    0, NULL,
    0,
    kSecProtocolTypeAny, kSecAuthenticationTypeAny,
    &passwordLength, &password_data,
    NULL);

Unfortunately, this has started returning errSecAuthFailed for reasons that are unclear to us.

A few additional details we've checked and things we've tried, to no avail:

The daemon binary is signed with a Developer Id certificate.
The daemon binary contains an embedded Info.plist section with a bundle ID and version.
I can see the daemon binary in the "Always allow access by these applications" list in the "Access Control" tab of the password item in Keychain Access.app.
If I manually switch to "Allow all applications to access this item" in Keychain Access, it works. This somewhat defeats the point of saving the password in the keychain, however.
We've tried playing around with the parameters to SecKeychainAddInternetPassword, but this doesn't seem to have made any difference.
We've tried explicitly unlocking the keychain with SecKeychainUnlock(), but as the documentation suggests, this seems to be superfluous.
Deleting the item in Keychain Access.app causes SecKeychainFindInternetPassword() to yield errSecItemNotFound, as you'd expect. So it can definitely find the saved item, it just isn't allowed to read it.

The keychain documentation isn't exactly easy to read and in parts rather tautological. ("In order to do Y, you need to do Y," without mentioning why you'd want to do Y.) Nevertheless, I think I've made it through and have understood most of it. Various aspects of our particular setup aren't covered in detail (access from a daemon), but it seems pretty clear that accessing an item previously saved by the same app should not require any special authorisation or authentication. Which is in direct contradiction to the behaviour we're seeing.

Any ideas?

After spending some more hours on this across several days, we finally worked out what was going on.

First, I tried to build a minimal example that would reproduce the problem. This did not fail with errSecAuthFailed and thus didn't reproduce the problem. So back to the original daemon, there must be something specifically about it that was going wrong.

The next idea was to check the system log for the time when SecKeychainFindInternetPassword() was called. This turned up some error messages:

securityd   CSSM Exception: -2147411889 CSSMERR_CL_UNKNOWN_TAG
securityd   MacOS error: -67063
securityd   MacOS error: -67063
securityd   code requirement check failed (-67063), client is not Apple-signed
securityd   CSSM Exception: 32 CSSM_ERRCODE_OPERATION_AUTH_DENIED
OurDaemon   subsystem: com.apple.securityd, category: security_exception, enable_level: 0, persist_level: 0, default_ttl: 0, info_ttl: 0, debug_ttl: 0, generate_symptoms: 0, enable_oversize: 0, privacy_setting: 2, enable_private_data: 0
OurDaemon   CSSM Exception: -2147416032 CSSMERR_CSP_OPERATION_AUTH_DENIED

This suggested the problem might be with code signing. Strange. Checking the code signature of the binary with codesign -vv returned no issues.

After hunting around the web for various parts of the error messages, I found -67063 corresponds to errSecCSGuestInvalid. The comment reads "code identity has been invalidated."

Okay, definitely some codesigning error, but what does it mean, and why did it occur?

Hunting around some more finally turned up the explanation, and also the solution: http://lists.apple.com/archives/apple-cdsa/2010/Mar/msg00027.html

It means that at some point since the program got started, something happened to it that made it invalid.

and

if you run a signed program, then replace it (by, say, building a new version in place :-), and then run the new version, the kernel will still hold the old signature attached to the executable's vnode. If that's your situation, just removing the executable and recreating it clears up the problem for good (until you overwrite the file again :-). We recommend that signed code always be replaced (mv(1), not cp(1), or equivalents).

This explained it. I was copying new versions of the daemon into place using

sudo cp path/to/built/daemon /usr/local/libexec/

Apparently, that overwrites the file in-place rather than creating a new vnode, writing that, and then renaming it over the old file. So the solution is to either cp to a temp directory first, and then mv into place. Or delete the destination file before using cp.

As soon as I did that, it worked!