Xcode Server Build: “Multiple matching codesigning

Question

All of a sudden, with no change to the Xcode build server, I started getting an error, even across different bots, branches, and apps (all are Mac apps). I didn't change any software or configuration on the build server (that I know of), or even sign into it. I get the error once during analysis:

Check dependencies

Code Sign error: Multiple matching codesigning identities found: Multiple codesigning identities (i.e. certificate and private key pairs) matching “Mac Developer: OS X Server (N________L)” were found.

And again during Testing (though my tests pass – the one above is what fails the build):

Testing failed:

Multiple matching codesigning identities found: Multiple codesigning identities (i.e. certificate and private key pairs) matching “Mac Developer: OS X Server (N________L)” were found.

I have tried the following, triggering a build after each step, with no change in the message:

1. Triggered another build
2. Updated to the latest Xcode on the build server (6.3.2)
3. Searched through Keychain Access for any "OS X Server" certificates or keys, finding none
4. Removed all Developer Teams from Server.app. Note: this acted weird. I had to remove it twice before the setting read "No Teams", but now it's stayed that way
5. I signed in to add my team again
6. Removed it again, which still had to be done twice before it "stuck"
7. Restarted the machine, multiple times
8. Removed all provisioning profiles from /Library/Developer/XcodeServer/ProvisioningProfiles
9. Signed in with my developer account again after removing the provisioning profiles. They have been recreated, but the error remains

I turned off "Perform analyze action" in the bot, which did get rid of the message, but turning it back on, or turning on "Perform archive action", reintroduce it, so that's not an option (also, I want static analysis performed, which is why it's turned on in the first place).

Assuming for the moment that Xcode Server's list of signing identities is corrupted, how would I go about resetting them, outside of Server.app (since I already removed the Development Team there)? I found a couple of directories that seem like candidates for some troubleshooting and/or backup restoration, but I'm hesitant to mess around without knowing for sure what they're for.

/Library/Developer/XcodeServer/Certificates
/Library/Developer/XcodeServer/Keychains

I noticed that my provisioning profiles all expire on May 20, 2016. This means my old ones expired two days ago (5/20/2015), which was between my last successful (5/19) and first failed (5/21) builds.

So now, knowing the source of these failures, the problem now becomes: how do I clear out the old provisioning profiles (or certificates, etc.) that are causing the duplicates?

The server is up-to-date on Yosemite (10.10.3), Server (4.1), and Xcode (6.3.2).

Update

I'm trying to list codesigning identities, but can't get even a single "Mac Developer: OS X Server" to show up, let alone multiples. These are the commands I've tried, which only list the non-Server identities in the keychain:

security find-identity -p codesigning
sudo -u _xcsbuildd security find-identity -p codesigning
sudo -u _xcsd security find-identity -p codesigning

Answer 1

It looks like a bug in Server did in fact introduce a duplicate signing identity. I reported it as rdar://21080937, if you'd like to dupe it.

In order to fix it, I had to learn about how Xcode Server stores signing identities (thanks entirely to an extremely helpful answer to an unrelated question).

Xcode Server stores Developer Program certificates in

/Library/Developer/XcodeServer/Keychains/Portal.keychain

Following the steps below, I was able to open the keychain, find the duplicate, remove it, and put it back. These steps are adapted from the answer linked above.

1. Sign out of the Developer Portal from Server.app by removing the developer team you're having issues with

2. Copy the Portal keychain to your desktop:

   > sudo cp /Library/Developer/XcodeServer/Keychains/Portal.keychain ~/Desktop/
   

   Password: your-administrator-password

   > sudo chown `whoami`:staff ~/Desktop/Portal.keychain 
   

3. Set the Portal keychain password to “123”

   > security set-keychain-password -o "`sudo cat /Library/Developer/XcodeServer/SharedSecrets/PortalKeychainSharedSecret`" ~/Desktop/Portal.keychain 
   

   New Password: 123

   Retype New Password: 123

4. Open the Keychain in Keychain Access:

   > open -b com.apple.keychainaccess ~/Desktop/Portal.keychain
   

5. Unlock the “Portal” keychain using password “123”

6. Find the duplicate keys with the name listed in the error

7. Remove one (I picked the one modified less recently, but it may not matter)

8. Lock the “Portal” keychain, quit “Keychain Access”

9. Reset the Portal keychain password:

   > security set-keychain-password -p "`sudo cat /Library/Developer/XcodeServer/SharedSecrets/PortalKeychainSharedSecret`" ~/Desktop/Portal.keychain 
   

   Password: your-administrator-password (optional step)

   Old Password: 123

   It may or may not ask you for your administrator password again, pay attention to the prompt.

10. Backup the original keychain

    > sudo cp /Library/Developer/XcodeServer/Keychains/Portal.keychain /Library/Developer/XcodeServer/Keychains/Portal.keychain.old
    

11. Copy the Portal keychain back

    > sudo cp ~/Desktop/Portal.keychain /Library/Developer/XcodeServer/Keychains/
    

12. Fix the new keychain's permissions

    > sudo chown _xcsbuildd:_xcs /Library/Developer/XcodeServer/Keychains/Portal.keychain
    

13. Since the system caches open keychains, restart the server

14. Add your developer team back, in Server.app

15. Run a build, verifying everything looks good. If so, remove the backup copies of the keychain

    > rm ~/Desktop/Portal.keychain
    > sudo rm /Library/Developer/XcodeServer/Keychains/Portal.keychain.old