NSButton subclass as colorwell & preventing NSColo

Question

I followed some examples for making an NSButton subclass work as an NSColorWell (since our NSButton subclass already gives us the appearance behavior we need), however I've noticed that after using the button to invoke the panel and changing the color, it's also changing the color of selected text in our document. If I instead subclassed NSColorWell with our appearance customizations, would it not have this problem?

However, I'm still hoping for a work-around that avoids that & still lets us use our button subclass. I've seen discussion threads suggest letting the button itself become the first responder, however with the button being in a separate palette I'm having trouble getting this to work. Also I'd prefer not to alter the responder chain or have the palette become the key window. How evil would a category on NSColorPanel be that overrode setColor:, making it post the expected notification but not touch the first responder?

(Note, rather than simply opening the color panel, I'm currently making use of BFColorPickerPopover by DrummerB https://github.com/DrummerB/BFColorPickerPopover. However I don't think that's much of a complication. I had the same NSColorPanel / first responder issue before integrating it).

Was asked to post source code, so here's the relevant bits from my NSButton subclass (note, uses the picker popover mentioned above rather than NSColorPanel directly):

.h:

@interface ...
@property (nonatomic, strong) NSColor *color;
@property (nonatomic, assign) BOOL active;
@property (nonatomic, strong) NSColor *buttonColor;
@property (nonatomic, weak) BFColorPickerPopover *popover;
- (void)activate:(BOOL)exclusive; // param ignored, always exclusive
- (void)activate;
- (void)deactivate;
- (void)takeColorFrom:(id)sender;
@end

.m:

@implementation ...
@dynamic color;
- (NSColor *)color
{
    return self.buttonColor;
}
- (void)setColor:(NSColor *)newColor
{
    self.buttonColor = newColor;
    [self generateSwatch];
    self.needsDisplay = YES;
    self.popover.color = newColor;
}
- (void)activate:(BOOL)exclusive
{
    [self activate]; // always exclusive
}
- (void)activate
{
    self.popover = [BFColorPickerPopover sharedPopover];
    self.popover.color = self.buttonColor;
    [self.popover showRelativeToRect:self.frame ofView:self.superview
                             preferredEdge:self.preferredEdgeForPopover];
    [[NSNotificationCenter defaultCenter] addObserver:self
                        selector:@selector(popoverDidClose:)
                            name:NSPopoverDidCloseNotification
                          object:self.popover];
    [[NSNotificationCenter defaultCenter] addObserver:self
                        selector:@selector(colorDidChange:)
                            name:NSColorPanelColorDidChangeNotification
                          object:self.popover.colorPanel];
    activeButton = self;
    self.active = YES;
}
- (void)deactivate
{
    if (self.popover)
    {
        [self.popover close];
        self.popover = nil;
    }
    [[NSNotificationCenter defaultCenter] removeObserver:self
           name:NSPopoverDidCloseNotification object:self.popover];
    [[NSNotificationCenter defaultCenter] removeObserver:self
                      name:NSColorPanelColorDidChangeNotification
                   object:self.popover.colorPanel];
    if (activeButton == self) activeButton = nil;
    self.active = NO;
}
- (void)popoverDidClose:(NSNotification *)notification
{
    self.popover = nil; // don't let deactivate ask it to close again
    [self deactivate];
}
- (void)colorDidChange:(NSNotification *)notification
{
    self.buttonColor = self.popover.colorPanel.color;
    [self generateSwatch];
    self.needsDisplay = YES;
    [self sendAction:self.action to:self.target];
}
- (void)mouseDown:(NSEvent *)theEvent
{
    if (self.isEnabled && !self.active)
        [self activate];
    else if (self.active)
        [self deactivate];
}
- (void)takeColorFrom:(id)sender
{
    if ([sender respondsToSelector:@selector(color)])
        self.color = [sender color];
}
@end

Addendum:

I tried using a normal NSColorWell in place of my NSButton subclass, and same issue. Colors chosen in the panel calls the first responder's changeColor: in addition to invoking the action method. So forgetting everything about NSButton in my question, how, in general, does one use a NSColorWell whose color mustn't also be pushed onto the first responder? Must one resort to customizing the expected first responder to selectively ignore changeColor:, or is making the NSColorWell the first responder really the thing to do, or something else?

Answer 1

Yes, a better web search term (NSColorWell "first responder") and I see others have struggled with NSColorWell and this same problem for a long time. Many old mailing list threads have covered this and saw 3 solutions:

1. http://www.cocoabuilder.com/archive/cocoa/82832-nscolorwell-changecolor-and-first-responder.html Douglas Davidson suggests subclassing the potential first responder(s) so they ignore changeColor: (probably what I'll do)

2. http://www.cocoabuilder.com/archive/cocoa/3263-your-nscolorwell-got-in-my-nstext.html Guy English suggests making the color well the first responder temporarily (what I tried but had problems due to color well being in a panel which I don't want becoming key)

3. http://www.cocoabuilder.com/archive/cocoa/180323-detecting-currently-active-nscolorwell.html Martin suggests cutting preventing the changeColor: call in the first place by posing as NSColorPanel and overriding a private method (closest to what I wanted, but more risk of app store rejection than I'm comfortable with)

UPDATE: I tried #1 but it turns out I can't override the first responder (WebView / WebHTMLView grrr). Going with #3, I put the following in a NSColorPanel category, made my color button set panel.avoidsChangingFirstResponder=YES, and it seems to work:

static char changeColorPatchAssociatedObjectKey; // address of this is used as a unique runtime value

- (BOOL)avoidsChangingFirstResponder
{
    NSNumber *changeColorPatchFlag = (NSNumber *)objc_getAssociatedObject(self, &changeColorPatchAssociatedObjectKey);
    return changeColorPatchFlag && changeColorPatchFlag.boolValue;
}

- (void)setAvoidsChangingFirstResponder:(BOOL)enablePatch
{
    NSNumber *changeColorPatchFlag = (NSNumber *)objc_getAssociatedObject(self, &changeColorPatchAssociatedObjectKey);
    if ((!changeColorPatchFlag && enablePatch) || (changeColorPatchFlag && changeColorPatchFlag.boolValue != enablePatch))
        objc_setAssociatedObject(self, &changeColorPatchAssociatedObjectKey, @( enablePatch ), OBJC_ASSOCIATION_RETAIN_NONATOMIC);
}

+ (void)load
{
    if (self == [NSColorPanel class])
    {
        // patch implementation of _forceSendAction:notification:firstResponder: (use swizzle technique from MAKVONotificationCenter.m)
        // for one that calls original but with the last BOOL parameter conditionally changed to NO
        SEL methodSel = NSSelectorFromString(@"_forceSendAction:notification:firstResponder:");
        Method method = class_getInstanceMethod(self, methodSel);
        IMP origImpl = method_getImplementation(method);
        IMP newImpl = imp_implementationWithBlock(^(void *obj, SEL s, BOOL isAct, BOOL isNotif, BOOL isFirstResp) {
            NSNumber *changeColorPatchFlag = (NSNumber *)objc_getAssociatedObject((__bridge id)(obj), &changeColorPatchAssociatedObjectKey);
            if (changeColorPatchFlag && changeColorPatchFlag.boolValue)
                isFirstResp = NO;
            ((void (*)(void *, SEL, BOOL, BOOL, BOOL))origImpl)(obj, s, isAct, isNotif, isFirstResp);
        });
        class_replaceMethod(self, methodSel, newImpl, method_getTypeEncoding(method));
    }
}

I hope someone else finds this useful.