Fabricjs - how do I deselect an object and select

I have 3 fabric objects on the canvas, a rect and text are grouped as 1 object, then I dynamically create a polygon, and redraw it on object:moving between the group object and another rect. These form what looks like a "speech bubble", with the group and the rect (pointer) both being movable.

What I'd like to achieve is: when the polygon is clicked with the mouse, you can move all 3 objects while holding the mouse down.

So far what I have groups the items together fine, but you can only move all the items if you select the polygon, then release the mouse button, then click the group again, hold down the mouse and move the group.

I've tried the following canvas methods, thinking they might "deselect" the polygon on click, but none of them appear to work.

canvas.deactivateAllWithDispatch();
canvas.deactivateAll();
canvas.discardActiveObject();

Jsfiddle of the prototype I have so far is here:

http://jsfiddle.net/beewayne/3y2x3vty/

I messed around a little and the best I got is to manually move the object and the group together until the user lets up the mouse for the first time.

I modified the mousedown listener to look more like this:

  shape.on('mousedown', function(evt) {

    var objs = canvas.getObjects();

    var group = new fabric.Group(objs, {status: 'moving'});

    // Relevant code
    var originalX = shape.left,
        originalY = shape.top,
        mouseX = evt.e.pageX,
        mouseY = evt.e.pageY;

    canvas.on('object:moving', function(evt) {
      shape.left = originalX;
      shape.top = originalY;
      group.left += evt.e.pageX - mouseX;
      group.top += evt.e.pageY - mouseY;
      originalX = shape.left;
      originalY = shape.top;
      mouseX = evt.e.pageX;
      mouseY = evt.e.pageY;
    });

    canvas.setActiveGroup(group.setCoords()).renderAll();
  });

  // clean up the listener
  shape.on('mouseup', function(evt) {
    canvas.off('object:moving');
  });

I used the mouse offset, since adding the object to the group makes it "jump" a little. Also, I continuously set the object to the same position, so that it doesn't move within the group.

http://jsfiddle.net/kqfswu4b/1/