Extra white space when using BoxLayout

Question

I'm using Java Swing to make a simple GUI using a BoxLayout LayoutManager. It is basically composed of a single JPanel, with additional JPanels able to be added by the user.

With two JPanels (the first one, and one additional), the format is proper:

But when even more JPanels are added, extra space is added to the left side of the frame:

Here's what I have:

public void updateListFrameContentPane(Container mainPane) {
    mainPane.setLayout(new GridLayout(1,1+chatPanels.size()));
    JPanel firstPanel = new JPanel();
    firstPanel.setLayout(new BoxLayout(firstPanel, BoxLayout.Y_AXIS));
    firstPanel.add(friendsLabel);        
    firstPanel.add(listScrollPane);
    mainPane.add(firstPanel);
    for(JPanel chatPanel : chatPanels)
       mainPane.add(chatPanel);
}

Where chatPanels is a HashSet<JPanel>. I tried using the setMaximumSize method on each JPanel to allow them to grow larger but that didn't help. How do I get rid of the extra space?