fragment remove issue

I am just learning fragments today. I press a button and it adds/removes a fragment. However if I try remove the fragment every fragment apart from the one I want removed is removed, why? The first press correctly adds a fragment.

Button2 fragment:

 Button button = (Button) view.findViewById(R.id.button2);
        button.setOnClickListener(new View.OnClickListener() {
          @Override
          public void onClick(View v) {

              ButtonFragment fragment = new ButtonFragment();
              if (fragment != null && fragment.isVisible()) {



                  FragmentManager fragmentManager = getFragmentManager();

                  FragmentTransaction transaction =  fragmentManager.beginTransaction(); 
                  transaction.remove(fragment).commit();

              }
              else if(!fragment.isVisible())
              {
                  FragmentManager fragmentManager = getFragmentManager();

                  FragmentTransaction transaction =  fragmentManager.beginTransaction(); 
                  transaction.add(R.id.fragment_container, fragment ).commit();
  }       

          }
        });
        return view;
      }
}

I have two fragments like this in xml: When I click the button I want the fragment not defined in xml to be added, and it is. However the next time i press the button, which should remove that fragment. Everything is removed apart from that fragment.

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:orientation="horizontal"
    android:background="#123456"
    android:id="@+id/fragment_container"  >


    <fragment
        android:id="@+id/TimeFragment"
        android:layout_width="0dp"
        android:layout_weight="2"
        android:layout_height="match_parent"
        class="com.example.myfragment.TimeFragment" >
        <!-- Preview: layout=@layout/details -->
    </fragment>



    <fragment
        android:id="@+id/Button2Fragment"
        android:layout_width="0dp"
        android:layout_weight="3"
        android:layout_height="match_parent"
        class="com.example.myfragment.Button2Fragment" >
        <!-- Preview: layout=@layout/details -->
    </fragment>

</LinearLayout>

You cant remove a Framgnet that you have added using the XML. If you want to remove the fragment via the .remove method you should first add it to your layout via the .add method, and not embed it into the XML file. in this case you can only .show or .hide the Fragments.

UPDATE:

To add the ButtonFragment dynamically do this:

ButtonFragment buttonsFragment = new ButtonFragment();
newfragmentTransaction = fragmentManager.beginTransaction();
newfragmentTransaction.add(R.id.containerForFragments, buttonsFragment ).commit();

UPDATE 2: This code:

  Button button = (Button) view.findViewById(R.id.button2);
    button.setOnClickListener(new View.OnClickListener() {
      @Override
      public void onClick(View v) {

          ButtonFragment fragment = new ButtonFragment();
          if (fragment != null && fragment.isVisible()) {



              FragmentManager fragmentManager = getFragmentManager();

              FragmentTransaction transaction =  fragmentManager.beginTransaction(); 
              transaction.remove(fragmentManager.findFragmentById(R.layout.activity_main)).commit();

          }
          else if(!fragment.isVisible())
          {
              FragmentManager fragmentManager = getFragmentManager();

              FragmentTransaction transaction =  fragmentManager.beginTransaction(); 
              transaction.add(R.layout.activity_main, fragment ).commit();
          }       

      }
    });

should be run from the Activity and not from the Fragment.