support.v7.app.AlertDialog throws NullPointerExcep

I updated AppCompat to the newly released revision 22.1.0 and changed my AlertDialog to support.v7.app.AlertDialog. But on a Lollipop device, it throws the following exception on dismissDialog().

java.lang.NullPointerException:   attempt to invoke virtual method 'java.lang.Class java.lang.Object.getClass()' on a null object reference
  at android.support.v7.internal.app.WindowDecorActionBar.getDecorToolbar(WindowDecorActionBar.java:248)
  at android.support.v7.internal.app.WindowDecorActionBar.init(WindowDecorActionBar.java:201)
  at android.support.v7.internal.app.WindowDecorActionBar.<init>(WindowDecorActionBar.java:184)
  at android.support.v7.app.AppCompatDeleg  ateImplV7.cre  ateSupportActionBar(AppCompatDelegateImplV7.java:176)
  at android.support.v7.app.AppCompatDelegateImplBase.getSupportActionBar(AppCompatDelegateImplBase.java:85)
  at android.support.v7.app.AppCompatDelegateImplV7.onStop(AppCompatDeleg  ateImplV7.java:221)
  at android.support.v7.app.AppCompatDialog.onStop(AppCompatDialog.java:108)
  at android.app.Dialog.dismissDialog(Dialog.java:438)
  at android.app.Dialog.dismiss(Dialog.java:414)
  at android.support.v7.app.AlertController$ButtonHandler.handleMessage(AlertController.java:157)
  at android.os.Handler.dispatchMessage(Handler.java:102)
  at android.os.Looper.loop(Looper.java:145)
  at android.app.ActivityThread.main(ActivityThread.java:5834)
  at java.lang.reflect.Method.invoke(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:372)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1388)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1183)

How do I fix it?

(Lower-version devices seems to work well. This just happens in Lollipop)

+ I don't call dismiss() explicitly in my code. The dialog throws the exception when it is dismissed by back-button or positive/negative button.

++ Here is my code that uses v7.app.AlertDialog. Thank you.

import android.app.Dialog;
import android.content.DialogInterface;
import android.content.DialogInterface.OnClickListener;
import android.os.Bundle;
import android.support.v4.app.DialogFragment;
import android.support.v7.app.AlertDialog;
import android.text.TextUtils;

public class SimpleYesNoFragment extends DialogFragment {

    public interface OnConfirmListner {
        public void onConfirm();
    }

    public static SimpleYesNoFragment newInstance(String title, String message) {
        SimpleYesNoFragment fragment = new SimpleYesNoFragment();
        Bundle args = new Bundle();
        args.putString("title", title);
        args.putString("message", message);
        fragment.setArguments(args);

        return fragment;
    }

    private OnConfirmListner mListener;

    public void setOnConfirmListener(OnConfirmListner l) {
        mListener = l;
    }

    @Override
    public Dialog onCreateDialog(Bundle savedInstanceState) {
        AlertDialog.Builder b=  new  AlertDialog.Builder(getActivity())
        .setPositiveButton(R.string.yes, new OnClickListener() {

            @Override
            public void onClick(DialogInterface dialog, int which) {
                if(mListener != null) {
                    mListener.onConfirm();
                }
            }
        })
        .setNegativeButton(R.string.no, new OnClickListener() {
            @Override
            public void onClick(DialogInterface dialog, int which) {
            }
        });

        Bundle args = getArguments();
        String title = args.getString("title",null);
        if(!TextUtils.isEmpty(title)) {
            b.setTitle(title);
        }
        String message = args.getString("message",null);
        if(!TextUtils.isEmpty(message)) {
            b.setMessage(message);
        }

        return b.create();
    }
}

By chance I've noticed that my project has values-v21/themes.xml and it applies android:Theme.Material.Light.Dialog.Alert to the support.v7.app.AlertDialog. This caused the weird bug.

Using a proper AppCompat Theme, such as Theme.AppCompat.Light.Dialog.Alert, to the support.v7.app.AlertDialog solved the problem. Thanks to everyone for helping me out.