New to Java

I've been trying to figure this out for a while, and I read a bunch of threads, all to no avail. So, don't think I haven't done my research, please!

I want to create a JFrame, add a JPanel to the Center spot, and swap it with another JPanel when I choose to. I have a button in the South spot that swaps it. I also put an "annoying" button in the middle of the JPanel's for fun. This is not a school assignment or anything, I'm building a game, and I want to use it for that.

I'd appreciate your comments (on all the code too if you have other comments). If you can help me to fill in the spots in the two panel methods where I need to remove the old panel, I would appreciate it greatly! Or, if there is a different way to do this, that is cool too. Basically, anything you can tell me.

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;

public class Test extends JFrame {

	public static void main(String[] args) {
		Test t = new Test();
	}

	public Test() {
		JButton swapButton = new JButton("Swap panels");
		setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
		swapButton.addActionListener(new buttonListener());
		add(swapButton, "South");
		setSize(300, 300);
		setResizable(false);
		setVisible(true);
	}

	private boolean red = false;
	private JPanel p;

	public void blackPanel() {
		// remove the other panel.
		p = new JPanel();
		JButton annoyingButton = new JButton("AnnoyingButton");
		p.setBackground(Color.BLACK);
		p.add(annoyingButton, "Center");
		add(p, "Center");
		pack();
	}

	public void redPanel() {
		// remove the other panel.
		p = new JPanel();
		JButton annoyingButton = new JButton("AnnoyingButton");
		p.setBackground(Color.RED);
		p.add(annoyingButton, "Center");
		add(p, "Center");
		pack();
	}

	class buttonListener implements ActionListener {
		public void actionPerformed(ActionEvent e) {
			red = !red;
			if (red) {
				redPanel();
			}
			else blackPanel();
		}
	}
}