The speaker has a round rubber thing around the outer edge.
remove the rubber thing and the speaker grille comes off. After that you have access to the 4 screws that you can use to remove the speaker.
It's pretty easy to do, and you don't need to remove the entire console.
Be careful with poking around the bottom part of the hole where the speaker is mounted, that's where the alarm reset switch comes up. If you break it you get all kinds of trouble with the alarm system. Trust a man with experience on this one...