It is a bitch!!! It comes off a certain way or you'll break it. I wish I could remember for you! I think it's just clips if I recall, but I swear...it was tight! I would recommend getting your hands on a service manual or DVD before you start so you don't break anything!
By the way, you can change mirrors without having to remove that console! If you need to run wires, run them from the driver's kick panel up the pillar post and tuck them under the lip of the headliner. I've done that on several cars.