I wouldn't take off any more than that if I were you. I suppose it could work if you did, but you'd need to epoxy it to something in order to hold it in place then. I would make sure you keep those 3 holes at the bottom.
I think maybe after several days of touching it I finally understand the concept...
I don't think you use the mini jack. Is there a blue/yellow wire coming out the same end as the mini jack?