It is highly advisable to take off the door panels because of the way that speaker grill "tabs" are designed. Very difficult to take off the grill from the outside without bending/snapping those tabs. More importantly, virtually impossible to put the grill back in place and bend the tabs back in place without working from behind the panel. It's not impossible, but taking off the panel will save you a lot of hassle/time in the long run. Also FYI you can buy new replacement speakers directly from Bose for ~$40 per speaker. Call their customer service directly at (800)231-2673. You might have to leave a msg w/your contact details and they will call back. I have a bad PS front speaker myself but am getting a free used replacement due to the generosity of ymisoqt!
Speaking of, that is what I am doing today! I bought my new ones directly from bose, which is advisable because it is way cheaper than through the dealer! I learned the hard way.... I will come back and let you know how hard it was to replace. Be back in a bit