I don't know what year you've got, but on my 89 Coupe taking out the door panel was kind of a PITA. You have to remove all the screws (I forget how many). Once you're comfortable that you've found them all, you need to give the bottom of the panel a few pulls to free it from the snap fittings which hold it in.
Once you freed it from those snap fittings, then you'll find that the top of the panel is still stuck. There are more snap fittings at the top of the door (attached to the window slot). These can be tough to pop off. You have to pull up to get them out. Be careful not to rip the door panel material when pulling.
Once you've got the panel completely off you'll see the screws that hold on the speaker grille.
Good luck!!
-Matt