I don't envy your job removing all that.
I remember replacing my speakers I took the cards off.
While the speakers were out I did put some material throught the speaker hole and onto the inside of the door to dampen the sounds to the outside a little. I wouldn't want to take all that off though