You mean apart from the fact I need at least 3 fillings replaced after having them shook out me head?

The harsher ride is the only down side, but you really do have to experience it. As a passenger you feel it more because as a driver you're usually braced against the wheel. With coilovers you can set the ride height and the softness or harshness of the ride too. Oh and you do have to take speed bumps carefully and keep an eye out for really bad pot holes.
I was in Pumabuild's up & running 2.3L Duratec engine swap ST and it was SO nice being in a car with standard suspension but it didnt handle that well.
There will always be a trade off.