Generally speaking, the main reason for keeping a track smooth is to keep the g's down. If there's the odd bump that gives 5/6 postive verical g's, then what the hell - it'd just make the ride more interesting.
It becomes a problem when the track is that rough that the rider is constantly under 6-8 g's. That'd just cause blackouts
In real life, you do find the odd coaster that has some rough bits, and in general, people will moan about this when they get off
The Shockwave in Drayton Manor is a perfect example. Built by Intamin, it gets a bit rough going into the loop and most of the riders bump their head in the process!