For wood:
I've been on lots of smooth and lots of rough wooden coasters. For the most part, any NEW wooden coaster is smooth. It's hard to determine which is the smoothest - Boulderdash and Knoebels Twister are smooth and well-maintained. But neither is that old yet. Looking at older woodies that have run long enough to require maintenance, I'll give a nod for smoothest to any woodie at Kennywood. They've all been taken care of.
For roughest, same story - but if I look at it in terms of which coaster I think is not even enjoyable because of roughness, I'll say Son of Beast.
For steel:
the smoothest - SROS, SFNE
roughest - Viper, Great Adventure