Speed limits, in addition to be base tyranny, encourage poor driving skills (along with all the other babying and directions drivers get) and thus decrease safety.
Edit: And life would be terribly boring if you were only allowed to do things you needed to do. Take one of my personal hobbies, riding ATVs (four-wheelers) offroad. It's noisy, dangerous, harmful to the environment, and the vast majority of people don't need them. Is it bad? Should it be banned?