Never thought about it until now, & it was actually interesting to think about; here's my .02:

Assume a gust of wind causes a 2 degree drift in your trajectory.
Now, assume a human reaction time to this gust of wind of 0.5 seconds.
At 20 mph (29 1/3 ft/sec), this .5 second reaction time combined with a 2 degree shift off-course results in a lateral displacement of ~1/2 foot; so you have essentially drifted in your lane by 6" before you can correct your vehicle's trajectory. Not too big a deal.
Now, consider the high speed case. The shift off course is linear with respect to speed; therefore, if your vehicle speed is 60 mph (3 times 20 mph), you have drifted in your lane by 3x0.5 = 1.5 feet in that same 1/2 second it took you to react to the wind gust. This is a much bigger drift & is more likely to induce an "Oh $hit!" type of reaction.
These numbers are a little big (1 degree, with 3" & 9" shifts, respectively, are probably more realistic), but they do help to get the point across.
There are certainly other factors involved (i.e. self-aligning torque vs. vehicle velocity), but this to me seems likely to be the dominant effect.