I can't find any patterns at all:
There were some trojans lurking, but I've since wiped them and done several virus scans since, and they haven't returned.
I haven't put in any new hardware, except my Logitech G15 keyboard.
It happens when the computer is idle and also when the CPU is under heavy use.
It's not when my dodgy iPod or hard drive is connected.
It's not the temperature - CPU temp is pretty consistent between 35 and 45 degrees.
And it's not an old computer - about 6 or so months old.
It's driving me crazy!
Edit: It had something to do with my webcam being plugged into my keyboard. My Logitech webcam being plugged into my Logitech keyboard, no less. Thanks for all the help to everyone who commented!