On the face of it, that’s a good candidate – but if it does this with the motor stopped, and you aren’t touching the control, it seems less likely. If the pot is faulty, there’s a good chance that if you turn it up slowly, the motor will be a bit ‘squirrely’ when it starts up. Another check you could try is to turn the pot up very slightly from the end stop and seeing if it still does this. By doing this, hopefully you get onto a less worn bit of track without the motor actually running, but if the same thing happens, then likely you’d be looking for a different cause, unless the pot is really bad.
Alternatively, just try changing the pot and see what happens.