I get this all the time with my Focusrite 18i20 but only when I set the latency buffer to the lowest possible value of 16. If I use anything higher than that like 32, 64, 128 etc I don't get that problem. It happens on multiple usb ports on my system that are 3.0 not 2.0, but I can't use the 2.0 ports on my machine because they're tied directly into the chipset, no internal hub.