If problems go away at 64 samples I would just stick with that, should be well beneath noticeable latency. Laptops can be finicky with extremely low latency. I have similar specs to yours, just a faster CPU and an RME card and my laptop's Touchpad creates relatively high DPC latency. It could be the midi/USB driver in your case. It may even be USB 1, forcing the USB 3 port into legacy mode or whatever.
I'd be happy you run stable at 64 samples, that's better than many people do and doesn't give you any problems for tracking or live performance.