Yes, you need to have your record latency compensation dialed in before complicating the picture with External Insert. I should have mentioned it but didn't because you were talking about errors of 4000 samples, and Manual Offset is usually on the order of 20-50 samples.
It sounds like you are now compensating for the some EI delay with Manual Offset, which you don't want to do, because Manual Offset will apply to compensation of 'normal' recordings, and they will end up over-compensated.
To set Manual Offset, use a tool like the free CEntrance Latency Tester to measure actual round trip latency, subtract whatever is reported in for Total Round-Trip in Preferences > Audio > Driver Settings (usually smaller), and enter that difference (usually positive) as the Manual Offset.
You can also measure the actual round trip using EI with nothing in the loopback path, and add one buffer to that number. It should come out the same as CEntrance.
Manual Offset should be fixed regardless of buffer size so you shouldn't have to touch it again.