I'm trying to squeeze as much out of my Audio interface as possible in terms of minimizing audio  latency.
 
I have been playing with buffer settings and resolutions in the setup and I'm getting mixed results. I tried a buffer of 512 at 24/48 on a 20 plus channel interface. The results were pretty bad at around 21ms round trip latency. My best result so far has been a setting of 24/96. This reduced the round trip latency to 10ms. I can go to 256 buffers, but haven't tried that yet. 5ms one way latency is what I get at 24/96. Not too shabby for simply playing with amp sims. Recording them while live is another story though. 
 
Are there any other things I could do to improve those numbers aside from getting another audio interface? What if I would eliminate all channels but the ones I'm recording with, would that minimize my latency numbers?

Another interface isn't likely to lower latency by much, perhaps by as little as half a millisecond or none at all.
 
While interface drivers do vary somewhat in efficiency, most of the latency is attributable to buffering, which has nothing to do with the interface itself, or its driver. Buffers fill at a constant rate determined by the sample rate, so a given buffer size and rate will always yield the same latency. The only way to lower latency is to reduce the buffer size or increase the sample rate.
 
How small a buffer you can get away with depends on your computer's power, the efficiency of its peripherals, the overhead incurred by background processes and hardware interrupt handlers, and the amount of processing done within the DAW.
 
With so many variables, there is no single recipe for making the computer more efficient and thereby allowing you to reduce buffer sizes. All you can do is determine the smallest buffer size that doesn't result in dropouts, and then gradually figure out what changes help in that regard. If you're lucky, you'll identify one major culprit, such as a wireless network adapter or a greedy video card that's sucking up CPU cycles. 
 
You may also decide to adopt a two-step method, wherein you do all your tracking at a very low buffer size (e.g. 64 bytes) before adding any effects to the project, and then raising the buffer to a higher value during mixing and mastering. Plugins not only increase demands on the CPU, but may also incur extra latency even with the same audio interface buffers.

The only means of reducing round-trip latency:

• Use smaller ASIO buffer size
• Reduce the size of the safety-buffer (most units don't allow this)
• Increasing the sample-rate (doubling the sample-rate will roughly cut RTL in half - at the expense of higher CPU use)

Nothing else will have an effect on round-trip latency (disabling channels on the unit will have no effect).

If you want to get lower than 5ms round-trip latency at a 64-sample ASIO buffer size, you have two options:

• PCIe audio interface (one that allows using a 32-sample or 16-sample ASIO buffer size)
• Thunderbolt audio interface with full "PCIe via Thunderbolt" drivers/support

 
Win10 now offers "PCIe via Thunderbolt" support for Thunderbolt 3.
Have to be running one of the latest generation Z170x or X99p motherboards that supports TB3 via USB-C
Microsoft claims that TB3 support should be backward compatible with TB2 and TB1.
Right now, all TB audio interfaces are TB2.
Thus, you need a USB-C to Thunderbolt adapter.  
USB-C to TB adapters have been announced... but I've yet to see one actually available.
 

You say you haven't tried a buffer of 256 yet.  Why not?  Generally, you want to work with the lowest buffer your system will allow and still perform without problems.
 
(This part edited):  Jim beat me to it (and said it better).

Jim Roseberry
If you want to get lower than 5ms round-trip latency at a 64-sample ASIO buffer size, you have two options:

• PCIe audio interface (one that allows using a 32-sample or 16-sample ASIO buffer size)
• Thunderbolt audio interface with full "PCIe via Thunderbolt" drivers/support

Or settle for around 6 or 7ms round-trips and get an RME USB/firewire interface.

To which has to be added 1ms for every foot between ears and monitors of course.

If someone can't cope with playing with their instrument speaker or stage foldback six feet away from them, reducing latency below 6ms is unlikely to help them much.

Thank you all for the help!! I'm not close to my DAW and won't be until late next week at the soonest. I'm anxious to try 256 sample rate at 24/96 . I believe if I can get down to around 8ms I'll be happy.
 
I guess there's no point in attempting to disable tracks.
 
tlw I would be overjoyed to get a 6ms round trip latency on usb. 
 
 

On my RME UFX @ 64 buffers @44.1K/24 (connected via UB) SONAR reports 5.1ms RTL.
 
Also if you must track with FXs on other tracks, you can always freeze those other tracks.

I have a Focusrite Scarlett 18i20 and get 9.2ms @64 buffers and 44.1khz
The 18i20 has three performance modes (recording, balanced and mixing), each increases the RTL accordingly at the same buffer and sample rate. I'am not sure if that is the safety buffer?
Also it allows me to select 32 and even 16 buffers, I have used it with such a small buffer size and It can work for recording, but I' am very happy working at 64 buffers with 9.2ms and I can mix without pops or dropouts.
Hope it helps! 

tlw

Jim Roseberry
If you want to get lower than 5ms round-trip latency at a 64-sample ASIO buffer size, you have two options:

• PCIe audio interface (one that allows using a 32-sample or 16-sample ASIO buffer size)
• Thunderbolt audio interface with full "PCIe via Thunderbolt" drivers/support

Or settle for around 6 or 7ms round-trips and get an RME USB/firewire interface.

To which has to be added 1ms for every foot between ears and monitors of course.

If someone can't cope with playing with their instrument speaker or stage foldback six feet away from them, reducing latency below 6ms is unlikely to help them much.

 
FWIW, I play out quite a bit.
Once you start to venture too far away (without IEMs), you can *really* start to feel the lag (latency).
Can you compensate? Yes.  Does it affect feel?  Absolutely.
Lower latency feels tighter.
I'm pleased with the RTL of my Fireface UFX.  
That said, if I can get the RTL lower (which feels even better), I'm all for it.
 
Don't agree that lower latency feels "better"?
Load up your favorite piano sample library.  Play it at a 256-sample ASIO buffer size.
Now drop the ASIO buffer size to 64-samples.
Which feels more immediate/responsive?
To my ears/sensibilities... it's easy to hear/feel the difference.