Dithering to 24 bit really doesn't matter:
1. First of all, stuff at the level of 24 bit dither isn't remotely close to being audible under any real world conditions whatsoever.
2. Even if it was, the 24 bit dither would be buried under other noise - the best 24 bit converters can do is equivalent to 20 bits or so, so the dither is always buried under the noise from the converters in addition to other noise and it's all below what your ears can hear anyway.
3. Your audio almost certainly already contains nothing but noise at the same level as the 24 bit dither, which means you have "self dither" and dithering thus provides no benefit because the noise in your signal is already doing the dither's job.
4. You don't need to worry about repeatedly dithering causing noise buildup, because of #1 and #2 and the fact that you'd have to apply 24 bit dither literally
dozens of times before it could possibly matter.
IOW 24 bit dither really just doesn't matter at all. You can dither or not dither or dither repeatedly at that 24 bit level and it will make no difference whatsoever.
You can think of it as changing the font on a street sign 2 miles away to make it "more legible" - it might make a difference from a distance that's just at the limits of your vision (like dither with 16 bit audio), but from miles away it's beyond irrelevant.