I often discover after the fact that my improvised compositions have interesting mixtures of "straight" and triplet timings, but this is a new one on me:
 
In a piece that quantizes nicely to 16ths for the most part, I was hearing a triplet figure that I thought was going to turn out to be 8th triplets (360 ticks), but when I tried to quantize it to that, it was getting all fouled up. On closer examination I determined that the three notes were actually evenly spaced across the last 5 16ths of the bar (i.e. 400 ticks = 5 32nd triplets).
 
Now that I understand it, I can quantize those notes to 32nd triplets or set up a Groove Quantize reference, but now I'm curious... How would you notate that and make it clear that the triplet is over the last 5 16ths rather than 8th triplets over the last quarter? Seems like it would be hard to see the difference.
 
Any notation wizards out there have an opinion?
 
 

I've only read about such stuff --- http://www2.siba.fi/muste...x.php?id=100&la=en

Thanks, Bob. I Googled several different tuplet notation references, but this one's better than most. Looks like the bottom line is you just put the bracket over the affected notes, and the notation of the preceding notes/rests in the bar should make it clear where the tuplet starts. I'll have to see if I can do it in Notion just for grins.
 
 

Edinburgh University has a Music Theory course on Coursera and one of them touched on music durations very hard core... made my head spin. The real "issue" for me is how many folks would read that score and play it properly? That particular lesson made a lot of the students incredibly irate as it was only a few minutes, but then delved into the math of the notations without great clarification. 

I think I would split the bar and put that trip figure into its own bar. Then the player can at least put it in the right place. It'd probably still take some coaching to get what you want at the performance.

Really, I think I'd notate as regular trips with an indication to play it the slightest bit early. Because math is hard :)

Seriously, I would try and find someone who has experience transcribing Zappa because that sounds like the kind of thing they've probably run up against, and solved, hundreds of times 
 
Notation bends my brain. I'm currently collaborating on a book of guitar transcriptions and we get into these long drawn out discussions on how to best notate certain weird figures or meters. We've found that these ambiguous cases come up quite often and it's a battle between neatness and readability on the one hand, and "technical correctness" on the other. What I've come to realize is that in some cases, you just have to do the best you can and add footnotes to explain yourself 

Thanks for the feedback, everyone. I played around with Notion a little, and below is the best I could come up with. Three dotted 8ths is 9 16ths in the time of 5. Makes some sense, and it sounds right so that makes it right in my book:
 

 
Key is Eb minor in case you're wondering. Maybe I'll throw up a little sample on my neglected Soundclick page so you can hear what all the fuss is about. 
 
 
 
 
 

I would like hear a sample when it's ready Dave.

What you did is essentially correct.  
 
There is no proper (traditional) way to notate an expansion, such as 3 in the space of 5.  Tuplets are only for compounding into a smaller space, so 9:5 works.  The only other option is to split the measure and change the meter, like it was suggested, to an 11/16 and a 9/16 measure.
 
I would prefer the latter approach if I were reading it.  Neither is too difficult to pull off by a good musician.  

Finally got around to throwing this up for your (very brief) listening pleasure. Just a short, improvised chord progression and rhythm that I didn't want to forget so I recorded it without developing it beyond the initial inspiration. I have about a thousand of these. 
 
There are several instances of the triplet in question. The first, and the one notated above, is bar 6.
 
http://www.soundclick.com/bands/page_songInfo.cfm?bandID=757783&songID=13249262