automated A/B testing?

Re:automated A/B testing? (ba_midi)

Tue, 09 Feb 2010 16:27:45 GMT

mike_mccue

I like the playlist idea, but I'd like it to be randomized and I'd like to synchronize my notes after the fact.

The idea is that I often convince myself that I am crazy. e.g. I ca spend hours tweaking the bass... and then a in a moment decide the bass track is two loud.

A day later I'm not sure what I like. :-)

Oh trust me, I know the groove too well. And I get ear fatigued quickly, which doesn't help !

Re:automated A/B testing? (The Maillard Reaction)

Tue, 09 Feb 2010 16:09:49 GMT

I like the playlist idea, but I'd like it to be randomized and I'd like to synchronize my notes after the fact.

The idea is that I often convince myself that I am crazy. e.g. I ca spend hours tweaking the bass... and then a in a moment decide the bass track is two loud.

A day later I'm not sure what I like. :-)

Re:automated A/B testing? (ba_midi)

Tue, 09 Feb 2010 16:05:36 GMT

mike_mccue

Thanks for the very thoughtful replies.

I feel I should clarify and describe what I am hoping to do.

I want to get a chance for myself and any visitors to listen to several cuts of a mix and have some way to take note of favorites without any regard for what version I am listening too.

To date I use the method Don described... but I feel that knowing which version I am listening too effects my judgment.

The ABX feature in Foobar2000 seems excellent, but it is more for comparing differences rather than preference.

I am looking for some automated way to record preferences in a A/B type blind listening circumstance.

thanks again,
mike

Mike,

If you'll be listening to entire mixes, why not just create like a playlist - or just copy the mixes back to back on one track in Sonar (use Markers to note which is which). They'd play one after the other in that case.

I don't know if the Matrix can hold long wav files, but you could also use it to just trigger mixes.

Re:automated A/B testing? (The Maillard Reaction)

Tue, 09 Feb 2010 15:46:06 GMT

Thanks for the very thoughtful replies.

I feel I should clarify and describe what I am hoping to do.

I want to get a chance for myself and any visitors to listen to several cuts of a mix and have some way to take note of favorites without any regard for what version I am listening too.

To date I use the method Don described... but I feel that knowing which version I am listening too effects my judgment.

The ABX feature in Foobar2000 seems excellent, but it is more for comparing differences rather than preference.

I am looking for some automated way to record preferences in a A/B type blind listening circumstance.

thanks again,
mike

Re:automated A/B testing? (tarsier)

Tue, 09 Feb 2010 15:01:55 GMT

I am not a fan of the old A/B method of comparison - shocking eh? I have been teaching against it now for a few years. Here is what I tell my students: #1 Listen very very carefully to version A at least four times and take notes with time code and make very very specific comments. #2 Do the same for every version. The important thing is to learn what is different over time not in an instant (A/B)

But with the A/B/X software, you can take as long as you want listening to them. Listen to A for a week, then B for a month if you want. Get comfortable and listen. Take a break. Listen some more, or don't. There doesn't have to be anything quick about it at all. The only pressure is whatever you put on yourself.

Besides which, what you described is an A/B test. " />

Re:automated A/B testing? (ba_midi)

Tue, 09 Feb 2010 14:58:30 GMT

For the fun of it, I'm going to opine a bit here..... I am not a fan of the old A/B method of comparison - shocking eh? I have been teaching against it now for a few years. Here is what I tell my students: #1 Listen very very carefully to version A at least four times and take notes with time code and make very very specific comments. #2 Do the same for every version. The important thing is to learn what is different over time not in an instant (A/B) and to know what you are listening to and what you are listening for. #3 Once you have educated yourself and can identify which is A, which is B, etc - then you can "A/B" in shorter time slices. But I prefer my A/B audition times to be no less than 45 seconds. Since if you move your head one inch inbetween the comparisons it can create a new comb even if your room is near perfect. #4 I know this seems too time consuming for some folks - but that is why we call it engineering - it takes time to do it well (notice I didn't say ... to do it 'right'). I want repeatable results with clear reasons, educating my listening improves my ability to hit the results I want faster over time. #5 I have been around long enough to remember when people would ask how many days it would take to get a mix done - now they ask how many mixes can I get done in a day. Ugh! -D

I more or less agree with what you said, but the main reason I do A/B'ing is just to make sure I'm not completely losing my mind lol.

I usually keep one mix to A/B with a new mix. I rarely keep a lot of mixes. If I do, I save them as a different name (project v1, project v2, etc).

But there are times I'm listenting to discover the 'overall'-ness rather than specifics. IOW, is it wide enough, is it too bright, etc... rather than "did the bass's A string come through loud enough at bar 22".

Re:automated A/B testing? (DonM)

Tue, 09 Feb 2010 13:52:31 GMT

mike_mccue

I just learned of some good ABX testing capabilities using with Foobar2000 and the ABX comparator plug in... but that seems to be suited for testing recognition of differences in sound files.

What sort of applications are available for testing for preference? For example lets say you have 5 versions of a mix and want to listen and select a favorite from them without having to press stop and play repeatedly.

Is there some sort of automated playback system that lets you listen and pick favorites while unaware of the results you are gathering?

thanks,
mike

For the fun of it, I'm going to opine a bit here.....

I am not a fan of the old A/B method of comparison - shocking eh? I have been teaching against it now for a few years. Here is what I tell my students:

#1 Listen very very carefully to version A at least four times and take notes with time code and make very very specific comments.
#2 Do the same for every version. The important thing is to learn what is different over time not in an instant (A/B) and to know what you are listening to and what you are listening for.
#3 Once you have educated yourself and can identify which is A, which is B, etc - then you can "A/B" in shorter time slices. But I prefer my A/B audition times to be no less than 45 seconds. Since if you move your head one inch inbetween the comparisons it can create a new comb even if your room is near perfect.
#4 I know this seems too time consuming for some folks - but that is why we call it engineering - it takes time to do it well (notice I didn't say ... to do it 'right'). I want repeatable results with clear reasons, educating my listening improves my ability to hit the results I want faster over time.
#5 I have been around long enough to remember when people would ask how many days it would take to get a mix done - now they ask how many mixes can I get done in a day. Ugh!

-D

Re:automated A/B testing? (bitflipper)

Tue, 09 Feb 2010 12:47:01 GMT

I use a tool called WinABX. It does have some limitations, though: it cannot play 24-bit waves. There is also an ABX plugin available for Foobar2000.

Re:automated A/B testing? (papa2005)

Tue, 09 Feb 2010 12:34:36 GMT

mike_mccue

Thanks Bristol Jonesy,

That's sort of what I've been doing... I was sort of hoping there was something where you can log "votes" and then review your choices after a half hour of listening to see if any particular mix suggested itself as the one you picked most often.

I haven't really thought it thru... ad was hoping someone already had. :-)

best,
mike

Mike,

I don't quite understand the end result you're looking for. Who is doing the voting?

I do multiple mix "passes" frequently but I'm doing them with the objectivity of getting it "right"...When I think I've "nailed" it I listen on different monitors in my studio and then on a different system (living room, boom box, car, etc.,)...

Re:automated A/B testing? (The Maillard Reaction)

Tue, 09 Feb 2010 12:21:03 GMT

Thanks Billy.

Re:automated A/B testing? (ba_midi)

Tue, 09 Feb 2010 11:16:07 GMT

mike_mccue

I just learned of some good ABX testing capabilities using with Foobar2000 and the ABX comparator plug in... but that seems to be suited for testing recognition of differences in sound files.

What sort of applications are available for testing for preference? For example lets say you have 5 versions of a mix and want to listen and select a favorite from them without having to press stop and play repeatedly.

Is there some sort of automated playback system that lets you listen and pick favorites while unaware of the results you are gathering?

thanks,
mike

Mike, you can easily A/B in Sonar using the Quick Group approach (explained below).
The problem with this one is that you are limited to A/B'ing only 2 mixes at a time. I do it often.

Make each mix track a quick group (click on the tiny QG selector near the track number) by CTRL Clicking so they both get grouped.
Then click the SOLO widget on both tracks.
Now click the MUTE on ONE of the tracks.
Start playback - you will hear the UNmuted track play first.
Click on the MUTE widget of either track to "toggle" which track is playing.
Thus, you can do quick A/B'ing easily.

I can't figure out a way to do more than two tracks as this technique depends on the toggling method.

Re:automated A/B testing? (The Maillard Reaction)

Tue, 09 Feb 2010 09:55:20 GMT

Thanks Bristol Jonesy,

That's sort of what I've been doing... I was sort of hoping there was something where you can log "votes" and then review your choices after a half hour of listening to see if any particular mix suggested itself as the one you picked most often.

I haven't really thought it thru... ad was hoping someone already had. :-)

best,
mike

Re:automated A/B testing? (Bristol_Jonesey)

Tue, 09 Feb 2010 09:26:17 GMT

If you're not bothered about what you play them back on Mike, make up a playlist with your 5 tracks in Media Player, set it to random, shut your eyes & listen.

Press stop when you hear the one that's 'best'

automated A/B testing? (The Maillard Reaction)

Tue, 09 Feb 2010 08:45:56 GMT

I just learned of some good ABX testing capabilities using with Foobar2000 and the ABX comparator plug in... but that seems to be suited for testing recognition of differences in sound files.

What sort of applications are available for testing for preference? For example lets say you have 5 versions of a mix and want to listen and select a favorite from them without having to press stop and play repeatedly.

Is there some sort of automated playback system that lets you listen and pick favorites while unaware of the results you are gathering?

thanks,
mike