Vlad,
I sympathize - I too love layered sounds and lots of them. My usual method of "writing" a song is to work off a main loop, lately it has been bass. Get 12 or 16 bars, put some drums underneath it, and then work out the structure of the song - ie., how many loops I need to string together for 4-6 minutes of a song. And change up those basic rhythms, add a second part (much like I did for the first section) as a middle section, just add some variety. I write a one page lyric sheet, since that seems to fit into right for the vocalist.
Then I get to my point - layering inharmonic synths into it. that is what I actually like doing - "playing" synth sounds. Again, I will typically use 3 or 4 different synth sounds I play throughout the whole piece. Then I bring in my guitarist, and he will lay 3-4 guitar tracks, more or less randomly playing rhythm and lead lines - whatever he feels like.
At that point I start culling the herd of sound, using all the methods described in earlier posts - EQ, panning, and yes, muting. Even trained musicians can only listen to 3-4 lines or instruments at once. Usually divided into a rhythmic blur and the lead element that stands out. And that means, for me, killing off a lot of nice bits and pieces that don't serve the song, as cool as that guitar lick is, or that blob of sound I've found. Because in the next 16 bars that blob of sound finds its own space where it sticks out. Or that phrase. And once it is established, the listener can find it again when it is more submerged, if they are listening for it. When I'm happy for the music, then I get out the knives again when the singer turns the piece into a song w/ the main melody.
So even if you don't want to go through the culling part throughout your piece, you have to introduce elements, and provide a space when they first appear or come to prominence. And get brutal about providing such space, even if you don't mute. Volume envelopes, subtractive EQ, panning all can carve out space. Even those pads, which I usually think of as more full range, can have the bottom cut off, esp. if you are mainly interested in spotlighting it for the filter cut off. And cutting out the bottom, even up to 400-500 Hz, leaves that less clogged for the drums and bass, which have more room to punch (tho I think punch needs dynamics, something relatively empty to punch into, but that doesn't seem to be the modern sensibility).
The other thing is it takes a lot of time, much less talent to get a handle on all the crafts involved. Learning an instrument, learning how to record it (even running soft synths through an amp and mic'ing it to add some "air" around the virtual thing can differentiate it in a busy song), writing the music, arranging the instrumentation and then mixing it is full-time work for many years. One of the funniest things I heard about that phenomenon was John Cale, classically-trained musician, co-founder of the Velvet Underground, record exec and solo artist, who was over twenty years into the business before he felt like he knew what he was doing "producing" an album (or CD at that point). And his work at that time wasn't as good as the earlier stuff (tho that was more a matter of the strength of his songwriting, not the technical stuff).
It sounds like you know what you want to do, it is merely a matter of buckling down and refining your techniques. That was one reason I suggested muting (or perhaps more a matter of not adding in elements until you get something simpler to punch etc. like you want). Experimenting on fewer tracks at once might make it quicker to learn what works for you, which you could then expand to more tracks. Or take a "full" song and spend a lot of time on it fooling w/ all the techniques others have written to get it closer to how you think it should sound.
@