This book may or may not provide some insight (and even if it doesn't, it's a very interesting read):
Beyond that, personally I believe that you're thinking about it too much. You're also asking too much of the human psyche in terms of consistency. Does "practically everyone" feel the same common emotion when hearing a particular combination of notes and rhythms? Of course not. There is likely to be
some common ground, especially among people of the same culture, but apart from that our emotional reactions to music are deeply personal and are tied to memory and experience. I think "The Lark Ascending" by Vaughan Williams is one of the most beautiful pieces of music ever written - it sends shivers down my spine that are almost unbearable and evokes all the magnificence and beauty of nature. Yet I remember years ago playing it to a group of friends and them asking me to put something else on after about 5 minutes. I just felt horrified that people I was friends with didn't "get it." There are many other people who feel the same way about that piece of music as I do, but there are just as many people for whom it does nothing. One of my friends told me he thought it was bland, like the incidental music in an old movie.
I don't think there is any formula written in stone which determines what notes and what rhythms evoke what emotions. There are certain general guidelines - happy major and sad minor, the dreamy and mysterious feel of the Lydian mode, the idea of tension/release, the movement of chords around a cycle back to a "home" chord which feels like the completion of a journey etc, but this is all pretty much broad brush stuff. Music, like any form of art, is the translation of inexplicable thoughts and feelings within your mind into an intermediate physical form which can be perceived by other people and
hopefully "reconstructed" into the same feelings within
their minds. It's the communication of your sense of life and your perceptions into a language that can be interpreted by others. But the likelihood of someone else reconstructing those thoughts in exactly the same way as they existed in the mind of the artist is very slim. Someone else's interpretation is always going to be colored by their own perceptions and experiences. At this point I'm talking pretentious BS at a level that I'm not comfortable with, so I'll leave it at that