I actually just finished reading pretty much the whole thing (the online book, not the one Dave just recommended). The author does a remarkable job of staying away from complex arithmetic (complex as in real and imaginary components to a number).
The final six or so chapters are supposed to get into the really deep stuff, but it turns out that the online version of the book has the content of those chapters mostly missing (mercifully missing, I might speculate).
This really is a pretty remarkable text for those with sufficient interest. I came away with a pretty solid understanding of how convolution works in the time domain. Of course, any software doing convolution on all but really short segments is going to use FFT. That is pretty thoroughly explained as well, but I can't claim more than a vague theoretical understanding of how convolution works in the frequency domain - coding this stuff must be a real <rhymes with ditch>.
By the way, thumbs up for Dave's recommendation as well.