well specific libraries is hard to define, that's a personal choice based on the music you create, however, I've become a big fan of small, specialized libraries, and for that sort of thing I highly recommend browsing the following sites:
ToneHammer MicroHammer Soniccouture Sonokinetic CineSamples You'll likely still need a couple of larger libraries for things like strings, brass, etc. I have not settled on any yet, but I've pretty much narrowed the field, for my purposes anyway, to:
Chris Hein- just amazing brass
Kirk Hunter - some of the coolest strings I've ever heard
Bela D Media - beautiful choirs and some really cool instruments
There are also some uber-large specialized libraries, like LA Scoring Strings, but I think that sort of thing probably needs to wait, at least for me, till I've covered all the other bases - besides which I still have Garritan's original Orchestral String Library for Giga, which is still amazing!
And, don't overlook Native Instrument's own offerings. I only own one so far, their Sax&Brass library, but it is absolutely amazing. You can get a cut down version for the free Kore player (that's what I did) but it will just leave you wanting more!
Have fun!