I use the Samsung 42". That is a SWEET setup!
I did try with a 55" but ended up looking all over the place... I consider that a good problem to have (if you're going to have a problem) but I will take the advice here and go with the 40" in our living room.
I can't see the placement of your ref monitors, how far away to you have to sit from your display.
One thing I'm going to dig is being able to lower my ref monitors a few inches to comfortably be in the sweet spot.
As it is now I have to raise my office chair all the way up and sit up as straight as I can to get up to their level.