Almost all commercial audio equipment has left and right signal ground shorted together internally, so you could get away with a 3PDT switch. The Electroswitch C4 series is supposed to be quite good:
http://www.mouser.com/catalog/catalogUSD/639/1674.pdf
Take a look at Nate's post #8 inthis thread.
Decent rotary switches? - Head-Fi: Covering Headphones, Earphones and Portable Audio
For switching between sources, you could certainly use a DPDT or 3- 4PDT toggle switch if you are certain that 2 inputs is the most you would ever need.
Rotary Switches - Head-Fi: Covering Headphones, Earphones and Portable Audio
Another good thread on rotary switches. Note that many of the quality switches will have a high rotational effort, so a large knob will be needed to make turning these easy.