Actually, if you're talking completely passive, it depends on the two components being switched, and the adapters you use. If they both result in using the "hot" and "inverted" signals (I.E. not ground/shield), then it's just like having two switches, and floating the ground. (And if they ground-ground [i.E. if they connect "ground" to ground], then you're not even floating the ground [well, you are, but who cares?]). Similarly, if they both use "hot" and ground (I.E. only one side of the balanced), then again, it's just like having two switches in the signal path, and you lose the balanced-ness of it.
I wouldn't mix the two, is all.