I have found that I am happier when I'm not completely out of shape.
And in my case, heart disease runs in the family (dad died at the age of 57), so 20 years off the end of my life could mean I'm dead tomorrow -- yesterday, even -- and that prospect is, indeed, a big turn-off.
There's fit, and there's "not completely out of shape". The CFS I went through several years ago (going on a decade, now) could largely be attributed to being significantly out of shape.