Up the coast is definitely nicer. I've never seen a nicer place to live that Santa Barbara. It's a backwater, which is a plus to me. But more importantly, it's a really nice backwater with lots of culture, a University of California campus and a nice city college, and the weather and geography is really nice. If you want more culture or excitement, we are only 1.5 - 2 hours away from LA. We are north and west of LA, so we very rarely see smog. It's been over a year and maybe longer. I'm a wine and motorcycle guy. There are great vineyards and great motorcycle roads no more than 30 minutes from my house. There's lots of upside here.
There are two downsides. The first is that it is growing (like every place else) and the quality of life is definitely slowing declining. It's still real nice though. The large downside is the cost of living here. Housing prices are just below San Francisco levels. Gas, food, etc. prices are among the highest in the state.
That's going to happen everywhere. I've lived in and out of California all my life. I'll probably live out of it again someday, but maybe not.
Once you get away from the coast in California, prices start going down until they are down right cheap. Of course, a lot of those places are cheap for a reason. Property taxes are relatively low (per $1000) here because of Prop 13 many years ago. Of course, if your house is assessed many $1000's, you still pay a lot of property tax. The people that have stayed in their houses for years pay very little property tax.