No. Unless you're listening to binaural recordings (and for that matter, binaural recordings of which you know the phase), you're not going to be able to hear the difference. And even in that 1% of the time where a recording is binaural, you may not be able to tell the difference. Absolute phase only matters for imaging, and since headphones screw up imaging in 99% of recordings (which are recorded to image well on speakers, if they are recorded to image at all, which most of them are not). Not only does phase vary from track to track, but it can vary on different tracks of a multi-track master, to which you don't have access.
Here's a test that you might have access to: Cowboy Junkies, Trinity Sessions. Listen to it on headphones, try flipping the phase. If you can tell the difference -- and you have a preference and the preference is not arbitrary (I.E. you can tell which is correct, not just that you have a preference for one or the other) -- then congratulations, you should care for this recording.