Honestly? No, there's nothing wrong with wanting time to yourself. Too much work can be a really, really bad thing. Not only does it take away from family time, but it can just really wear on you. So yeah, I'm with the above. Just clarify your stance, and stick to your guns. Me? Personally any overtime I could get my gruby paws on I would have taken. Not because I didn't enjoy my family time, but because we needed the money. Since the split with my son's mother, well... Yeah I still need the money, but who am I coming home to now? Exactly. My roommates... Nothing special.
Just make sure your boss has an understanding of your needs.