DH *used* to be that bad. After every trip he would always say, "Do you know how much money we spent...!!!" On our last trip in December, it was just going to be me and Travis (DS). Terry said he didn't want to go because he couldn't afford it. (The man makes enough money!) So, I told him that I was going with Travis in December, I was paying for it myself, it wouldn't cost him a dime and if he wanted to go with us, he could. Needless to say, he ended up going on the trip, paying his own way.
Before one trip, he sounded just like your DH. I told him I would rather go without him and have a good time with my son than for him to come, be miserable and make us all miserable. It wouldn't be a family trip with him not there, but I was making "family" memories and I wanted good memories for my son. (I've learned in life if I don't speak my mind at the right time, I lose out.)
Good Luck on whatever you choose to do!!!
