How can I deal with just a casual relationship?

I think you should just move on. You don't know when, if ever, he will commit. You can't put your life on hold for him. You already unhappy with the situation, get out now before you become any more attached to him.