Well, until money has exchanged hands, ultimately I'm sure the seller can do whatever they want. Is this right or fair? Heck no! I say this as someone who is currently selling a vehicle and I have a verbal sales agreement with the guy. He can't pick it up until Saturday as he is from out of town. Now, it is still my vehicle, no money has changed hands and no deposit was left so if someone offered me more than him I could sell it to them on the spot and inform the other person that it's sold. This however, would be a jerk move and I have more integrity than that. However, if Saturday comes and goes and I have not heard from him, I feel like my conscience is free to sell the vehicle the next day if someone offered me the money at that point because the agreement feel through and how long am I expected to hold onto something for someone.
Anyway, long story short, if you agreed on a price and date, the guy should show some integrity and hold it for you otherwise he's just being a prick. But, if you didn't let him know you are busy and he was expecting a call the next day, I could see him getting cold feet as who knows if other people agreed to buy the car before you and never showed.
Just my three cents.
BTW, did you get the car? Did this get resolved? I hope the guy is a stand up guy and didn't screw you over.