Making a Wait

I do not actually like people waiting for me but most of the time I land up making people wait for me. I would have said "yes" to my friends to leave home and the very next minute either someone would have called me or stopped by at my desk. I can not be rude to the person talking to me by saying "hey listen I got to pack home now, talk to you later". I have to spend couple of minutes chit-chatting and later convey the message you know what my friends are waiting for my arrival to disperse home. Today I disappointed two of my friends by making them wait for 20-30 minutes.