Anger, hurt, frustration

I work for 2 companies. One of which is part time at best, usually 2 - 4 hours a week sporadically. The last week or two I have had to take a couple calls while on shift at my other 'regular' job. The total amount of time is quite limited, maybe half an hour to an hour of time spent, taken away from my regular job. I keep close track of the time and make it up in the morning or at night if it is there. If anything is of immediate concern with my regular job, it always takes priority. However some of my coworkers seem to think I don't work 'for the company' anymore. I have had some absences that I have not clearly defined for them (multiple doctors appt's for my wife, a recurring psychiatrist appointment for myself, a root canal etc) that I have not wanted to share. It appears I will have to. ugh. I hate having to defend myself when I have proven myself over and over to be a reliable employee with one exception: communicating what is going on with me to those who have no business caring about it since they don't report to me and I don't report to them.