Should I quit my job?

I am a babysitter for a blended family and a mixed race couple (mom is white/dad is black). Yesturday the mom called me and said that she had concerns that I was racist and the kids have been complaining about me. Now, I have never had complaints for as long as I have been a babysitter (14 years) and I am definately not racist. I am as left wing and liberal as they come, and now I feel uncomfortable being there because of these acccusations. She wont tell me what I did to cause them to think this about me, or allow me to ask her husband because he told her not to say anything. I am really uncomfortable there now, and I dont know what to do.