No, that's not the only problem there. He's doing it in reverse, too. It's like he won't believe anything you say about your feelings is true no matter what. He was sure you're just trying to trick him or something. Even showing it with your actions hasn't helped, he acts like you're just waiting for your chance to abandon him.

I don't... know how to fix that.