I am a married woman. He is a married man. I love my husband. He loves his wife. I love him. He loves me. We are staying with our spouses...but will always love each other, even if it means never acting on it or being together. Love is so unfair.
Sometimes it does work that way.. unfair and all. But if he loves you and you love him, how is it fair to your husband or his wife for you two to stay with them? Emotional affairs are so, so, so much worse than physical ones.
You can't help what your heart wants or does. And no matter what action is taken in this situation, someone gets hurt. And really, in my opinion (and this is my opinion, doesn't make it right or wrong), everyone will hurt no matter the decision. Do what you feel is best for you.
Many people don't understand that you CAN love more than one person...and they can all be valid loves. I understand what you're saying and it sounds like you're both dealing with it the best you can.