My daughter broke up with her boyfriend and we are still close she says I am not suppose to talk to him?

My granddaughter and I are very close to her ex boyfriend and have been for over 4 years. He has gotten married and has a baby and when he comes home it is to see the granddaughter. My daughter who is 21 now came over with her finance while they were here and after they left she exploded and said it wasn't right for us to still have contact with him. I can see the awkwardness for them but I don't think it is fair to the granddaughter who has been close to him. Am I suppose to give up the one I call my adopted son because my daughter doesn't like him?