Developing a crush on a co-worker can be distracting and inappropriate, especially when both individuals are married. It's important to maintain professionalism in the workplace. Keeping emotional distance can prevent complications in a career.
Poll the other members of your group about how they feel about the younger adults and the baby being with you. You may discover you are not the only ones who aren't comfortable with it. If that's the case, then someone is going to have to speak up and object.