Without being rude, how can i tell my father-in-law he needs to start seeing my husbands and my children?

My husband, kids and myself live in a duplex next door to my husband's sister. But our duplexes are not connected. We literaly share a backyard. When my father-in-law visits my sister-in-law you know hes really just wants to visit his grandchildren. But my husband and i have a son and his first Granddaughter. Shes the only girl out of 9 grandboys. All he ever wanted was a granddaughter I dont see how he can visit with his daughter and 2 grandboys for 2 hours and not take atleast 5 mins of his time to see our children cuz he has 3 other children that have kids as well he doesnt even make an effort to see those grandchildren either. He knows were home and when were outside and he leaving he just keeps walking and waves hi and bye to us. My husband knows its messed up too. But can anyone give me advice on what we should say or do? Thanks I really appreciate the feedback. No mean answers please.