I am the wrong side of life to look for anyone?

I have a 15month old daughter and am the wrong side of life to be looking for another man. I have just had a night last weekend with my first love (we remained close friends for 23yrs!) He has a long term girlfriend and has admitted he loves me but he cannot leave her and so we are back being friends again. Of course I know our relationship could not fire up from when we were teenagers and also know he wants kids. I only just made it to have mine before forty! I have always loved him and when I have been single in my life he has been in a relationship and vice versa. My partner left me after having an affair for 8 months and treating me really badly, this man is now emotionally bullying me and despite never spending anytime with his daughter has now decided he would like to have her for weekends. He has never even changed her nappy, bathed her or put her to bed. When she cries he shouts at her to shut up. SO I doubt he would manage and I feel she deserves better. I am now in limbo in my life - what to do???