Hug_Me_Tighter,
I agree with the above to trust your instincts. Your gut has spoken, thank goodness you have listened or you could be waking up to a whole lot of trouble.
People who are eager to get into another relationship without truly finishing with the last and not bothering to take time to heal or find out who they are on their own are bad news waiting to happen for so many reasons it is tough to even know where to start. Trust that when we are truly available and seek truly available people for love interests, things will be less complicated, confusing and conflicting.
If he is truly a good friend he wouldn't put you or your and his children in such an awkward position in the first place. If you can't walk up to his door, ring the bell and say hi honey, and sit down to dinner or help cook the dinner, it is a go no where proposition destined to be filled with pain. I don't understand why people willingly and knowingly sign up for that. Keep a clear head and don't be manipulated. I know there are strong words said so far in these posts, but better to hear and really listen now than be kicking yourself later.
Beware, he may just want a safe, convenient place to land when things finally blow up at home as they are destined to do in such a situation. But deep down I believe you already know this. Take care of, trust and honour yourself.