If he's flirting why get so upset when I ask him about it?

I work with a male colleague and we are friends and get along well, we are both married people. He has always been very touchy feely and touches my arms, shoulders, back and cupped my face and touched my hair once and winks at me. He has been clearly flirting. He also asks me often if I'm going to have any more children. Recently I decided to talk to him about this, I asked him in a nice way: "What is all this touchy feely business about? What are you after?" And why do you keep asking me about having another baby? I said it light heartedly but I think I caught him off guard. He got extremely upset and defensive saying: "It's just the way I am! I'll make a concious effort not to touch you again!" I'm not flirting or being affectionate!" "And I'm not asking about you having another baby because I want to see what you and your husband are up to!" Then he stormed off!!! He ignored me for the rest of the day. I have seen him since and we had a normal conversation in passing. I guess what I'm asking is why would he get so upset when I asked him about his flirty behaviour? Is it because he is attracted to me but wants to disguise his feelings?