being a girl is dumb sometimes

So there's a guy at work. I am about 75% percet sure he is flirting with me. Today he suggested lunch for all of the interns, but I am not suitable for social contact right now, so I turned him down. He responded something about how he hoped I would change my mind, and then before he left with the others he kicked the back of my chair to get my attention (I habitually have earbuds in) and tried to talk me into it again. And then afterwards told me I'd missed out.
I'm pretty sure he's flirting.
Now, 1. He's older than me, 2. He's a coworker, 3. I'm a lesbian. So he's so far out of the running that he's not even on the same race track. And his flirting is starting to make me uncomfortable, so clearly I should ask him to stop.
Except.
I feel guilty for the fact that he's flirting. I feel guilty for being uncomfortable. This is fucked up.
It isn't my fault. I'm uncomfortable and I have the right to bring that up. My comfort is not secondary to his ego. If he's a decent human being he'll apologize and stop.
But I don't have the guts to do it. Every chance I get I freeze up and shut up and shuffle awkwardly. I don't know how to say this. I feel guilty.