How is denying gay marriage a civil rights violation?

Not that I don't see your point, but just for the sake of argument, who has the right to marry you? A man, a woman? Why doesn't the opposite gender have that same right?