I think what is missing from both points of view is the question of what people want sexually. That is not determined by a person's sexual identity. A man, for instance, might be a pedophile. So if he goes into a toilet and there are little boys (same gender), those little boys are in danger, despite it being the same gender.

Perhaps the answer might be to have little girl's toilets or little boy's toilets or children's toilets, or just wait until you get home.