I don’t think you’re being transphobic, but I don’t think it’s possible for us trans folks to determine exactly who belongs under the “trans umbrella” either. I too prefer the currently disfavored term transsexual to describe the physical dysphoria I experience with my assigned sex. But I also identify as agender, so I can see why non-binary people do not want to be referred to as cis, regardless of whether or not they want to physically transition. As long as they’re sincere about their gender identity, I’m fine calling them trans even if they don’t experience the same medical and social obstacles that I do.

I know some non-binary people who identify as neither cis nor trans, as well. There are no easy answers here, from my perspective.