Would it bother you if someone called you "Ma'am" or "Miss", or referred to you with she/her pronouns? Would you correct them if they did? If so, you do care about labels.