When Transparent first came out, I really enjoyed the show, and defended the casting of Jeffrey Tambor on the basis of, as you point out, there being trans actors, writers, and consultants involved with the show. I also felt that his casting was appropriate because he was playing an older trans woman at the beginning of her transition. But listening to criticism from trans women in particular, I saw and agreed with their points about reinforcing the “man in a dress” trope, about trans actors needing work, and about trans actors being perfectly capable of playing pre- and early-transition scenes. I still think their side has a lot of merit, but I agree with your point that we shouldn’t focus too narrowly or exclusively on these issues.
I now feel that if a cis person is going to play a trans role, they should at the very least be of the appropriate gender. One example I liked was the 2011 German film Romeos, which had a cis male actor playing a gay trans male character, in what I (as a queer transmasculine person) felt was a sensitive portrayal. While I understand Tambor was cast in Transparent in part because of name recognition, it would have been better to choose a woman for the part if a cis person had to be cast. In flashback scenes they could have perhaps had a cis male actor, as they did with Laverne Cox’s twin brother in Orange is the New Black. While I know there are other trans people in the cast of Transparent, (though I stopped watching after the first episode of the second season as I found it too depressing and triggering), Maura is the lead role, so having a cis actor play that part can overshadow the others.