Love this idea :-D Though then again, I wouldn't want note-taking to be seen as a "punishment". It's an important job, even though it does get assigned unfairly.
FWIW, I was assigned female and often took notes at meetings before my gender transition, and continued to do so afterward. I agree that taking good notes makes it much more difficult to contribute to the discussion. At my previous job, I was glad when auto-generated transcripts eventually eliminated the need to take notes during Zoom meetings, at least.