You are not selfish in wanting this and you are not alone.
You've made the right start in that you both need to talk about it.
The solution will depend on you both and your specific needs. Might be as simple as a schedule, agree manly nights and girlie nights.
Or maybe a bit of different spice when you want the man to come out. You be the maid and get your botty spanked.
Or keep the "maid" locked up in chastity fo a while. "Play the maid and you stay in the cage, be the man for the day and I'll fetch the key". This really does depend on what you are into.
As for him wanting to be seen and you being afraid of the neighbours. Don't ignore it or he'll just start taking silly risks hoping for an "accidental" exposure. Better to let him do it in a planned and safe way, in America Halloween is a good excuse for him to go out dressed as a sissy, over here less so. If you have a local Pride event (or not so local if you're really worried by neighbours) then it's another place he can safely wear a frock for the day.
Don't give up on him. You've got a man who will like to go shopping, pays attension to what you wear, knows that high heels can't be worn for ten mile hikes etc.