Here south of Sacramento bananas are fairly rare and almost everyone that hears about my bananas is surprised that they can even be grown here. Somoe of the other farmers laugh and say I'm crazy, but that's part of the fun of it.
If I was in the situation of being around people that hated bananas I'd probably make up something weird to get them talking. Maybe I'd say I had a problem losing my temper and that the doctors attributed it to a chemical imbalance and that my nutritionist suggested that I add more fresh bananas to my diet and then tell them that it seemed to be helping. I might also add that, since I'm a guy, it seemed to work better if I ate the male flowers as well.
Maybe you could also sneak into other yards at night time and plant corms in the yards of your neighbors and later on put out word that someone has been stealing some of your rare bananas. Get creative!