If you guys aren't wearing ear protection every time you throw your leg over your bike, you are definitely damaging your ears and no, it doesn't matter how quiet you think your helmet is. Search around on here for hearing protection threads and you'll find some very solid and informative threads on this subject.
I'd sooner ride without my helmet than my hearing protection. Tinnitus, hearing aids, and not being able to understand anything in a crowded room - all that stinks and wouldn't wish it anyone. If you're riding without hearing protection, all of that is in your future.
'16 R1200GS Triple Black
'16 KTM Super Duke 1290R
'18 Yamaha WR250R