"I’ve always said, ‘Use it or you lose it,’" he says. "But you can use it too much and once people do that, I wish they’d shut up, and there are a few out there who should”