At some point in history, someone told you that you were erm-ing too much. Someone did. Or maybe something happened to convince you that it was a bad thing. Or do we just know instinctively, because ERM is the word most people are afraid of?
Let’s get one thing clear: To “erm” is human. To “erm” repeatedly when you are trying to make a point, can become distracting.
As with all vocal or physical gestures, once you start repeating them THAT’s when they become annoying.