I think of it like this, for every 1 that you see you probably have 5 that you can't see. When I see them out at night I use a pair of tweezers and pull them out gently so they don't break and split. They are great at consuming uneaten food and detritus, but as mentioned they can over populate and become an eyesore. So kept in check I think they are fine, when you start to see them on every rock I'd start with some manual removal.