As an example of what I consider to be very good hand-held camera work, watch the scenes on the boat in JAWS. They contributed positively to the tension and excitement that those scenes needed and were clearly justified, not gratuitous, and not so shaky as to induce vertigo. I also think the technique was used judiciously in that it was not overused and was used in a relatively small portion of the film.
Potential downsides to the technique especially when it becomes what a lot of viewers would call "shaky-cam": a) it turns off potential viewers who are sensitive to vertigo induced by a shaky image (which could lead to a loss in box office revenue), b) it will lead some viewers to think that the technique was used simply because it was faster, cheaper, or easier to do it that way (not necessarily bad reasons, but it may leave a bad impression on some viewers).
My $0.02 is that if the hand-held technique is not clearly well-motivated and appropriate to the scene, it shouldn't be used. Think of the possible alternatives that will still satisfy your storytelling objective or solve your filming problem before you use it.