I find him downright odd looking, but, weirdly, that's the appeal. I've heard people who really can't stand the look of him say that he is 'horse faced', and I do sort of see where they are getting that from, however unflattering such a reference is.

Honestly, I've seen him in a few other things, and he doesn't seem as intriguing or appealing. So, for me at least, I think that most of the attraction comes from the personality traits of the Sherlock character. I'm sure that is some sort of disturbing statement on my psychological issues, but there it is...