Is Tarkovsky really all that unknown to the public? Certainly, like all major foreign directors, he's not known well enough to the general public. But at least here in the states, I think he's part of the second tier of foreign film directors when it comes to name-recognition, which is to say that is to say that you can expect most pretentious college students who have any interest in the cinema to at least have heard of him or his work. He's not up there with people like Bergman, Kurosawa, and Fellini, but his films are pretty readily available (at least on video and DVD) and they're being seen by those with some interest in foreign cinema. Plus, Tarkovksy made a few films that fit into (or at least can be marketed as though they fit into) a genre, sci-fi, with some popularity, and so his work may be benefiting from exposure to a group of people who wouldn't have much interest in foreign films otherwise. I think there is evidence for this in the top 100 here, since each of his seven features--no, I'm not including juvenilia like The Steamroller and the Violin--has a place on that list. (I like Tarkovsky, but that does seem a bit much.)
I think there are other directors who have it far worse in these parts, including Dreyer, Bresson, Ozu, Rossellini, Ophuls, Tati, Mizoguchi, etc. These are some of the greatest directors in the cinema, and the attention given to their work is far too scant. Some of these directors (e.g. Bresson, Tati, and Ophuls) are more or less completely unknown. My experience suggests to me that you really have to be pretty serious about your film-watching to even find out about these people, and when you do find out about them, it can be a chore to find their work. The problem for these other directors is not that their work is completely ignored but instead that the vast majority of critical attention is on a few exemplary works at the expense of the rest of their oeuvre. People know about M and Tokyo Story and Ugetsu and The Passion of Joan of Arc, but how many people know Scarlet Street or Late Spring or Sisters of the Gion or Ordet? Look at the ratings here, and you'll get a distressing sense of what the answer must be. (By the way, I wholeheartedly agree that Mizoguchi is criminally ignored. There's no question in my mind that he belongs on the list of the five-or-so greatest directors whose work I'm familiar with.)
Ok, I'm sure this is enough prefatory crap for all of you. Now I'll just throw out a list of some films I think are underrated: Jour de Fete, The Joyless Street, Kino-Eye, Kuhle Wampe, Land without Bread, The Long Day Closes, Mikey and Nicky, New Earth, Numero Deux, The Nun, Park Row, Queen Kelly, Rancho Notorious, Sisters of the Gion, The Sun's Burial.
I'm sure I could go further with this, but that'll have to do for now. I'm aware that some of these films are already considered masterpieces in some circles, and for those films, I'm suggesting that the circle be widened.