I'll admit I'm not expert in this area--am I anywhere?--but I think the last question is fairly easily answered in some cases: some directors probably continued to use black and white, since it was cheaper. That, I suppose, was a primary motivation in a lot of decisions to use black and white rather than color.
The debate about whether black and white is better than color strikes me as about as pointless as the debate about whether silent film is better than sound film. The categories seem largely incommensurable: each has its benefits and drawbacks, and they do quite different things. It appears one would have to appeal to some overarching normative theory about film aethetics to make an argument on behalf of either black and white or color, and the arguments I've heard on this matter have all been quite jejune and unconvincing (e.g. film creates a dreamscape, and so black and white is better; film ought to be realistic, and so color is better; and so on). I'm sure people have come up with better reasons than these, but I've yet to encounter them.
Here's the only method I'd suggest for settling this (I think, silly) debate: go and look at the films themselves. To my eye, there are some visually magnificent color films that would gain nothing by being in black and white, and there are some equally beautiful films in black and white that would gain nothing from being in color. Take some lovely black and white films--for instance (and in no particular order), Ordet, Day of Wrath, Andrei Rublev, Queen Kelly, The Scarlet Empress, The Magnificent Ambersons, Ivan the Terrible, The Night of the Hunter, Sunrise, Tabu, L'Eclisse, L'Atalante, Last Year at Marienbad, Sansho the Bailiff, Ugetsu, The Life of Oharu, etc. Would they really gain anything by being in color? I fail to see how, and, moreover, I think they'd be likely to lose a lot of what is most distinctive about them--at least visually.
The evidence of the value of black-and-white cinematography is there on the screen. Just look.