In case you didn't feel like digging through all of the comments, drplokta concludes that the smallest city to host a Worldcon is Heidelberg in 1970. Incidentally, I think that in such discussions, the question is "population at the time the place hosted the Worldcon," not "what is the place's population today?"