Leave had supporters on the fringe, among the poor and even in the City