Cost-volume filtering is one of the most widely known tech-niques to solve general multi-label problems, however it is problematically inefficient when the label space size is ex-tremely large. This paper presents a coarse-to-fine strategy of the cost-volume filtering that handles efficiently and ac-curately multi-label problems with a large label space size. Based upon the observation that true labels at the same im-age coordinate of different scales are highly correlated, we truncate unimportant labels for the cost-volume filtering by leveraging the labeling output of lower scales. Experimen-tal results show that our algorithm achieves much higher effi-ciency than the original cost-volume filtering while enjoying the comparable accuracy to it. Index Terms — cost-volume filtering, Markov random fields, label selection, coarse-to-fine