We propose a novel complementarity model for a general three-dimensional manipulation system with rolling and sliding contacts to solve the forward dynamics problem. The key idea is to derive complementary constraints for a rolling contact by decomposing the tangential properties. We also discuss the existence and uniqueness of a solution to the model, and algorithms for solving the problem. We show examples of computer simulation for dynamics of multifingered hand manipulation and discuss how the transition of contact modes varies for applied joint driving torques.