In this report I present hierarchical parallel Markov models for the creation of interactive social agents for video games and entertainment. The approach extends existing, popular character technologies for social, communicative interaction. First, adding the knowledge of temporal interaction structure enables natural language interaction on a time scale much longer than current chatterbot technologies. Second, adding support for hierarchical interaction decomposition, where an interaction is represented as a collection of smaller, simpler elements, simplifies the authoring of complex engagement. Third, adding support for the concurrent engagement of these elements enables engagement in interleaved, naturalistic communication. The resulting decomposition supports redundancy of representation, graceful performance degradation through the simultaneous engagement of behaviors on different levels of abstraction, and the stochastic approximation mechanism increases robustness in the face of noise and ambiguity. In this report, I present the details of hierarchical parallel Markov models, I examine two entertainment agents that use this technique, and explain the implementational details of how such an approach can be used in the development of future systems.