Most of the current methods to assess effective connectivity from functional magnetic resonance imaging (fMRI) rely on the assumption that all relevant brain regions are entered into the analysis. If this assumption is untenable, which we believe is most often the case, then spurious connections between brain regions can appear. In this paper we propose to use an ancestral graph to model connectivity, which provides a way to avoid spurious connections. The ancestral graph is determined from trial-by-trial variation and not from the time series. A random effects model is defined for ancestral graphs which allows for individual differences. The framework of local misspecification in the random effects model is used, which allows for modeling errors in connections and brain regions. The framework of local misspecification additionally provides a test on parameters in the graph which is robust against model misspecification. The test can be used to find differences in connection strength between, for example, conditions. Monte Carlo simulations show that the ancestral graph is appropriate to use even with modeling errors. To assess the accuracy further, the proposed method was applied to real fMRI data to determine how brain regions interact during speech monitoring.