Non-Markovian dynamics are ubiquitous across physics, biology, and engineering. Yet our understanding of non-Markovian processes significantly lags that of simpler Markovian processes, due largely to a lack of tractable models. In this article, we present a minimal model of non-Markovian dynamics in which the current state copies past states with arbitrary history dependence. We show that many properties of this process can be studied analytically, providing insight into the relationships between history dependence, autocorrelations, and information-theoretic metrics like entropy and dynamical information. Strikingly, we find that autocorrelations can fail, even qualitatively, to capture the underlying dependencies. Ultimately, this model serves as a tractable sandbox for exploring non-Markovian dynamics.