We extend the classical test particle insertion (TPI) method to anisotropic systems, and develop an anisotropic test particle insertion (ATPI) framework that efficiently extracts anisotropic interaction potentials from dense fluids using a small number of equilibrium snapshots. The method leverages the parallelizability of the insertion process, enabling rapid reconstruction of complex anisotropic interactions via high-performance computing. The ATPI method is quantitatively validated for liquids in both simulations and colloidal experiments. With a few modifications, this method is able to extract anisotropic interactions from solid lattices. By exploiting the theoretical equivalence between interaction measurement and design, we further demonstrate that ATPI can be used to construct interaction potentials that drive the system into target structures such as honeycomb lattices, kagome lattices, and dodecagonal quasicrystals. Our approach opens a new avenue for the inverse design of self-assembling systems.