Plants employ diverse mating strategies, including self- and cross-fertilization or their combination. Despite the ubiquity of mixed mating – the use of both self- and cross-fertilization within the same population – its evolutionary origins and dynamic stability remain elusive. Here, we study self-incompatibility, in which self-fertilization is disabled by molecular recognition and enabled when mutations disrupt this recognition. Using population-level stochastic simulations, we model the effects of two main parameters on the prevailing mating mode: promiscuity of molecular recognition and inbreeding depression, which penalizes self-compatibility. We reveal a phase diagram with three phases: complete self-incompatibility, complete self-compatibility, and a mixture combining both. The mixed phase exhibits newly-described vigorous, non-decaying fluctuations in the proportions of individuals occupying either mating mode. Pollen limitation extends the parameter range over which complete self-incompatibility is favored, contrary to the commonly held view. This study offers new insight into the evolutionary dynamics of plant reproductive systems.