Determining crystal symmetry from powder X-ray diffraction (PXRD) is a central problem in materials characterization, yet multiple space groups produce indistinguishable patterns, and automated methods plateau at low accuracy on real data. The open question is where the recoverable symmetry information is lost. We cast symmetry determination as decoding through a noisy measurement channel and measure its capacity directly. On our real-data benchmark, a Bayes-optimal decoder that is given the pattern of systematically absent reflections recovers 67% Top-1 at the benchmark’s class distribution, yet end-to-end model accuracy is only ~10%. An explicit two-stage decomposition shows that nearly all of this gap comes from determining which reflections are truly absent in a noisy, degraded diffraction pattern, not from mapping a known absence pattern to the correct symmetry. To probe this limit with a strong decoder, we introduce a physics-informed transformer, with an explicit sin²θ coordinate channel, physics-aware positional encoding, and a structured multi-task decoder that separates geometric rule learning from holistic pattern recognition, trained on a synthetic-distribution curriculum chosen to reduce train-test overlap risk. This model is competitive with or stronger than published baselines and still hits the same wall, showing that the limit is structural rather than a deficiency of any one architecture. We also show that several recent state-of-the-art models share this wall. Mapping residual errors onto the directed acyclic graph of maximal symmetry-reducing (translationengleiche) subgroups shows that the errors are physically structured, local on the hierarchy and biased toward lower-symmetry descendants, a signature of erased absence cues, and that this hierarchy makes classical Pawley verification tractable by bounding the search. The bottleneck is measurement, not classification: the productive lever is better absence recovery, not larger classifiers.