We present the Sum-Product Probabilistic Language (SPPL), a new system that automatically delivers exact solutions to a broad range of probabilistic inference queries. SPPL uses a new class of symbolic expressions to represent the distribution on execution traces of a probabilistic program that generalize sum-product networks by handling mixed-type distributions, numeric transformations, logical formulas, and pointwise and set-valued constraints. We formalize SPPL in terms of a novel translation strategy from probabilistic programs to sum-product expressions and present new and sound algorithms for exactly conditioning on and computing probabilities of events. We present new techniques for improving the scalability of translation and inference by automatically exploiting conditional independences and repeated structure in SPPL programs. We implement a prototype of SPPL with a modular architecture and evaluate it on a suite of benchmarks that the system is designed to solve, which establish that SPPL is up to 3500x faster than state-of-the-art systems for fairness verification; up to 1000x faster than state-of-the-art symbolic algebra techniques; and can compute exact probabilities of rare events in milliseconds.