We have developed a framework for spectroscopic inference that we believe is a significant step towards fulfilling the astronomer’s dream of using all of the data, called Starfish. Our statistical framework attempts to overcome many of the difficulties inherent to deriving fundamental stellar properties from high resolution spectra. Principally, at high resolution and high sensitivity, model systematics—such as inaccuracies in the strengths of particular lines—will dominate the noise budget. We address these problems by accounting for the covariant structure of the residuals that can result from fitting models to data in this high signal-to-noise, high spectral resolution regime. Using some of the machinery developed by the field of Gaussian processes, we can parameterize the covariant structure both due to general line mis-matches as well as specific “outlier” spectral lines due to pathological errors in the atomic and molecular line databases.