Add some assertions about outliers (#10773)

I think I have finally teased apart the codepaths which handle outliers, and those that handle non-outliers. 
Let's add some assertions to demonstrate my newfound knowledge.