Stop setting the outlier flag for things that aren't (#10614)

Marking things as outliers to inhibit pushes is a sledgehammer to crack a
nut. Move the test further down the stack so that we just inhibit the thing we
want.