There was a bunch of unnecessary conversion back and forth between dict and
list going on here. We can simplify a bunch of the code.