This reverts commit e0c01296.

As discussed at
https://github.com/matrix-org/synapse/pull/7801#pullrequestreview-444652786, I
don't think this is an improvement.