Try harder when sending leave events

When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.