Be less strict about validation of UI authentication sessions during
registration to match client expecations.

Since we only use pubsub, the dbid is irrelevant.

Add dummy_events_threshold which allows configuring the number of forward extremities a room needs for Synapse to send forward extremities in it.

For the record, the reason we need this is as follows:

each RDATA command comes down the redis pipe as a subscription message. txredisapi as written needs at least three reactor ticks to read each subscription message from the tcp buffer. Hence, once the process gets loaded, it starts getting behind, and eventually redis knifes the connection. it then takes ages for the master to work its way through the backlog, before it reconnects again, during which any commands from any workers are dropped.

Speed up fetching device lists changes in sync.

Fix lost events on replication reconnection

... otherwise we can believe we're up to date when we're not.

Currently we copy `users_who_share_room` needlessly about three times,
which is expensive when the set is large (which it can easily be).

This caused `prev_state_ids` to be incorrect if the state event was not
replacing an existing state entry.

Fix MANIFEST.in

An update of check-manifest shone some light on some issues with MANIFEST.in, specifically that we didn't ignore/prune the contrib directory, and that we were using prune instead of exclude for files. This fixes both issues.

Fixes #7403

We forgot to set the password on the subscriber connection, as well as
not calling super methods for overridden connectionMade/connectionLost
functions.

populate_stats_process_rooms was added in #5971 / v1.4.0; current_state_events_membership was added in #5706 / v1.3.0.

Fixes #7380.

For in memory streams when fetching updates on workers we need to query the source of the stream, which currently is hard coded to be master. This PR threads through the source instance we received via `POSITION` through to the update function in each stream, which can then be passed to the replication client for in memory streams.

We move the processing of typing and federation replication traffic into their handlers so that `Stream.current_token()` points to a valid token. This allows us to remove `get_streams_to_replicate()` and `stream_positions()`.

Further improvements to requesting the public rooms list on a homeserver which has it set to private (#7368)

Hopefully this is no worse than what we have on master...

By persisting the user interactive authentication sessions to the database, this fixes
situations where a user hits different works throughout their auth session and also
allows sessions to persist through restarts of Synapse.

This is primarily for allowing us to send those commands from workers, but for now simply allows us to ignore echoed RDATA/POSITION commands that we sent (we get echoes of sent commands when using redis). Currently we log a WARNING on the master process every time we receive an echoed RDATA.