Work on https://github.com/matrix-org/matrix-doc/pull/2716

Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823)

* Simplify `start_listening` callpath

* Correctly check the size of uploaded files

First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.

Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl&gt;`>

Signed-off-by: Dan Callahan <danc@element.io>

Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>

Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.

Type hint fixes due to Twisted 21.2.0 adding type hints.

* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.

 - Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](https://github.com/matrix-org/synapse/blob/80d6dc9783aa80886a133756028984dbf8920168/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version

This was never used, so let's get rid of it.

The two are equivalent, but really we want to check the HTTP result that got
returned to the channel, not the code that the Request object *intended* to
return to the channel.

This defaults `ip_range_blacklist` to reserved IP ranges and also adds an
`ip_range_whitelist` setting to override it.

Authentication is done by checking a shared secret provided
in the Synapse configuration file.

Pusher URLs now must end in `/_matrix/push/v1/notify` per the
specification.

A few test cases were relying on being able to mount non-client servlets on the
test resource. it's better to give them their own Resources.

Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).

Changes `@cache_in_self` to use underscore-prefixed attributes.

remove the stubbing out of `request.process`, so that `requestReceived` also renders the request via the appropriate resource.

Replace render() with a stub for now.

The root resource isn't necessarily a JsonResource, so rename this method
accordingly, and update a couple of test classes to use the method rather than
directly manipulating self.resource.

Where we want to render a request against a specific Resource, call the global
make_request() function rather than the one in HomeserverTestCase, allowing us
to pass in an appropriate `Site`.

Fixes #6755

We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).

A future PR will add an API for creating such a token.

When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.

This allows trailing commas in multi-line arg lists.

Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.

Signed-off-by: Dan Callahan <danc@element.io>

This implements a more standard API for instantiating a homeserver and
moves some of the dependency injection into the test suite.

More concretely this stops using `setattr` on all `kwargs` passed to `HomeServer`.

These are tests for #8439

Update `EventCreationHandler.create_event` to accept an auth_events param, and
use it in `_locally_reject_invite` instead of reinventing the wheel.

All handlers now available via get_*_handler() methods on the HomeServer.