Except `synapse/events/__init__.py`, which will be done in a follow-up.

Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

Mark Module API error imports as re-exported and mark Synapse as containing type annotations (#11054)

Instead of only dumping them if trial passes.

* support ubuntu 21.10 indri

* add changelog

* update to correct codename

Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>

Co-authored-by: Brendan Abolivier <github@brendanabolivier.com>

The race allowed the current position to advance too far when stream IDs
are still being persisted.

This happened when it received a new stream ID from a remote write
between a new stream ID being allocated and it being added to the set of
unpersisted stream IDs.

Fixes #9424.

This reverts #11019 and structures the code a bit more like it was before #10985.

The global cache state must be reset before running the tests since other test
cases might have configured caching (and thus touched the global state).

Make `get_last_client_by_ip` return the same dictionary structure
regardless of whether the data has been persisted to the database.

This change will allow slightly cleaner type hints to be applied later
on.

This commit fixes two bugs to do with decorators not instrumenting
`ReplicationEndpoint`'s `send_request` correctly. There are two
decorators on `send_request`: Prometheus' `Gauge.track_inprogress()`
and Synapse's `opentracing.trace`.

`Gauge.track_inprogress()` does not have any support for async
functions when used as a decorator. Since async functions behave like
regular functions that return coroutines, only the creation of the
coroutine was covered by the metric and none of the actual body of
`send_request`.

`Gauge.track_inprogress()` returns a regular, non-async function
wrapping `send_request`, which is the source of the next bug.
The `opentracing.trace` decorator would normally handle async functions
correctly, but since the wrapped `send_request` is a non-async function,
the decorator ends up suffering from the same issue as
`Gauge.track_inprogress()`: the opentracing span only measures the
creation of the coroutine and none of the actual function body.

Using `Gauge.track_inprogress()` as a context manager instead of a
decorator resolves both bugs.

Pass through `SynapseError`s that are raised from experimental `check_event_allowed` callback of the module API (#11042)

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Searches the returned HTML for an oEmbed endpoint using the
autodiscovery mechanism (`<link rel=...>`), and will request it
to generate the preview.

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

Also mark `synapse.streams` as having has no untyped defs

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html

 for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.

The shared ratelimit function was replaced with a dedicated
RequestRatelimiter class (accessible from the HomeServer
object).

Other properties were copied to each sub-class that inherited
from BaseHandler.

Use `PreserveLoggingContext()` to ensure that logging contexts are not
lost when exiting a read/write lock.

When exiting a read/write lock, callbacks on a `Deferred` are triggered
as a signal to any waiting coroutines. Any waiting coroutine that
becomes runnable is likely to follow the Synapse logging context rules
and will restore its own logging context, then either run to completion
or await another `Deferred`, resetting the logging context in the
process.

This also turns off calculating code coverage, as we didn't use it and it was a lot of noise

Hopefully this fixes these tests sometimes failing in CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Include the event ids being peristed

Looks like the wrong exception type was caught in #10932.

* `disallow-untyped-defs` for `synapse.state`

Much smaller than I was expecting!