If we have recently seen a valid well-known for a domain we want to
retry on (non-final) errors a few times, to handle temporary blips in
networking/etc.

Retry well-known lookup before expiry. 

Fix mediarepo worker startup

This gives a bit of a grace period where we can attempt to refetch a
remote `well-known`, while still using the cached result if that fails.

Hopefully this will make the well-known resolution a bit more torelant
of failures, rather than it immediately treating failures as "no result"
and caching that for an hour.

Handle pusher being deleted during processing.

Add a lower bound to well-known TTL.

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Don't unnecessarily block notifying of new events.

Whitelist history visbility sytests for worker mode

Fix curl command typo in purge_remote_media.sh

Fix handling of redactions of redactions

Was verbose option instead of -X, command didn't work

Signed-off-by: Thomas Citharel <tcit@tcit.fr>

It costs both us and the remote server for us to fetch the well known
for every single request we send, so we add a minimum cache period. This
is set to 5m so that we still honour the basic premise of "refetch
frequently".

Handle TimelineBatch being limited and empty.

Part of fixing matrix-org/sytest#652

Sytest PR: matrix-org/sytest#667

When persisting events we calculate new stream orderings up front.
Before we notify about an event all events with lower stream orderings
must have finished being persisted.

This PR moves the assignment of stream ordering till *after* calculated
the new current state and split the batch of events into separate chunks
for persistence. This means that if it takes a long time to calculate
new current state then it will not block events in other rooms being
notified about.

This should help reduce some global pauses in the events stream which
can last for tens of seconds (if not longer), caused by some
particularly expensive state resolutions.

This hopefully addresses #5407 by gracefully handling an empty but
limited TimelineBatch. We also add some logging to figure out how this
is happening.

Account validity: allow defining HTML templates to serve the user on account renewal attempt

Return 502 not 500 when failing to reach any remote server.

Fixes #5803

Handle incorrectly encoded query params correctly

Validate well known state events are state events.

Don't allow clients to send tombstones that reference the same room