Skip to content
Snippets Groups Projects
upgrade.md 55.26 KiB

Upgrading Synapse

Before upgrading check if any special steps are required to upgrade from the version you currently have installed to the current version of Synapse. The extra instructions that may be required are listed later in this document.

  • Check that your versions of Python and PostgreSQL are still supported.

    Synapse follows upstream lifecycles for Python and PostgreSQL, and removes support for versions which are no longer maintained.

    The website https://endoflife.date also offers convenient summaries.

  • If Synapse was installed using prebuilt packages, you will need to follow the normal process for upgrading those packages.

  • If Synapse was installed from source, then:

    1. Activate the virtualenv before upgrading. For example, if Synapse is installed in a virtualenv in ~/synapse/env then run:

      source ~/synapse/env/bin/activate
    2. If Synapse was installed using pip then upgrade to the latest version by running:

      pip install --upgrade matrix-synapse

      If Synapse was installed using git then upgrade to the latest version by running:

      git pull
      pip install --upgrade .
    3. Restart Synapse:

      ./synctl restart

To check whether your update was successful, you can check the running server version with:

# you may need to replace 'localhost:8008' if synapse is not configured
# to listen on port 8008.

curl http://localhost:8008/_synapse/admin/v1/server_version

Rolling back to older versions

Rolling back to previous releases can be difficult, due to database schema changes between releases. Where we have been able to test the rollback process, this will be noted below.

In general, you will need to undo any changes made during the upgrade process, for example:

  • pip:

    source env/bin/activate
    # replace `1.3.0` accordingly:
    pip install matrix-synapse==1.3.0
  • Debian:

    # replace `1.3.0` and `stretch` accordingly:
    wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
    dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb

Upgrading to v1.44.0

The URL preview cache is no longer mirrored to storage providers

The url_cache/ and url_cache_thumbnails/ directories in the media store are no longer mirrored to storage providers. These two directories can be safely deleted from any configured storage providers to reclaim space.

Upgrading to v1.43.0

The spaces summary APIs can now be handled by workers

The available worker applications documentation has been updated to reflect that calls to the /spaces, /hierarchy, and /summary endpoints can now be routed to workers for both client API and federation requests.

Upgrading to v1.42.0

Removal of old Room Admin API

The following admin APIs were deprecated in Synapse 1.25 (released on 2021-01-13) and have now been removed:

  • POST /_synapse/admin/v1/purge_room
  • POST /_synapse/admin/v1/shutdown_room/<room_id>

Any scripts still using the above APIs should be converted to use the Delete Room API.

User-interactive authentication fallback templates can now display errors

This may affect you if you make use of custom HTML templates for the reCAPTCHA or terms fallback pages.

The template is now provided an error variable if the authentication process failed. See the default templates linked above for an example.

Removal of out-of-date email pushers

Users will stop receiving message updates via email for addresses that were once, but not still, linked to their account.

Upgrading to v1.41.0

Add support for routing outbound HTTP requests via a proxy for federation

Since Synapse 1.6.0 (2019-11-26) you can set a proxy for outbound HTTP requests via http_proxy/https_proxy environment variables. This proxy was set for:

  • push
  • url previews
  • phone-home stats
  • recaptcha validation
  • CAS auth validation
  • OpenID Connect
  • Federation (checking public key revocation)

In this version we have added support for outbound requests for:

  • Outbound federation
  • Downloading remote media
  • Fetching public keys of other servers

These requests use the same proxy configuration. If you have a proxy configuration we recommend to verify the configuration. It may be necessary to adjust the no_proxy environment variable.

See using a forward proxy with Synapse documentation for details.

Deprecation of template_dir

The template_dir settings in the sso, account_validity and email sections of the configuration file are now deprecated. Server admins should use the new templates.custom_template_directory setting in the configuration file and use one single custom template directory for all aforementioned features. Template file names remain unchanged. See the related documentation for more information and examples.

We plan to remove support for these settings in October 2021.

/_synapse/admin/v1/users/{userId}/media must be handled by media workers

The media repository worker documentation has been updated to reflect that calls to /_synapse/admin/v1/users/{userId}/media must now be handled by media repository workers. This is due to the new DELETE method of this endpoint modifying the media store.

Upgrading to v1.39.0

Deprecation of the current third-party rules module interface

The current third-party rules module interface is deprecated in favour of the new generic modules system introduced in Synapse v1.37.0. Authors of third-party rules modules can refer to this documentation to update their modules. Synapse administrators can refer to this documentation to update their configuration once the modules they are using have been updated.

We plan to remove support for the current third-party rules interface in September 2021.

Upgrading to v1.38.0

Re-indexing of events table on Postgres databases

This release includes a database schema update which requires re-indexing one of the larger tables in the database, events. This could result in increased disk I/O for several hours or days after upgrading while the migration completes. Furthermore, because we have to keep the old indexes until the new indexes are ready, it could result in a significant, temporary, increase in disk space.

To get a rough idea of the disk space required, check the current size of one of the indexes. For example, from a psql shell, run the following sql:

SELECT pg_size_pretty(pg_relation_size('events_order_room'));

We need to rebuild four indexes, so you will need to multiply this result by four to give an estimate of the disk space required. For example, on one particular server:

synapse=# select pg_size_pretty(pg_relation_size('events_order_room'));
 pg_size_pretty
----------------
 288 MB
(1 row)

On this server, it would be wise to ensure that at least 1152MB are free.

The additional disk space will be freed once the migration completes.

SQLite databases are unaffected by this change.

Upgrading to v1.37.0

Deprecation of the current spam checker interface

The current spam checker interface is deprecated in favour of a new generic modules system. Authors of spam checker modules can refer to this documentation to update their modules. Synapse administrators can refer to this documentation to update their configuration once the modules they are using have been updated.

We plan to remove support for the current spam checker interface in August 2021.

More module interfaces will be ported over to this new generic system in future versions of Synapse.

Upgrading to v1.34.0

room_invite_state_types configuration setting

The room_invite_state_types configuration setting has been deprecated and replaced with room_prejoin_state. See the sample configuration file.

If you have set room_invite_state_types to the default value you should simply remove it from your configuration file. The default value used to be:

room_invite_state_types:
   - "m.room.join_rules"
   - "m.room.canonical_alias"
   - "m.room.avatar"
   - "m.room.encryption"
   - "m.room.name"

If you have customised this value, you should remove room_invite_state_types and configure room_prejoin_state instead.

Upgrading to v1.33.0

Account Validity HTML templates can now display a user's expiration date

This may affect you if you have enabled the account validity feature, and have made use of a custom HTML template specified by the account_validity.template_dir or account_validity.account_renewed_html_path Synapse config options.

The template can now accept an expiration_ts variable, which represents the unix timestamp in milliseconds for the future date of which their account has been renewed until. See the default template for an example of usage.

ALso note that a new HTML template, account_previously_renewed.html, has been added. This is is shown to users when they attempt to renew their account with a valid renewal token that has already been used before. The default template contents can been found here, and can also accept an expiration_ts variable. This template replaces the error message users would previously see upon attempting to use a valid renewal token more than once.

Upgrading to v1.32.0

Regression causing connected Prometheus instances to become overwhelmed

This release introduces a regression that can overwhelm connected Prometheus instances. This issue is not present in Synapse v1.32.0rc1.

If you have been affected, please downgrade to 1.31.0. You then may need to remove excess writeahead logs in order for Prometheus to recover. Instructions for doing so are provided here.

Dropping support for old Python, Postgres and SQLite versions

In line with our deprecation policy, we've dropped support for Python 3.5 and PostgreSQL 9.5, as they are no longer supported upstream.

This release of Synapse requires Python 3.6+ and PostgresSQL 9.6+ or SQLite 3.22+.

Removal of old List Accounts Admin API

The deprecated v1 "list accounts" admin API (GET /_synapse/admin/v1/users/<user_id>) has been removed in this version.

The v2 list accounts API has been available since Synapse 1.7.0 (2019-12-13), and is accessible under GET /_synapse/admin/v2/users.

The deprecation of the old endpoint was announced with Synapse 1.28.0 (released on 2021-02-25).

Application Services must use type m.login.application_service when registering users

In compliance with the Application Service spec, Application Services are now required to use the m.login.application_service type when registering users via the /_matrix/client/r0/register endpoint. This behaviour was deprecated in Synapse v1.30.0.

Please ensure your Application Services are up to date.

Upgrading to v1.29.0

Requirement for X-Forwarded-Proto header

When using Synapse with a reverse proxy (in particular, when using the [x_forwarded]{.title-ref} option on an HTTP listener), Synapse now expects to receive an [X-Forwarded-Proto]{.title-ref} header on incoming HTTP requests. If it is not set, Synapse will log a warning on each received request.

To avoid the warning, administrators using a reverse proxy should ensure that the reverse proxy sets [X-Forwarded-Proto]{.title-ref} header to [https]{.title-ref} or [http]{.title-ref} to indicate the protocol used by the client.

Synapse also requires the [Host]{.title-ref} header to be preserved.

See the reverse proxy documentation, where the example configurations have been updated to show how to set these headers.

(Users of Caddy are unaffected, since we believe it sets [X-Forwarded-Proto]{.title-ref} by default.)

Upgrading to v1.27.0

Changes to callback URI for OAuth2 / OpenID Connect and SAML2

This version changes the URI used for callbacks from OAuth2 and SAML2 identity providers:

  • If your server is configured for single sign-on via an OpenID Connect or OAuth2 identity provider, you will need to add [synapse public baseurl]/_synapse/client/oidc/callback to the list of permitted "redirect URIs" at the identity provider.

    See the OpenID docs for more information on setting up OpenID Connect.

  • If your server is configured for single sign-on via a SAML2 identity provider, you will need to add [synapse public baseurl]/_synapse/client/saml2/authn_response as a permitted "ACS location" (also known as "allowed callback URLs") at the identity provider.

    The "Issuer" in the "AuthnRequest" to the SAML2 identity provider is also updated to [synapse public baseurl]/_synapse/client/saml2/metadata.xml. If your SAML2 identity provider uses this property to validate or otherwise identify Synapse, its configuration will need to be updated to use the new URL. Alternatively you could create a new, separate "EntityDescriptor" in your SAML2 identity provider with the new URLs and leave the URLs in the existing "EntityDescriptor" as they were.

Changes to HTML templates

The HTML templates for SSO and email notifications now have Jinja2's autoescape enabled for files ending in .html, .htm, and .xml. If you have customised these templates and see issues when viewing them you might need to update them. It is expected that most configurations will need no changes.

If you have customised the templates names for these templates, it is recommended to verify they end in .html to ensure autoescape is enabled.

The above applies to the following templates:

  • add_threepid.html
  • add_threepid_failure.html
  • add_threepid_success.html
  • notice_expiry.html
  • notice_expiry.html
  • notif_mail.html (which, by default, includes room.html and notif.html)
  • password_reset.html
  • password_reset_confirmation.html
  • password_reset_failure.html
  • password_reset_success.html
  • registration.html
  • registration_failure.html
  • registration_success.html
  • sso_account_deactivated.html
  • sso_auth_bad_user.html
  • sso_auth_confirm.html
  • sso_auth_success.html
  • sso_error.html
  • sso_login_idp_picker.html
  • sso_redirect_confirm.html

Upgrading to v1.26.0

Rolling back to v1.25.0 after a failed upgrade

v1.26.0 includes a lot of large changes. If something problematic occurs, you may want to roll-back to a previous version of Synapse. Because v1.26.0 also includes a new database schema version, reverting that version is also required alongside the generic rollback instructions mentioned above. In short, to roll back to v1.25.0 you need to:

  1. Stop the server

  2. Decrease the schema version in the database:

    UPDATE schema_version SET version = 58;
  3. Delete the ignored users & chain cover data:

    DROP TABLE IF EXISTS ignored_users;
    UPDATE rooms SET has_auth_chain_index = false;

    For PostgreSQL run:

    TRUNCATE event_auth_chain_links;
    TRUNCATE event_auth_chains;

    For SQLite run:

    DELETE FROM event_auth_chain_links;
    DELETE FROM event_auth_chains;
  4. Mark the deltas as not run (so they will re-run on upgrade).

    DELETE FROM applied_schema_deltas WHERE version = 59 AND file = "59/01ignored_user.py";
    DELETE FROM applied_schema_deltas WHERE version = 59 AND file = "59/06chain_cover_index.sql";
  5. Downgrade Synapse by following the instructions for your installation method in the "Rolling back to older versions" section above.

Upgrading to v1.25.0

Last release supporting Python 3.5

This is the last release of Synapse which guarantees support with Python 3.5, which passed its upstream End of Life date several months ago.

We will attempt to maintain support through March 2021, but without guarantees.

In the future, Synapse will follow upstream schedules for ending support of older versions of Python and PostgreSQL. Please upgrade to at least Python 3.6 and PostgreSQL 9.6 as soon as possible.

Blacklisting IP ranges

Synapse v1.25.0 includes new settings, ip_range_blacklist and ip_range_whitelist, for controlling outgoing requests from Synapse for federation, identity servers, push, and for checking key validity for third-party invite events. The previous setting, federation_ip_range_blacklist, is deprecated. The new ip_range_blacklist defaults to private IP ranges if it is not defined.

If you have never customised federation_ip_range_blacklist it is recommended that you remove that setting.

If you have customised federation_ip_range_blacklist you should update the setting name to ip_range_blacklist.

If you have a custom push server that is reached via private IP space you may need to customise ip_range_blacklist or ip_range_whitelist.

Upgrading to v1.24.0

Custom OpenID Connect mapping provider breaking change

This release allows the OpenID Connect mapping provider to perform normalisation of the localpart of the Matrix ID. This allows for the mapping provider to specify different algorithms, instead of the default way.

If your Synapse configuration uses a custom mapping provider ([oidc_config.user_mapping_provider.module]{.title-ref} is specified and not equal to [synapse.handlers.oidc_handler.JinjaOidcMappingProvider]{.title-ref}) then you must ensure that [map_user_attributes]{.title-ref} of the mapping provider performs some normalisation of the [localpart]{.title-ref} returned. To match previous behaviour you can use the [map_username_to_mxid_localpart]{.title-ref} function provided by Synapse. An example is shown below:

from synapse.types import map_username_to_mxid_localpart

class MyMappingProvider:
    def map_user_attributes(self, userinfo, token):
        # ... your custom logic ...
        sso_user_id = ...
        localpart = map_username_to_mxid_localpart(sso_user_id)

        return {"localpart": localpart}

Removal historical Synapse Admin API

Historically, the Synapse Admin API has been accessible under:

  • /_matrix/client/api/v1/admin
  • /_matrix/client/unstable/admin
  • /_matrix/client/r0/admin
  • /_synapse/admin/v1

The endpoints with /_matrix/client/* prefixes have been removed as of v1.24.0. The Admin API is now only accessible under:

  • /_synapse/admin/v1

The only exception is the [/admin/whois]{.title-ref} endpoint, which is also available via the client-server API.

The deprecation of the old endpoints was announced with Synapse 1.20.0 (released on 2020-09-22) and makes it easier for homeserver admins to lock down external access to the Admin API endpoints.

Upgrading to v1.23.0

Structured logging configuration breaking changes

This release deprecates use of the structured: true logging configuration for structured logging. If your logging configuration contains structured: true then it should be modified based on the structured logging documentation.

The structured and drains logging options are now deprecated and should be replaced by standard logging configuration of handlers and formatters.

A future will release of Synapse will make using structured: true an error.

Upgrading to v1.22.0

ThirdPartyEventRules breaking changes

This release introduces a backwards-incompatible change to modules making use of ThirdPartyEventRules in Synapse. If you make use of a module defined under the third_party_event_rules config option, please make sure it is updated to handle the below change:

The http_client argument is no longer passed to modules as they are initialised. Instead, modules are expected to make use of the http_client property on the ModuleApi class. Modules are now passed a module_api argument during initialisation, which is an instance of ModuleApi. ModuleApi instances have a http_client property which acts the same as the http_client argument previously passed to ThirdPartyEventRules modules.

Upgrading to v1.21.0

Forwarding /_synapse/client through your reverse proxy

The reverse proxy documentation has been updated to include reverse proxy directives for /_synapse/client/* endpoints. As the user password reset flow now uses endpoints under this prefix, you must update your reverse proxy configurations for user password reset to work.

Additionally, note that the Synapse worker documentation has been updated to

state that the /_synapse/client/password_reset/email/submit_token endpoint can be handled

by all workers. If you make use of Synapse's worker feature, please update your reverse proxy configuration to reflect this change.

New HTML templates

A new HTML template, password_reset_confirmation.html, has been added to the synapse/res/templates directory. If you are using a custom template directory, you may want to copy the template over and modify it.

Note that as of v1.20.0, templates do not need to be included in custom template directories for Synapse to start. The default templates will be used if a custom template cannot be found.

This page will appear to the user after clicking a password reset link that has been emailed to them.

To complete password reset, the page must include a way to make a [POST]{.title-ref} request to /_synapse/client/password_reset/{medium}/submit_token with the query parameters from the original link, presented as a URL-encoded form. See the file itself for more details.

Updated Single Sign-on HTML Templates

The saml_error.html template was removed from Synapse and replaced with the sso_error.html template. If your Synapse is configured to use SAML and a custom sso_redirect_confirm_template_dir configuration then any customisations of the saml_error.html template will need to be merged into the sso_error.html template. These templates are similar, but the parameters are slightly different:

  • The msg parameter should be renamed to error_description.
  • There is no longer a code parameter for the response code.
  • A string error parameter is available that includes a short hint of why a user is seeing the error page.

Upgrading to v1.18.0

Docker [-py3]{.title-ref} suffix will be removed in future versions

From 10th August 2020, we will no longer publish Docker images with the [-py3]{.title-ref} tag suffix. The images tagged with the [-py3]{.title-ref} suffix have been identical to the non-suffixed tags since release 0.99.0, and the suffix is obsolete.

On 10th August, we will remove the [latest-py3]{.title-ref} tag. Existing per-release tags (such as [v1.18.0-py3]{.title-ref}) will not be removed, but no new [-py3]{.title-ref} tags will be added.

Scripts relying on the [-py3]{.title-ref} suffix will need to be updated.

Redis replication is now recommended in lieu of TCP replication

When setting up worker processes, we now recommend the use of a Redis server for replication. The old direct TCP connection method is deprecated and will be removed in a future release. See workers for more details.

Upgrading to v1.14.0

This version includes a database update which is run as part of the upgrade, and which may take a couple of minutes in the case of a large server. Synapse will not respond to HTTP requests while this update is taking place.

Upgrading to v1.13.0

Incorrect database migration in old synapse versions

A bug was introduced in Synapse 1.4.0 which could cause the room directory to be incomplete or empty if Synapse was upgraded directly from v1.2.1 or earlier, to versions between v1.4.0 and v1.12.x.

This will not be a problem for Synapse installations which were:
  • created at v1.4.0 or later,
  • upgraded via v1.3.x, or
  • upgraded straight from v1.2.1 or earlier to v1.13.0 or later.

If completeness of the room directory is a concern, installations which are affected can be repaired as follows:

  1. Run the following sql from a [psql]{.title-ref} or [sqlite3]{.title-ref} console:

    INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
       ('populate_stats_process_rooms', '{}', 'current_state_events_membership');
    
    INSERT INTO background_updates (update_name, progress_json, depends_on) VALUES
       ('populate_stats_process_users', '{}', 'populate_stats_process_rooms');
  2. Restart synapse.

New Single Sign-on HTML Templates

New templates (sso_auth_confirm.html, sso_auth_success.html, and sso_account_deactivated.html) were added to Synapse. If your Synapse is configured to use SSO and a custom sso_redirect_confirm_template_dir configuration then these templates will need to be copied from synapse/res/templates into that directory.

Synapse SSO Plugins Method Deprecation

Plugins using the complete_sso_login method of synapse.module_api.ModuleApi should update to using the async/await version complete_sso_login_async which includes additional checks. The non-async version is considered deprecated.

Rolling back to v1.12.4 after a failed upgrade

v1.13.0 includes a lot of large changes. If something problematic occurs, you may want to roll-back to a previous version of Synapse. Because v1.13.0 also includes a new database schema version, reverting that version is also required alongside the generic rollback instructions mentioned above. In short, to roll back to v1.12.4 you need to:

  1. Stop the server

  2. Decrease the schema version in the database:

    UPDATE schema_version SET version = 57;
  3. Downgrade Synapse by following the instructions for your installation method in the "Rolling back to older versions" section above.

Upgrading to v1.12.0

This version includes a database update which is run as part of the upgrade, and which may take some time (several hours in the case of a large server). Synapse will not respond to HTTP requests while this update is taking place.