Skip to content
Snippets Groups Projects
  • Eric Eastwood's avatar
    1ca70fd3
    Allow room creator to send MSC2716 related events in existing room versions (#10566) · 1ca70fd3
    Eric Eastwood authored
    * Allow room creator to send MSC2716 related events in existing room versions
    
    Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869
    
    Restoring `get_create_event_for_room_txn` from,
    https://github.com/matrix-org/synapse/pull/10245/commits/44bb3f0cf5cb365ef9281554daceeecfb17cc94d
    
    * Add changelog
    
    * Stop people from trying to redact MSC2716 events in unsupported room versions
    
    * Populate rooms.creator column for easy lookup
    
    > From some [out of band discussion](https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`).
    >
    > https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642
    
    * Remove and switch away from get_create_event_for_room_txn
    
    * Fix no create event being found because no state events persisted yet
    
    * Fix and add tests for rooms creator bg update
    
    * Populate rooms.creator field for easy lookup
    
    Part of https://github.com/matrix-org/synapse/pull/10566
    
     - Fill in creator whenever we insert into the rooms table
     - Add background update to backfill any missing creator values
    
    * Add changelog
    
    * Fix usage
    
    * Remove extra delta already included in #10697
    
    * Don't worry about setting creator for invite
    
    * Only iterate over rows missing the creator
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898
    
    * Use constant to fetch room creator field
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029
    
    * More protection from other random types
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853
    
    * Move new background update to end of list
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181
    
    * Fix query casing
    
    * Fix ambiguity iterating over cursor instead of list
    
    Fix `psycopg2.ProgrammingError: no results to fetch` error
    when tests run with Postgres.
    
    ```
    SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room
    ```
    
    ---
    
    We use `txn.fetchall` because it will return the results as a
    list or an empty list when there are no results.
    
    Docs:
    
    > `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used:
    >
    > https://www.psycopg.org/docs/cursor.html#cursor-iterable
    
    And I'm guessing iterating over a raw cursor does something weird when there are no results.
    
    ---
    
    Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530
    ```
    tests.test_visibility.FilterEventsForServerTestCase.test_large_room
    ===============================================================================
    [FAIL]
    Traceback (most recent call last):
      File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column
        self.get_success(
      File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success
        return self.successResultOf(d)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf
        self.fail(
    twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead:
    Traceback (most recent call last):
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback
        self._startRunCallbacks(fail)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
        self._runCallbacks()
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
        current.result = callback(  # type: ignore[misc]
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult
        current_context.run(_inlineCallbacks, r, gen, status)
    --- <exception caught here> ---
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
        result = current_context.run(
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
        return g.throw(self.type, self.value, self.tb)
      File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update
        await self._do_background_update(desired_duration_ms)
      File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update
        items_updated = await update_handler(progress, batch_size)
      File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column
        end = await self.db_pool.runInteraction(
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction
        result = await self.runWithConnection(
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection
        return await make_deferred_yieldable(
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
        current.result = callback(  # type: ignore[misc]
      File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda>
        d.addCallback(lambda x: function(*args, **kwargs))
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
        compat.reraise(excValue, excTraceback)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
        return function(*args, **kwargs)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
        raise exception.with_traceback(traceback)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
        result = func(conn, *args, **kw)
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func
        return func(db_conn, *args, **kwargs)
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction
        r = func(cursor, *args, **kwargs)
      File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn
        for room_id, event_json in txn:
    psycopg2.ProgrammingError: no results to fetch
    ```
    
    * Move code not under the MSC2716 room version underneath an experimental config option
    
    See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909
    
    * Add ordering to rooms creator background update
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277
    
    * Add comment to better document constant
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458
    
    * Use constant field
    Allow room creator to send MSC2716 related events in existing room versions (#10566)
    Eric Eastwood authored
    * Allow room creator to send MSC2716 related events in existing room versions
    
    Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869
    
    Restoring `get_create_event_for_room_txn` from,
    https://github.com/matrix-org/synapse/pull/10245/commits/44bb3f0cf5cb365ef9281554daceeecfb17cc94d
    
    * Add changelog
    
    * Stop people from trying to redact MSC2716 events in unsupported room versions
    
    * Populate rooms.creator column for easy lookup
    
    > From some [out of band discussion](https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`).
    >
    > https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642
    
    * Remove and switch away from get_create_event_for_room_txn
    
    * Fix no create event being found because no state events persisted yet
    
    * Fix and add tests for rooms creator bg update
    
    * Populate rooms.creator field for easy lookup
    
    Part of https://github.com/matrix-org/synapse/pull/10566
    
     - Fill in creator whenever we insert into the rooms table
     - Add background update to backfill any missing creator values
    
    * Add changelog
    
    * Fix usage
    
    * Remove extra delta already included in #10697
    
    * Don't worry about setting creator for invite
    
    * Only iterate over rows missing the creator
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898
    
    * Use constant to fetch room creator field
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029
    
    * More protection from other random types
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853
    
    * Move new background update to end of list
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181
    
    * Fix query casing
    
    * Fix ambiguity iterating over cursor instead of list
    
    Fix `psycopg2.ProgrammingError: no results to fetch` error
    when tests run with Postgres.
    
    ```
    SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room
    ```
    
    ---
    
    We use `txn.fetchall` because it will return the results as a
    list or an empty list when there are no results.
    
    Docs:
    
    > `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used:
    >
    > https://www.psycopg.org/docs/cursor.html#cursor-iterable
    
    And I'm guessing iterating over a raw cursor does something weird when there are no results.
    
    ---
    
    Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530
    ```
    tests.test_visibility.FilterEventsForServerTestCase.test_large_room
    ===============================================================================
    [FAIL]
    Traceback (most recent call last):
      File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column
        self.get_success(
      File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success
        return self.successResultOf(d)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf
        self.fail(
    twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead:
    Traceback (most recent call last):
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback
        self._startRunCallbacks(fail)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
        self._runCallbacks()
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
        current.result = callback(  # type: ignore[misc]
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult
        current_context.run(_inlineCallbacks, r, gen, status)
    --- <exception caught here> ---
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
        result = current_context.run(
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
        return g.throw(self.type, self.value, self.tb)
      File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update
        await self._do_background_update(desired_duration_ms)
      File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update
        items_updated = await update_handler(progress, batch_size)
      File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column
        end = await self.db_pool.runInteraction(
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction
        result = await self.runWithConnection(
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection
        return await make_deferred_yieldable(
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
        current.result = callback(  # type: ignore[misc]
      File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda>
        d.addCallback(lambda x: function(*args, **kwargs))
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
        compat.reraise(excValue, excTraceback)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
        return function(*args, **kwargs)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
        raise exception.with_traceback(traceback)
      File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
        result = func(conn, *args, **kw)
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func
        return func(db_conn, *args, **kwargs)
      File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction
        r = func(cursor, *args, **kwargs)
      File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn
        for room_id, event_json in txn:
    psycopg2.ProgrammingError: no results to fetch
    ```
    
    * Move code not under the MSC2716 room version underneath an experimental config option
    
    See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909
    
    * Add ordering to rooms creator background update
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277
    
    * Add comment to better document constant
    
    See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458
    
    * Use constant field