Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • tedomum/matrix-media-repo
1 result
Show changes
Commits on Source (49)
Hide whitespace changes
Inline Side-by-side
Showing
with 658 additions and 114 deletions
.github/workflows/main.yml
View file @ 85fabea5
......@@ -44,5 +44,5 @@ jobs:
- name: "Run: compile assets"
run: "$PWD/bin/compile_assets"
- name: "Run: tests"
run: "go test -c -v ./test && ./test.test '-test.v'" # cheat and work around working directory issues
run: "go test -c -v ./test && ./test.test '-test.v' -test.parallel 1" # cheat and work around working directory issues
timeout-minutes: 30
CHANGELOG.md
View file @ 85fabea5
......@@ -7,9 +7,53 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
## [Unreleased]
*Nothing yet.*
*Nothing yet*
## [v1.3.4] - February 9, 2024
## [1.3.7] - July 30, 2024
### Added
* A new global config option, `repo.freezeUnauthenticatedMedia`, is supported to enact the unauthenticated media freeze early. See `config.sample.yaml` for details.
### Changed
* The default leaky bucket capacity has changed from 300mb to 500mb, allowing for more downloads to go through. The drain rate and overflow limit are unchanged (5mb/minute and 100mb respectively).
## [1.3.6] - July 10, 2024
### Fixed
* Ensure a `boundary` is set on federation downloads, allowing the download to work.
## [1.3.5] - July 10, 2024
### Added
* New datastore option to ignore Redis cache when downloading media served by a `publicBaseUrl`. This can help ensure more requests get redirected to the CDN.
* `HEAD /download` is now supported, as per [MSC4120](https://github.com/matrix-org/matrix-spec-proposals/pull/4120).
* S3 datastores can now specify a `prefixLength` to improve S3 performance on some providers. See `config.sample.yaml` for details.
* Add `multipartUploads` flag for running MMR against unsupported S3 providers. See `config.sample.yaml` for details.
* A new "leaky bucket" rate limit algorithm has been applied to downloads. See `rateLimit.buckets` in `config.sample.yaml` for details.
* Add support for [MSC3916: Authentication for media](https://github.com/matrix-org/matrix-spec-proposals/pull/3916).
* To enable full support, use `signingKeyPath` in your config. See `config.sample.yaml` for details.
* Server operators should point `/_matrix/client/v1/media/*` and `/_matrix/federation/v1/media/*` at MMR.
### Changed
* The leaky bucket rate limiting introduced above is turned on by default. Administrators are encouraged to review the default settings and adjust as needed.
### Fixed
* Metrics for redirected and HTML requests are tracked.
* Fixed more issues relating to non-dimensional media being thumbnailed (`invalid image size: 0x0` errors).
* Long-running purge requests no longer fail when the requesting client times out. They are continued in the background.
* Purging old media has been fixed to actually identify old media.
* JPEG thumbnails will now use sensible extensions.
* Fixed directory permissions when exporting MMR to Synapse.
* In some rare cases, memory usage may have leaked due to thumbnail error handling. This has been fixed.
* Synapse signing keys with blank lines can now be decoded/combined with other keys.
## [1.3.4] - February 9, 2024
### Added
......@@ -496,7 +540,10 @@ a large database (more than about 100k uploaded files), run the following steps
* Various other features that would be expected like maximum/minimum size controls, rate limiting, etc. Check out the
sample config for a better idea of what else is possible.
[unreleased]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.4...HEAD
[unreleased]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.7...HEAD
[1.3.7]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.6...v1.3.7
[1.3.6]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.5...v1.3.6
[1.3.5]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.4...v1.3.5
[1.3.4]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.3...v1.3.4
[1.3.3]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.2...v1.3.3
[1.3.2]: https://github.com/t2bot/matrix-media-repo/compare/v1.3.1...v1.3.2
......
api/_apimeta/auth.go
View file @ 85fabea5
......@@ -16,6 +16,19 @@ type UserInfo struct {
IsShared bool
}
type ServerInfo struct {
ServerName string
}
type AuthContext struct {
User UserInfo
Server ServerInfo
}
func (a AuthContext) IsAuthenticated() bool {
return a.User.UserId != "" || a.Server.ServerName != ""
}
func GetRequestUserAdminStatus(r *http.Request, rctx rcontext.RequestContext, user UserInfo) (bool, bool) {
isGlobalAdmin := util.IsGlobalAdmin(user.UserId) || user.IsShared
isLocalAdmin, err := matrix.IsUserAdmin(rctx, r.Host, user.AccessToken, r.RemoteAddr)
......
api/_auth_cache/auth_cache.go
View file @ 85fabea5
......@@ -12,7 +12,7 @@ import (
"github.com/t2bot/matrix-media-repo/matrix"
)
var tokenCache = cache.New(0*time.Second, 30*time.Second)
var tokenCache = cache.New(cache.NoExpiration, 30*time.Second)
var rwLock = &sync.RWMutex{}
var regexCache = make(map[string]*regexp.Regexp)
......
api/_routers/97-require-server-auth.go 0 → 100644
View file @ 85fabea5
package _routers
import (
"errors"
"net/http"
"github.com/t2bot/matrix-media-repo/api/_apimeta"
"github.com/t2bot/matrix-media-repo/api/_responses"
"github.com/t2bot/matrix-media-repo/common"
"github.com/t2bot/matrix-media-repo/common/rcontext"
"github.com/t2bot/matrix-media-repo/matrix"
)
type GeneratorWithServerFn = func(r *http.Request, ctx rcontext.RequestContext, server _apimeta.ServerInfo) interface{}
func RequireServerAuth(generator GeneratorWithServerFn) GeneratorFn {
return func(r *http.Request, ctx rcontext.RequestContext) interface{} {
serverName, err := matrix.ValidateXMatrixAuth(r, true)
if err != nil {
ctx.Log.Debug("Error with X-Matrix auth: ", err)
if errors.Is(err, matrix.ErrNoXMatrixAuth) {
return &_responses.ErrorResponse{
Code: common.ErrCodeUnauthorized,
Message: "no auth provided (required)",
InternalCode: common.ErrCodeMissingToken,
}
}
if errors.Is(err, matrix.ErrWrongDestination) {
return &_responses.ErrorResponse{
Code: common.ErrCodeUnauthorized,
Message: "no auth provided for this destination (required)",
InternalCode: common.ErrCodeBadRequest,
}
}
return &_responses.ErrorResponse{
Code: common.ErrCodeForbidden,
Message: "invalid auth provided (required)",
InternalCode: common.ErrCodeBadRequest,
}
}
return generator(r, ctx, _apimeta.ServerInfo{
ServerName: serverName,
})
}
}
api/_routers/98-use-rcontext.go
View file @ 85fabea5
......@@ -35,6 +35,12 @@ func NewRContextRouter(generatorFn GeneratorFn, next http.Handler) *RContextRout
}
func (c *RContextRouter) ServeHTTP(w http.ResponseWriter, r *http.Request) {
defer func() {
if c.next != nil {
c.next.ServeHTTP(w, r)
}
}()
log := GetLogger(r)
rctx := rcontext.RequestContext{
Context: r.Context(),
......@@ -95,20 +101,24 @@ func (c *RContextRouter) ServeHTTP(w http.ResponseWriter, r *http.Request) {
beforeParseDownload:
log.Infof("Replying with result: %T %+v", res, res)
if downloadRes, isDownload := res.(*_responses.DownloadResponse); isDownload {
ranges, err := http_range.ParseRange(r.Header.Get("Range"), downloadRes.SizeBytes, rctx.Config.Downloads.DefaultRangeChunkSizeBytes)
if errors.Is(err, http_range.ErrInvalid) {
proposedStatusCode = http.StatusRequestedRangeNotSatisfiable
res = _responses.BadRequest("invalid range header")
goto beforeParseDownload // reprocess `res`
} else if errors.Is(err, http_range.ErrNoOverlap) {
proposedStatusCode = http.StatusRequestedRangeNotSatisfiable
res = _responses.BadRequest("out of range")
goto beforeParseDownload // reprocess `res`
}
if len(ranges) > 1 {
proposedStatusCode = http.StatusRequestedRangeNotSatisfiable
res = _responses.BadRequest("only 1 range is supported")
goto beforeParseDownload // reprocess `res`
var ranges []http_range.Range
var err error
if downloadRes.SizeBytes > 0 {
ranges, err = http_range.ParseRange(r.Header.Get("Range"), downloadRes.SizeBytes, rctx.Config.Downloads.DefaultRangeChunkSizeBytes)
if errors.Is(err, http_range.ErrInvalid) {
proposedStatusCode = http.StatusRequestedRangeNotSatisfiable
res = _responses.BadRequest("invalid range header")
goto beforeParseDownload // reprocess `res`
} else if errors.Is(err, http_range.ErrNoOverlap) {
proposedStatusCode = http.StatusRequestedRangeNotSatisfiable
res = _responses.BadRequest("out of range")
goto beforeParseDownload // reprocess `res`
}
if len(ranges) > 1 {
proposedStatusCode = http.StatusRequestedRangeNotSatisfiable
res = _responses.BadRequest("only 1 range is supported")
goto beforeParseDownload // reprocess `res`
}
}
contentType = downloadRes.ContentType
......@@ -252,10 +262,6 @@ beforeParseDownload:
if expectedBytes > 0 && written != expectedBytes {
panic(errors.New(fmt.Sprintf("mismatch transfer size: %d expected, %d sent", expectedBytes, written)))
}
if c.next != nil {
c.next.ServeHTTP(w, r)
}
}
func GetStatusCode(r *http.Request) int {
......
api/custom/federation.go
View file @ 85fabea5
......@@ -33,7 +33,10 @@ func GetFederationInfo(r *http.Request, rctx rcontext.RequestContext, user _apim
}
versionUrl := url + "/_matrix/federation/v1/version"
versionResponse, err := matrix.FederatedGet(versionUrl, hostname, rctx)
versionResponse, err := matrix.FederatedGet(rctx, versionUrl, hostname, serverName, matrix.NoSigningKey)
if versionResponse != nil {
defer versionResponse.Body.Close()
}
if err != nil {
rctx.Log.Error(err)
sentry.CaptureException(err)
......
api/custom/purge.go
View file @ 85fabea5
......@@ -141,13 +141,13 @@ func PurgeOldMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta.
"include_local": includeLocal,
})
domains := make([]string, 0)
excludeDomains := make([]string, 0)
if !includeLocal {
domains = util.GetOurDomains()
excludeDomains = util.GetOurDomains()
}
mediaDb := database.GetInstance().Media.Prepare(rctx)
records, err := mediaDb.GetOldExcluding(domains, beforeTs)
records, err := mediaDb.GetOldExcluding(excludeDomains, beforeTs)
if err != nil {
rctx.Log.Error(err)
sentry.CaptureException(err)
......
api/r0/download.go
View file @ 85fabea5
......@@ -18,7 +18,11 @@ import (
"github.com/t2bot/matrix-media-repo/common/rcontext"
)
func DownloadMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
func DownloadMediaUser(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
return DownloadMedia(r, rctx, _apimeta.AuthContext{User: user})
}
func DownloadMedia(r *http.Request, rctx rcontext.RequestContext, auth _apimeta.AuthContext) interface{} {
server := _routers.GetParam("server", r)
mediaId := _routers.GetParam("mediaId", r)
filename := _routers.GetParam("filename", r)
......@@ -53,30 +57,51 @@ func DownloadMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta.
return _responses.BadRequest("timeout_ms does not appear to be an integer")
}
recordOnly := false
if r.Method == http.MethodHead {
rctx.Log.Debug("HEAD request received - changing parameters")
//downloadRemote = false // we allow the download to go through to ensure proper metadata is returned
recordOnly = true
}
rctx = rctx.LogWithFields(logrus.Fields{
"mediaId": mediaId,
"server": server,
"filename": filename,
"allowRemote": downloadRemote,
"allowRedirect": canRedirect,
"mediaId": mediaId,
"server": server,
"filename": filename,
"allowRemote": downloadRemote,
"allowRedirect": canRedirect,
"authUserId": auth.User.UserId,
"authServerName": auth.Server.ServerName,
})
if !util.IsGlobalAdmin(user.UserId) && util.IsHostIgnored(server) {
rctx.Log.Warn("Request blocked due to domain being ignored.")
return _responses.MediaBlocked()
if auth.User.UserId != "" {
if !util.IsGlobalAdmin(auth.User.UserId) && util.IsHostIgnored(server) {
rctx.Log.Warn("Request blocked due to domain being ignored.")
return _responses.MediaBlocked()
}
}
media, stream, err := pipeline_download.Execute(rctx, server, mediaId, pipeline_download.DownloadOpts{
FetchRemoteIfNeeded: downloadRemote,
BlockForReadUntil: blockFor,
CanRedirect: canRedirect,
RecordOnly: recordOnly,
AuthProvided: auth.IsAuthenticated(),
})
if err != nil {
var redirect datastores.RedirectError
if errors.Is(err, common.ErrMediaNotFound) {
return _responses.NotFoundError()
} else if errors.Is(err, common.ErrRestrictedAuth) {
return _responses.ErrorResponse{
Code: common.ErrCodeNotFound,
Message: "authentication is required to download this media",
InternalCode: common.ErrCodeUnauthorized,
}
} else if errors.Is(err, common.ErrMediaTooLarge) {
return _responses.RequestTooLarge()
} else if errors.Is(err, common.ErrRateLimitExceeded) {
return _responses.RateLimitReached()
} else if errors.Is(err, common.ErrMediaQuarantined) {
rctx.Log.Debug("Quarantined media accessed. Has stream? ", stream != nil)
if stream != nil {
......
api/r0/thumbnail.go
View file @ 85fabea5
......@@ -20,7 +20,11 @@ import (
"github.com/t2bot/matrix-media-repo/common/rcontext"
)
func ThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
func ThumbnailMediaUser(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
return ThumbnailMedia(r, rctx, _apimeta.AuthContext{User: user})
}
func ThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, auth _apimeta.AuthContext) interface{} {
server := _routers.GetParam("server", r)
mediaId := _routers.GetParam("mediaId", r)
allowRemote := r.URL.Query().Get("allow_remote")
......@@ -55,15 +59,19 @@ func ThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta
}
rctx = rctx.LogWithFields(logrus.Fields{
"mediaId": mediaId,
"server": server,
"allowRemote": downloadRemote,
"allowRedirect": canRedirect,
"mediaId": mediaId,
"server": server,
"allowRemote": downloadRemote,
"allowRedirect": canRedirect,
"authUserId": auth.User.UserId,
"authServerName": auth.Server.ServerName,
})
if !util.IsGlobalAdmin(user.UserId) && util.IsHostIgnored(server) {
rctx.Log.Warn("Request blocked due to domain being ignored.")
return _responses.MediaBlocked()
if auth.User.UserId != "" {
if !util.IsGlobalAdmin(auth.User.UserId) && util.IsHostIgnored(server) {
rctx.Log.Warn("Request blocked due to domain being ignored.")
return _responses.MediaBlocked()
}
}
widthStr := r.URL.Query().Get("width")
......@@ -124,6 +132,7 @@ func ThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta
BlockForReadUntil: blockFor,
RecordOnly: false, // overridden
CanRedirect: canRedirect,
AuthProvided: auth.IsAuthenticated(),
},
Width: width,
Height: height,
......@@ -134,8 +143,16 @@ func ThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta
var redirect datastores.RedirectError
if errors.Is(err, common.ErrMediaNotFound) {
return _responses.NotFoundError()
} else if errors.Is(err, common.ErrRestrictedAuth) {
return _responses.ErrorResponse{
Code: common.ErrCodeNotFound,
Message: "authentication is required to download this media",
InternalCode: common.ErrCodeUnauthorized,
}
} else if errors.Is(err, common.ErrMediaTooLarge) {
return _responses.RequestTooLarge()
} else if errors.Is(err, common.ErrRateLimitExceeded) {
return _responses.RateLimitReached()
} else if errors.Is(err, common.ErrMediaQuarantined) {
rctx.Log.Debug("Quarantined media accessed. Has stream? ", stream != nil)
if stream != nil {
......
api/r0/versions.go 0 → 100644
View file @ 85fabea5
package r0
import (
"net/http"
"slices"
"github.com/getsentry/sentry-go"
"github.com/t2bot/matrix-media-repo/api/_apimeta"
"github.com/t2bot/matrix-media-repo/api/_responses"
"github.com/t2bot/matrix-media-repo/matrix"
"github.com/t2bot/matrix-media-repo/common/rcontext"
)
func ClientVersions(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
versions, err := matrix.ClientVersions(rctx, r.Host, user.UserId, user.AccessToken, r.RemoteAddr)
if err != nil {
rctx.Log.Error(err)
sentry.CaptureException(err)
return _responses.InternalServerError("unable to get versions")
}
// This is where we'd add our feature/version support as needed
if versions.Versions == nil {
versions.Versions = make([]string, 1)
}
// We add v1.11 by force, even though we can't reliably say the rest of the server implements it. This
// is because server admins which point `/versions` at us are effectively opting in to whatever features
// we need to advertise support for. In our case, it's at least Authenticated Media (MSC3916).
if !slices.Contains(versions.Versions, "v1.11") {
versions.Versions = append(versions.Versions, "v1.11")
}
return versions
}
api/routes.go
View file @ 85fabea5
......@@ -18,6 +18,7 @@ import (
const PrefixMedia = "/_matrix/media"
const PrefixClient = "/_matrix/client"
const PrefixFederation = "/_matrix/federation"
func buildRoutes() http.Handler {
counter := &_routers.RequestCounter{}
......@@ -32,16 +33,27 @@ func buildRoutes() http.Handler {
// Standard (spec) features
register([]string{"PUT"}, PrefixMedia, "upload/:server/:mediaId", mxV3, router, makeRoute(_routers.RequireAccessToken(r0.UploadMediaAsync), "upload_async", counter))
register([]string{"POST"}, PrefixMedia, "upload", mxSpecV3Transition, router, makeRoute(_routers.RequireAccessToken(r0.UploadMediaSync), "upload", counter))
downloadRoute := makeRoute(_routers.OptionalAccessToken(r0.DownloadMedia), "download", counter)
register([]string{"GET"}, PrefixMedia, "download/:server/:mediaId/:filename", mxSpecV3Transition, router, downloadRoute)
register([]string{"GET"}, PrefixMedia, "download/:server/:mediaId", mxSpecV3Transition, router, downloadRoute)
register([]string{"GET"}, PrefixMedia, "thumbnail/:server/:mediaId", mxSpecV3Transition, router, makeRoute(_routers.OptionalAccessToken(r0.ThumbnailMedia), "thumbnail", counter))
register([]string{"GET"}, PrefixMedia, "preview_url", mxSpecV3TransitionCS, router, makeRoute(_routers.RequireAccessToken(r0.PreviewUrl), "url_preview", counter))
downloadRoute := makeRoute(_routers.OptionalAccessToken(r0.DownloadMediaUser), "download", counter)
register([]string{"GET", "HEAD"}, PrefixMedia, "download/:server/:mediaId/:filename", mxSpecV3Transition, router, downloadRoute)
register([]string{"GET", "HEAD"}, PrefixMedia, "download/:server/:mediaId", mxSpecV3Transition, router, downloadRoute)
register([]string{"GET"}, PrefixMedia, "thumbnail/:server/:mediaId", mxSpecV3Transition, router, makeRoute(_routers.OptionalAccessToken(r0.ThumbnailMediaUser), "thumbnail", counter))
previewUrlRoute := makeRoute(_routers.RequireAccessToken(r0.PreviewUrl), "url_preview", counter)
register([]string{"GET"}, PrefixMedia, "preview_url", mxSpecV3TransitionCS, router, previewUrlRoute)
register([]string{"GET"}, PrefixMedia, "identicon/*seed", mxR0, router, makeRoute(_routers.OptionalAccessToken(r0.Identicon), "identicon", counter))
register([]string{"GET"}, PrefixMedia, "config", mxSpecV3TransitionCS, router, makeRoute(_routers.RequireAccessToken(r0.PublicConfig), "config", counter))
configRoute := makeRoute(_routers.RequireAccessToken(r0.PublicConfig), "config", counter)
register([]string{"GET"}, PrefixMedia, "config", mxSpecV3TransitionCS, router, configRoute)
register([]string{"POST"}, PrefixClient, "logout", mxSpecV3TransitionCS, router, makeRoute(_routers.RequireAccessToken(r0.Logout), "logout", counter))
register([]string{"POST"}, PrefixClient, "logout/all", mxSpecV3TransitionCS, router, makeRoute(_routers.RequireAccessToken(r0.LogoutAll), "logout_all", counter))
register([]string{"POST"}, PrefixMedia, "create", mxV1, router, makeRoute(_routers.RequireAccessToken(v1.CreateMedia), "create", counter))
register([]string{"GET"}, PrefixClient, "versions", mxNoVersion, router, makeRoute(_routers.OptionalAccessToken(r0.ClientVersions), "client_versions", counter))
register([]string{"GET"}, PrefixClient, "media/preview_url", mxV1, router, previewUrlRoute)
register([]string{"GET"}, PrefixClient, "media/config", mxV1, router, configRoute)
authedDownloadRoute := makeRoute(_routers.RequireAccessToken(v1.ClientDownloadMedia), "download", counter)
register([]string{"GET"}, PrefixClient, "media/download/:server/:mediaId/:filename", mxV1, router, authedDownloadRoute)
register([]string{"GET"}, PrefixClient, "media/download/:server/:mediaId", mxV1, router, authedDownloadRoute)
register([]string{"GET"}, PrefixClient, "media/thumbnail/:server/:mediaId", mxV1, router, makeRoute(_routers.RequireAccessToken(v1.ClientThumbnailMedia), "thumbnail", counter))
register([]string{"GET"}, PrefixFederation, "media/download/:mediaId", mxV1, router, makeRoute(_routers.RequireServerAuth(v1.FederationDownloadMedia), "download", counter))
register([]string{"GET"}, PrefixFederation, "media/thumbnail/:mediaId", mxV1, router, makeRoute(_routers.RequireServerAuth(v1.FederationThumbnailMedia), "thumbnail", counter))
// Custom features
register([]string{"GET"}, PrefixMedia, "local_copy/:server/:mediaId", mxUnstable, router, makeRoute(_routers.RequireAccessToken(unstable.LocalCopy), "local_copy", counter))
......@@ -134,12 +146,16 @@ var (
mxR0 matrixVersions = []string{"r0"}
mxV1 matrixVersions = []string{"v1"}
mxV3 matrixVersions = []string{"v3"}
mxNoVersion matrixVersions = []string{""}
)
func register(methods []string, prefix string, postfix string, versions matrixVersions, router *httprouter.Router, handler http.Handler) {
for _, method := range methods {
for _, version := range versions {
path := fmt.Sprintf("%s/%s/%s", prefix, version, postfix)
if version == "" {
path = fmt.Sprintf("%s/%s", prefix, postfix)
}
router.Handler(method, path, http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
defer func() {
// hopefully the body was already closed, but maybe it wasn't
......
api/v1/download.go 0 → 100644
View file @ 85fabea5
package v1
import (
"bytes"
"net/http"
"github.com/t2bot/matrix-media-repo/util/ids"
"github.com/t2bot/matrix-media-repo/api/_apimeta"
"github.com/t2bot/matrix-media-repo/api/_responses"
"github.com/t2bot/matrix-media-repo/api/_routers"
"github.com/t2bot/matrix-media-repo/api/r0"
"github.com/t2bot/matrix-media-repo/common/rcontext"
"github.com/t2bot/matrix-media-repo/util/readers"
)
func ClientDownloadMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
r.URL.Query().Set("allow_remote", "true")
r.URL.Query().Set("allow_redirect", "true")
return r0.DownloadMedia(r, rctx, _apimeta.AuthContext{User: user})
}
func FederationDownloadMedia(r *http.Request, rctx rcontext.RequestContext, server _apimeta.ServerInfo) interface{} {
query := r.URL.Query()
query.Set("allow_remote", "false")
query.Set("allow_redirect", "true") // we override how redirects work in the response
r.URL.RawQuery = query.Encode()
r = _routers.ForceSetParam("server", r.Host, r)
res := r0.DownloadMedia(r, rctx, _apimeta.AuthContext{Server: server})
boundary, err := ids.NewUniqueId()
if err != nil {
rctx.Log.Error("Error generating boundary on response: ", err)
return _responses.InternalServerError("unable to generate boundary")
}
if dl, ok := res.(*_responses.DownloadResponse); ok {
return &_responses.DownloadResponse{
ContentType: "multipart/mixed; boundary=" + boundary,
Filename: "",
SizeBytes: 0,
Data: readers.NewMultipartReader(
boundary,
&readers.MultipartPart{ContentType: "application/json", Reader: readers.MakeCloser(bytes.NewReader([]byte("{}")))},
&readers.MultipartPart{ContentType: dl.ContentType, FileName: dl.Filename, Reader: dl.Data},
),
TargetDisposition: "attachment",
}
} else if rd, ok := res.(*_responses.RedirectResponse); ok {
return &_responses.DownloadResponse{
ContentType: "multipart/mixed; boundary=" + boundary,
Filename: "",
SizeBytes: 0,
Data: readers.NewMultipartReader(
boundary,
&readers.MultipartPart{ContentType: "application/json", Reader: readers.MakeCloser(bytes.NewReader([]byte("{}")))},
&readers.MultipartPart{Location: rd.ToUrl},
),
TargetDisposition: "attachment",
}
} else {
return res
}
}
api/v1/thumbnail.go 0 → 100644
View file @ 85fabea5
package v1
import (
"bytes"
"net/http"
"github.com/t2bot/matrix-media-repo/util/ids"
"github.com/t2bot/matrix-media-repo/api/_apimeta"
"github.com/t2bot/matrix-media-repo/api/_responses"
"github.com/t2bot/matrix-media-repo/api/_routers"
"github.com/t2bot/matrix-media-repo/api/r0"
"github.com/t2bot/matrix-media-repo/common/rcontext"
"github.com/t2bot/matrix-media-repo/util/readers"
)
func ClientThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, user _apimeta.UserInfo) interface{} {
r.URL.Query().Set("allow_remote", "true")
r.URL.Query().Set("allow_redirect", "true")
return r0.ThumbnailMedia(r, rctx, _apimeta.AuthContext{User: user})
}
func FederationThumbnailMedia(r *http.Request, rctx rcontext.RequestContext, server _apimeta.ServerInfo) interface{} {
query := r.URL.Query()
query.Set("allow_remote", "false")
query.Set("allow_redirect", "true") // we override how redirects work in the response
r.URL.RawQuery = query.Encode()
r = _routers.ForceSetParam("server", r.Host, r)
res := r0.ThumbnailMedia(r, rctx, _apimeta.AuthContext{Server: server})
boundary, err := ids.NewUniqueId()
if err != nil {
rctx.Log.Error("Error generating boundary on response: ", err)
return _responses.InternalServerError("unable to generate boundary")
}
if dl, ok := res.(*_responses.DownloadResponse); ok {
return &_responses.DownloadResponse{
ContentType: "multipart/mixed; boundary=" + boundary,
Filename: "",
SizeBytes: 0,
Data: readers.NewMultipartReader(
boundary,
&readers.MultipartPart{ContentType: "application/json", Reader: readers.MakeCloser(bytes.NewReader([]byte("{}")))},
&readers.MultipartPart{ContentType: dl.ContentType, FileName: dl.Filename, Reader: dl.Data},
),
TargetDisposition: "attachment",
}
} else if rd, ok := res.(*_responses.RedirectResponse); ok {
return &_responses.DownloadResponse{
ContentType: "multipart/mixed; boundary=" + boundary,
Filename: "",
SizeBytes: 0,
Data: readers.NewMultipartReader(
boundary,
&readers.MultipartPart{ContentType: "application/json", Reader: readers.MakeCloser(bytes.NewReader([]byte("{}")))},
&readers.MultipartPart{Location: rd.ToUrl},
),
TargetDisposition: "attachment",
}
} else {
return res
}
}
api/webserver.go
View file @ 85fabea5
......@@ -2,7 +2,6 @@ package api
import (
"context"
"encoding/json"
"errors"
"net"
"net/http"
......@@ -10,12 +9,12 @@ import (
"sync"
"time"
"github.com/didip/tollbooth"
"github.com/didip/tollbooth/v7"
"github.com/getsentry/sentry-go"
sentryhttp "github.com/getsentry/sentry-go/http"
"github.com/sirupsen/logrus"
"github.com/t2bot/matrix-media-repo/api/_responses"
"github.com/t2bot/matrix-media-repo/common/config"
"github.com/t2bot/matrix-media-repo/limits"
)
var srv *http.Server
......@@ -35,17 +34,7 @@ func Init() *sync.WaitGroup {
if config.Get().RateLimit.Enabled {
logrus.Debug("Enabling rate limit")
limiter := tollbooth.NewLimiter(0, nil)
limiter.SetIPLookups([]string{"X-Forwarded-For", "X-Real-IP", "RemoteAddr"})
limiter.SetTokenBucketExpirationTTL(time.Hour)
limiter.SetBurst(config.Get().RateLimit.BurstCount)
limiter.SetMax(config.Get().RateLimit.RequestsPerSecond)
b, _ := json.Marshal(_responses.RateLimitReached())
limiter.SetMessage(string(b))
limiter.SetMessageContentType("application/json")
handler = tollbooth.LimitHandler(limiter, handler)
handler = tollbooth.LimitHandler(limits.GetRequestLimiter(), handler)
}
// Note: we bind Sentry here to ensure we capture *everything*
......
archival/entity_export.go
View file @ 85fabea5
......@@ -44,6 +44,7 @@ func ExportEntityData(ctx rcontext.RequestContext, exportId string, entityId str
FetchRemoteIfNeeded: false,
BlockForReadUntil: 10 * time.Minute,
RecordOnly: false,
AuthProvided: true, // it's for an export, so assume authentication
})
if errors.Is(err, common.ErrMediaQuarantined) {
ctx.Log.Warnf("%s is quarantined and will not be included in the export", mxc)
......
assets/providers.json
View file @ 85fabea5
......@@ -11,6 +11,19 @@
}
]
},
{
"provider_name": "3Q",
"provider_url": "https://3q.video/",
"endpoints": [
{
"schemes": [
"https://playout.3qsdn.com/embed/*"
],
"url": "https://playout.3qsdn.com/oembed",
"discovery": true
}
]
},
{
"provider_name": "Abraia",
"provider_url": "https://abraia.me",
......@@ -136,10 +149,12 @@
{
"schemes": [
"https://amtraker.com/trains/*",
"https://beta.amtraker.com/trains/*"
"https://amtraker.com/trains/*/*",
"https://*.amtraker.com/trains/*",
"https://*.amtraker.com/trains/*/*"
],
"url": "https://api.amtraker.com/v2/oembed",
"discovery": false
"url": "https://api.amtraker.com/v3/oembed",
"discovery": true
}
]
},
......@@ -402,6 +417,19 @@
}
]
},
{
"provider_name": "biqnetwork",
"provider_url": "https://biqapp.com/",
"endpoints": [
{
"schemes": [
"https://cloud.biqapp.com/*"
],
"url": "https://biqapp.com/api/v1/video/oembed",
"discovery": true
}
]
},
{
"provider_name": "Blackfire.io",
"provider_url": "https://blackfire.io",
......@@ -430,6 +458,19 @@
}
]
},
{
"provider_name": "Bluesky Social",
"provider_url": "https://bsky.app",
"endpoints": [
{
"schemes": [
"https://bsky.app/profile/*/post/*"
],
"url": "https://embed.bsky.app/oembed",
"discovery": true
}
]
},
{
"provider_name": "Bookingmood",
"provider_url": "https://www.bookingmood.com",
......@@ -482,6 +523,25 @@
}
]
},
{
"provider_name": "Bunny",
"provider_url": "https://bunny.net/",
"endpoints": [
{
"schemes": [
"https://iframe.mediadelivery.net/*",
"http://iframe.mediadelivery.net/*",
"https://video.bunnycdn.com/*",
"http://video.bunnycdn.com/*"
],
"url": "https://video.bunnycdn.com/OEmbed",
"formats": [
"json"
],
"discovery": true
}
]
},
{
"provider_name": "Buttondown",
"provider_url": "https://buttondown.email/",
......@@ -565,6 +625,22 @@
}
]
},
{
"provider_name": "Celero",
"provider_url": "https://www.celero.io",
"endpoints": [
{
"schemes": [
"https://embeds.celero.io/*"
],
"url": "https://api.celero.io/api/oembed",
"discovery": true,
"formats": [
"json"
]
}
]
},
{
"provider_name": "Ceros",
"provider_url": "http://www.ceros.com/",
......@@ -869,7 +945,8 @@
"endpoints": [
{
"schemes": [
"https://www.dailymotion.com/video/*"
"https://www.dailymotion.com/video/*",
"https://geo.dailymotion.com/player.html?video=*"
],
"url": "https://www.dailymotion.com/services/oembed",
"discovery": true
......@@ -1223,6 +1300,19 @@
}
]
},
{
"provider_name": "Figma",
"provider_url": "https://www.figma.com",
"endpoints": [
{
"schemes": [
"https://www.figma.com/file/*"
],
"url": "https://www.figma.com/api/oembed",
"discovery": true
}
]
},
{
"provider_name": "Firework",
"provider_url": "https://fireworktv.com/",
......@@ -1526,6 +1616,7 @@
"endpoints": [
{
"schemes": [
"https://gumlet.tv/watch/*",
"https://www.gumlet.com/watch/*",
"https://play.gumlet.io/embed/*"
],
......@@ -1635,6 +1726,19 @@
}
]
},
{
"provider_name": "Hopvue",
"provider_url": "https://www.hopvue.com",
"endpoints": [
{
"schemes": [
"https://*.hopvue.com/*"
],
"url": "https://portal.hopvue.com/api/oembed/",
"discovery": true
}
]
},
{
"provider_name": "HuffDuffer",
"provider_url": "http://huffduffer.com",
......@@ -1659,6 +1763,19 @@
}
]
},
{
"provider_name": "Ideamapper",
"provider_url": "https://ideamapper.com/",
"endpoints": [
{
"schemes": [
"https://oembed.ideamapper.com/*"
],
"url": "https://oembed.ideamapper.com/oembed",
"discovery": true
}
]
},
{
"provider_name": "Idomoo",
"provider_url": "https://idomoo.com/",
......@@ -1827,6 +1944,19 @@
}
]
},
{
"provider_name": "Insight Timer",
"provider_url": "https://insighttimer.com/",
"endpoints": [
{
"schemes": [
"https://insighttimer.com/*"
],
"url": "https://widgets.insighttimer.com/services/oembed",
"discovery": true
}
]
},
{
"provider_name": "Instagram",
"provider_url": "https://instagram.com",
......@@ -1893,6 +2023,19 @@
}
]
},
{
"provider_name": "Itabtech infosys",
"provider_url": "https://samay.itabtechinfosys.com/",
"endpoints": [
{
"schemes": [
"https://samay.itabtechinfosys.com/*"
],
"url": "https://samay.itabtechinfosys.com/oembed/",
"discovery": true
}
]
},
{
"provider_name": "itemis CREATE",
"provider_url": "https://play.itemis.io",
......@@ -2469,17 +2612,27 @@
"endpoints": [
{
"schemes": [
"https://ndla.no/*"
"https://ndla.no/*",
"https://ndla.no/article/*",
"https://ndla.no/audio/*",
"https://ndla.no/concept/*",
"https://ndla.no/image/*",
"https://ndla.no/video/*"
],
"url": "https://ndla.no/oembed",
"discovery": false
},
}
]
},
{
"provider_name": "neetoRecord",
"provider_url": "https://neetorecord.com",
"endpoints": [
{
"schemes": [
"https://liste.ndla.no/*"
"https://*.neetorecord.com/watch/*"
],
"url": "https://liste.ndla.no/oembed",
"discovery": false
"url": "https://api.neetorecord.com/api/v1/oembed"
}
]
},
......@@ -3257,6 +3410,33 @@
}
]
},
{
"provider_name": "SharedFile",
"provider_url": "https://shared-file-kappa.vercel.app/file/",
"endpoints": [
{
"schemes": [
"https://shared-file-kappa.vercel.app/file/*"
],
"url": "https://shared-file-kappa.vercel.app/file/api/oembed",
"discovery": true
}
]
},
{
"provider_name": "Shopshare",
"provider_url": "https://shopshare.tv",
"endpoints": [
{
"schemes": [
"https://shopshare.tv/shopboard/*",
"https://shopshare.tv/shopcast/*"
],
"url": "https://shopshare.tv/api/shopcast/oembed",
"discovery": true
}
]
},
{
"provider_name": "ShortNote",
"provider_url": "https://www.shortnote.jp/",
......@@ -3848,6 +4028,18 @@
}
]
},
{
"provider_name": "Trackspace",
"provider_url": "http://trackspace.upitup.com/",
"endpoints": [
{
"schemes": [
"http://trackspace.upitup.com/*"
],
"url": "https://trackspace.upitup.com/oembed"
}
]
},
{
"provider_name": "Trinity Audio",
"provider_url": "https://trinityaudio.ai",
......@@ -4142,7 +4334,12 @@
"schemes": [
"https://share.viostream.com/*"
],
"url": "https://play.viostream.com/oembed"
"url": "https://play.viostream.com/oembed",
"discovery": true,
"formats": [
"json",
"xml"
]
}
]
},
......@@ -4259,6 +4456,21 @@
}
]
},
{
"provider_name": "Web3 is Going Just Great",
"provider_url": "https://www.web3isgoinggreat.com/",
"endpoints": [
{
"schemes": [
"https://www.web3isgoinggreat.com/?id=*",
"https://www.web3isgoinggreat.com/single/*",
"https://www.web3isgoinggreat.com/embed/*"
],
"url": "https://www.web3isgoinggreat.com/api/oembed",
"discovery": true
}
]
},
{
"provider_name": "wecandeo",
"provider_url": "https://www.wecandeo.com/",
......@@ -4388,7 +4600,9 @@
"https://youtu.be/*",
"https://*.youtube.com/playlist?list=*",
"https://youtube.com/playlist?list=*",
"https://*.youtube.com/shorts*"
"https://*.youtube.com/shorts*",
"https://youtube.com/shorts*",
"https://*.youtube.com/embed/*"
],
"url": "https://www.youtube.com/oembed",
"discovery": true
......
cmd/homeserver_offline_exporters/import_to_synapse/main.go
View file @ 85fabea5
......@@ -78,7 +78,7 @@ func main() {
ctx.Log.Infof("Copying %s", mxc)
directories := path.Join(cfg.ExportPath, "local_content", record.MediaId[0:2], record.MediaId[2:4])
err = os.MkdirAll(directories, 0655)
err = os.MkdirAll(directories, 0755)
if err != nil {
return err
}
......@@ -119,6 +119,12 @@ func main() {
thumb, err := thumbnailing.GenerateThumbnail(src, record.ContentType, s.width, s.height, s.method, false, ctx)
if err != nil {
if thumb.Reader != nil {
err2 := thumb.Reader.Close()
if err2 != nil {
ctx.Log.Warn("Non-fatal error cleaning up thumbnail stream: ", err2)
}
}
ctx.Log.Debug("Error generating thumbnail (you can probably ignore this). ", s, err)
return
}
......@@ -134,7 +140,7 @@ func main() {
dirLock.Lock()
defer dirLock.Unlock()
thumbDir := path.Join(cfg.ExportPath, "local_thumbnails", record.MediaId[0:2], record.MediaId[2:4], record.MediaId[4:])
err = os.MkdirAll(thumbDir, 0655)
err = os.MkdirAll(thumbDir, 0755)
if err != nil {
ctx.Log.Warn("Error creating thumbnail directories. ", s, err)
return
......
cmd/utilities/generate_signing_key/main.go
View file @ 85fabea5
package main
import (
"crypto/ed25519"
"crypto/rand"
"flag"
"fmt"
"os"
"sort"
"strings"
"github.com/sirupsen/logrus"
"github.com/t2bot/matrix-media-repo/cmd/utilities/_common"
......@@ -27,16 +22,7 @@ func main() {
if *inputFile != "" {
key, err = decodeKey(*inputFile)
} else {
keyVersion := makeKeyVersion()
var priv ed25519.PrivateKey
_, priv, err = ed25519.GenerateKey(nil)
priv = priv[len(priv)-32:]
key = &homeserver_interop.SigningKey{
PrivateKey: priv,
KeyVersion: keyVersion,
}
key, err = homeserver_interop.GenerateSigningKey()
}
if err != nil {
logrus.Fatal(err)
......@@ -47,28 +33,6 @@ func main() {
_common.EncodeSigningKeys([]*homeserver_interop.SigningKey{key}, *outputFormat, *outputFile)
}
func makeKeyVersion() string {
buf := make([]byte, 2)
chars := strings.Split("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789", "")
for i := 0; i < len(chars); i++ {
sort.Slice(chars, func(i int, j int) bool {
c, err := rand.Read(buf)
// "should never happen" clauses
if err != nil {
panic(err)
}
if c != len(buf) || c != 2 {
panic(fmt.Sprintf("crypto rand read %d bytes, expected %d", c, len(buf)))
}
return buf[0] < buf[1]
})
}
return strings.Join(chars[:6], "")
}
func decodeKey(fileName string) (*homeserver_interop.SigningKey, error) {
f, err := os.Open(fileName)
if err != nil {
......
cmd/workers/media_repo/reloads.go
View file @ 85fabea5
......@@ -9,6 +9,8 @@ import (
"github.com/t2bot/matrix-media-repo/common/runtime"
"github.com/t2bot/matrix-media-repo/database"
"github.com/t2bot/matrix-media-repo/errcache"
"github.com/t2bot/matrix-media-repo/limits"
"github.com/t2bot/matrix-media-repo/matrix"
"github.com/t2bot/matrix-media-repo/metrics"
"github.com/t2bot/matrix-media-repo/pgo_internal"
"github.com/t2bot/matrix-media-repo/plugins"
......@@ -28,7 +30,9 @@ func setupReloads() {
reloadPluginsOnChan(globals.PluginReloadChan)
reloadPoolOnChan(globals.PoolReloadChan)
reloadErrorCachesOnChan(globals.ErrorCacheReloadChan)
reloadMatrixCachesOnChan(globals.MatrixCachesReloadChan)
reloadPGOOnChan(globals.PGOReloadChan)
reloadBucketsOnChan(globals.BucketsReloadChan)
}
func stopReloads() {
......@@ -53,8 +57,12 @@ func stopReloads() {
globals.PoolReloadChan <- false
logrus.Debug("Stopping ErrorCacheReloadChan")
globals.ErrorCacheReloadChan <- false
logrus.Debug("Stopping MatrixCachesReloadChan")
globals.MatrixCachesReloadChan <- false
logrus.Debug("Stopping PGOReloadChan")
globals.PGOReloadChan <- false
logrus.Debug("Stopping BucketsReloadChan")
globals.BucketsReloadChan <- false
}
func reloadWebOnChan(reloadChan chan bool) {
......@@ -203,6 +211,20 @@ func reloadErrorCachesOnChan(reloadChan chan bool) {
}()
}
func reloadMatrixCachesOnChan(reloadChan chan bool) {
go func() {
defer close(reloadChan)
for {
shouldReload := <-reloadChan
if shouldReload {
matrix.FlushSigningKeyCache()
} else {
return // received stop
}
}
}()
}
func reloadPGOOnChan(reloadChan chan bool) {
go func() {
defer close(reloadChan)
......@@ -220,3 +242,17 @@ func reloadPGOOnChan(reloadChan chan bool) {
}
}()
}
func reloadBucketsOnChan(reloadChan chan bool) {
go func() {
defer close(reloadChan)
for {
shouldReload := <-reloadChan
if shouldReload {
limits.ExpandBuckets()
} else {
return // received stop
}
}
}()
}