Adding metric and alert to notify promtail falling behind (#412)
* Adding the metric promtail_file_bytes_total which will report on the total size of each file with an active tailer. Added an alert which will fire whenever the total bytes for a file is 100kb or more ahead the read bytes for more than 15mins. Added an additional alert to track client bytes ready to send (encoded bytes) vs actually sent so we can add an alert on promtail send failures and still track stats on encoded bytes being sent. * Moving where we update the total file bytes outside the tailer so it shouldn't get hung up by tailer issues. * Recording file size based on Glob matches rather than what we are tailing to give even better insight
Showing
- pkg/promtail/client/client.go 9 additions, 1 deletionpkg/promtail/client/client.go
- pkg/promtail/targets/filetarget.go 19 additions, 0 deletionspkg/promtail/targets/filetarget.go
- pkg/promtail/targets/tailer.go 4 additions, 1 deletionpkg/promtail/targets/tailer.go
- production/loki-mixin/alerts.libsonnet 15 additions, 0 deletionsproduction/loki-mixin/alerts.libsonnet
Loading
Please register or sign in to comment