add external_labels support to promtail ksonnet

add container_name label to logs

fixes #190
Pods can have multiple containers. This fix ensures that we collect
logs for all containers.

* allow promtail target path to be a glob match

Promtail now expects __path__ to be a glob. All files that match
the glob expression will be tailed.  If new files are created that
match the glob, they will immediately be tailed.

* remove entry from positions file when file is deleted

* fix syncing the current positions to positions file

* handle file renames

When a file is renamed a "rename" and a "create" event are sent.
We need to treat the rename as a remove.  If the new filename matches
the path glob, then we will start tailing it when the "create" event
is reveived.

* lint

Signed-off-by: Xiang Dai <764524258@qq.com>

* add beginnings of fluentd plugin

* working fluentd output plugin for loki

* refactor fluentd plugin

- update the plugin to support extra_labels, label_keys, line_format
  and drop_single_key options
- update rspec tests
- update readme
- build custom docker image for docker-compose
- default to systemd input by default.

* update readme todo section

* remove extra license file

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* all: update dependencies

* Update some more dependancies and pin the right versions for Prometheus deps.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

* tools/promatail.sh: add DATAROOT option

Signed-off-by: Xiang Dai <764524258@qq.com>

* ksonnet: support custom data root

Signed-off-by: Xiang Dai <764524258@qq.com>

* Move dataroot setting to $._config.promtail_config.dataroot

* whitespace fix

* Update promtail.libsonnet

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Improve ksonnet to be more manageable

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Compress config into a single structure.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Various fixes

- s/config/$._config/

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

either https://www.youtube.com/watch?v=U7C5SpRtK74&list=PLj6h78yzYM2PZf9eA7bhWnIh_mK1vyOfU&index=346 or https://www.youtube.com/watch?v=U7C5SpRtK74

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Cut new blocks once the existing ones are full

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Address PR comments

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Clarify what we mean by 'like Prometheus, but for logs'.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Typos.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Support for JSON, not just protos.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* s/timestamp/ts/

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Add link to protobuf and client library to API docs.

* Link to API docs from README

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Add more words about auth.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

* Fix stream tracking metric

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Track lines and bytes received per user

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Use instance instead of org

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Nuke dead code

Usage has been removed as part of perf-optimisation:
https://github.com/cortexproject/cortex/commit/3006e3901d607f25fa2e213ae05d00380e80a8bc#diff-180f56d9aaf5a9aa079e6fc9cfcf1bc8L365



Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Loki is a component, not the entire stack

* Loki has a capital el.

* Add loki helm chart

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* Add promtail helm chart

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* Update loki conf from comment

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* Update promtail conf from comment

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* Update values from comment

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* adding resources limit and request

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* update README

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* rename to daemonset promtail

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* use masger for loki

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* Move into production repo, re-add readme.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Move to production folder

Signed-off-by: Julien Garcia Gonzalez <julien@giantswarm.io>

* Review feedback; add instructions for installing Grafana with helm.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Move all the config for various ways of deploying Loki into one place.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Fix reference to consul ksonnet module.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Move getting started docs into the production dir.

Remove the raw docker commands, only use the compose file.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Spell production correctly.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Tweaks

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Tweak docs and docker compose setup; put config in images.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* typo

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* fix deploy issue

Signed-off-by: Xiang Dai <764524258@qq.com>

* whitespace

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>