Added option that loki service run as nodeport

Fix missing volume after refactor

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

TestClient used for watching log file events appends logs to a list
Due to race condition some of the logs added to it goes missing
Adding a Mutex to it to lock messages before adding logs to it

Add link to chunk format doc

Signed-off-by: Xiang Dai <764524258@qq.com>

Add help info for users uses high k8s version

Separate loki & promtail charts

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

Log error to stderr

Improve documentation based on what I learned when I did loki setup.

* Explain that api needs the start parameter set if you want to see logs older than one hour
* Explain that S3 and DynamoDB can also use instance roles in EC2
* Add couple of hints how to debug Promtail
* Add documentation how the Promtail prometheus scrape_configs work

Signed-off-by: Xiang Dai <764524258@qq.com>

Add dashboard example

adding labels design doc

Fix serviceaccount error

remove logcli config part

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Steven Sheehy <ssheehy@firescope.com>

Fixing lagging alert to also catch when the read_bytes is past the end of the file.
Adding an alert to let us know if we are missing tailing of any files.

Update logcli doc

Signed-off-by: Xiang Dai <764524258@qq.com>

Since kinpin.v2 not support use config like this, remove them.

Signed-off-by: Xiang Dai <764524258@qq.com>

Add health check

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

Signed-off-by: Xiang Dai <764524258@qq.com>

Change hpcloud tailer library to poll mode to workaround bugs in the event driven code within the library
Had to rework tests with some waits for messages because polling is slower
There are several longer sleeps in the promtail_test which are not ideal and have been marked with FIXME to remove once we have a better solution

Add help about how to get metrics