* handle symbolic links correctly

When tailing a file that is rotated we expect that a fsnotify.Rename
event will be sent when the old file is renamed and a fsnotif.Create
event will be sent when the replacement file is created. This allows
us to close and re-open the file.  However if the file we are
tailing is a symbolic link (and the target is in another directory)
we wont get these events.  To address this we need to tail the target
file directly.

fixes #274

* handle symlinks being re-linked

When a symbolic link is re-linked we will only get a fsnotify.Create
event. No rename or delete event will have been sent to trigger
the tailer to be stoped.  When this happens we should stop the current
tailer for the path and start a new one. This will allow us to perform
a new Readink on the path to get the updated target filename.

* Flush idle chunks properly

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Check the correct chunk is full before appending a new chunk.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Add the config to helm and ksonnet too.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Also, make ErrOutOfOrder a HTTP 4xx error and continue to append even when we encourter an out of order entry.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

removing incorrect calling of positions.Stop from within a target as it runs single instance within promtail, moved the call to Stop up to promtail.go within the Shutdown() method.  Should Fix #249 (#252)

* Adding a test for target.go
Fixing some shutdown issues where tailers were stopped but didn't properly wait for the current position to be saved to file
Fixed an issue where an empty directory would lead to nothing being tailed

* adding more tests for target.go

* renaming quitComplete -> done per feedback
making PositionsFile private again, no reason for it to be public
removed unnecessary save of empty file on creation of positions file

* fixing lint errors

* cleaning up some comments
removing TODO around directories, added a test to verify behavior
some more cleanup per review

* fixing imports

* allow promtail target path to be a glob match

Promtail now expects __path__ to be a glob. All files that match
the glob expression will be tailed.  If new files are created that
match the glob, they will immediately be tailed.

* remove entry from positions file when file is deleted

* fix syncing the current positions to positions file

* handle file renames

When a file is renamed a "rename" and a "create" event are sent.
We need to treat the rename as a remove.  If the new filename matches
the path glob, then we will start tailing it when the "create" event
is reveived.

* lint

* all: update dependencies

* Update some more dependancies and pin the right versions for Prometheus deps.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Cut new blocks once the existing ones are full

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Address PR comments

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Support for JSON, not just protos.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* s/timestamp/ts/

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Fix stream tracking metric

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Track lines and bytes received per user

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Use instance instead of org

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Nuke dead code

Usage has been removed as part of perf-optimisation:
https://github.com/cortexproject/cortex/commit/3006e3901d607f25fa2e213ae05d00380e80a8bc#diff-180f56d9aaf5a9aa079e6fc9cfcf1bc8L365



Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Egor Kovetskiy <e.kovetskiy@gmail.com>

Signed-off-by: alei <rayingecho@gmail.com>

Fix chunk retain period

Signed-off-by: alei <rayingecho@gmail.com>

Fix undefined symbol

Signed-off-by: alei <rayingecho@gmail.com>

* Add line length to batchSize

Signed-off-by: Eugen Biegler <nupagadi@posteo.de>

* Reset batchSize in timer condition too

Signed-off-by: Eugen Biegler <nupagadi@posteo.de>

Signed-off-by: yeya24 <ben.ye@daocloud.io>

Persistence chunks using the Cortex chunk store.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Rename Tempo to Loki.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Use new build image, don't delete generated files on clean.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Make promtail read all its config from a file.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Use gokit logging everywhere.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Start work to parse normal log lines.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* update common and kuberesolver.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* lint

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Get EntryParser stuff working, so we can tail raw log files.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Make sure filename label works.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* lint, remove interfacer.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Check in generated files, to make getting started easier.

Add gometalinter, and a lint check for changes to the generated files.

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

* Make lint pass?

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Add checkenc without serialisation for now.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Simplify interface and add serialisatio`

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Move away from \xFF magic to something simple

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Add serialisation and Deserialisation

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Modify interface to be closer to logish interface.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* chunkenc: Fix race b/w append and iteration.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* chunkenc: Make iterators honour bounds

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* chunkenc: Remove locks as safety is assured externally

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* chunkenc: Add checksums

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Add code quotes around block design.

* Split headBlock into it's own type.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Simplify encoding and decoding.

Signed-off-by: Goutham Veeramachaneni <cs14btech11014@iith.ac.in>

* Expose flags.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Use the already existing EntryIterator interface

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Use existing Chunk interface.

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Review feedback

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

* Integrate the compressed chunk and add metrics

Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com>

Signed-off-by: Tom Wilkie <tom.wilkie@gmail.com>