diff --git a/README.md b/README.md index 2023e077331d03bc3aeec30b2bfd1ad92acb7506..af9ee2a25360451bdee7e31d5495fd241eafbf79 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,80 @@ [](https://circleci.com/gh/grafana/loki/tree/master) [Design doc](https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3-q6vBAsZYIQ5ZeYBkyM/edit) -Loki is a horizontally-scalable, highly-available, multi-tenant, log aggregation -system inspired by Prometheus. It is designed to be very cost effective, as it does -not index the contents of the logs, but rather a set of labels for each log stream. +Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of labels for each log stream. -## Run it locally +Compared to other log aggregation systems, Loki: + +- does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing the metadata, Loki is simpler to operate and cheaper to run. +- indexes and groups log streams using the same labels you’re already using with Prometheus. +- ia an especially good fit for storing Kubernetes logs. It is seamless to switch between metrics and logs using the same Kubernetes labels that you’re already using with Prometheus. +- has native support in Grafana (already in the nightly builds, will be included in Grafana 6.0). + +Loki consists of 3 components: + +- `loki` is the main server, responsible for storing logs and processing queries. +- `promtail` is the agent, responsible for gathering logs and sending them to loki. +- Grafana for the UI. + +## Install + +Currently there are two ways to install Loki, docker and building from source (precompiled binaries coming soon). + +### Install Using Docker + +The Docker images for [Loki](https://hub.docker.com/r/grafana/loki/) and [Promtail](https://hub.docker.com/r/grafana/promtail/) are available on DockerHub. + +To test locally using `docker run`: + +1. Create a docker network that the docker containers can share: + ```bash + docker network create loki + ``` +2. Start the Loki server: + ```bash + docker run --name loki --network=loki -p 3100:3100 --volume "$PWD/docs:/etc/loki" grafana/loki:master-8fa9461 -config.file=/etc/loki/loki-local-config.yaml + ``` +3. Then start the Promtail agent. The default config polls the contents of your `/var/log` directory. + ```bash + docker run --name promtail --network=loki --volume "$PWD/docs:/etc/promtail" --volume "/var/log:/var/log" --network="container:loki" grafana/promtail:make-images-static-26a87c9 -config.file=/etc/promtail/promtail-local-config.yaml + ``` +4. If you also want to run Grafana in docker: + ```bash + docker run --name grafana --network=loki -p 3000:3000 -e "GF_EXPLORE_ENABLED=true" grafana/grafana:master + ``` +5. Follow the steps for configuring the datasource in Grafana in the section below and set the URL field to: `http://loki:3100` + +Another option is to use the docker-compose file in the docs directory: + +1. git clone this repo locally (or just copy the contents of the docker-compose file locally into a file named `docker-compose.yaml`) +2. `cd loki/docs` +3. `docker-compose up` + +If you have have an older cached version of the grafana/grafana:master container then start by doing either: + +```bash +docker pull grafana/grafana:master +``` + +Or for docker-compose: + +```bash +docker-compose pull +``` + +### Configuring the Loki Datasource in Grafana + +Grafana ships with built-in support for Loki in the [latest nightly builds](https://grafana.com/grafana/download). Loki support will be officially released in Grafana 6.0. + +1. Open the side menu by clicking the Grafana icon in the top header. +2. In the side menu under the Dashboards link you should find a link named Data Sources. +3. Click the `+ Add data source` button in the top header. +4. Choose Loki from the list. +5. The http URL field should be the address of your Loki server e.g. `http://localhost:3100` + +Read more about the Explore feature in the [Grafana docs](http://docs.grafana.org/features/explore) and on how to search and filter logs with Loki. + +### Build and Run Loki Locally Loki can be run in a single host, no-dependencies mode using the following commands. @@ -16,7 +85,7 @@ and sending them to loki and `grafana` as the UI. To run loki, use the following commands: -``` +```bash $ go build ./cmd/loki $ ./loki -config.file=./docs/loki-local-config.yaml ... @@ -24,7 +93,7 @@ $ ./loki -config.file=./docs/loki-local-config.yaml To run promtail, use the following commands: -``` +```bash $ go build ./cmd/promtail $ ./promtail -config.file=./docs/promtail-local-config.yaml ... @@ -32,8 +101,8 @@ $ ./promtail -config.file=./docs/promtail-local-config.yaml Grafana is Loki's UI, so you'll also want to run one of those: -``` +```bash $ docker run -ti -p 3000:3000 -e "GF_EXPLORE_ENABLED=true" grafana/grafana:master ``` -In the Grafana UI (http://localhost:3000), log in with "admin"/"admin", add a new "Grafana Logging" datasource for `http://host.docker.internal:3100`, then go to explore and enjoy! \ No newline at end of file +In the Grafana UI (http://localhost:3000), log in with "admin"/"admin", add a new "Grafana Loki" datasource for `http://host.docker.internal:3100`, then go to explore and enjoy! diff --git a/docs/docker-compose.yaml b/docs/docker-compose.yaml new file mode 100644 index 0000000000000000000000000000000000000000..0cd32791ef2fb4f217484216d255afc4ce5d1977 --- /dev/null +++ b/docs/docker-compose.yaml @@ -0,0 +1,34 @@ +version: "3" + +networks: + loki: + +services: + loki: + image: grafana/loki:master-8fa9461 + ports: + - "3100:3100" + volumes: + - $PWD:/etc/loki + command: -config.file=/etc/loki/loki-local-config.yaml + networks: + - loki + + promtail: + image: grafana/promtail:make-images-static-26a87c9 + volumes: + - $PWD:/etc/promtail + - /var/log:/var/log + command: + -config.file=/etc/promtail/promtail-docker-config.yaml + networks: + - loki + + grafana: + image: grafana/grafana:master + ports: + - "3000:3000" + environment: + GF_EXPLORE_ENABLED: "true" + networks: + - loki diff --git a/docs/promtail-docker-config.yaml b/docs/promtail-docker-config.yaml new file mode 100644 index 0000000000000000000000000000000000000000..8bc5f24ca7ebe241e4232549aea60e0f9109c85b --- /dev/null +++ b/docs/promtail-docker-config.yaml @@ -0,0 +1,19 @@ +server: + http_listen_port: 0 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +client: + url: http://loki:3100/api/prom/push + +scrape_configs: +- job_name: system + entry_parser: raw + static_configs: + - targets: + - localhost + labels: + job: varlogs + __path__: /var/log