processing-log-lines.md

Processing Log Lines

A detailed look at how to setup promtail to process your log lines, including extracting metrics and labels.

• Pipeline
• Stages

Pipeline

Pipeline stages implement the following interface:

type Stage interface {
  Process(labels model.LabelSet, extracted map[string]interface{}, time *time.Time, entry *string)
}

Any Stage is capable of modifying the labels, extracted data, time, and/or entry, though generally a Stage should only modify one of those things to reduce complexity.

Typical pipelines will start with a regex or json stage to extract data from the log line. Then any combination of other stages follow to use the data in the extracted map. It may also be common to see the use of match at the start of a pipeline to selectively apply stages based on labels.

The example below gives a good glimpse of what you can achieve with a pipeline :

scrape_configs:
- job_name: kubernetes-pods-name
  kubernetes_sd_configs: ....
  pipeline_stages:
  - match:
      selector: '{name="promtail"}'
      stages:
      - regex:
          expression: '.*level=(?P<level>[a-zA-Z]+).*ts=(?P<timestamp>[T\d-:.Z]*).*component=(?P<component>[a-zA-Z]+)'
      - labels:
          level:
          component:
      - timestamp:
          format: RFC3339Nano
          source: timestamp
  - match:
      selector: '{name="nginx"}'
      stages:
      - regex:
          expression: \w{1,3}.\w{1,3}.\w{1,3}.\w{1,3}(?P<output>.*)
      - output:
          source: output
  - match:
      selector: '{name="jaeger-agent"}'
      stages:
      - json:
          expressions:
            level: level
      - labels:
          level:
- job_name: kubernetes-pods-app
  kubernetes_sd_configs: ....
  pipeline_stages:
  - match:
      selector: '{app=~"grafana|prometheus"}'
      stages:
      - regex:
          expression: ".*(lvl|level)=(?P<level>[a-zA-Z]+).*(logger|component)=(?P<component>[a-zA-Z]+)"
      - labels:
          level:
          component:
  - match:
      selector: '{app="some-app"}'
      stages:
      - regex:
          expression: ".*(?P<panic>panic: .*)"
      - metrics:
        - panic_total:
            type: Counter
            description: "total count of panic"
            source: panic
            config:
              action: inc