Skip to main content

Fluent Bit


설치

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update fluent \
&& helm search repo fluent/fluent-bit -l | head -n 10
helm show values fluent/fluent-bit \
--version 0.21.7 \
> fluent-bit-values.yaml
fluent-bit-values.yaml
resources:
requests:
cpu: 10m
memory: 64Mi
limits:
cpu: 100m
memory: 128Mi

labels: {}

luaScripts:
# https://docs.fluentbit.io/manual/pipeline/filters/lua
tag.lua: |
function add_tag(tag, timestamp, record)
new_record = record
new_record["tag"] = tag
return 2, timestamp, new_record
end

config:
# https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file#config_section
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
# https://docs.fluentbit.io/manual/administration/buffering-and-storage
storage.path /var/log/flb-storage/
storage.metrics On

filters: |
[FILTER]
# log키에 있는 json 텍스트를 파싱하여 키/값으로 분리 후 log 키 삭제
Name parser
Match *
Parser docker
Key_Name log

[FILTER]
# docker parser가 실패한 경우 level=warn 추가
Name modify
Match *
Condition Key_Exists log
Add level warn

[FILTER]
# fluent-bit tag를 tag키의 값으로 추가
Name lua
Match *
script /fluent-bit/scripts/tag.lua
call add_tag
helm template fluent-bit fluent/fluent-bit \
--version 0.21.7 \
-n monitoring \
-f fluent-bit-values.yaml \
> fluent-bit.yaml
helm upgrade fluent-bit fluent/fluent-bit \
--install \
--version 0.21.7 \
-n monitoring \
--history-max 3 \
-f fluent-bit-values.yaml

Configuration

info

Fluent Bit에서 사용하는 Regex 테스트는 https://rubular.com/ 에서 가능합니다.

Format

Format은 기본적으로 out_file으로 데이터는 아래와 같은 포맷을 갖게됩니다.

<tag>: [<time>, {"key1": "value1", "key2": "value2"}]

INPUT

Tail

[INPUT]
Name tail
# <tag>: [<time>, {"log": "<message>"}]
multiline.parser docker, cri
# /var/log/containers/<pod-name>_<pod-namespace>_<container-name>-<hash>.log
# * wildcard 사용 가능
Path /var/log/containers/xxx*.log
Exclude_Path /var/log/containers/xxx*istio*.log
# 파일의 이름(절대 경로)에 정규표현식을 사용하여 패턴 매칭
Tag_Regex /(?<project>[a-z0-9][-a-z0-9]*)-(?<env>\bdev|stag|prod\b)(.*)_(?<namespace_name>.+)_(?<container_name>.+)-
Tag <project>.<env>
Mem_Buf_Limit 5MB
storage.type filesystem

FILTER

FILTER는 선언된 순서대로 적용됩니다.

Validating data

OUTPUT

Forward

env:
- name: FLUENTD_HOST
value: "fluentd-forwarder.monitoring.svc"
- name: FLUENTD_PORT
value: "24224"
[OUTPUT]
Name forward
Match *
Host ${FLUENTD_HOST}
Port ${FLUENTD_PORT}
Retry_Limit 5