Alertmanager Slack연동 시 빈내용 반환

상황

아래 이미지와 같이 Text부분만 전달되고, 템플릿 변수는 빈값으로 넘어오는 상황

 

해결과정

1. 로그 확인 ( 딱히 이상 없고 잘 전달되고있음)

# 명령어
k logs alertmanager-kube-prometheus-stack-1706-alertmanager-0 -n monitoring

# 결과값
ts=2024-01-29T07:36:41.238Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml
ts=2024-01-29T07:48:20.187Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml
ts=2024-01-29T07:48:20.188Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config_out/alertmanager.env.yaml

 

  2. Prometheus UI 탐색

kubectl port-forward svc/kube-prometheus-stack-1706-prometheus 9090:9090 -n monitoring

 

  3. 슬랙 웹훅 URL에 직접 curl 명령어를 날려 테스트 메세지를 보내봄
 Text 부분이 전달되기에 이 문제는 아닐꺼라 생각했지만 확인차 진행함
(아래 이미지와 같이 메세지가 전달되는 걸 확인한 후 네트워크 나 방화벽 문제는 아니라고 판단함)

curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, this is a test message from Alertmanager!"}' https://hooks.slack.com/services/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

 

  4. Alertmanager 구성 확인 values값을 수정을 했는데 alertmanager.yml 에서는 변경이 안되길래 이부분이 문제인가 싶었음

# 명령어
kubectl exec -n monitoring alertmanager-kube-prometheus-stack-1706-alertmanager-0 -- cat /etc/alertmanager/alertmanager.yml

# 결과값
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

# 설정한 values값
config:
    global:
      resolve_timeout: 5m
      slack_api_url: https://hooks.slack.com/services/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
    inhibit_rules:
    - equal:
      - namespace
      - alertname
      source_matchers:
      - severity = critical
      target_matchers:
      - severity =~ warning|info
    - equal:
      - namespace
      - alertname
      source_matchers:
      - severity = warning
      target_matchers:
      - severity = info
    - equal:
      - namespace
      source_matchers:
      - alertname = InfoInhibitor
      target_matchers:
      - severity = info
    - target_matchers:
      - alertname = InfoInhibitor
    receivers:
    - name: slack-notifications
      slack_configs:
      - channel: '#k8s-alert'
        send_resolved: true
        text: |
          *Alert:* {{ .CommonLabels.alertname }}
          *Severity:* {{ .CommonLabels.severity }}
          *Instance:* {{ .CommonLabels.instance }}
          *Description:* {{ .CommonAnnotations.description }}
          *Summary:* {{ .CommonAnnotations.summary }}
        title: '[{{.Status | toUpper}}] {{ .CommonLabels.alertname }}'
    route:
      group_by:
      - alertname
      group_interval: 1m
      group_wait: 10s
      receiver: slack-notifications
      repeat_interval: 5h
      routes:
      - matchers:
        - alertname = "InfoInhibitor|Watchdog"
        receiver: slack-notifications
    templates:
    - /etc/alertmanager/config/*.tmpl

 

 

---

해결

위에 values값에 group_by 값을 alertmaneger.yml 값과 일치시켜줌 이렇게 일치시켜주니까 템플릿 변수 값이 잘 전달됨

# alertmaneger.yml
route:
  group_by: ['alertname']

# values.yaml
route:
      group_by:
      - alertname