Grafana + Prometheus + Exporter로 서버 모니터링 구성하기

Grafana + Prometheus + Exporter로 서버 모니터링 구성하기

1.리눅스 서버 모니터링

1-1.준비할것

1.Grafana  설치

2.Prometheus 설치

3.exporter 설치

4.Prometheus 환경파일(prometheus.yml)

위의 경우 docker compose로 한번에 세팅 가능

docker-compose.yml

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - /data/jyyoon/prometheus:/etc/prometheus
      - prometheus_data:/data/jyyoon/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - "9091:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/data/jyyoon/grafana
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    pid: "host"
    network_mode: "host"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($|/)'
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.51.152:9100']

모두 준비되었다면 docker compose up -d

1-2. Grafana 세팅

Load 의 ID 값은 템플릿 ID를 사용하자.

1860: 기본 리눅스 서버 모니터링
10991: RabbitMQ 모니터링

2.RabbitMQ 모니터링

나는 RabbitMQ 도 상세히 모니터링 하고 싶어서 추가했다.

docker-compose.yml(rabbitmq)

services:
  rabbitmq:
    image: rabbitmq:3-management
    container_name: rabbitmq
    ports:
      - "5672:5672"      # AMQP
      - "15672:15672"    # Management UI
      - "15692:15692"    # 프로메테우스 플러그인 매트릭 포트
    volumes:
      - /data/jyyoon/rabbitmq:/var/lib/rabbitmq
    environment:
      - RABBITMQ_DEFAULT_USER=guest
      - RABBITMQ_DEFAULT_PASS=guest

prometheus.yml(rabbitmq)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.51.152:9100']

  - job_name: 'rabbitmq'
    metrics_path: /metrics
    static_configs:
      - targets: ['192.168.51.152:15692']

 

 

rabbitmq 의 경우 자체적으로 metrics 를 지원하기 prometheus.yml에 주소만 선언해주면 되지만 만약에 db를 추가하고 싶다면 db는 자체적으로 내부 정보를 알려주는 api가 없기 때문에 exporter 인스턴스를 만들어줘야한다.

ex)db의 경우 아래와 같이 선언

docker-compose.yml(db)

services:
  mysql:
    container_name: mysql
    image: mysql:8.0.34
    volumes:
      - /data/jyyoon/db/mysql/mysql-data:/var/lib/mysql
      - /data/jyyoon/db/mysql/init-exporter.sql:/docker-entrypoint-initdb.d/init-exporter.sql:ro
    environment:
      TZ: "Asia/Seoul"
      MYSQL_DATABASE: mysqldb
      MYSQL_ROOT_PASSWORD: root
      MYSQL_USER: exporter
      MYSQL_PASSWORD: export_pwd
    ports:
      - "3306:3306"
    restart: always

  mysqld-exporter:
    container_name: mysqld_exporter
    image: prom/mysqld-exporter
    command:
      - "--config.my-cnf=/etc/mysql_exporter.cnf"
      - "--web.listen-address=:9104"
    volumes:
      - /data/jyyoon/db/mysql/mysql_exporter.cnf:/etc/mysql_exporter.cnf:ro
    links:
      - mysql
    ports:
      - 9104:9104
    depends_on:
      - mysql
    restart: unless-stopped

init-exporter.sql

CREATE USER 'exporter'@'%' IDENTIFIED BY 'export_pwd';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
FLUSH PRIVILEGES;

mysql-exporter.cnf

[client]
user=exporter
password=export_pwd
host=mysql

 

prometheus.yml(db)

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['192.168.51.152:9100']

  - job_name: 'rabbitmq'
    metrics_path: /metrics
    static_configs:
      - targets: ['192.168.51.152:15692']

  - job_name: 'mysqld_exporter'
    metrics_path: /metrics
    static_configs:
      - targets: ['서버주소:9104']

**그라파나 템플릿ID**

입맛에 맞게 이~쁜거 골라서 쓰자(14031, 7362, 20016)