Prometheus-服务监控系统

简介

Prometheus 是一个开源的服务监控系统和时间序列数据库

安装

1. 下载

wget https://github.com/prometheus/prometheus/releases/download/v2.42.0/prometheus-2.42.0.linux-arm64.tar.gz

2. 解压

tar zxvf prometheus-2.42.0.linux-arm64.tar.gz

3. 配置

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']
#应用
  - job_name: 'order'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['127.0.0.1:8080']
#数据库
  - job_name: 'mysql_export'
    static_configs:
      - targets: ['127.0.0.1:9104']
#主机
  - job_name: 'node_export'
    static_configs:
      - targets: ['127.0.0.1:9100']

4. 运行

./prometheus --config.file=prometheus.yml

架构

查询

Prometheus提供了一种称为PromQL（PrometheusQueryLanguage）的函数查询语言，它允许用户实时选择和聚合时间序列数据。表达式的结果可以显示为图形，在Prometheus的表达式浏览器中显示为表格数据，也可以由外部系统通过HTTP API使用。

基础

表达式类型

• 瞬时向量（Instant vector）：一组时间序列，包含每个时间序列的单个样本，所有样本共享相同的时间戳
• 范围向量（Range vector）：一组时间序列，包含每个时间序列随时间变化的数据点范围
• 标量（Scalar）：一个简单的数字浮点值
• 字符串（String）：一个简单的字符串值；当前未使用

操作

函数

案例

HTTP API

REMOTE READ API

应用

将应用接入prometheus，在github找到对应语言的client，一些以go为案例

client：https://github.com/prometheus/client_golang

1. 构建应用

go mod init
go get github.com/prometheus/client_golang/prometheus
go get github.com/prometheus/client_golang/prometheus/promauto
go get github.com/prometheus/client_golang/prometheus/promhttp

2. 编写http指标采集入口

package main

import (
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

func recordMetrics() {
    go func() {
        for {
            opsProcessed.Inc()
            time.Sleep(2 * time.Second)
        }
    }()
}

var (
    opsProcessed = promauto.NewCounter(prometheus.CounterOpts{
        Name: "app_processed_ops_total",
        Help: "The total number of processed events",
    })
)

func main() {
    recordMetrics()
    http.Handle("/metrics", promhttp.Handler())
    http.ListenAndServe(":8088", nil)
}

3. 配置prometheus.yaml

scrape_configs:
- job_name: myapp
  scrape_interval: 10s
  static_configs:
  - targets:
    - localhost:8088

4. 运行应用

nohup ./main 2>1 1>/dev/null &

5. 导入大盘，当然你也可以自定义

官网搜索地址：https://grafana.com/grafana/dashboards/

6. 查看大盘

参考资料：https://prometheus.io/docs/guides/go-application/

案例

配置RT/TPS

应用提供指标

package main

import (
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	httpRequestDurationHistogram = prometheus.NewHistogram(
		prometheus.HistogramOpts{
			Name:    "http_request_duration_milliseconds",
			Help:    "HTTP request duration distribution.",
			Buckets: []float64{10, 50, 100, 200, 500, 1000, 2000, 5000},
		},
	)
	httpRequestsTotal = prometheus.NewCounter(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "Total number of HTTP requests.",
		},
	)
)

func init() {
	prometheus.MustRegister(httpRequestsTotal, httpRequestDurationHistogram)
}

func main() {
	http.Handle("/metrics", promhttp.Handler())
	http.Handle("/ok", http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
		httpRequestsTotal.Inc()
		startTime := time.Now()
		//DO SOMETHING
		service()
		duration := time.Since(startTime)
		ms := float64(duration.Nanoseconds()) / 1e6
		// 将请求处理时间作为指标值增加到Prometheus的histogram类型指标中
		httpRequestDurationHistogram.Observe(ms)
		writer.WriteHeader(http.StatusOK)
		writer.Write([]byte(time.Now().Format("2006-01-02 15:04:05")))
	}))
	http.ListenAndServe(":8088", nil)
}

func service() {
	log.Println(time.Now().Format("2006-01-02 15:04:05"))
	startTime := time.Now()
	fib(40)
	duration := time.Since(startTime)
	fmt.Println(duration)
}

func fib(n int) int {
	if n == 1 || n == 2 {
		return n
	}
	return fib(n-1) + fib(n-2)
}

配置查询指标

sum(rate(http_requests_total{job="app"}[1m])) 
sum(http_request_duration_milliseconds_sum) / sum(http_request_duration_milliseconds_count)

资料

1. https://prometheus.io/
2. https://github.com/prometheus