🕵️ Grafana et les logs / traces
Estimated time to read: 3 minutes
Quand on parle de monitoring, on distingue souvent trois types de données :
- Les métriques
- Les logs
- Les traces
Nous avons déjà vu comment afficher les métriques dans Grafana en s'appuyant sur Prometheus, mais il est aussi possible d'afficher les logs et les traces.
Ajout des DS Loki et Tempo
Pour celà on vas configurer Grafana pour qu'il puisse se connecter à Loki et Tempo.
Objectifs:
- Si ce n'est pas déjà fait la datasource pour
Prometheusici -
Une data source pour se connecter à l'instance
Loki(Le storage pour nos logs)Information Value Name loki URL http://loki:3100 Authentication methods No Authentication Skip TLS certificate validation true -
Une data source pour se connecter à l'instance
Tempo(le storage pour les traces)Information Value Name tempo URL http://tempo:3200 Authentication methods No Authentication Skip TLS certificate validation true Trace to metrics DS prometheus Trace to Logs DS loki
Les logs sont ici gérés par Grafana Loki et les traces par Grafana Tempo. Nous allons importer le dashboard suivant pour manipuler les logs et les traces.
Observability Sample.json
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
},
{
"name": "DS_LOKI",
"label": "loki",
"description": "",
"type": "datasource",
"pluginId": "loki",
"pluginName": "Loki"
},
{
"name": "DS_TEMPO",
"label": "tempo",
"description": "",
"type": "datasource",
"pluginId": "tempo",
"pluginName": "Tempo"
}
],
"__elements": {},
"__requires": [
{
"type": "panel",
"id": "bargauge",
"name": "Bar gauge",
"version": ""
},
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "11.6.0"
},
{
"type": "panel",
"id": "logs",
"name": "Logs",
"version": ""
},
{
"type": "datasource",
"id": "loki",
"name": "Loki",
"version": "1.0.0"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "stat",
"name": "Stat",
"version": ""
},
{
"type": "panel",
"id": "table",
"name": "Table",
"version": ""
},
{
"type": "datasource",
"id": "tempo",
"name": "Tempo",
"version": "11.6.0-pre"
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"description": "Observe Spring Boot application with three pillars of observability: Traces (Tempo), Metrics (Prometheus), Logs (Loki) on Grafana through OpenTelemetry and OpenMetrics.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 28,
"panels": [],
"title": "Metrics",
"type": "row"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 4,
"x": 0,
"y": 1
},
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(http_server_request_duration_seconds_bucket{service_name=\"$app_name\", http_status_code=~\".*\", http_route!~\"$url_filter_regex\"})",
"interval": "",
"legendFormat": "{{span_kind}}",
"range": true,
"refId": "A"
}
],
"title": "Total Requests",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"noValue": "0",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 3,
"x": 4,
"y": 1
},
"id": 22,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(http_server_request_duration_seconds_bucket{service_name=\"$app_name\", http_response_status_code=~\"2.*\", http_route!~\"$url_filter_regex\"})",
"interval": "",
"legendFormat": "",
"range": true,
"refId": "A"
}
],
"title": "Total 2XX",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"noValue": "0",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 3,
"x": 7,
"y": 1
},
"id": 31,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(http_server_request_duration_seconds_bucket{service_name=\"$app_name\", http_response_status_code=~\"3.*\", http_route!~\"$url_filter_regex\"})",
"interval": "",
"legendFormat": "",
"range": true,
"refId": "A"
}
],
"title": "Total Exceptions ( 3XX )",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"noValue": "0",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 3,
"x": 10,
"y": 1
},
"id": 32,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(http_server_request_duration_seconds_bucket{service_name=\"$app_name\", http_response_status_code=~\"4.*\", http_route!~\"$url_filter_regex\"})",
"interval": "",
"legendFormat": "",
"range": true,
"refId": "A"
}
],
"title": "Total Exceptions ( 4XX )",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"noValue": "0",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 3,
"x": 13,
"y": 1
},
"id": 33,
"options": {
"colorMode": "value",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"percentChangeColorMode": "standard",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showPercentChange": false,
"textMode": "auto",
"wideLayout": true
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum(duration_count{service_name=\"$app_name\", http_response_status_code=~\"5.*\", http_route!~\"$url_filter_regex\"})",
"interval": "",
"legendFormat": "",
"range": true,
"refId": "A"
}
],
"title": "Total Exceptions ( 5XX )",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "continuous-GrYlRd"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "ms"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 4
},
"id": 6,
"options": {
"displayMode": "lcd",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 300,
"minVizHeight": 10,
"minVizWidth": 0,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "auto",
"valueMode": "color"
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "sum by(http_route)(http_server_request_duration_seconds_sum{service_name=\"$app_name\", http_route!~\"$url_filter_regex\"}) / \r\nsum by(http_route)(http_server_request_duration_seconds_count{service_name=\"$app_name\", http_route!~\"$url_filter_regex\"})",
"interval": "",
"legendFormat": "{{ span_kind}}",
"range": true,
"refId": "A"
}
],
"title": "Requests Average Duration",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"fixedColor": "blue",
"mode": "fixed"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 4
},
"id": 37,
"options": {
"displayMode": "lcd",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 300,
"minVizHeight": 10,
"minVizWidth": 0,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "auto",
"valueMode": "color"
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "http_server_request_duration_seconds_count{service_name=\"$app_name\", http_route!~\"$url_filter_regex\"}",
"interval": "",
"legendFormat": "{{ http_route }}",
"range": true,
"refId": "A"
}
],
"title": "Number of requests",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"fixedColor": "blue",
"mode": "fixed"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 13
},
"id": 24,
"options": {
"displayMode": "basic",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 300,
"minVizHeight": 10,
"minVizWidth": 0,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "auto",
"valueMode": "color"
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "http_server_request_duration_seconds_sum{service_name=\"$app_name\", http_response_status_code=~\"2.*\", http_route!~\"$url_filter_regex\"}",
"interval": "",
"legendFormat": "{{http_route}}",
"range": true,
"refId": "A"
}
],
"title": "Number of 2xx Requests",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"fixedColor": "red",
"mode": "fixed"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
},
"unit": "none"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 12,
"y": 13
},
"id": 36,
"options": {
"displayMode": "basic",
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom",
"showLegend": false
},
"maxVizHeight": 300,
"minVizHeight": 10,
"minVizWidth": 0,
"namePlacement": "auto",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"range"
],
"fields": "",
"values": false
},
"showUnfilled": true,
"sizing": "auto",
"valueMode": "color"
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"exemplar": true,
"expr": "http_server_request_duration_seconds_sum{service_name=\"$app_name\", http_response_status_code!~\"2.*\", http_route!~\"$url_filter_regex\"}",
"interval": "",
"legendFormat": "{{http_route}}",
"range": true,
"refId": "A"
}
],
"title": "Number of none 2xx Requests",
"type": "bargauge"
},
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 22
},
"id": 30,
"panels": [],
"title": "Logs / Traces",
"type": "row"
},
{
"datasource": {
"type": "loki",
"uid": "${DS_LOKI}"
},
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 23
},
"id": 2,
"options": {
"dedupStrategy": "exact",
"enableInfiniteScrolling": false,
"enableLogDetails": true,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": false
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "loki",
"uid": "${DS_LOKI}"
},
"editorMode": "code",
"expr": "{application=\"$app_name\"} | json | line_format \"{{.severity}} {{.body}}\"",
"hide": false,
"queryType": "range",
"refId": "A"
}
],
"title": "Log of All Spring Boot Apps",
"type": "logs"
},
{
"datasource": {
"type": "tempo",
"uid": "${DS_TEMPO}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"custom": {
"align": "auto",
"cellOptions": {
"type": "auto"
},
"inspect": false
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green"
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 26,
"w": 24,
"x": 0,
"y": 31
},
"id": 26,
"options": {
"cellHeight": "sm",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true
},
"pluginVersion": "11.6.0",
"targets": [
{
"datasource": {
"type": "tempo",
"uid": "${DS_TEMPO}"
},
"filters": [
{
"id": "5578df66",
"operator": "=",
"scope": "span",
"value": [],
"valueType": "int"
},
{
"id": "span-name",
"operator": "=",
"scope": "span",
"tag": "name",
"value": [],
"valueType": "string"
},
{
"id": "service-name",
"operator": "=",
"scope": "resource",
"tag": "service.name",
"value": [
"$app_name"
],
"valueType": "string"
}
],
"groupBy": [
{
"id": "8dcc5ceb",
"scope": "span"
}
],
"hide": false,
"limit": 20,
"query": "$trace_id",
"queryType": "traceqlSearch",
"refId": "A",
"tableType": "traces"
}
],
"title": "Traces",
"type": "table"
}
],
"refresh": "",
"schemaVersion": 41,
"tags": [],
"templating": {
"list": [
{
"current": {},
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"definition": "label_values(http_server_request_duration_seconds_sum,service_name)",
"includeAll": false,
"label": "Application Name",
"name": "app_name",
"options": [],
"query": {
"qryType": 1,
"query": "label_values(http_server_request_duration_seconds_sum,service_name)",
"refId": "PrometheusVariableQueryEditor-VariableQuery"
},
"refresh": 1,
"regex": "",
"type": "query"
},
{
"current": {
"text": "/management/health|/management/info",
"value": "/management/health|/management/info"
},
"description": "Regular expression for excluse URL to monitor",
"label": "URL Filter Regex",
"name": "url_filter_regex",
"options": [
{
"selected": true,
"text": "/management/health|/management/info",
"value": "/management/health|/management/info"
}
],
"query": "/management/health|/management/info",
"type": "textbox"
},
{
"current": {},
"datasource": {
"type": "tempo",
"uid": "${DS_TEMPO}"
},
"definition": "",
"description": "",
"label": "Span Names",
"name": "trace_id",
"options": [],
"query": {
"label": "kind",
"refId": "TempoDatasourceVariableQueryEditor-VariableQuery",
"type": 1
},
"refresh": 1,
"regex": "",
"type": "query"
}
]
},
"time": {
"from": "now-30m",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Observability Sample",
"uid": "aeap3x0fe720wf",
"version": 2,
"weekStart": ""
}
Deux visualizations sont disponibles avec ce Dashboard.
Logs
La première est la wid Log qui va permettre d'afficher les résultats de nos requêtes vers Loki
Utilisez la fonction Explain Query pour obtenir de l'aide sur ce que fait cette requête
Pour aller plus loin sur la configuration des logs c'est par ici.
On utilise Loki comme stockage mais il est possible de faire la meme chose avec Elastic, InfluxD ou autre.
On n'affiche pas les logs INFO
On va ici modifier la requête pour masquer les logs au niveau INFO
ℹ️ C'est surement case sensitive
Spoiler la solution est là
On ajoute simplement un filtre sur le label level pour exclure la valeur INFO
Traces
Les traces permettent de visualiser une succession d'évènements.
Une trace est consitituée de N Spans qui correspondent à une action.
Ici nous utilisons l'agent OpenTelemetry sur les applications Java pour les outiller. C'est l'instrumentation automatique de l'agent qui définit les spans.
Il est aussi possible de définir ces spans par développement dans l'application en utilisant la librairie OTEL. Elle est disponible dans plusieurs langages.
Pour afficher les traces, pas de visualization spécifique mais un tableau est suffisant. L'idée ici est de lister l'ensemble des traces que l'on à collecté.
Il est possible de configuer la query pour ne retenir que certains traces. Le builder permet de faire facilement des filtres, sans forcement maitriser la syntaxe de Tempo. Plus de détails sur cet écran dans la doc officiel sur traceql de Grafana.
On va filtrer uniquement sur les traces d'erreur
On va dupliquer la visualization et rajouter un filtre pour n'afficher que les traces avec un status code différent de 200.
Spoiler la solution est là
ou alors directement en traceQL : {span.http.response.status_code!=200}
On explore les traces ?
Pour aller plus loin avec les traces on peux, en cliquant sur le lien avec l'ID de la Trace ou de la Span acceder à la page de détail de la trace.
On retrouve ici les différents spans qui compose la trace, les temps de traitement de chaque span. Assez pratique pour diagnostiquer les problèmes en production :
🛫 Prochaine étape : Plugin Infinity & API ➡️