KtorのアプリからMicrometer+PrometheusでJVMのメトリクスを取得する
はじめに
Ktorのドキュメント呼んでいたらMicrometerに対応してそうだというのを見かけてちょっと動かしてみようかと思います。
基本的にはドキュメントに書かれた流れを沿う感じでやろうかと思います。
あと、PrometheusはDockerを用いて起動します。
やってみる
環境
$ lsb_release -a LSB Version: core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch Distributor ID: Ubuntu Description: Ubuntu 20.04.2 LTS Release: 20.04 Codename: focal $ uname -srvmpio Linux 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux $ docker version Client: Docker Engine - Community Version: 20.10.6 API version: 1.41 Go version: go1.13.15 Git commit: 370c289 Built: Fri Apr 9 22:47:17 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.6 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: 8728dd2 Built: Fri Apr 9 22:45:28 2021 OS/Arch: linux/amd64 Experimental: true containerd: Version: 1.4.4 GitCommit: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e runc: Version: 1.0.0-rc93 GitCommit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec docker-init: Version: 0.19.0 GitCommit: de40ad0 $ java --version openjdk 16 2021-03-16 OpenJDK Runtime Environment (build 16+36-2231) OpenJDK 64-Bit Server VM (build 16+36-2231, mixed mode, sharing)
プロジェクトの作成
プロジェクトはIntelliJのKtorプラグインを使って作成します。
設定は以下のようにします。
依存の追加のところでMicrometer
を追加することもできますが、今回ははRouting
だけを選択します。
出来上がったプロジェクトのPomの抜粋を以下に示します。
<properties> <ktor_version>1.5.3</ktor_version> <kotlin.code.style>official</kotlin.code.style> <kotlin_version>1.4.32</kotlin_version> <logback_version>1.2.3</logback_version> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <kotlin.compiler.incremental>true</kotlin.compiler.incremental> <main.class>hirooka.dev.ApplicationKt</main.class> </properties>
<dependencies> <dependency> <groupId>io.ktor</groupId> <artifactId>ktor-server-core</artifactId> <version>${ktor_version}</version> </dependency> <dependency> <groupId>io.ktor</groupId> <artifactId>ktor-server-netty</artifactId> <version>${ktor_version}</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>${logback_version}</version> </dependency> <dependency> <groupId>io.ktor</groupId> <artifactId>ktor-server-tests</artifactId> <version>${ktor_version}</version> <scope>test</scope> </dependency> </dependencies>
Ktorのバージョンは1.4.32
が選択されているのと、依存にはテスト関連の者Lockbackそして、ktor-server-core
が追加されていました。
Prometheus形式のメトリクスを取得できるようにする
まずは以下の依存をPomに追加する必要があります。
<dependency> <groupId>io.ktor</groupId> <artifactId>ktor-metrics-micrometer</artifactId> <version>${ktor_version}</version> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <version>1.6.6</version> </dependency>
1つ目はKtorのMicrometerのサポートを提供してくれるプロジェクトで、2つ目はPrometheusのレジストリーです。
次にApplication.kt
を以下のように修正します。
import io.ktor.server.engine.* import io.ktor.server.netty.* import hirooka.dev.plugins.* import io.ktor.application.* import io.ktor.metrics.micrometer.* fun main() { val prometheusMeterRegistry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT) embeddedServer(Netty, port = 8080, host = "0.0.0.0") { install(MicrometerMetrics) { registry = prometheusMeterRegistry } configureRouting() }.start(wait = true) }
追加したのはinstall(MicrometerMetrics)
のところです。
KtorでMicromerを使う場合installでMicrometerMwetricsのクラスをインストールする必要があります。
次にJVMのメトリクスを公開するために更に以下のような修正をくわえます。
fun main() { val prometheusMeterRegistry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT) embeddedServer(Netty, port = 8080, host = "0.0.0.0") { install(MicrometerMetrics) { registry = prometheusMeterRegistry meterBinders = listOf( JvmMemoryMetrics(), JvmGcMetrics(), ProcessorMetrics(), ) } configureRouting() }.start(wait = true) }
最後のProcessorMetrics()
はシステムのメトリクスなので今回はいらないかもしれませんが一応追加しておきます。
最後に取得したメトリクスを公開するRoutingを記述します。
fun main() { val prometheusMeterRegistry = PrometheusMeterRegistry(PrometheusConfig.DEFAULT) embeddedServer(Netty, port = 8080, host = "0.0.0.0") { install(MicrometerMetrics) { registry = prometheusMeterRegistry } routing { get("/metrics") { call.respond(prometheusMeterRegistry.scrape()) } } configureRouting() }.start(wait = true) }
Mainの頭のところでイニシャライズしたprometheusMeterRegistry
を使う必要があったため、今回はApplication.kt
にRoutingを記述しました。
具体的には/metrics
というPathに対してprometheusMeterRegistry.scrape()
で取得できるメトリクスを返すようにしています。
これでアプリケーション側の設定は完了です。
アプリケーションを起動して、cURLでメトリクスを取得してみます。
$ curl localhost:8080/metrics # HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool # TYPE jvm_buffer_memory_used_bytes gauge jvm_buffer_memory_used_bytes{id="mapped",} 0.0 jvm_buffer_memory_used_bytes{id="direct",} 3.3562632E7 # HELP system_cpu_usage The "recent cpu usage" for the whole system # TYPE system_cpu_usage gauge system_cpu_usage 0.06208306434258662 # HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management # TYPE jvm_memory_max_bytes gauge jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 1.22908672E8 jvm_memory_max_bytes{area="heap",id="G1 Survivor Space",} -1.0 jvm_memory_max_bytes{area="heap",id="G1 Old Gen",} 8.321499136E9 jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0 jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 5836800.0 jvm_memory_max_bytes{area="heap",id="G1 Eden Space",} -1.0 jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9 jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 1.22912768E8 # HELP jvm_gc_pause_seconds Time spent in GC pause # TYPE jvm_gc_pause_seconds summary jvm_gc_pause_seconds_count{action="end of minor GC",cause="Metadata GC Threshold",} 1.0 jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Metadata GC Threshold",} 0.006 # HELP jvm_gc_pause_seconds_max Time spent in GC pause # TYPE jvm_gc_pause_seconds_max gauge jvm_gc_pause_seconds_max{action="end of minor GC",cause="Metadata GC Threshold",} 0.006 # HELP jvm_gc_max_data_size_bytes Max size of long-lived heap memory pool # TYPE jvm_gc_max_data_size_bytes gauge jvm_gc_max_data_size_bytes 8.321499136E9 # HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time # TYPE system_load_average_1m gauge system_load_average_1m 2.18 # HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use # TYPE jvm_memory_committed_bytes gauge jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 2686976.0 jvm_memory_committed_bytes{area="heap",id="G1 Survivor Space",} 6291456.0 jvm_memory_committed_bytes{area="heap",id="G1 Old Gen",} 4.5088768E8 jvm_memory_committed_bytes{area="nonheap",id="Metaspace",} 2.5608192E7 jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 2555904.0 jvm_memory_committed_bytes{area="heap",id="G1 Eden Space",} 6.291456E7 jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",} 2883584.0 jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 2555904.0 # HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool # TYPE jvm_buffer_total_capacity_bytes gauge jvm_buffer_total_capacity_bytes{id="mapped",} 0.0 jvm_buffer_total_capacity_bytes{id="direct",} 3.3562631E7 # HELP system_cpu_count The number of processors available to the Java virtual machine # TYPE system_cpu_count gauge system_cpu_count 8.0 # HELP ktor_http_server_requests_seconds_max # TYPE ktor_http_server_requests_seconds_max gauge ktor_http_server_requests_seconds_max{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",} 0.021426668 # HELP ktor_http_server_requests_seconds # TYPE ktor_http_server_requests_seconds histogram ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001048576",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001398101",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.001747626",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.002097151",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.002446676",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.002796201",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.003145726",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.003495251",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.003844776",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.004194304",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.005592405",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.006990506",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.008388607",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.009786708",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.011184809",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.01258291",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.013981011",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.015379112",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.016777216",} 0.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.022369621",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.027962026",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.033554431",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.039146836",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.044739241",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.050331646",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.055924051",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.061516456",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.067108864",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.089478485",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.1",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.111848106",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.134217727",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.156587348",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.178956969",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.20132659",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.223696211",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.246065832",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.268435456",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.357913941",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.447392426",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.5",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.536870911",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.626349396",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.715827881",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.805306366",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.894784851",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="0.984263336",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="1.073741824",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="1.431655765",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="1.789569706",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="2.147483647",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="2.505397588",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="2.863311529",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="3.22122547",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="3.579139411",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="3.937053352",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="4.294967296",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="5.726623061",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="7.158278826",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="8.589934591",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="10.021590356",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="11.453246121",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="12.884901886",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="14.316557651",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="15.748213416",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="17.179869184",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="20.0",} 1.0 ktor_http_server_requests_seconds_bucket{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",le="+Inf",} 1.0 ktor_http_server_requests_seconds_count{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",} 1.0 ktor_http_server_requests_seconds_sum{address="localhost:8080",method="GET",route="/metrics",status="200",throwable="n/a",} 0.021426668 # HELP ktor_http_server_requests_active # TYPE ktor_http_server_requests_active gauge ktor_http_server_requests_active 1.0 # HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool # TYPE jvm_buffer_count_buffers gauge jvm_buffer_count_buffers{id="mapped",} 0.0 jvm_buffer_count_buffers{id="direct",} 6.0 # HELP jvm_memory_used_bytes The amount of used memory # TYPE jvm_memory_used_bytes gauge jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'",} 2629376.0 jvm_memory_used_bytes{area="heap",id="G1 Survivor Space",} 6291456.0 jvm_memory_used_bytes{area="heap",id="G1 Old Gen",} 0.0 jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 2.4776112E7 jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 1170432.0 jvm_memory_used_bytes{area="heap",id="G1 Eden Space",} 1.2582912E7 jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 2673248.0 jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 507008.0 # HELP jvm_gc_live_data_size_bytes Size of long-lived heap memory pool after reclamation # TYPE jvm_gc_live_data_size_bytes gauge jvm_gc_live_data_size_bytes 0.0 # HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process # TYPE process_cpu_usage gauge process_cpu_usage 9.6815834767642E-4 # HELP jvm_gc_memory_allocated_bytes_total Incremented for an increase in the size of the (young) heap memory pool after one GC to before the next # TYPE jvm_gc_memory_allocated_bytes_total counter jvm_gc_memory_allocated_bytes_total 2.3068672E7 # HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC # TYPE jvm_gc_memory_promoted_bytes_total counter jvm_gc_memory_promoted_bytes_total 0.0
Promehteusから収集する
最後に公開されているメトリクスをPrometheus側から取得したいと思います。
まずは
global: scrape_interval: 15s evaluation_interval: 15s rule_files: scrape_configs: - job_name: prometheus static_configs: - targets: ['host.docker.internal:8080/metric']
余談ですがDocker 20.10系からLinuxで--add-host=host.docker.internal:host-gateway
みたいな感じで起動時にオプションを渡してやるとhost.docker.internal
でホストマシンにアクセスできるようになったみたいですね。
PrometheusをDockerで起動します。この際に上記の設定ファイルをマウントするようにします。
docker run \ --add-host=host.docker.internal:host-gateway \ -p 9090:9090 \ -v /path/to/four/config/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus
これでPrometheusが起動し、localhost:9090
でUIに接続できます。
ターゲットの一覧を確認するときちんとアプリが認識されているのがわかります。
以下のようにjvm_buffer_count_buffers
も可視化できました。