- Example Spring application with logging
- Deploy a Spring Boot Application on Kubernetes
- Deploy a Grok Exporter as a sidecar on Kubernetes
- Deploy a Prometheus on Kubernetes
In previous post, you have looked into Prometheus basics. In this part, it is time to create Prometheus service on the Kubernetes which can scrape metrics from additional metric exporter. It’s the perfect solution when you can’t export metrics from the application source code and your application logs the information you want to expose as metrics.
Example Spring application with logging
To understand how an grok exporter works, we must first create an application that logs some data to files.
I have prepared a very simple application in Spring Boot that has two functionalities:
Welcomes users entering the home page. If the user does not provide his name in
@RequestParam- the application will answer: Hello Guest
HelloController source code: Github
Every 10 seconds creates a log in the form: DATE TIME SERVICE_NAME CONNECTION_TIME.
Service names are randomly selected from 3 defined names. Response times are also random. The logs are supposed to resemble real logs of the application connecting to other services and measuring the time of connection. The source code is on the Github
Logs are saved to the
logs/applog.log file. Log4j configuration: log4j.xml code: Github
Deploy a Spring Boot Application on Kubernetes
Firstly, we have to build docker image with application. I have already done it - image is pushed to the DockerHub. You can just import the image in the deployment definition (
image: docker.io/michf/spring-logging-app:latest) for Kubernetes to deploy application.
If you want to create an image of your own application you can create a Dockerfile based on mine and push to the Dockerhub.
Now, we have to prepare Kubernetes deployment configuration. Let’s start from the Deployment config:
That’s all. You can deploy logging application on Kubernetes.
Deploy a Grok Exporter as a sidecar on Kubernetes
We will be collecting data from our logs using a sidecar container. A sidecar container is a secondary container which is run within the same Pod. In our case
grok exporter will be a sidecar container.
To configure sidecar pattern, we will create a volume (with /logs path) for our Pod to be shared by all containers in the Pod. Then we will configure grok exporter to read logs from shared location.
Firstly, we have to create grok service. Service will be a target in Prometheus:
9144 is a default grok exporter port.
Log capture using grok exporter in Kubernetes
To capture logs we need to create a configuration file. For this purpose we will use Kubernetes
Configmap will create and mount the config.yml file (configuration for grok exporter). In config.yml we set the path to the mounted volume (/logs/applog.log) and two types of metrics. The histogram will inform you about how many connections to external services took less than 100 ms, how many calls took more than 100, but less than 1000, and how many calls lasted more than 3000 ms. Counter will inform you how many http requests has served our application for a given user.
match: property is an expression pattern. You can find all grok patterns on the Github. Then you can adjust the pattern to catch your logs.
Let’s add volume and grok exporter image to the deployment definition from previous point:
Now, we can deploy above Deployment. After entering the pod via Kubernetes Dashboard, you should see information about two containers (from grok and from the logging application):
Let’s check if grok expese endpoint wiht metrics. I used
kubectl port-forward pod-name port:port to forward a grok port to localhost
You can go to
127.0.0.1:9144/metrics and check the application metrics based on the logs:.
Deploy a Prometheus on Kubernetes
Firstly, create a configmap with the Prometheus scrape config:
Prometheus will collect information from the grok-exporter service every 15 seconds.
In Prometheus deployment config We have to mount created a moment ago config map as a file inside
To acces the Prometheus GUI over an IP/DNS, we need to expose it as a service:
Now we can access the Prometheus dashboard using kubectl port-forward
Thank you for reading! I hope you enjoyed the post.