Centralized logging with Kafka – transparency, efficiency and fewer outages

Distributed log files delay troubleshooting and increase downtime. With a central logging system based on Apache Kafka, jovoco creates transparency, accelerates analyses and reduces development and operating costs.

Customer

Customer benefits

Technology stack

"The central logging system allows us to detect errors immediately and we can react before entire departments come to a standstill. This saves time and ensures stable processes."
Profil
Company management

Challenges of the customer

Before the start of the project, log data was scattered in files, databases or in the working memory of individual applications. This meant that error analyses took a long time, as developers first had to find out where the relevant logs were stored. Each application used its own procedures – sometimes files, sometimes databases, sometimes pure in-memory logs.

This fragmentation led to a high level of dependency on individual developers. If a responsible person was absent, error diagnosis was considerably delayed. At the same time, undetected system failures caused expensive downtime: If several employees were unable to work due to missing applications, costs and productivity losses were incurred.

In addition to monitoring availability with Uptime Kuma ( to the case study ), central logging covers root cause analysis and makes fault diagnosis much faster.

With a central logging system based on Kafka, jovoco pursued the goal of capturing logs in real time, standardizing them and making them searchable. This allows errors to be detected more quickly, development costs to be reduced and operational failures to be avoided.

Our approach

Data consolidation

The logging system is based on Apache Kafka. Different log sources - including file logs, database transaction logs and application logs - were connected with Vector and Debezium and consolidated in Kafka topics.
1

Data modeling

Separate topics were defined for each system. These can be managed and access rights granularly controlled via a specially developed Python API. This ensures that sensitive information remains protected and that developers only see the relevant data.
2

Transformation & KPIs

ElasticSearch complements Kafka as a search backend. In combination with Kibana, filter options are available according to log level, timestamp or service. This allows developers to immediately track error rates, system failures or trends.
3

Reporting & Visualization

Dashboards have been created in Kibana that visualize log data in real time. Errors can be localized within seconds. In addition, jovoco configured notifications to automatically forward critical events to development teams. While Uptime Kuma already reliably reports availability and failures (see case study "Proactive system monitoring"), Kafka logging provides the necessary level of detail to trace causes in the code or in database processes.
4

Automation

Notifications and forwarding have been automated via Kafka-Consumer. At the same time, the log index is updated daily so that manual corrections are immediately incorporated into the database and system quality is continuously improved.
5

Results for the customer

Annual time savings through central log search
0 Hours
Development time / application saved for notification functions
> 0 Hours

Further results:

Do you have a specific project in mind?

Your decision would be the same as that of renowned companies:

Geschaftsfuhrer Matthias vom IT Dienstleister jovoco

Similar case studies

Schadensfälle BI Reporting

BI reporting in Excel: consolidated data, clear KPIs and automated reports

Inconsistent data and manual evaluations cost time. With a consolidated BI data model in Excel, jovoco creates transparency, provides clear KPIs and enables reports at the touch of a button.

Migration from Google to Microsoft 365 – how jovoco ensures a secure and smooth email transition

Standardizing IT infrastructure and reducing costs - how a medium-sized company switched from Google Workspace to Microsoft 365 with the support of jovoco GmbH and migrated emails, calendars and contacts without any downtime.

Proactive system monitoring with Azure & Co: avoid outages, increase security, reduce costs

Unnoticed failures and certificate errors lead to costs and risks. With a monitoring solution provided in Azure, jovoco enables early detection, automatic notification and continuous system availability.