Applications need to log activities and provide traceability. This has become more important because of the distributed nature of both end-user front-ends and server-side components. Only the simplest applications with low loads work on just one server.

Therefore, a centralised log repository, with logging of all activities from all components, is a fundamental component of modern application architectures. All application components must integrate with this repository and insert log entries at speed but without incurring the latencies which synchronous writes introduce. Applications written in various programming languages must be able to log to a common repository, to allow the repository to be a powerful shared resource. Application code must not have the access to modify or delete log entries even if the application developer chooses to attempt this. And finally, log entries must be decipherable, at least to some extent, without the application developer reading the entry and interpreting it in the context of her code.

This sort of logging framework cannot be designed as an afterthought. Nor is it application-specific. Almost all the technology components are available as mature components in the open-source world, but they need to be integrated into a coherent product. LogHarbour addresses this universal unmet need.

LogHarbour has client libraries which integrate with server-side application code in Go or Java. If needed, the logging end-points may be extended to remote clients by wrapping the client libraries in REST API. Internally, the client libraries push out the log entries as messages in a Kafka stream, and one host runs a Kafka consumer to read the stream and add records into an ElasticSearch database.

Three types of logs are supported, for three purposes. The activity log holds a record of all activities the application intends to log. The data-change log holds before-after values of all fields of any record which changes. If the data-change log is used correctly and all changes are logged, it should be possible to reconstruct data objects at any point in time. The third type of log is for debug messages, used only by developers. All log entries have a standardised structure with metadata and payload, and the LogHarbour UI can extract log entries for viewing, without dependence on application code.