Logging, oh sweet, logging. I have never fully understood why people spend hours discussing logging. And above all logging frameworks. And as people tend to be so religious about it and approach… let’s take it away from them. And here’s where we wrapped logging in several annotations, fluent API and released so everyone can be angry about not using their favourite logging framework anymore. After having it for some time it is merely a common idiom even newbies will get. And speaking in idioms is a dream come true. Except… unless you’re doing this on purpose, for commoditisation of the technology and expressing idioms, don’t. Here’s a sample of what we wanted to achieve - standard log format and standard way to log:

@Log(level=Level.INFO)
public Foo bar(Baz baz){ 
    ...
}

To use the other, more customizable API, you simply make:

public Foo bar(Baz baz){ 
    log.info().message("message {} {}",1,"123");  //logs  INFO   - requestId    | callerId  | userId    |message 1 123
    log.error().requestId("123").message("error"); //logs ERROR - requestId     | callerId  | userId    |error
}

And still get the standard Ops-friendly format.

At the stabilization time we knew that for future’s sake we’ll need to apply way more sanity checks than we initially put. This is where the safety was born. A library that implements Michael Nygard’s Release it! concepts. And boy, you’ll need one of those as your number of production services and interactions grows. Hopefully Netflix shared a great safety library Hystrix. Example of safety is a circuit breaker pattern annotation. Each integration point is guarded by a circuit breaker that is triggered after a defined number of exceptions and locked for predefined time:

@GuardedByCircuitBreaker(exceptionsThreshold=5,retryTimeout=3000) 
public Foo bar(URL url){
    ...
}

Monitoring in a heterogenous, distributed environment has a lot of challanges. As we decided to have the services running in a common webserver clusters, the technology the company was using for years, some of the tools have been already available - runtime profiling, request tracking, migration to name a few. However as metrics freaks we needed more. And again it had to be a common idiom. Declarative and transparent. Kind of like Metrics by Coda Hale. Exactly - Metrics. We put some effort to integrate it with our idea of the metrics and monitoring, defined a common concept JSON-based status page holding all the information. To get a standard set of metrics we use for each service, you’d simply:

@DefaultRequestMetrics(id = "Foo") 
public Bar bar(
    Foo parameters) {
    ...
}

Sample status page parsed by monitoring:

{"version":"3.0.0","gauges":{"FooService.counterGauge":{"value":1},"FooService.heavyCounterGauge":{"value":1001}},"counters":{},"histograms":{},"meters":{},"timers":{}}

Before the idea of the service oriented middleware the company had been primarily a Java shop. They’ve been successfully using Maven for a couple of years, had internal repositories, mirrors, yada yada yada. Aside from all the baadddd, bad vibes maven has, for the straight-forward cases and archetype system it felt the tool to use. The preparation of the uberverbose maven archetype w/ all the modularization we wanted took a bit, yet it was totally worth it. A template with just enough stubbed classes, structure, dependencies set up is a huge value. Just to it.

Git was not used at the company before we introduced the approach. However for our purposes Gitflow was a match made in heaven. The way it played with the environment of change felt just right. We needed a well-defined flow that would fit company’s release cycle compatible approach. We knew Git well enough to share the knowledge with the company’s employees. Currently each service has it’s own repository. Each repository has it’s collaborators. People outside of the collaborators group are always welcome to fork and pull-request the repository. Now that the components are so simple, peer reviews may be done by forking a repository and submitting a pull request.

The great thing about having an automated gitflow is that the CI is capable of pushing the latest, generated docs into company’s Confluence. The Confluence contains service profiles describing service metadata (metrics, thresholds), APIs, third party interactions. All the data is generated and pushed into the wiki by Jenkins. Most of the time we simply use an annotation processor for metrics, reaction thresholds etc. APIs however are being documented with Swagger-compatible Enunciate maven plugin. The template usually contains API methods w/ Javadocs, latest API/client maven dependency at times containing samples. Of course you could say that all the data is either way available through repository or it’s web front. However there are several client systems and service consumers that look for summary about service portfolio and services’ capabilities. And for DRY purposes we never edit the description manually.

A monitoring tool is being used as an active status pages consumer. It reads JSON pages and pushes notifications according to thresholds set in the service profiles. It is also fed with external data. One particular thing that we should have had implemented is the ‘phone home’ pattern. The pattern assumes that each service should actively ping back the ‘mothership’ monitoring tool with a heartbeat. The failure discovery approach along the status pages would have provided enough information on application status. Both Nagios and Zabbix provide a comprehensive APIs for implementing such integration. Previously I have also mentioned the classic solutions that had existed in the company and needed only a limited effort to get them working for the distributed approach. Each incoming request was marked with an ID that is stored in request header. The ID may be then traced through each service and network component it passes.

Having a centralized error catcher (Sentry in this case) enables distributed applications to proactively push each exception to a single WebAPI. The catcher acts as a central storage and dispatcher for issues among applications. As it matches and aggregates exceptions, notifications are distributed according to defined thresholds until fixed (or marked false positive).