This post describes the usage of Vault in Fyber as well as our newly open-sourced “hashi-tools” library.

The Security challenge

About a year ago, we deployed Vault to our production environment.

We embraced Vault to get rid of some bad practices that are quite common in the industry; storing credentials statically in a configuration file, or even hard-coded within the project’s repository.

Aside from the obvious security issue, this practice usually leads to a situation where multiple services share the same credentials, which means that these credentials need to have access to many resources. This breaks the least-privileges principle and makes the revoking process of these keys - in case they are compromised - very challenging.

For this reason (and others..), we wanted to change our way of working with secrets. We figured that it would be better to have:

• Secrets for applications and systems in one secure, centralized place.

• Dynamic and programmatic application access; services can get a temporary, unique credentials(called a lease) on demand, on runtime, per service.

• A solution for the “first introduction” problem, where a service needs to hold a secret to be able to access other systems, whether directly or through a third party.

• Easy key revocation.

• Secure auditing.

Vault seemed like a natural fit, as it technically provides all of the above; it supports dynamic secrets for all popular systems and tools (cloud providers, databases, ssh, etc.), it provides a central, secure, configurable place to store secrets, it offers a variety of authentication methods which perform authentication and assigning policies to a user and services, and it has automatic revocation and auditing.

If you are not familiar with Vault, take a few minutes to review the documentation here.

The plan

In Vault, we use policies to restrict access, enforcing the “need to know” principle and instrument a “Role-Based Access Control” by specifying access privileges.

For each backend (AWS/Mysql, etc.), we define roles; each has a different set of permissions granted.

We want that each service would be able to get credentials on demand, for each system, generated by the Vault server with the appropriate permissions. When a service is replaced, rebooted, or dies, its credentials should be revoked.

We also like to use Vault as, well, a “secret vault” to keep constant sensitive information there, just like as in Lastpass.

The flow should then be something like this:

The implementation challenge

So we constructed a Vault cluster. The remaining implementation of the above workflow is on the developer’s side. I feel safe in claiming that for most developers, dealing with secrets is no brainer. Credentials with more than enough permissions either find their way somehow to a configuration file inside the node, or they just all have access to them and they can put them wherever they like in the Git repo. With Vault, to be able to comunnicate with the Vault server, the developer will have to implement an authentication mechanism in each project. The way to do it is to:

1. Store a pre-defined Vault token. Most of our services are dynamic in the sense that they scale up and down, so creating a mechanism where we generate and place a Vault token in the service config file is not ideal, and anyway we prefer not to keep any secret physically inside a node.

2. Use a Vault authentication method. Vault has pluggable authentication methods, making it easy to authenticate with Vault using whatever form works best for you. For example, you can authenticate using your personal GitHub access token. However, as in this example, there is a need to store a secret physically, and the implementation is not always trivial. In both cases, the temporary token needs to be periodically renewed by the application.

After logging-in, in order to obtain credentials from the Vault server, an HTTP call with the appropriate properties is needed; and then parse the response and periodically renew the lease, along with the regulars, such as handle failures, monitoring, etc.

The ‘hashi-tools’ library

To ease the migration of the secrets management to Vault, I wrote an SDK library. Most of our services are written in Scala, and so is the client.

The library contains two clients: ‘Secrets’ for Vault and ‘Discovery’ for Consul. The library is used by all of our backend applications that have been written in the past year.

So then let’s update the workflow:

Main features

The main features for ‘Secrets’ are:

• Scala-based.

• Friendly DSL (with Java applications as well).

• Implementation of the AWS AMI method.

• Renews its Vault token automatically if needed.

• Creates leases for Secret Engines and renews them automatically.

• It exposes operational metrics.

• It can be easily configured with the ‘application.conf’ file or with a Typesafe object.

• Async functionality.

The full documentation can be found in the project’s repository.

Usage examples

You can contribute!

There is still a long way to go to making the SKD more complete. The main tasks are here.

The syncing can be done by running a process that is not part of the Kubernetes cluster itself; in Hashicorp’s consul-k8s project, or, preferably, by installing the Consul-Helm Chart.

Hashicorp released a Consul-Helm Chart for installing, configuring, and upgrading Consul on Kubernetes.

There are decisions to be made regarding the nature of the syncing, but the first step would always be to clone the Consul-Helm project.

Before installing the Helm Chart, let’s review some of the essential configurations that are found in the standard helm values file, “values.yaml”.

Configure consul-helm

By default, the Chart resolution installs everything: a Consul server cluster, client agents on all nodes, and feature components.

If you already maintain a Consul cluster and are interested in joining the Kubernetes services to your existing cluster, then the “enable” property in the “server” section should be set to “false”:

We’ll also need to enable the Consul client and tell it what the Consul-server address is, so it can join the cluster:

We’ll also need to specify the datacenter:

Now we have to choose whether to sync to Kubernetes or Consul (or both!).

Sync to Consul means that the Kubernetes services will appear in the Consul catalog, and they could be available via HTTP or Consul-DNS. Later on, we’ll describe how to configure that.

If you already maintain a Consul cluster, you probably want to sync to Consul.

Sync to Kubernetes means that services in Consul can be made available as first-class Kubernetes services, and thus able to access them through Kubernetes Core-DNS or with any Kubernetes way. If you are not planning on using it, you might want to set the sync to Kubernetes to “false”, since you might get lost with all the Consul services that would suddenly appear as Kubernetes’ components. If you enable it for every namespace, well, it can really get messy.

For example, in the image below we’ll define a syncing to Consul and not to Kubernetes.

I recommend going over the entire “values.yaml” file and set the relevant values.

Now for the installation itself:

Clone the repository from here and perform “helm install”.

Accessing the Consul HTTP API

Access to the Consul HTTP API is through the consul-agent, Pod, we’ve created.

Every Node has a Consul agent, and there are a couple of ways to expose access to them.

One way is to create a Nodeport service. A Nodeport defines a static port, meaning that it opens a specific port on all nodes so any traffic from the outside to this port would get to this service.

The service, in turn, would forward traffic to an application labeled “consul”, which is a Consul agent.

Below is an example of such Consul Service. ‘Kubectl apply’ it to spin the service.

apiVersion: v1
kind: Service
metadata:
  labels:
    app: consul
  name: consul-http-api
  namespace: <namespace>
spec:
  selector:
    app: consul
    hasDNS: "true"
    release: consul
  ports:
  - name: http-api
    protocol: TCP
    port: 8500
    targetPort: 8500
  type: NodePort

Configure the Consul domain with the CoreDNS

CoreDNS is a DNS server that commonly serves as the Kubernetes cluster DNS. It can be configured via its “Corefile”, which is defined in the “coredns” ConfigMap. If we like to use the Consul-DNS to call our external Consul services from a Kubernetes component (instead of or along the CoreDNS addresses for Kubernetes services), we’ll need to configure Consul in the “Corefile” section.

In the CoreDNS .yaml file below, the “consul:53” section is what we are interested in.

To force all non-cluster DNS lookups to go through our consul-sync Kubernetes service so we will be able to use the Consul names. The key “consul:53” means that names that end with the string “consul” will go through the consul-sync service. Since the address of the Consul services is of this template:

<service name>.service.<datacenter name>.consul

This is exactly what we want.

apiVersion: v1
kind: ConfigMap
metadata:
 name: coredns
 namespace: kube-system
data:
 Corefile: |
   .:53 {
       errors
       log
       health
       kubernetes cluster.local 10.0.0.0/24
       prometheus :9153
       proxy . /etc/resolv.conf
       cache 30
       reload
   }
   consul:53 {
       log
       errors
       cache 30
       proxy . ${CONSUL_DNS_SERVER_ADDR} {
        # policy round_robin
        # health_check /health:8080
       }
   }

Note that we configured “proxy”. Any queries that are not within the cluster domain of Kubernetes will be forwarded to predefined resolvers, in our case the Consul service.

The address of the Consul service can be invoked with this script:

CONSUL_DNS_SERVER_ADDR=kubectl get svc consul-sync-dns -o jsonpath='{.spec.clusterIP}' -n <namespace>

To reconfigure CoreDNS with the additionals to the ConfigMap, perform:

kubectl get pods -n kube-system -oname |grep coredns |xargs kubectl apply -f kube-system

Configurations of the Consul-K8s sync components are implemented as Jenkins-Pipeline methods on my GitHub.

Identify implementation methods

You may recall we mentioned that Jbehave registered the Java implementation methods on start up. But how does it identify an implementation step?

Basically, these methods would have to override the configuration() method, which is part of the jbehave library. We have however, let ‘thucydides’ work for us. The ‘thucydides-core’ library (that comes with the ‘jbehave-plugin’), provides the ‘ThucydidesJUnitStories’ class. This identifies the ‘.story’ files and the links for the Java implementations. All we need is to have a Java class that inherits from it in the Java implementation root package:

import net.thucydides.jbehave.ThucydidesJUnitStories;
public class ApiBDDTestSuite extends ThucydidesJUnitStories {}

Final project structure:

Running the tests

At this point, we can run the tests as Junit.

Running the tests manually

In order for the black-box tests to pass, we need to have a server up and running. To run the tests with the IDE, right click on the ‘jbehave’ package.

In this fashion, we can run the tests quickly, debugging them and the server.

Automate the tests with Jetty(Maven plugin)

What we really want is to test the Volcano server as part of the maven build, the ‘maven compile’ task. In this case, we need maven to start a Volcano HTTP server, run the tests, and get Volcano down.

Eclipse Jetty is an open-source project providing an HTTP server and javax.servlet container We’ll use a Jetty maven-plugin, to start a Volcano server on which the tests will be running.

The maven plugin should be configured as follows:

<plugin>
   <groupId>org.eclipse.jetty</groupId>
   <artifactId>jetty-maven-plugin</artifactId>
   <version>${jetty.version}</version>
   <configuration>
       <httpConnector>
           <port>8083</port>
       </httpConnector>
       <scanIntervalSeconds>0</scanIntervalSeconds>
       <stopKey>stopApiTestEnv</stopKey>
       <stopPort>9999</stopPort>
       <jvmArgs>-Xms1024m -Xmx2048m -XX:PermSize=256M -XX:MaxPermSize=512M</jvmArgs>
       <contextHandlers>
           <contextHandler implementation="org.eclipse.jetty.maven.plugin.JettyWebAppContext">
               <war>${project.basedir}/../volcano_core/target/volcano_core-1.2-SNAPSHOT.war</war>
               <contextPath>/volcano</contextPath>
           </contextHandler>
       </contextHandlers>
   </configuration>
   <executions>
       <execution>
           <id>start-jetty</id>
           <phase>pre-integration-test</phase>
           <goals>
               <goal>start</goal>
           </goals>
       </execution>
       <execution>
           <id>stop-jetty</id>
           <phase>post-integration-test</phase>
           <goals>
               <goal>stop</goal>
           </goals>
       </execution>
   </executions>
</plugin>

Note that:

1. The server’s port needs to be available, you may want to declare a different port from the default that your server is using.

2. The memory you declare allocated with the jvmargs needs to be enough to get the server up

3. In ‘contextHandlers’ we provide the web application archive to be used. If you have multiple web modules in your project that need to be tested, provide them all.

4. In the executions section we tell jetty to start before the tests are executed and stop afterwards. If we don’t explicitly ask that, then the jetty would stay up.

Test Summary Reports

Test Summary Reports are an important deliverable. They are needed to reflect test results in a clear way, allowing them to be analyzed quickly.

Jbehave storyReporter supports many report formats and will generate the reports for each step and story automatically after the tests have been completed in the /target/jbehave folder.

We’ll use another Thucydides maven plugin to generate some better-looking reports:

<plugin>
   <groupId>org.apache.maven.plugins</groupId>
   <artifactId>maven-site-plugin</artifactId>
   <version>3.2</version>
   <configuration>
       <reportPlugins>
           <plugin>
               <groupId>net.thucydides.maven.plugins</groupId>
               <artifactId>maven-thucydides-plugin</artifactId>
               <version>${thucydides.version}</version>
           </plugin>
       </reportPlugins>
   </configuration>
</plugin>

This will generate various reports automatically at the end of the Maven run, in /target/site.

The ‘index.html’ file that has been created is the most comprehensive, interactive report:

A click on the ‘change password’ test (story) for example, would provide some additional data on each step result.

You can zoom in with a click on the various test components to get results and statistics.

Wrapping up

Behavior Driven Development is a methodology for developing software through example-based communication among developers, QAs, and project managers.

With Jbehave, this means writing Given-When-Then scenarios to illustrate examples in a plain text, which links to Java implementation methods.

As part of our mission to create Blackbox tests in a BDD fashion for the Volcano application, we’ve covered some test cases, discussed design and challenges, and reviewed some post-compilation reports.

The complete code can be found in my GitHub.

Registration story

Let’s zoom in on the first story, which is the Volcano registration story file:

Narrative:
As a Volcano enthusiast Dani would like to register to Volcano social network

Scenario: A new user is signing up for the Volcano system
Given dani shemesh is a Volcano enthusiast
When dani is signing up for Volcano with the user-name: dani and the password: 123456
Then dani is able to log in with the user-name: dani and the password: 123456

Note that the story is written in the third person. This will let us capture the subject (i.e. the user) later.

The ‘Given’ is linked with a Java method. The method should:

1. Carry the Jbehave @Given annotation

2. The String parameter of the annotation should match the text described in the step, while the user name (i.e.) ‘dani shemesh’ is replaced with a variable $user

3. In order to capture the user parameter, the function receives a String parameter, following the @Named annotation

The @Given implementation of this scenario is a dummy for obvious reasons.

public class UserAccount {
    Cache cache;
...
    @Given("$user is a Volcano enthusiast")
        public String user(@Named("user")String user) {
        return "let " + user + "be a volcano enthusiast";
}

Next is the implementation of the registration step where we cache the user details:

@When("$user is signing up for Volcano with the user-name: $user and the password: $password")
public void  newUser(@Named("user")String user, @Named("password") String password) {
   RequestDispatcher.createUser(user, password);
   cache.getUsers().put(user, password);
}

And the verification step, where the user should be able to log in and have a live token.

@Then("$user is $ableorNot to log in with the user-name: $user and the password: $password")
public void logIn(@Named("user")String user, @Named("ableorNotAble")String ableOrNot, @Named("password") String password) throws IOException {
   String token = RequestDispatcher.logIn(user, password).getResponseBody();
   if (Objects.equals(ableOrNot, "able")){
       assert !token.isEmpty();
       cache.getToken().put(user, token);
   }
   else assert token.isEmpty();
}

Implementation challenges

Note that, we have used a cache to store the new user’s details in the registration step.

Considering other future user actions, like “change password” and “add a friend”, the user will be required to be logged-in, means that this would be a given step:

Given dani is logged in

Now, we like to:

1. Re-use the java implementation of the log-in method (reuse @Given methods)

2. Avoid creating a new user and log-in before each scenario (reuse @When actions)

This brings up unexpected challenges from an OOP point of view. Before discussing them, let’s explain why

The Jbehave environment

Though we are writing in Java, we are not in an OOP scope, but in Jbehave’s.

When JBehave starts up, it registers all the implemented methods and their parameters, then reads the steps in each scenario for each story and maps them to the appropriate method in the code behind. Here all their values are in memory and are executed one by one. The memory is cleaned after each story.

Re-use the @Given methods

This one is easy. Instead of inheritance/composition, we can place the @Given methods wherever we like, and Jbehave will identify it. So, we’ll create a dedicated class for given methods, dealing with user scenarios:

public class GivenUserSteps {
Cache cache;
...
@Given("$user is a Volcano enthusiast")
public String user(@Named("user") String user) {
   return "let " + user + "be a volcano enthusiast";
}

@Given("$user is logged in")
public void logIn(@Named("user") String user) throws IOException {
   String session = RequestDispatcher.logIn(user, cache.getUsers().get(user)).getResponseBody();
   cache.getToken().put(user, session);
}

Implement tests for similar entities in the same class

We can once again take advantage of the Jbehave environment to place test implementations with similar characters together, even if they implement steps of different stories, for example:

public class UserAccount {

   Cache cache;

   @When("$user is signing up for Volcano with the user-name: $user and the password: $password")
   public void  newUser(@Named("user")String user, @Named("password") String password) {
       RequestDispatcher.createUser(user, password);
       cache.getUsers().put(user, password);
   }

   @When("$user is changing his password to $newPassword")
   public void  changePassword(@Named("user")String user, @Named("newPassword") String newPassword) throws IOException {
       String response = RequestDispatcher.changePassword(cache.getToken().get(user), user, cache.getUsers().get(user), newPassword).getResponseBody();
       if (response.equals("OK"))
           cache.getUsers().put(user, newPassword);
   }
….
}

Re-use actions with Dependency Injection using Spring

In order to reuse actions, in our case the user that we have created and logged in with, in the “registration” step, we need to cache. We cannot put the cache in a global variable or static at some place, since we are working with different classes and the memory cleans up after each story.

Luckily, Jbehave supports some of the most popular, Java based, dependency injection plugins. The most popular is probably Spring, and we’ll use it to inject the resources (i.e. the cache) we need for reusing actions.

Configure Spring

We’ll use ‘org.springframework’ artifacts for the Spring integration:

<artifactId>spring-context</artifactId>

To make our test classes to be singleton spring beans, using the @Service annotation

<artifactId>spring-test</artifactId>

to identify the test classes beans under com.fullgc.jbehave namespace, using @ContextConfiguration annotation and a spring-context xml file.

* <artifactId>spring-beans</artifactId>

To inject the resources to the test classes, using the @Autowired annotation

And the following Thucydides artifact

<groupId>net.thucydides</groupId>

<artifactId>thucydides-junit</artifactId>

For the integration of the bean classes and with Jbehave, using SpringIntegration class.

The UserAccount test class now looks like this:

@ContextConfiguration(locations = "/spring-context.xml")
@Service
public class UserAccount {

   @Rule
   public SpringIntegration springIntegration = new SpringIntegration();

   @Autowired
   CacheBean cache;

   @When("$user is signing up for Volcano with the user-name: $user and the password: $password")
   public void  newUser(@Named("user")String user, @Named("password") String password) {
       RequestDispatcher.createUser(user, password);
       cache.getUsers().put(user, password);
   }

   @When("$user is changing his password to $newPassword")
   public void  changePassword(@Named("user")String user, @Named("newPassword") String newPassword) throws IOException {
       String response = RequestDispatcher.changePassword(cache.getToken().get(user), user, cache.getUsers().get(user), newPassword).getResponseBody();
       if (response.equals("OK"))
           cache.getUsers().put(user, newPassword);
   }
...
}

Note that the new user is cached and reused for a “change password” action.

(We use the cache in the same fashion as the ‘add friend’ story, which we won’t cover here, but can be found in the ‘Volcano’ repository)

The other test classes should be modified in a similar fashion.

The spring context:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
      xmlns:context="http://www.springframework.org/schema/context"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xmlns:task="http://www.springframework.org/schema/task"
      xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
       http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd">

   <context:component-scan base-package="com.fullgc.jbehave"/>
   <task:annotation-driven />
</beans>

Next

In Part-3 we’ll learn how to launch the web app in compile time, run the tests and generate summary reports.

In this guide, we’ll learn to design, develop and automate Black-box tests for a JEE web application in a BDD fashion. We’ll develop on top of Jbehave framework.

Glossary

Black-box testing

According to Wikipedia, “Black-box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings”. As such, Black-box testing focuses entirely on the inputs and outputs of the software system – the “black box”.

Behavior-driven development(BDD)

Behavior-driven development is an extension of test-driven development that makes use of a simple, domain-specific scripting language.

In BDD, you describe what you want the system to do by talking through example behavior. Work from the outside-in to implement those behaviors using examples to validate you’re what you’re building.

The customer (who may be a Scrum Product Owner) describes what they want, and the developers ask questions to flesh out enough detail about the behavior to be able to implement it”. (BDD in a nutshell)

Structure:

• A BDD story is a description of a requirement and its business benefit, and a set of criteria by which we all agree that it is “done”. There should be a story for each feature.

• A Story consists of a narrative and one or more scenarios.

• A narrative is a short, simple description of a feature told from the perspective of a person or role that requires the new functionality. The narrative shifts the focus from writing features to discussing them (technologyconversations.com).

• A scenario consists of steps, in the format of ‘Given-When-Then’. ‘Given’ and ‘When’ trigger actions, and ‘Then’ is the verification:

Frameworks and Tools

Jbehave in a nutshell

JBehave is an open-source framework for Behavior-Driven Development.

It supports Java-based development, and plain English is used to form the story.

The steps in the story are visually linked to a corresponding Java method:

JBehave supports multiple mechanisms for parameter injection. In the above example, the ‘user’ and ‘password’ as arguments extracted from the @When step, with the @named annotation, following a natural order to the parameters in the annotated Java method.

In addition, Jbehave provides an easy way to create more intelligent data types than these strings. There are multiple plugins for generating comprehensive and interactive reports.

There are many (many!) advanced features that are worth checking out (see advanced topics in the Jbehave site); we’ll only be using a few of them. We’ll explain how Jbehave works at a lower level later.

Thucydides

Thucydides is a tool designed to make writing automated acceptance tests easier.

Thucydides and JBehave work well together. Thucydides uses simple conventions to make it easier to get started in writing and implementing JBehave stories. It reports on both JBehave and Thucydides steps, which can be seamlessly combined in the same class.

Requirements and tools

• There are JBehave plugins for IntelliJ-Idea and Eclipse. Both come with a custom JBehave Story Editor which provides a syntax highlighting, step hyperlink detection and link to corresponding Java method, step autocompletion, detecting both unimplemented steps and more. Hence, one of these IDE is required.

• We use maven for import libraries, build, run and automated tests

• You can download and follow the source code through the guide. The dispatcher of the ‘tests’ module is written in Scala. To run it you’ll need a Scala SDK.

Jbehave Plugin and the ‘Volcano’ Stories

We use Idea IntelliJ-IDE with ‘Jbehave support’ plugin for writing the stories and the code behind.

Volcano

‘Volcano’ is intended to be a social network for Volcano enthusiasts.

The project manager of ‘Volcano’ would like to add some basic features and behaviors:

1. User account

   1. Registration to ‘Volcano’.

   2. Change password.

2. User network

   1. Add a new friend.

Each feature will be described in it’s own Jbehave story file. Here is how the “Registration” story looks like before applying the ‘‘Jbehave support’’ plugin:

And after:

At this stage the steps are marked in red, and we get a message when the mouse hovers that there is no Java method that linked with the steps.

Initial Dependencies

Here Thucydides kicks in.

We’ll use ‘net.thucydides’ libraries for the implementation.

The following Jbehave plugin artifact:

<groupId>net.thucydides</groupId>
<artifactId>thucydides-jbehave-plugin</artifactId>

This includes the Jbehave libraries that are required for the Java implementation code behind:

‘Jbehave-core’ provides the basic Jbehave BDD building blocks: The @Given @When @Then Annotations, and the annotations responsible for the parameter injection (i.e. @Named) ‘Jbehave-junit-runner’ provides functionality for the story and scenarios lifecycle and reporting.

Later on, we’ll use it to identify the stories and run the test.

Next

In Part-2 we’ll implement the ‘Registration’ story, and review solutions to the implementation challenges.

Before we’ll begin writing the Jenkinsfile, keep in mind that there are many ways to implement a CI/CD process. The flow we’ll discuss and implement is just one approach. Moreover, there are usually multiple ways of writing a command in the Jenkinsfile: Native Groovy (Pipeline plugin DSL is groovy based), use a shell script, a Pipeline script code, external libraries, etc..

Multibranch Pipeline

In a Multibranch Pipeline project, Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control (It is possible to write a Pipeline script directly in the job configuration though). It enables an implementation of different Jenkinsfiles for different branches. However, here we’re going to implement a single Jenkinsfile for multiple branches.

Configuration

The entire definition of the Pipeline would be written in the Jenkinsfile, except the following: In the ‘Branch Source’ we’ll declare the source control and repository that we’ll work with (this can also be done with code). In addition, we’ll set the branch discovery strategy to “All Branches”, meaning the job would start for every modification of the repository (i.e. for every push).

Then we’ll exclude the release and ‘hotfix’ branches (this will be explained later).

Writing the Jenkinsfile, step-by-step

Context

The Pipeline job should be run on a dedicated Jenkins slave, ‘server CICD’, hence the script would be written inside a node context:

node('Server CICD') { }

Checkout

This step checkouts code from source control. Scm is a special variable which instructs the checkout step to clone the specific revision which triggers this Pipeline run.

stage('Checkout') {
   checkout([
           $class           : 'GitSCM',
           branches         : scm.branches,
           extensions       : scm.extensions + [[$class: 'LocalBranch', localBranch: '']],
           userRemoteConfigs: scm.userRemoteConfigs
   ])
}

Build

a. Maven build: We are using the maven build tool, and trigger a maven build with a shell command. We like to get a detailed report from Pipeline on a failure, including failed tests, links to them, and statistics. Moreover, we like the job status to become automatically ‘unstable’ if there were failed tests. These are provided by the Pipeline Maven plugin, which wraps the maven build command.

withMaven(jdk: 'JDK 8 update 66', maven: 'Maven 3.0.5') {
           sh "mvn -Dmaven.test.failure.ignore=true clean install"
}

b. Handle build exceptions and test failures: On maven build failure:

If Pipeline checked out a feature branch (triggered by a push to a branch which starts with ‘ST-‘ ), a notification email should be sent to the feature owner only. We’ll use the Mailer plugin for that.

Otherwise, we like an email notification to be sent to all server members, and a notification to the slack channel as well (Slack plugin). This should include a list of the last Git commits with the committer name, so that we can get an idea of what code modification broke the build.

If an exception has been thrown during the build, we like to:

1. Catch it

2. Change the build status to ‘failure’

3. Send the appropriate notifications

4. Throw the exception

The final script for the build looks like this:

String branch = env.BRANCH_NAME.toString()
stage('Maven build') {

    //returns a set of git revisions with the name of the committer
   @NonCPS
   def commitList = {
       def changes = ""
       currentBuild.changeSets.each { set ->
           set.each { entry ->
               changes += "${entry.commitId} - ${entry.msg} \n by ${entry.author.fullName}\n"
           }
       }
       return changes
   }

   def handleFailures = {
       if (branch.startsWith("ST-")) {
           step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: emailextrecipients([[$class: 'RequesterRecipientProvider']]), sendToIndividuals: true])
       } else {
           step('Send notification') {
               mail(to: 'server@fullgc.com',
                       subject: "Maven build failed for branch: " + env.BRANCH_NAME.toString(),
                       body: "last commits are: " + commitList().toString() + " (<$BUILD_URL/console|Job>)");
           }
           slackSend channel: 'server', color: 'warning', message: "Maven build failed for branch ${branch} \". \nlast Commits are: \n" + commitList().toString() + "\n (<$BUILD_URL/console|Job>)"
       }
   }

   try {
       withMaven(jdk: 'JDK 8 update 66', maven: 'Maven 3.0.5') {
           sh "mvn -Dmaven.test.failure.ignore=true clean install"
       }
       if (currentBuild.result("UNSTABLE")) {
           handleFailures()
       }
   } catch (Exception e) {
       manager.buildFailure()
       handleFailures()
       throw e
   }
}

Release process

In this process, we’ll upload a tar (the maven build output) to s3, where the environment depends on the git branch we’re working on. The code would be placed in the ‘process’ step:

a. Release tar name. The release file is a tar file (the maven build output). Its name should represent the release version. The release version is found in the root pom.xml file, and we’ll extract it from there

b. Release candidate tar name. This is somewhat tricky. On release/hotfix only, we like to create a release candidate for QA. The release candidate holds the name:

<volcano version>RC-<RC number>

In our story, the first release candidate would be: ‘volcano-1.2.0-RC-1’(‘volcano-1.2.1-RC-1’ in the case of a hotfix).

The RC number starts with 1. If a QA person found a bug, we’d need to fix it, and then increment the RC number, i.e. ‘volcano-1.2.0-RC-2’ and so on.

We’ll use a text file with the current RC number to know what the next version should be for release. We then update the file, commit changes and create a new tart with the correct name.

def pom = readFile 'pom.xml'
def project = new XmlSlurper().parseText(pom)
String version = project.version.toString()
String tarName = "volcano-${version}-release-pack.tar.gz"

stage('Release'){

@NonCPS
def newVersion = {
   def doesFileExist = fileExists 'releases.txt'
   if (doesFileExist) {
       echo "file releases.txt exists"
       def file = readFile 'releases.txt'
       String fileContents = file.toString()
       Integer newTarVersion = fileContents.toInteger() + 1
       writeFile file: 'releases.txt', text: newTarVersion.toString()
       echo "new tar version is: " + newTarVersion
   } else {
       echo "file releases.txt does not exist"
       writeFile file: 'releases.txt', text: "1"
   }
   readFile 'releases.txt'
}

if (branch.startsWith("release") || branch.startsWith("hotfix")) {
   String tarRCVersion = newVersion()
   newTarName = "volcano${version}-RC-${tarRCVersion}-release-pack.tar.gz"
   sh "mv ./volcano/target/${tarName} ./viper/target/${newTarName}"
   tarName = newTarName
}

step('Commit and push releases file') {
   sh "git remote set-url origin git@bitbucket.org:fullgc/volcano.git"
   sh "git add -A"
   sh "git commit -m 'update volcano version to '${tarRCVersion}"
   sh "git push"
}

Note that we did exclude the release/hotfix branches. This allows a couple of team members to work on the branch when QA has made a rejection or there is a bug to fix, without the new version being released with every push.

Upload tar to s3

We won’t implement the deployment process with the Pipeline script, and leave it for the deployment tool, ‘Chef’ for the sake of this illustration. Chef will deploy a new volcano app, with the appropriate version in s3. This would require amazon s3 credentials. For the upload itself, there is a pipeline script. Nevertheless, we’ll implement it here using Amazon CLI commands, using the shell.

step('Upload tar to s3 cli') {
   withCredentials([[$class: 'AmazonWebServicesCredentialsBinding', credentialsId: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', accessKeyVariable: 'AWS_ACCESS_KEY_ID', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
       sh "pip install --user awscli"
       sh "sudo apt-get -y install awscli"
       sh "aws s3 cp ./volcano/target/${tarName} s3://fullgc/tars/"
   }
}

Deployment process

We won’t be diving too deeply into how Chef performs a deployment, but suffice to say this: In order for Chef to know that there is a new ‘volcano’ version it needs to deploy, the version in the environment (qa or development or production) file needs to be updated to the new version.

a. First, we’ll check out the Chef repository and ‘cd’ in the environments directory which the environment files rely on.

b. Replace the current version of the appropriate environment for the new version. This can be done by shell tools like jq. Here we’ll use Groovy.

c. Commit and push changes

d. Send a slack notification

e. If the branch is develop or master, it removes the releases.txt file

if (!branch.startsWith('ST-')){
  stage('Deploy') {

  def incrementVersion = {
     def environment = {
     switch(branch) {
         case 'develop': 'development.json'
             break
         case 'master': 'production.json'
             break
         default: 'qa.json'
     }
  }
   def environmentFileContent = readFile environment
   def environmentJson = new groovy.json.JsonSlurper().parseText(environmentFileContent)
   environmentJson.default_attributes.volcano.version = "volcano-${version}"
   String environmentPrettyJsonString = new groovy.json.JsonBuilder(environmentJson).toPrettyString()
   environmentJson = null
   writeFile file: $environment, text: environmentPrettyJsonString

  sh "git commit -am 'Changed volcano version in $environment env to '${version}"
  sh "git push"
}

checkout changelog: false, poll: false, scm: [$class: 'GitSCM', browser: [$class: 'BitbucketWeb', repoUrl: 'https://bitbucket.org/fullgc/chef'], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'LocalBranch', localBranch:
  '**']], submoduleCfg: [], userRemoteConfigs: [[url: 'git@bitbucket.org:fullgc/chef']]]
   dir('environments/') {
       incrementVersion()
   }
slackSend channel: 'server', color: 'good', message: " New volcano version ${version} is being deployed to $environment"
  }
if (branch == 'develop' || branch == 'master') sh "rm releases.txt"
}

In the image below an example of a Pipeline run:

Tips

• To save a lot of time, use the Pipeline Syntax generator in your own Jenkins for every pipeline command

• While working on the Jenkinsfile, you don’t have to commit and push every modification just to test an execution. You can use run an execution using ‘replay’ until everything works.

• Jenkins Blue Ocean plugin provides a new awesome user experience, check it out.

• Pipeline is still new, but you shouldn’t get too frustrated by weird errors you may get while writing the script, most of them are common and the solution can be easily found on the web.

• The Pipeline Unit Testing Framework allows you to unit test Pipelines before running them in full.

Wrapping up

Pipeline as a code is pretty much a game changer, in the sense that it is now in the hands of every programmer, allowing them to write a full release (and deployment) process, that can fit the development workflow easily.

The complete code can be found in my GitHub.

This section covers the green and purple steps in the workflow graph (from part-1).

The following is the basic plugin configuration needed for our story. It gives a name to the Git branches and tags. The ‘scmCommentPrefix’ would be the prefix for the commits performed by ‘jGit-flow’(i.e. the Git ‘squash’ operation)

<configuration>
  <flowInitContext>
     <masterBranchName>master</masterBranchName>
     <developBranchName>develop</developBranchName>
     <featureBranchPrefix>ST-</featureBranchPrefix>
     <releaseBranchPrefix>release-</releaseBranchPrefix>
     <hotfixBranchPrefix>hotfix-</hotfixBranchPrefix>
     <versionTagPrefix>volcano</versionTagPrefix>
  </flowInitContext>
  <scmCommentPrefix>JgitFLow step: </scmCommentPrefix>
</configuration>

A Feature Lifecycle:

Start a feature git flow process

A new feature starts with the command mvn jgitflow:feature-start.

• This prompts the user for the feature-branch name, which should carry the ticket ID (in our story, ‘ST-145’):

A new feature branch is then checked out from ‘develop’ and the ticket status is updated to ‘IN PROGRESS’:

Complete a feature git flow process

The command mvn jgitflow:feature-finish ends the feature lifecycle.

It prompts the user for the desired feature name to finish, where the default is the current branch.

The feature branch is then merged into ‘develop’, and ‘develop’ is pushed.

Before the merge, we like branch ‘develop’ to be pulled and the commits in the feature branch to be squashed, and so cleaner.

To achieve that, we’ll add the followings configuration to jgit-flow plugin:

<pullDevelop>true</pullDevelop>
<squash>true</squash>

When the feature is done, we like to resolve it and pass to the QA guy. Hence, it’s resolution should be switched to ‘Done’, and it’s status ‘QA’

A Release Lifecycle:

After all the features of the next version have been completed and merged to ‘develop’ branch, it’s time for the git flow release process to kick in.

Start a release git flow process

A release process starts with the command mvn jgitflow:release-start on branch ‘develop’, which prompts the user for the version name.

The default is the next major version (according to the pom file), in this case, 1.2.0. Then it performs the following actions:

• Pulls from the remote ‘develop’ branch (we’ve already configured).

• Updates and commits the poms with the new version. This requires the following tag attribute:

<autoVersionSubmodules>true</autoVersionSubmodules>

• Checks out to the new ‘release’ branch, called release-VERSION (e.g. release-1.2.0).

• Pushes (this would trigger a release process of a ‘release candidate’ for QA machines. (The release processes will be described in the next section, ‘Pipeline’)

Complete a release git-flow process

Once the version is approved by QA, the ‘git-flow’ release can be completed. The command mvn jgitflow:release-finish performs the following:

• Merges the ‘release’ branch back into ‘master’

• Tags the ‘release’ with its name.

• Back merges the ‘release’ into ‘develop’

• Updates ‘develop’s poms with ‘-SNAPSHOT'. (1.3-SNAPSHOT).

• Removes the ‘release’ branch.

Checkout and push ‘master’ branch would start the release process

A Hotfix Lifecycle

When there is a need for a quick fix for a code that is already in production, the ‘git-flow’ hotfix comes to the rescue.

Start a hotfix git flow process

The ‘master’ branch is always synced with the latest deployment code. The Feature starts with the command mvn jgitflow:hotfix-start prompts the user for the version name.

The default is the next minor version (according to the pom file), in this case, 1.2.1

• Pulls from the remote ‘master’ branch (done by adding the configuration).

<pullMaster>true</pullMaster>

• Updates and commits the poms with the new version.

• Checks out to the new ‘hotfix’ branch, called hotfix-VERSION (e.g. release-1.2.1).

• Pushes (like the stage that triggers a release of a hotfix candidate’ for QA machines).

Complete a release git-flow process

Once the version is approved by QA, the ‘hotfix’ git flow can be completed. The command mvn jgitflow:hotfix-finish performs actions which are quite similar to the ‘release-finish’:

• Merges the ‘release’ branch back into ‘master’

• Tags the release with its name.

• Back merges the release into ‘develop’

• Restores poms version in ‘develop’ (the version is still 1.3-SNAPSHOT)

• Removes the ‘release’ branch.

Checkout and push ‘master’ branch would start the release process

Summary

We’ve reviewed the git workflow of a new feature, ‘release’ and ‘hotfix’, and ended up with the following ‘jgit-flow’ configuration:

<configuration>
    <flowInitContext>
         <masterBranchName>master</masterBranchName>
         <developBranchName>develop</developBranchName>
         <featureBranchPrefix>ST-</featureBranchPrefix>
         <releaseBranchPrefix>release-</releaseBranchPrefix>
         <hotfixBranchPrefix>hotfix-</hotfixBranchPrefix>
         <versionTagPrefix>volcano</versionTagPrefix>
    </flowInitContext>
    <scmCommentPrefix>JgitFLow step: </scmCommentPrefix>
    <pullDevelop>true</pullDevelop>
    <pullMaster>true</pullMaster>
    <squash>true</squash>
    <autoVersionSubmodules>true</autoVersionSubmodules>
</configuration>

In part-3 we’ll review the Jenkins Pipeline script, which compiles, runs the tests and performs the release process.

Let’s start with a quick review of the tools we’ll use for the workflow implementation

Jira

Atlassian Jira is a popular proprietary issue tracking system.

We’ll manipulate Atlassian Jira feature tickets along the flow. This can be skipped if you don’t use Jira.

The project we’ll manage would be part of the Server team (ST) and the feature that we like to implement and deploy would be ST-145.

Its initial ticket status is ‘open’, the resolution is ‘unresolved’:

GitFlow

GitFlow is a branching model for Git, created by Vincent Driessen.

The GitFlow workflow defines a strict branching model designed around the project release. It uses the following branches:

• Master: Stores the official release history. The origin/master is the main branch where the source code of HEAD always reflects a production-ready state.

• Develop: Serves as an integration branch for features

• Feature: Each new feature resides in its own branch. Feature branches use ‘develop’ as their parent branch. When a feature is complete, it gets merged back into ‘develop’

• Release: Supports preparation of a new production release.

• Hotfix: When a critical bug in a production version must be resolved immediately, a ‘hotfix’ branch may be branched off from the corresponding tag on the ‘master’ branch that marks the production version.

If you’re new to git-flow, please take some time to read about it in Driessen’s post or in Atlassian’s Guide.

Jgit-flow (Maven plugin)

JGit-Flow maven plugin is a Java implementation of GitFlow, and like Jira, it was published by Atlassian. It designs for releasing a maven-based project and includes many other useful features.

‘jGit-flow’ provides the following git-flow basic functionality:

• jgitflow:feature-start Starts a feature branch

• jgitflow:feature-finish Merges a feature branch

• jgitflow:release-start Starts a release

• jgitflow:release-finish Merges a release

• jgitflow:hotfix-start Starts a hotfix

• jgitflow:hotfix-finish Merges a hotfix

Each feature contains many attributes, providing very useful functionality (described in the links), that we’ll use later on.

Jgit-flow-jira

JGit-Flow-Jira is a fork that I made for ‘jgit-flow’, which uses a Jira client to change the state of a Jira ticket during the lifecycle of a feature. Unfortunately, jgit-flow is not bug-free, and currently maintained mostly by the users and not by Atlassian. It is, however, published as open source and written very clearly. Jgitflow-jira contains a fix for this open bug as well.

Jenkins(Pipeline)

Jenkins Pipeline (or simply “Pipeline”) is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins.

As opposed to the historic Gui-driven CI/CD tools for Jenkins jobs, the definition of a Pipeline is written into a text file (called a Jenkinsfile) as a code. This in turn can be committed to a project’s source control repository.

We will use Pipeline for build, tests and release.

The Pipeline script would be written in Groovy and would use Jenkins syntax and shell commands.

Complete development, release and deployment plan

The flow-chart below describes the entire workflow, from the Jira ticket to deployment, that we’ll learn how to implement in the following sections.

We’ll review a development flow of a feature that was assigned to the ‘server team’ called ‘ST-145’, and the process of releasing and deploying the next version: v 1.2.0, of an application called ‘volcano’.

There are many shapes and arrows in the graph, but there’s no need to make sense of them all right now, since we’re going to do exactly that in the following sections.

This part focuses on Akka metrics, meaning a high-level data on the Akka Objects that we’ve configured (i.e. Dispatchers, Routes, Actors (Routees), and Messages). We will consider how we gather them at Inneractive and give some useful tips.

Monitoring Tools for Akka

The Akka library does not include a native monitoring tool. However, there are a few tools that provide additional metrics for an Akka-based application (i.e. memory usage, trace information), profiling and further capabilities that help identify performance issues or identify a bottleneck.

To name a few:

• ‘Lightbend Monitoring’

  Provides all necessary features, including key Akka metrics and span traces.

  Takipi plugin provides actor events that can trigger debug snapshots of the stack trace, i.e. the state at the time of the error. By the way… It’s not free…

• Newrelic

  A powerful performance monitoring and management framework, with Spray and

  Akka-Http instrumented features. However it’s most important feature, at least regarding our purpose, the X-ray, which gives deeper insight into key transactions, and is available only with a ‘Pro’ subscription.

• Flame Graphs / VirtualVm or any other JVM profiling tool

• Kamon

Kamon

Overview

Kamon is an open source tool for monitoring applications running on the JVM. It supports various backends and has modules that integrate and gather metrics for Akka, Play, Spray/Akka-Http and more.

Kamon uses Aspectj to add a layer of code before and after a method is executed, and in this aspect- oriented fashion records metrics data (e.g. Records time before and after processing of messages/futures/routing (See Appendix).

Here at Inneractive we use Kamon to gather all metrics, including application custom metrics, even in applications that do not use Akka, because it proves to perform better (CPU wise) than other alternatives.

It contains metric types functionalities like .histogram(..), .counter(..), .gauge(..) and .minMaxCounter(..)

Configuration for Kamon Akka metrics

Akka integration has a collection of metrics for actor, router and dispatcher objects. Firstly, they are all collected in our Exchange server. Soon enough our monitoring tools, Prometheus and Datadog went down / froze because of a crazy load of metrics. The reason is that we have 1000 to 1500 Actor instances. The metrics’ names include the instance name (it is not a Tag as you might expect). We do have our own tags like ‘environment’ and ‘host’, and about ~600 Exchange instances. There you get the vast number of metrics that our monitors could not handle.

In addition, the ‘aspect’ way of gathering metric’s data is quite expensive, performance wise.

Our metric configuration looks as follows:

"metric": {
      "filters": {
       "akka-actor": {
         "excludes": ["**"],
         "includes": []
       },
       "akka-dispatcher": {
         "excludes": [],
         "includes": ["**"]
       },
       "akka-router": {
         "excludes": [],
         "includes": ["**"]
       },
       "trace": {
         "excludes": [],
         "includes": [ "**"]
       }
     }
    ...
    }

As you may notice, we exclude akka-actor, which means all routees metrics (time-in-mailbox, processing-time, mailbox-size), and gather metrics for actors ourselves. This is because:

• Routees metrics outnumbered all the others combined, since we have lots of actor instances, and we desperately needed our monitors to be up and running…

• We don’t really mind about a single routee in pretty much all cases, so gathered information like average, sum, 95 can do the trick

• The metrics’ names are somewhat awkward (partially because the name of the routees is like $a $b… and naming them requires them to be created explicitly in the code)

• We know the names of our actors, and have a foothold in our own actor’s code; so we could easily gather metrics ourselves, and with a more concise information. In addition, we wanted metrics about the messages, processing time and the time it took the message to arrive.

Tips

1. When you route manually, meaning when you don’t have a router, or whenever you have created the routees explicitly, you can give them names yourself(see actorOf parameters).

2. Akka does not provide an API to know the size of the mailbox. Instead, you can monitor by:

• Have a messages counter (like Kamon does)

• Override the MessageQueue. Google it.

Configuration for Kamon Akka ActorSystem

As stated, the aspectj operating by Kamon is quite expensive, even when excluding of all routees metrics. By default, Kamon uses the default-dispatcher. If you don’t set a default-dispatcher yourself, the threadpool size of the default-dispatcher will be the number of cores. In practice Kamon told us that about 75% of the running threads were Kamon’s..

You can set a dispatcher for Kamon as follows:

"internal-config": {
 "akka": {
   "actor": {
     "default-dispatcher": {
       "fork-join-executor": {
         "parallelism-factor": 0.25,
         "parallelism-max": 2,
         "parallelism-min": 1
       }
     }
   }
 }
}

Here we allow up to two threads (but even one would probably be enough), with a parallelism-factor of 0.25, meaning up to two threads when the machine has at least eight cores. This does the job.

Appendix

class ActorMonitor.scala from the module kamon-akka

83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
109
110

 def processMessage(pjp: ProceedingJoinPoint, envelopeContext: EnvelopeContext): AnyRef = {
       val timestampBeforeProcessing = RelativeNanoTimestamp.now

      try {
        Tracer.withContext(envelopeContext.context) {
          pjp.proceed()
        }

      } finally {
        val timestampAfterProcessing = RelativeNanoTimestamp.now
        val timeInMailbox = timestampBeforeProcessing - envelopeContext.nanoTime
        val processingTime = timestampAfterProcessing - timestampBeforeProcessing

        actorMetrics.foreach { am =>
          am.processingTime.record(processingTime.nanos)
          am.timeInMailbox.record(timeInMailbox.nanos)
          am.mailboxSize.decrement()
        }
       ...
      }
    }

…
  }

Line 83 - 'processMessage' executes 'around'(before and after) message processing by the actor(jpg, line 88)
Line 84 - Takes start time before the message being processed
Line 88 - Invokes the method that handles message is processing
Lines 92-94 - Latency calculations
Lines 96-99 Updates metrics cache

Monitor with Stackable Actor Traits

We like to collect aggregated data for the routees and messages then.

To monitor time-in-mailbox, processing-time, mailbox-size that we excluded from Kamon metric configuration, we need to monitor ‘around’ the message processing, i.e. around the ‘receive’ method

It can be achieved by imitating Kamon’s usage of Aspectj as we just saw. However, we use stackable-traits mixed to the Actors, for a monitoring layer around ‘receive’.

I wrote about the stackable traits pattern. In Part-2 I described (a simplified version of) how we use stackable actor traits at Inneractive, with code samples.

Dashboard Overview

Our dashboard consists of Kamon’s Akka metrics and custom metrics.

Akka metrics collection is explained in detail in the docs

Among all metrics, put special attention to:

• ‘active-threads’ and ‘running-threads’, these give a very good view of your threads distribution so you can tune the dispatcher’s configuration if needed

• ‘pool-size’, when you set upper, lower and increment factor and not a fixed size. This tells you how many threads are allocated in the pool, in practice.

• ‘processed-tasks’ - How busy is the executor, maybe more threads are needed

• ‘routing-time’ - indicates if the routing strategy fits.

• Custom metrics - ‘time-in-mailbox by actor/message’ for routees. Maybe a specific message handling is causing the problem.

• Custom metrics ‘message-start-process’ for routees. When the value is high, it can indicate that the receiver needs more resources or there are too many instances. This leads to lots of context-switches. An increase of the “throughput” parameter may help.

Wrapping up

Monitoring of the Akka objects plays an important and integral part of the tuning step (and is generally important for keeping track of the system behavior). There are various tools and ways to do so, Kamon is friendly and recommended, and you can gather the metrics yourself.

References

Kamon Documentation

Kamon-Akka repository

Stackable Actor Traits

Let’s start with Akka configuration, specifically the configuration of actor-instances, routing strategy and dispatchers & executors. Below is the relevant section of the application.conf

 {
  akka {
    actor {
      akka.actor.deployment {
        /my-service {
           nr-of-instances = ???
           router = ???
           dispatcher = "my-dispatcher"
    }
  my-dispatcher {
    executor = ???
    type = ???
}

The number of actor instances

I like to start by thinking about how many instances of an actor are suitable?

 akka.actor.deployment {
    /my-service {
      nr-of-instances = ???
    }
}

The size may depend on other configurations like routing strategy, dispatcher, threadpool size and more. Nevertheless, the nr-of-actor ‘strategy’ can already be decided at this point. Let’s review our options and use cases:

Single instance (or- Domain actor)

• A dedicated actor for low-priority side-effects like sending metrics, write to a log or to a cache and so forth.
• A mutable, single-source that needs to be handled(Cache)
• When you need to work sequentially for whatever reason

Fixed number of instances

• Instance per a copy of resource, or per a mutable resource
• For sharding, i.e. when you manage a distributed key-value cache and want to shard the inputs, then you may want an actor to manage each shard
• To execute tasks in parallel, and you don’t think you’ll need to manage Back-Pressure nor to scale up

Resizeable number of instances(when using a router)

akka.actor.deployment {
  /parent/router {
    resizer {
      lower-bound = ???
      upper-bound = ???
      pressure-threshold = ???
      messages-per-resize = ???
      ...
    }
  }
}

It is possible to configure resizable routees (actor instances managed by a router).

Routees can be added or removed dynamically, based on performance. You can configure specifically how much to scale up and down in case of unusual behavior.

Scale / Back-Pressure DIY!

When one component is struggling to keep-up, the entire system needs to respond in a sensible way.

You’re might be somewhat familiar wit Akka-Streams, widely known as a framework that manages your back-pressure. It’s possible to imitate the general behavior by yourself.

Let’s review some scenarios in which you may want to scale your routees:

The producer(In our use case, one of your actors), can produce faster than the received consumer(actor or any other source) can handle.

In this case you may:

• Back-pressure the producer, i.e. reduce the number of producer’s routees.

• Add more consumers(routees…)!

• Leave it. You don’t necessarily need to back-pressure. It may lead to a loss of messages (bounded mailbox) or running out of memory…

The consumer is faster than the producer.

Here the consumer will block waiting for the next item.

• Remove some consumers(routees…)!

• Add more producers if your system can theoretically produce faster.

• Leave it. Then you may not get the most from your machine.

Actor per-request

“You press, you make a request, the Meeseeks fulfills the request, and then it stops existing”(Rick Sanchez)

Actor per request works very similarly. An instance is created for every request, process it and then it will be destroyed.

You can configure Spray/Akka-HTTP to work in actor-per-request mode or do it yourself. However, it is not part of the Akka configuration, so I won’t go into too much detail. In a nutshell:

• Easy to manage state in the actor, because the context is always of a specific request, hence you don’t have to maintain any mapping of State => Request

• And here are some more insights

Note that there is a context-switches overhead which could theoretically lead to memory issues

Routing

Akka provides “strategies” for the Akka router to define the workload distribution among actors.

akka.actor.deployment {
    /my-service {
      router = ???
    }
}

Strategies Overview

Let’s quickly review the the routing strategies

• Random - Distributes messages randomly

• Round-Robin - Distributes messages in sequence

• Smallest-Mailbox - Sends the message to the smallest mailbox

• Broadcast - Distributes every message to all routees.

• Scatter-Gather-First - Distributes every message to all routees. Only the first to respond will execute the task.

• Tail-Chopping - Sends the message to one, randomly picked, routee and then after a small delay to a second routee.

• Consistent-hashing - Uses consistent hashing to select a routee based on the sent message

• In-Code - Custom your own routing by routing it yourself

Strategies Cheatsheet

*Can be solved by increasing the number of routees (which may cost in context-switches overhead)

**As a replacement for ‘smallest mailbox’. Latency differences could be high among connections to remote actors)

***The overhead depends on the task, whether it on the same machine or not

Dispatchers and Executors

akka.actor.deployment {
    /my-service {
      dispatcher = ???
      type = ???
    }
}

Dispatchers are what makes Akka actors “tick”, means put messages in mailboxes and route them. In addition, they are also an implementation of ExecutionContext, means they can execute Runnables and so a Scala Future.

my-dispatcher {
  executor = ???
  throughput = ???
....
}

Fork-Join-executor

my-dispatcher {
  executor = "fork-join-executor"
....
}

Java 7 introduced the Fork-Join executor.

As the name suggests, it forks a task into subtasks, each executed by a different thread, and joined the results.

There are two main characters that are worth mentioning here. According to Oracle docs -

1. “It is designed for work that can be broken into smaller pieces recursively”.

   Hence it is best for recursive problems - where a task can be broken into sub-tasks such that they would be executed in parallel and their results would be collected.

2. “The fork/join framework is distinct because it uses a work-stealing algorithm. Worker threads that run out of things to do can steal tasks from other threads that are still busy”

   Fork-Join shows better performance in most cases, compared to old Thread-Pool-Executor. It makes a better use of resources, since the idle threads can steal tasks from busier threads. However, there is a built-in danger here.

   Regarding the first statement, when a ‘Fork’ is performed, we have multiple threads and each of them is responsible for running a task. From the second, when a thread is finished it can take another task. But what if he got stuck performing this task? The other threads will wait on the ‘Join’ at some point, which is a threads starvation.

my-dispatcher {
  executor = "fork-join-executor"
  fork-join-executor {
    # Min number of threads to cap factor-based parallelism number to
    # Note that these threads will be created anyway on fork'
    # so try to avoid an unnecessary overhead.
    parallelism-min = ???
    # Parallelism (threads) ... ceil(available processors * factor)
    parallelism-factor = ???
    # This is NOT an upper bound on the total number of threads!
    # Max number of threads to cap factor-based parallelism number to
    parallelism-max = ???
....
  }
}

A common case is to use Fork-Join executor for future tasks inside an actor. Here, the dispatcher’s configuration of the actor should be considered as well. For example, the more threads you have for the actor, the more ‘future’ tasks will be performed, and you may want more threads for them.

Thread-pool-executor

The old Java 5 executor for asynchronous task execution can still fit in some cases and without the Fork-Join overhead.

While Fork-Join breaks the task for you, if you know how to break the task yourself then your code should be already built as a minimal task, executed by a single thread, which fits a thread-pool-executor.

The thread-pool executor is used by Akka Dispatcher and PinnedDispatcher.

Dispatcher allows you to define ‘min’, ‘max’ and increase ‘factor’ / ‘fixed’ size for your thread pool.

my-dispatcher {
  type = Dispatcher
  executor = "thread-pool-executor"
  thread-pool-executor {
    # Min number of threads to cap factor-based parallelism number to
    parallelism-min = ???
    # Parallelism (threads) ... ceil(available processors * factor)
    parallelism-factor = ???
    # Max number of threads to cap factor-based parallelism number to
    parallelism-max = ???
  }
....
}

The key is to find the right balance for actor instances to work in parallel and use the threads as much as they are need so other actors and processes can work as well. It’s also true for the Fork-Join executor and needs to be quite accurate.

PinnedDispatcher dedicates a unique thread to each actor. This is usually not the pattern you want for the machine, given the limited resources. Hence, it makes sense for the actor to share a pool of threads. However, if your actor performs a preferred task, you won’t want its instances to share the pool.

Do not use it if you have more instances than the number of cores in the machine. It is also not recommended for Futures, because you’ll probably need more than 1 thread…

Affinity-pool-executor

my-dispatcher {
  executor = "affinity-pool-executor"
....
}

This executor tries its best to have your actor instance always schedule with the same thread, which should increase throughput.

This is recommended for a small number of actor instances, where you have much more instances than threads, it is just not possible.

Tips

• Don’t use the Akka default dispatcher for your actorSystem, or for the actors themselves. Note that external Akka based frameworks use it as default, and you should configure a dedicated dispatcher for them as well.

• Have a different dispatcher for each actor, and for Futures inside an actor.

• Dispatchers have a ‘throughput’ parameter, which “defines the maximum number of messages to be processed per actor before the thread jumps to the next actor” Setting It to a higher value than the default, 1, is likely to improve performance so long as it is not part of the Affinity-pool dispatcher, and your actors are generally not very busy (otherwise the lack of fairness can cause a high load in some mailboxes).

• Read this terrific post in the ScalaC blog. It explains dispatcher’s internals in details.

Next

In Part-2 I will show how we monitor and analyze our actor-based system.