Performance Evaluation of Microservices
Architectures using Containers
Marcelo Amaral, Jord`a Polo, David Carrera
Technical University of Catalonia (UPC)
Barcelona Supercomputing Center (BSC)
Email: marcelo.amaral, jorda.polo, david.carrera@bsc.es
Iqbal Mohomed, Merve Unuvar,
Malgorzata Steinder
IBM T.J. Watson Research Center, Yorktown Heights, NY
Email: iqbal, munuvar, steinder@us.ibm.com
Abstract—Microservices architecture has started a new trend
for application development for a number of reasons: (1) to
reduce complexity by using tiny services; (2) to scale, remove
and deploy parts of the system easily; (3) to improve flexibility
to use different frameworks and tools; (4) to increase the overall
scalability; and (5) to improve the resilience of the system. Con-
tainers have empowered the usage of microservices architectures
by being lightweight, providing fast start-up times, and having
a low overhead. Containers can be used to develop applications
based on monolithic architectures where the whole system runs
inside a single container or inside a microservices architecture
where one or few processes run inside the containers. Two models
can be used to implement a microservices architecture using
containers: master-slave, or nested-container. The goal of this
work is to compare the performance of CPU and network running
benchmarks in the two aforementioned models of microservices
architecture hence provide a benchmark analysis guidance for
system designers.
I.
I
NTRODUCTION
Virtual Machines are a widely used building block of
workload management and deployment. They are heavily
used in both traditional data center environments and clouds
(private, public and hybrid clouds). The commonly used term
Virtual Machine (VM) refers to server virtualization, which can
be accomplished via full virtualization or paravirtualization.
In recent months, there has been a resurgence of interest
in container technology, which provides a more lightweight
mechanism - operating system level virtualization. Containers
are lightweight and fast - a single x86 server can reasonably
have 100s of containers running (memory usually ends up
being the scarce resource); moreover, containers start up very
quickly - under 1 to 2 seconds in most cases. There are many
reasons for this resurgence but from a technical perspective,
two of the biggest reasons are (i) the improvements in names-
pace support in the Linux kernel are available in popular
distributions, and (ii) a specific implementation of containers
- Docker - has successfully created an attractive packaging
format, useful tools and diverse ecosystem.
Containers are operating-system-level virtualization under
kernel Linux that can isolate and control resources for a set
of processes. Because of container does not emulate a full
virtualization of the physical hardware as VM does, it is
lightweight with less overhead. The core of containers rely
on Linux namespace [1] and cGroups [2]. The former is
an abstraction that wraps a set of processes appearing that
they are isolated instance. Linux namespace isolates the set
of filesystem mount points seen by the group of processes.
cGroups organize the processes in a hierarchy tree; they also
limit, police and account the resource usage of process group.
One can run a single application within a container whose
namespaces are isolated from other processes on the system.
Notwithstanding, the main capability of container is to allow
to run a complete copy of the Linux OS within it without the
overhead of running hypervisor. Although the kernel is shared,
they have limited access to the modules and drivers to the ones
that it has leaded. Despite a container has limited access, it can
have full access to the host when created as “privileged”. Such
a privileged container might run also another daemon inside to
create other containers as a nested-container approach; but can
emerge security problems when sharing the infrastructure with
other users. Currently, there exist many container distributions
such as OpenVZ [3], and Docker [4].
At a high-level, current Docker usage can be categorized
into two classes: (i) Docker container as a lightweight server,
and (ii) one process (or few related processes) per Docker
container. There have been extensive studies on (i), such as
[5]; in this paper, we concern ourselves with (ii). To our
knowledge, this approach (related processes per container)
does not have a widely used name even though the technique
itself is common. For purposes of this paper, we will call it
Related Processes Per Container or RPPC. RPPC is sensible
from the perspective of deployment, and is also a useful
building block in microservices architectures. We explain both
in turn.
Consider a traditional application server that implements
some business logic and talks to a remote database server.
Along with the core application server, one would install
auxiliary software or sidecars that provide facilities such as
logging, performance monitoring, configuration management,
proxying, and so on. Rather than packaging all of this software
into a single unit, one can group related processes in containers
and then deploy the ensemble. When some functionality needs
to be updated, one need only deploy a subset of the original
containers. Thus, the RPPC approach typically speeds up
deployment, reduces disruption and generally empowers the
devops team. It should be noted that this concept is not new
to even containers - configuration management tools such as
Puppet and Chef have provided this capability for years - only
update what needs to be changed. What’s different is that
individual containers are the unit of deployment.
The RPPC concept is also useful as a building block for
1
arXiv:1511.02043v1 [cs.DC] 6 Nov 2015
microservices. Microservices is a new trend in architecting
large software systems wherein a system is designed as a
set (dozens or even hundreds) of microservices. Microservices
can be developed, managed and scaled independently. There
is typically some kind of routing fabric that gets requests
to a specific instance of a microservices; this routing fabric
often provides load-balancing and can isolate microservices
that are in a failed state. One system that provides these
capabilities in a cloud or clustered environment is Google’s
Kubernetes. In Kubernetes, a pod is a group of containers that
is a deployable unit - moreover, all containers of a pod share
the same fate. While this doesn’t always make sense (e.g. in the
application server example above), it does make a lot of sense
in microservices and especially with containers (destroying a
faulty pod does no harm since the routing fabric will route to
healthy one; also, since starting up containers is a lot faster
than booting Virtual Machines, pod startup can be very fast).
We investigated two distinct ways to implement the RPPC
concept. In the first approach, all child containers are peers
of each other and a parent container (which serves to manage
them). We call this implementation as master-slave throughout
the paper. The second approach, where we refer as nested-
container, involves the parent container being a privileged
container and the child containers being inside its namespace.
Our purpose is to understand the performance differences of
the two approaches hence system designers can benefit from
our analysis in the future.
II.
M
ICROSERVICES
A
RCHITECTURE USING CONTAINERS
A. Containers
Containers are a mechanism that provide operating sys-
tem level virtualization, in that they can isolate and control
resources for a set of processes. Because a container does not
emulate the physical hardware as a virtual machine does, it is
lightweight with less overhead. While the concept of operating
system level virtualization is not new (e.g. chroot/jails in BSD),
there has been a great deal of industry interest in Linux
containers and Docker Inc’s implementation in particular.
The core of containers rely on Linux namespaces [1] and
cGroups [2]. Linux namespaces isolate the set of filesystem
mount points seen by a group of processes. cGroups organize
the processes in a hierarchy tree; it also limits, polices and
accounts for the resource usage of process groups. One can
run a single application within a container whose namespaces
are isolated from other processes on the system. On the other
hand, one can use a container as a lightweight Linux server,
by setting up facilities such as a process control system (e.g.
supervisord) and and ssh server. Currently, there exist many
container implementations such as OpenVZ [3], Rocket [6]
and Docker [4]. In this paper, we primarily focus on Docker
containers.
Normally, a Docker container has limited access to resource
on the host, but full access can be provided by creating it as a
”privileged” container. Such a privileged container can access
host devices and also run another Docker daemon inside itself
to create child containers (we call this the nested-container
approach). One must be careful with this approach as security
challenges may emerge when sharing system resources with
other users.
B. Key virtual networking differences
In recent years, there have been significant advances in vir-
tual networking features in Linux. Some notable mechanisms
include network namespaces, veth pairs, tap devices as well
as virtual switches such as OpenvSwitch and Linux Bridges.
The mechanisms can be used to provide a high degree of flex-
ibility and control of networking, and form the basis of SDN
(software defined networking) technologies such as Openstack
Neutron. For instance, a physical host might have just one
physical network interface, while a number of guest containers
can run in isolation by having their own network namespaces,
with tap devices wired into an OpenvSwitch. Moreover, one
could setup tunnels between the OpenvSwitch instances on
different machines, and can enable communication between
instances. In practice, there are a great many ways of setting up
virtual networking, with varying effects on throughput, latency
and CPU utilization.
In this paper, we mainly focus our attention to communi-
cation that takes place on a single host. Even in this restricted
case, there are many options to consider. Figure 2 shows 4
different communication setups that we explore later in the
paper, along with the various layers that must be traversed. Of
course, best performance is achieved when a process runs on
bare-metal and can utilize the native networking stack in the
OS. The default setup with Docker is that containers are wired
into a virtual switch such as a Linux Bridge or OpenvSwitch
(which is wired into the physical NIC/OS networking stack).
In the nested-container scenario, there is a second level of
virtual switching that takes place inside the container. In the
virtual machine case, there is typically a virtual switch as well
as virtualized network drivers.
C. Microservices
Microservices have gained much popularity in industry
in the last few years. This architecture can be considered a
refinement and simplification of Service-oriented Architecture
(SOA) [7]. The key idea is that rather than architecting
monolithic applications, one can achieve a plethora of benefits
by creating many independent services that work together in
concert. The benefits accrued include simpler codebases for
individual services, ability to update and scale the services in
isolation, enabling services to be written in different languages
if desired and utilize varying middleware stacks and even data
tiers for different services [8]. There are costs to this approach
as well [9] such as the computational overhead of running an
application in different processes and having to pay network
communication costs rather than simply making function calls
within a process. The proliferation of service processes almost
requires automated mechanisms for deployment (sometimes
called continuous delivery or continuous integration). Other
challenges include deciding what the boundaries of different
services are and determining when a service is too big.
D. Achieving Microservices with Containers
As previously stated, Docker containers are an excellent
match for building microservices. They are lightweight, start
very fast, and can wrap dependencies and vagaries of imple-
mentation inside themselves. For instance, a developer can
start dozens of containers on a modest laptop of the day. They
2
can go to a source such as DockerHub to download images
of containers with pre-configured middleware, databases and
applications. Such images can be extended with additional
customization of the developer’s choosing.
When one considers how services developed as ensem-
bles of containers can be deployed, two different approaches
emerge: master-slave or nested-containers as illustrated in
Figure 1.
Slave
|
