Comparing CloudLinux to Open Source Alternatives – Part 1
CloudLinux OS is an operating system designed for shared hosting providers; it makes use of a number of proprietary and custom technologies to offer fine-grained customer isolation and resource limiting.
Lightweight Virtualized Environments (LVE), essentially containers, are used to limit the resource usage of individual users and websites. This operates at the thread level and claims to be more efficient than other containerisation options. At the file system level, a virtualised file system dubbed CageFS is used to isolate users' files.
Without going into too much detail at this time, the approach taken by CloudLinux OS has significant drawbacks. It requires the kernel and applications to be modified to work correctly, and it makes use of proprietary technologies and patches.
Over the course of a number of posts, I intend to examine replicating the benefits provided by CloudLinux OS using open source software. In this first post I will benchmark a web server on CentOS. Performance will be tested with both static and dynamic (PHP) content, and testing will be carried out to evaluate how resource sharing is handled.
First, let's define the test environment; I'll be using a single server and three clients. For both the server and clients I'll be using DigitalOcean droplets with private networking configured. The droplets will be located in the same datacenter and all tests will be carried out over the private network.
Server Environment:
- CentOS 7.4 x86
- 2 CPU Cores - Intel Xeon E5-260Lv2 @ 2.4GHz
- 2GB RAM
- 40GB SSD
- Apache 2.4.29
- PHP 7.1.12 via PHP-FPM
Client Environment (x3):
- CentOS 7.4 x86
- 1 CPU Core - Intel Xeon E5-2630Lv2 @ 2.4GHz
- 1GB RAM
- 30GB SSD
- Siege 4.0.4
The web server will be configured with three virtual hosts, each of which will contain a static HTML page and a PHP page that executes phpinfo(). So that the clients can access the virtual hosts their /etc/hosts files will be modified so that the virtual hosts point to the server's private IP.
I'll be monitoring the CPU and memory usage throughout the tests using sar from the sysstat package.
Test 1
For the first test I'll benchmark a single client accessing the web server, simulating 100 concurrent users hitting the server for 5 minutes.
Static HTML File
Client:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/index.html
Lifting the server siege...
Transactions: 697405 hits
Availability: 100.00 %
Elapsed time: 299.75 secs
Data transferred: 92.45 MB
Response time: 0.04 secs
Transaction rate: 2326.62 trans/sec
Throughput: 0.31 MB/sec
Concurrency: 95.35
Successful transactions: 697405
Failed transactions: 0
Longest transaction: 3.05
Shortest transaction: 0.00
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 16.23 0.00 23.91 0.04 43.05 16.77
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirt
Average: 929651 953761 50.64 0 646913 5545184 294.42 155975 585730 424
Dynamic PHP File
Client:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/info.php
Lifting the server siege...
Transactions: 57997 hits
Availability: 100.00 %
Elapsed time: 299.57 secs
Data transferred: 3304.95 MB
Response time: 0.52 secs
Transaction rate: 193.60 trans/sec
Throughput: 11.03 MB/sec
Concurrency: 99.89
Successful transactions: 57997
Failed transactions: 0
Longest transaction: 1.40
Shortest transaction: 0.01
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 12.27 0.00 8.41 0.00 25.28 54.04
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 790413 1092999 58.03 0 702236 5169678 274.48 283914 588661 446
While not shown above, the client was generally fully loaded throughout testing with monitoring showing fairly consistent CPU usage at ~98%. The next set of tests should mitigate against hitting any client limits as I'll be using multiple clients to hit the server.
Test 2
For the second test I'll benchmark two clients accessing the server, with each simulating 100 concurrent users hitting two different virtual hosts for 5 minutes.
Static HTML (two virtual hosts loaded equally)
Clients:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/index.html
Lifting the server siege...
Transactions: 383472 hits
Availability: 100.00 %
Elapsed time: 299.97 secs
Data transferred: 50.83 MB
Response time: 0.08 secs
Transaction rate: 1278.37 trans/sec
Throughput: 0.17 MB/sec
Concurrency: 99.80
Successful transactions: 383472
Failed transactions: 0
Longest transaction: 7.09
Shortest transaction: 0.00
[root@centos-client2 ~]# siege -b -c100 -t5m http://test2.example.com/index.html
Lifting the server siege...
Transactions: 381380 hits
Availability: 100.00 %
Elapsed time: 299.53 secs
Data transferred: 50.56 MB
Response time: 0.08 secs
Transaction rate: 1273.26 trans/sec
Throughput: 0.17 MB/sec
Concurrency: 99.83
Successful transactions: 381380
Failed transactions: 0
Longest transaction: 7.45
Shortest transaction: 0.00
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 19.31 0.00 28.81 0.00 50.58 1.3
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 465032 1418380 75.31 0 864091 11042542 586.31 479695 594277 4001
Dynamic PHP File (two virtual hosts loaded equally)
Clients:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/info.php
Lifting the server siege...
Transactions: 53960 hits
Availability: 100.00 %
Elapsed time: 299.53 secs
Data transferred: 3074.90 MB
Response time: 0.55 secs
Transaction rate: 180.15 trans/sec
Throughput: 10.27 MB/sec
Concurrency: 99.79
Successful transactions: 53960
Failed transactions: 0
Longest transaction: 2.29
Shortest transaction: 0.03
[root@centos-client2 ~]# siege -b -c100 -t5m http://test2.example.com/info.php
Lifting the server siege...
Transactions: 54471 hits
Availability: 100.00 %
Elapsed time: 299.93 secs
Data transferred: 3104.12 MB
Response time: 0.55 secs
Transaction rate: 181.61 trans/sec
Throughput: 10.35 MB/sec
Concurrency: 99.87
Successful transactions: 54471
Failed transactions: 0
Longest transaction: 2.32
Shortest transaction: 0.03
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 22.63 0.00 15.55 0.01 47.43 14.39
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 452142 1431270 75.99 0 919709 7601917 403.62 549392 594342 802
The results here show that resources are being shared equally between the virtual hosts, which as the server isn't fully loaded isn't surprising. Before we can draw any conclusions we need to perform further testing.
Test 3
For the third test I'll benchmark three clients, each accessing a different virtual host simulating 100 concurrent users hitting the server for 5 minutes.
Static HTML (three virtual hosts loaded equally)
Clients:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/index.html
Lifting the server siege...
Transactions: 281721 hits
Availability: 100.00 %
Elapsed time: 299.88 secs
Data transferred: 37.35 MB
Response time: 0.11 secs
Transaction rate: 939.45 trans/sec
Throughput: 0.12 MB/sec
Concurrency: 99.82
Successful transactions: 281721
Failed transactions: 0
Longest transaction: 15.30
Shortest transaction: 0.00
[root@centos-client2 ~]# siege -b -c100 -t5m http://test2.example.com/index.html
Lifting the server siege...
Transactions: 283625 hits
Availability: 100.00 %
Elapsed time: 299.27 secs
Data transferred: 37.60 MB
Response time: 0.11 secs
Transaction rate: 947.72 trans/sec
Throughput: 0.13 MB/sec
Concurrency: 99.81
Successful transactions: 283625
Failed transactions: 0
Longest transaction: 15.48
Shortest transaction: 0.00
[root@centos-client3 ~]# siege -b -c100 -t5m http://test3.example.com/index.html
Lifting the server siege...
Transactions: 216490 hits
Availability: 100.00 %
Elapsed time: 299.54 secs
Data transferred: 28.70 MB
Response time: 0.14 secs
Transaction rate: 722.74 trans/sec
Throughput: 0.10 MB/sec
Concurrency: 99.79
Successful transactions: 216490
Failed transactions: 0
Longest transaction: 15.06
Shortest transaction: 0.00
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 17.44 0.00 28.15 0.01 52.14 2.27
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 327592 1555820 82.61 0 988454 11379007 604.17 604217 596546 5339
Dynamic PHP File (three virtual hosts loaded equally)
Clients:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/info.php
Lifting the server siege...
Transactions: 45605 hits
Availability: 100.00 %
Elapsed time: 299.57 secs
Data transferred: 2598.79 MB
Response time: 0.66 secs
Transaction rate: 152.23 trans/sec
Throughput: 8.68 MB/sec
Concurrency: 99.83
Successful transactions: 45605
Failed transactions: 0
Longest transaction: 7.68
Shortest transaction: 0.02
[root@centos-client2 ~]# siege -b -c100 -t5m http://test2.example.com/info.php
Lifting the server siege...
Transactions: 47425 hits
Availability: 100.00 %
Elapsed time: 299.01 secs
Data transferred: 2702.59 MB
Response time: 0.63 secs
Transaction rate: 158.61 trans/sec
Throughput: 9.04 MB/sec
Concurrency: 99.82
Successful transactions: 47425
Failed transactions: 0
Longest transaction: 4.90
Shortest transaction: 0.04
[root@centos-client3 ~]# siege -b -c100 -t5m http://test3.example.com/info.php
Lifting the server siege...
Transactions: 31178 hits
Availability: 100.00 %
Elapsed time: 299.39 secs
Data transferred: 1776.70 MB
Response time: 0.95 secs
Transaction rate: 104.14 trans/sec
Throughput: 5.93 MB/sec
Concurrency: 99.12
Successful transactions: 31178
Failed transactions: 0
Longest transaction: 7.86
Shortest transaction: 0.04
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 25.11 0.00 18.21 0.00 49.65 7.03
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 218294 1665118 88.41 0 1046513 11254070 597.54 702859 596587 936
Here we can see that the third virtual host is starting to suffer and resources are not being shared equally across all three virtual hosts. These results will be useful to test the benefits of CloudLinux OS going forward. However, these tests have loaded each virtual host equally which is not always the case in a shared hosting environment.
In the next test I'll try to simulate the 'noisy neighbour' effect common in shared hosting, and one of the main problems CloudLinux OS claims to assist with.
Test 4
For this test I'll benchmark three clients accessing the server; one simulating 100 concurrent users hitting one virtual host for 5 minutes and two simulating the 'noisy neighbour' effect by hitting the second virtual host with 500 concurrent users for 5 minutes each.
Static HTML ('noisy neighbour' effect)
Clients:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/index.html
Lifting the server siege...
Transactions: 84855 hits
Availability: 99.98 %
Elapsed time: 299.07 secs
Data transferred: 11.25 MB
Response time: 0.34 secs
Transaction rate: 283.73 trans/sec
Throughput: 0.04 MB/sec
Concurrency: 96.99
Successful transactions: 84855
Failed transactions: 19
Longest transaction: 27.26
Shortest transaction: 0.00
[root@centos-client2 ~]# siege -b -c500 -t5m http://test2.example.com/index.html
Lifting the server siege...
Transactions: 388019 hits
Availability: 99.97 %
Elapsed time: 299.59 secs
Data transferred: 51.44 MB
Response time: 0.37 secs
Transaction rate: 1295.17 trans/sec
Throughput: 0.17 MB/sec
Concurrency: 482.28
Successful transactions: 388019
Failed transactions: 128
Longest transaction: 55.22
Shortest transaction: 0.00
[root@centos-client3 ~]# siege -b -c500 -t5m http://test2.example.com/index.html
Lifting the server siege...
Transactions: 291324 hits
Availability: 99.95 %
Elapsed time: 300.10 secs
Data transferred: 38.65 MB
Response time: 0.49 secs
Transaction rate: 970.76 trans/sec
Throughput: 0.13 MB/sec
Concurrency: 474.25
Successful transactions: 291539
Failed transactions: 137
Longest transaction: 27.79
Shortest transaction: 0.00
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 17.57 0.00 27.61 0.02 53.04 1.76
root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 107829 1775583 94.27 0 1206337 11320294 601.05 822322 594765 5297
Dynamic PHP File ('noisy neighbour' effect)
Clients:
[root@centos-client1 ~]# siege -b -c100 -t5m http://test1.example.com/info.php
Lifting the server siege...
Transactions: 23951 hits
Availability: 99.74 %
Elapsed time: 299.98 secs
Data transferred: 1364.84 MB
Response time: 1.16 secs
Transaction rate: 79.84 trans/sec
Throughput: 4.55 MB/sec
Concurrency: 92.73
Successful transactions: 23951
Failed transactions: 62
Longest transaction: 26.96
Shortest transaction: 0.12
[root@centos-client2 ~]# siege -b -c500 -t5m http://test2.example.com/info.php
Lifting the server siege...
Transactions: 56031 hits
Availability: 99.30 %
Elapsed time: 300.50 secs
Data transferred: 3193.02 MB
Response time: 2.23 secs
Transaction rate: 186.46 trans/sec
Throughput: 10.63 MB/sec
Concurrency: 416.39
Successful transactions: 56031
Failed transactions: 394
Longest transaction: 108.77
Shortest transaction: 0.09
[root@centos-client3 ~]# siege -b -c500 -t5m http://test2.example.com/info.php
Lifting the server siege...
Transactions: 38318 hits
Availability: 98.97 %
Elapsed time: 300.89 secs
Data transferred: 2186.89 MB
Response time: 3.20 secs
Transaction rate: 127.35 trans/sec
Throughput: 7.27 MB/sec
Concurrency: 407.79
Successful transactions: 38376
Failed transactions: 399
Longest transaction: 108.80
Shortest transaction: 0.23
Server:
[root@centos-server ~]# sar 5 60
CPU %user %nice %system %iowait %steal %idle
Average: all 24.32 0.00 17.05 0.00 49.88 8.75
[root@centos-server ~]# sar -r 5 60
kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 81491 1801921 95.67 0 1163671 11790209 626.00 843283 578357 889
From these tests it is clear that resources are not being shared equally, availability is being affected and significant server resources are being used. This is not an ideal scenario for a shared hosting provider, and one of the reasons for CloudLinux OS popularity among shared hosting providers.
In the next post I will evaluate how CloudLinux OS performs in the same scenarios, and I will also look at how CloudLinux OS performs when resource limits are let using LVE. This will enable comparisons to be made to alternative approaches, such as cgroups or LXC.