More tests!

By principle, software engineering states that the amount of time dedicated to testing the product should be atleast 40% of the total timeline. Even though we had not planned to do so, we did practice this principle.

Bombarding the servers with 1000 connections concurrently was not as easy as it seemed to be. Having various restrictions on system limits, proved to be a deterrent. Changing the ulimit helped to an extent. But having tested on systems which were not supposed to be used as servers, led to complications.

The number of times our laptops hung and had to be restarted by the hard way is more than our fingers can count. This led us to conclude that we had to find another way of bench marking our server.
This was the effect of huge files on the system, but using smaller files did not exactly make sense, since there was no guarantee that the desired level of concurrency would be maintained. Using commands like netstat helped us to find out the actual number of connections active (ESTABLISHED).
Hence, we knew we had to use larger files so that they would take more time to download.


Over the course of the tests, ab spewed out a multitude of errors few of them being...
         apr_poll: timeout specified has expired
This error would pop up whenever the server took too long to respond. We couldn't find a way to increase the hard-coded timeout value of 30s.
When testing on a Mac, this error usually implied that the system had "crashed" and was no longer responding to any user action.
         apr_socket_recv: connection reset by peer
This error is caused by the server sending a RST signal. It occurred whenever the server did not have sufficient resources to process the client requests.

Large file, High concurrency - exactly the conditions we want, for a  test, but a bad recipe for our machines.
Other options being limited (subjected to the ulimit), we started looking at different directions. Also for ab, we could not find out a way to limit the rate of the request to be fetched as we could do in wget. Again, acting upon the advice of an expert in the field, we changed the mtu.
This made the time taken for a single request longer and hence ensured the level of concurrency to be maintained to an extent. And, one more parameter gets added!

Initially we were benchmarking by only varying the total number of requests and the number of concurrent connections. Now we had two more. One was the mtu and the other, the file size. Instead of taking a specific file of an average size, we decided to vary that too, to test the server.

So we continued our tests, hoping to see the light at the end of the tunnel.

Performance analysis

Now, let us look at things a bit more practically.
After all the expected reality is always different from the actual one! Even though one's definitions of reality itself may vary.

Lets look at the performance of boa. So we decided. Historically, boa has been bench marked with ab and zb. We decided to go ahead with the Apache Benchmark.

A simple command:
   ab -n 1000 -c 500 http://10.0.0.1:12345/file.txt
   ab -n <total number of requests> -c <concurrency level> http://<ip addr>:<port>/<location>
and lots of time, is what we required

Repeatedly performing the same on epoll implementation gave us quite surprising results. No, not surprising to the extent of select performing better, but both of them performing at about the same level.

               Time taken

Concurrency Level

epoll vs select

When the lines coincide so perfectly, you wonder if this was the result expected! But Alas! It was not. We expected epoll to perform better, not just better much better!

But theoretically, we did expect select and epoll to perform just about the same for a lesser number of concurrent connections. Upon consulting an expert for advice on these matters, we were told that a concurrency level of 250 is peanuts!

Planning to try out a concurrency of a 1000 clients we wondered how loaded our machines would be.
So we decided to find out!

Great Expectations

Performance analysis

Relying on the competence of Apache Benchmark, we decided to run performance tests on our version of Boa implemented with epoll and the original one implemented with select.
Theoretically, we expected to observe the following:

• With large number of active connections, select may perform better as the possibility of activity on an fd in the fd set is high. Hence traversing the fd set gives rise to a low "miss" rate.
• Similarly, with low concurrency, select's bit comparison will yield better results than the looping structure and word comparison in epoll.
• A better performance of epoll arises when there are a large number of inactive connections. This is a scenario that is difficult to simulate.
• Only a hard-limit (depending on the value specified by RLIMIT_NOFILE) persists for the maximum number of concurrent connections for epoll. For select, it is 1024. Although, in select, some workarounds are possible to change this maximum value by manipulating FD_SETSIZE (system dependent)
• Within select and epoll, increasing the number of connections while keeping concurrency constant should not yield a varying value. The performance is expected to remain constant.

With this frame of reference in mind, we proceed to perform the tests.

Wish us Luck!