IHttpAsyncHandler vs IHttpHandler Performance, Take 2

In our last post we took a look at performance between async/sync handlers for tasks that are intrinsically synchronous. What about tasks that were built with async in mind? Typically these sort of tasks are things that involve IO, like file access, a db query, or a call out to a website. Does this influence performance?

IHttpAsyncHandler Code and Performance

Performance Results from 2000 requests at a concurrency level of 50

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       8
Processing:   275 2423 1352.5   2047    9957
Waiting:      259 2421 1353.1   2046    9957
Total:        276 2424 1352.5   2048    9957

Percentage of the requests served within a certain time (ms)
  50%   2048
  66%   2744
  75%   3195
  80%   3574
  90%   4405
  95%   4865
  98%   5104
  99%   6389
 100%   9957 (longest request)

Performance Results from 3000 requests at a concurrency level of 50

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       4
Processing:   174 2992 1476.8   2879   12689
Waiting:      172 2991 1477.4   2878   12688
Total:        174 2993 1476.8   2880   12689

Percentage of the requests served within a certain time (ms)
  50%   2880
  66%   3605
  75%   4042
  80%   4245
  90%   4790
  95%   5059
  98%   6352
  99%   7524
 100%  12689 (longest request)

Performance Results from 3000 requests at a concurrency level of 100

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       2
Processing:   273 5501 2835.9   5641   20909
Waiting:      270 5499 2836.7   5638   20908
Total:        273 5501 2835.9   5641   20910

Percentage of the requests served within a certain time (ms)
  50%   5641
  66%   6939
  75%   7432
  80%   7785
  90%   8900
  95%  10091
  98%  11678
  99%  12784
 100%  20910 (longest request)

IHttpHandler Code and Performance

Performance Results from 2000 requests at a concurrency level of 50

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       2
Processing:   273 2419 856.3   1965    5861
Waiting:      271 2417 856.4   1964    5859
Total:        273 2420 856.3   1965    5862

Percentage of the requests served within a certain time (ms)
  50%   1965
  66%   2388
  75%   2928
  80%   3255
  90%   3816
  95%   4121
  98%   4592
  99%   4788
 100%   5862 (longest request)

Performance Results from 3000 requests at a concurrency level of 50

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       7
Processing:   269 3608 3519.8   2080   23885
Waiting:      267 3606 3520.4   2078   23885
Total:        270 3608 3519.8   2080   23885

Percentage of the requests served within a certain time (ms)
  50%   2080
  66%   2535
  75%   3155
  80%   3850
  90%   8223
  95%  11518
  98%  16501
  99%  18591
 100%  23885 (longest request)

Performance Results from 3000 requests at a concurrency level of 100

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       7
Processing:   494 5188 1228.9   4750    9665
Waiting:      492 5185 1228.9   4746    9662
Total:        494 5188 1228.9   4750    9666

Percentage of the requests served within a certain time (ms)
  50%   4750
  66%   5064
  75%   5536
  80%   5924
  90%   6938
  95%   7858
  98%   8910
  99%   9450
 100%   9666 (longest request)

Bonus Tests! With IsReusable=false on both types

This surprsingly reversed the results we saw above, async was better--

IHttpAsyncHandler with 3000 requests, 100 concurrent, !IsReusable:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       7
Processing:   199 6295 2689.3   6507   16925
Waiting:      196 6293 2690.2   6506   16925
Total:        199 6295 2689.3   6507   16925

Percentage of the requests served within a certain time (ms)
  50%   6507
  66%   7408
  75%   7970
  80%   8365
  90%   9498
  95%  10853
  98%  12395
  99%  13295
 100%  16925 (longest request)

IHttpHandler with 3000 requests, 100 concurrent, !IsReusable:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0      12
Processing:  2833 10871 4946.1  11540   25468
Waiting:     2833 10870 4946.8  11540   25468
Total:       2833 10872 4946.1  11541   25468

Percentage of the requests served within a certain time (ms)
  50%  11541
  66%  12492
  75%  13254
  80%  13950
  90%  17213
  95%  19879
  98%  21980
  99%  23041
 100%  25468 (longest request)

Analysis

Given this data, it appears as though IHttpAsyncHandlers seem to only pay off when you're designing a stateful handler (when you're using IsReusable=false), IHttpHandler conquers all other scnearios. Strange.

IHttpAsyncHandler vs IHttpHandler Performance

Got into a conversation today with my brother (and tech-blog editor extraordinaire) Justin Smith, about the performance of async vs non-async handlers/pages. Thought it'd be interesting to test out if you only reap the benefits of async handlers/pages when you're passing work off to a webservice or db. So, we're going to test throughput/response time of asnyc vs synchronous Fibonacci calculations.

Note all tests were ran on my localhost using in debug="false" mode, which is very important as with it on test results are completely different. Here are the tests and results:

IHttpAsyncHandler Example and Performance

The TestMethods class at the top was used in both tests.

Performance results for 2000 requests at a concurrency level of 50:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       1
Processing:     6   31   6.4     30      60
Waiting:        5   31   6.3     30      60
Total:          6   31   6.4     31      60

Percentage of the requests served within a certain time (ms)
  50%     31
  66%     32
  75%     35
  80%     36
  90%     39
  95%     42
  98%     47
  99%     48
 100%     60 (longest request)

Performance results for 5000 requests at a concurrency level of 50:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       2
Processing:     5   29   5.7     30      81
Waiting:        5   29   5.7     29      79
Total:          5   30   5.7     30      81

Percentage of the requests served within a certain time (ms)
  50%     30
  66%     31
  75%     33
  80%     33
  90%     37
  95%     39
  98%     42
  99%     44
 100%     81 (longest request)

Performance results for 5000 requests at a concurrency level of 100:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       4
Processing:    12   61  11.1     62      88
Waiting:       12   61  11.1     62      87
Total:         12   62  11.1     62      88

Percentage of the requests served within a certain time (ms)
  50%     62
  66%     66
  75%     68
  80%     69
  90%     75
  95%     80
  98%     83
  99%     85
 100%     88 (longest request)

IHttpHandler Example and Performance

Performance results for 2000 requests at a concurrency level of 50:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       2
Processing:     6   23   4.6     23      55
Waiting:        6   22   4.6     22      55
Total:          6   23   4.6     23      55

Percentage of the requests served within a certain time (ms)
  50%     23
  66%     24
  75%     25
  80%     26
  90%     28
  95%     30
  98%     34
  99%     37
 100%     55 (longest request)

Performance results for 5000 requests at a concurrency level of 50:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       4
Processing:     8   23   4.3     22      52
Waiting:        7   22   4.2     22      52
Total:          8   23   4.3     22      52

Percentage of the requests served within a certain time (ms)
  50%     22
  66%     24
  75%     24
  80%     25
  90%     27
  95%     30
  98%     36
  99%     40
 100%     52 (longest request)

Performance results for 5000 requests at a concurrency level of 100:

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       5
Processing:     7   47   6.9     45     138
Waiting:        7   46   6.8     45     114
Total:          7   47   6.9     45     138

Percentage of the requests served within a certain time (ms)
  50%     45
  66%     48
  75%     50
  80%     51
  90%     55
  95%     59
  98%     66
  99%     68
 100%    138 (longest request)

Analysis

So synchronous seems like the way to go-- no real gains on async. The next tests will be for IO bound tasks (web/db calls).

.NET Rx Driven Web Server, Take 2

Feeling like my code was rather crappy, i thought i'd take another stab at the Rx Webserver problem. I was especially unhappy with the async signature of Socket.BeginRecieve. A little research online and i found HttpListener, a .NET class that does the same thing except with a usable Async function (one that returns something meaningful) for my Observeable.FromAsyncPattern.

A Hyper-fast Rx Web Server

This one actually performs! Here's the code:

Edit: I found this after making mine-- José F. Romaniello's implentation of a .NET Rx Web Server. His code looks much more useful, and is a better example of Rx. My version hardly Rx, really. Anyways, i performance tested his as well and the figures were similar to the tests ran on this one below.

Performance Tests-- MVC3, Nodejs, and Our Rx Server

MVC3

Using a generic controller that returns "Thanks!":

Server Software:		ASP.NET
Server Hostname:		localhost
Server Port:		31362
Document Path:		/Test/
Document Length:		11 bytes
Concurrency Level:		1000
Time taken for tests:		42.845 seconds
Complete requests:		10000
Failed requests:		0
Total transferred:		2700000 bytes
HTML transferred:		110000 bytes
Requests per second:		233397.92
Transfer rate:		63017.44 kb/s received
Connnection Times (ms)
	min	avg	max
Connect:	0	0	590
Processing:	688	4077	4649
Total:	688	4077	5239

Our server

Using the code above:

Server Software:		Microsoft-HTTPAPI/2.0
Server Hostname:		localhost
Server Port:		8081
Document Path:		/Test/
Document Length:		7 bytes
Concurrency Level:		1000
Time taken for tests:		1.624 seconds
Complete requests:		10000
Failed requests:		0
Total transferred:		1500000 bytes
HTML transferred:		70000 bytes
Requests per second:		6157021.28
Transfer rate:		923553.19 kb/s received
Connnection Times (ms)
	min	avg	max
Connect:	0	0	2
Processing:	39	155	180
Total:	39	155	182

Node.js

Using the node.js homepage hello world sample:

Server Software:
Server Hostname:		localhost
Server Port:		1337
Document Path:		/
Document Length:		8 bytes
Concurrency Level:		1000
Time taken for tests:		2.066 seconds
Complete requests:		10000
Failed requests:		0
Total transferred:		720000 bytes
HTML transferred:		80000 bytes
Requests per second:		4839481.65
Transfer rate:		348442.68 kb/s received
Connnection Times (ms)
	min	avg	max
Connect:	0	7	94
Processing:	1	35	244
Total:	1	42	338

The Actual Node.js Beater

Really, there's a good bit of apples to oranges here, but still--can you believe that? I think that means that our little Rx server is C10K compliant! Further, many of our server's metrics beat node.js-- such as transfer rate, requests per second, max times... pretty awesome!

I've still got a ton to learn about Rx... back to the books now.

A .NET Rx Driven Web Server

Edit: please see my other post on creating a .NET Web Server from Rx (Reactive Extensions) since it contains better code.

Although I've seen .Net Rx (Reactive Extensions) around, I never messed with them until today. To me, the concepts behind Rx always seemed self explanatory--perhaps because i have accomplished concurrent apps in .NET 1.0/2.0 without them. However, having spent a little time with them today, I think Rx is good stuff. Honestly, I'm impressed with the interfaces and the services they provide. Let's check it out:

What are the .Net Reactive Extensions(Rx)?

Short answer: pubsub.

Long answer: a ton of sugar over top of .Net streams, async, TPL, and pubsub. I'm not going to get into the generic intros you can find elsewhere that involve streaming enumerables to the console. Instead I'd prefer to create the argument for Rx as such-- when given the need for "X", it is better to provide "the ability to provide X" than "X" itself. The Reactive Extensions give you a ton of really helpful methods to aide you in implementing "the ability to create X" over "X" itself. Allow me to explain--

If i asked you to write me a function that gave me the first 1 million numbers, how would you implement it? I know a younger me would've started on cranking out a for loop, not taking into consideration that decision's implications upon the system's memory. A smarter implementation would be to give me a function/object that gives me the ability to create the first million numbers, perhaps through iterating through the set. Such an object could then forgo the previously mentioned memory issues. The idea of giving "the ability to create/observe X" over "X" itself is arguably the conceptual basis of functional programming's lazy evaluation, which is also what Rx aims to help the user create (to me, at least). So, out of the box you get a ton of ways to create and enable the push/pull of streaming data and/or events.

An Rx TCP Server

The first thing i could think of to make with Rx is a single-threaded TCP server. Maybe that's because when i think of streaming data these days, i tend to think of a node.js style web server. How hard could it be? (And what would the performance be like...?

Version One: A Single-Threaded Non-Rx Readonly TCP Server

If you run the following code, and make a request on your web browser to http://localhost:8081 you'll see the GET request come through to the app.

Version Two: A Single-Threaded Rx Enabled TCP Server

In this version I added two properties to the NetActor-- Incoming and Outgoing. Both are based on new Rx interfaces that allow the client to tap into the push/pull of data to the client. So if you open your web browser, open up the localhost site, and then type into the console app and press enter, it will get delivered to the web page:

Version Three: The Node.js Killer

Ok, so in order to get apache bench to recognize my console app as a web server i had to bind the NetActor's Ip to something other than localhost... not sure why. Once i got that working, i had intermittent failure until I implemented part of the HTTP spec-- at least the response code and connection closed. After that, and also after creating the ability for the NetActor to shut itself down and start itself up, here is what i was left with:

Apache Bench Results

At 500ms+ with a concurrency level of 1, this is definitely not a node.js killer..... ;-)