Poor performance with the web-frameworks benchmark

[pushkarnk]

Kitura with Kitura-NIO seems to performs badly on the web-frameworks benchmarks

Reported over slack. Copied from there:

Language (Runtime)	Framework (Middleware)	Average	50th percentile	90th percentile	99th percentile	99.9th percentile	Standard deviation
swift (4.2)	perfect (3.1)	22.11 ms	21.81 ms	26.35 ms	33.46 ms	56.41 ms	3607.00
swift (5.0)	vapor (3.3)	65.25 ms	27.59 ms	49.31 ms	1354.51 ms	3496.16 ms	228498.67
swift (5.0)	kitura (2.6)	35.61 ms	33.10 ms	48.56 ms	74.07 ms	647.15 ms	26 433.67
swift (5.0)	kitura (2.7)	62.88 ms	61.14 ms	83.65 ms	115.44 ms	533.98 ms	21219.00
swift (5.0)	kitura-nio (2.7)	177.09 ms	65.25 ms	115.28 ms	3343.02 ms	6831.05 ms	555543.00

Language (Runtime)	Framework (Middleware)	Requests / s	Throughput
swift (4.2)	perfect (3.1)	44599.00	41.93 MB
swift (5.0)	vapor (3.3)	32438.67	54.37 MB
swift (5.0)	kitura (2.6)	28 626.67	53.09 MB
swift (5.0)	kitura (2.7)	15597.00	29.04 MB
swift (5.0)	kitura-nio (2.7)	13049.00	24.67 MB

[weissi]

@pushkarnk this looks like some kind of stall to me. On the 50th percentile (median) kitura/kitura are pretty much the same, but the standard deviation is huge! So either the machine that was running the tests was super busy at some point when running kitura-nio or something stalled/got stuck for a while?

[waghanza]

FYI -> docker is in use on above results, then metrics are not very useful (could vary by ~40% )

I'm working on the-benchmarker/web-frameworks#632 to use an isolated droplet and therefore to reduce docker noise

PS : If I can also have sponsored machine from IBM, I'll be happy to run Kitura benchmarks on ❤️

[weissi]

oh docker with networking i/egress networking? Yeah, that'll give you a bunch of variance.

[waghanza]

In the future, it will be on dedicated VM with a 2 GB/s link

[djones6]

FYI, we can replicate a gap between Kitura-NIO and Kitura-net, and that definitely needs investigation. Previous investigations have shown Kitura-NIO to scale better than Kitura-net, but it's more CPU hungry so slower at lower CPU counts - we need to find out where that CPU is being burnt.

However, @helenmasters and I couldn't replicate the general slowness of Kitura vs. Vapor running the benchmark on a bare-metal Ubuntu 16.04 server, once we removed the Ubuntu version penalty (the-benchmarker/web-frameworks#1346). We limited the Docker containers to a subset of CPUs to emulate the 8-cpu environment, and Vapor and Kitura came out similar in performance (results summarised in Kitura/Kitura#1448).

Perfect did show a lead over the others in our environment, similar to the one reported here, so the puzzle is why our local results for Kitura are substantially better than the ones published above.