I’ve held the course more times than I can remember, and usually done improvements after every course. Others have also held the course, including Redpill Linpro’s Kacper Wysocki, maintainer of security.vcl and Magnus Hagander (Postgresql god/swede). Feedback and gradual improvements have turned a set of slides into a pretty good course material.

We recently started holding on-line courses too. This revealed several new challenges. The obvious challenges are things like getting basic voice communication working (it sounds easy, but you’d be amazed…). It was also interesting when I held the course in my apartment on Australian time, and my ISP decided to perform maintenance on my cable modem (it was 2AM local, after all). So I’ve had to hold the course on a 3G connection, communicating with Australia. Fun. Then there’s the lack of or severely reduced feedback, which presents challenges in how we do exercises and generally deal with the course. In a class room I can easily determine if the participants are able to keep up, if I’m going too slow or too fast and whether or not a subject is more interesting than an other. All of that is, at best, very difficult in an on-line session.

The last few weeks I’ve finally gotten around to merging the sysadmin course with the web development course that Jérôme Renard has written for us. It proved the perfect opportunity to give the course an other big update. While the course was already updated for Varnish 3, I’ve made several other Varnish 3-related additions. More importantly is that the flow of the course has changed from one oriented on Varnish functionality to tasks you wish to accomplish with Varnish. Instead of teaching you about Varnish architecture first, then Varnish parameters, the course now has a chapter devoted to Tuning.

Instead of just throwing in purging or banning when talking about VCL, there’s now a chapter called Cache Invalidation, that attempts to give broader understanding of the alternatives you have and when to use which solution. Similarly, there’s a chapter called Saving The Request, which starts out with the core Grace mechanisms, moves on to health checks, saint mode, req.hash_always_miss, directors and more.

There are several reasons I write about this. First of all: I’m very excited about the material. I’ve worked on it regularly for several years, doing everything from hacking rst2s5, tweaking the type setting and design to updating the content, reorganizing the course and of course holding it. It may seem like one big marketing stunt, but I can promise you that I never blog about something I’m not passionate about, regardless of whether it is work-related or not.

The other reason is that I’m holding the course next week. This will be the first time we hold it using the changed structure. I would have wanted to hold it in a class room first, but holding it on-line is still exciting.

If you wish to participate, head over to https://www.varnish-software.com/products-services/training/varnish-administration-course and convince your boss it will be awesome!

The release date is set for Thursday the 16th (next week(June, 2011, for the potential future archive crawlers)), and several release parties are planned. Varnish Software will be present in Santa Clara, London and Oslo, but numerous parties are planned all around the world.

This will be a very special day for everyone in Varnish Software. Varnish 2.0.0 was released roughly the same week I started working at Redpill Linpro, before I was really involved with Varnish, and I still regret that I didn’t snatch one of those fancy 2.0-t-shirts.

What’s new?

While there are too many news to mention, there are two that stand out more than anything.

The first is gzip compression and the second is the Varnish module-architecture, or simply vmods. That most of ESI has been re-factored under the hood will be evident in future releases, and the same goes for streaming delivery.

Compression

- “What, you don’t already have compression?!”

Remember, Varnish is a caching service. Most of the time you don’t need it to do the compression, because the web server will do it for you. But there are good reasons for why you want it. ESI is the primary use-case.

With ESI, Varnish needs to parse the content, and it can’t do that if it doesn’t understand the transfer encoding. With Varnish 2.1, you have to send uncompressed data to Varnish if you want to use ESI. This means you have to either deliver uncompressed content to your clients, or use yet an other service in front of Varnish.

But let me get a bit technical, because Varnish 3.0′s compression is pretty awe-inspiring.

So with ESI, you have multiple, individually cached elements that make up a single user-visible page. So what we could do, is glue it all together and compress it before we send it. The downside is that we’ll do the compression over and over. An alternative would be to cache the compressed result as long as the individual elements are unchanged, but that will require more space and more complexity.

So what does Varnish 3.0 does? It stores the elements compressed, modifies the right gzip-bits and glues it all together on the fly, without decompressing it. If a single element is changed, only that element needs to be updated. This is probably the best solution, even if the complexity of meddling with binary gzip headers directly can lead to some pretty tricky code. I challenge you to find a solution that handles compression in a smarter way.

Varnish 3.0 also does decompression. If your browser doesn’t support compression (Possibly a script or other tool, real browsers support compression), Varnish will decompress the object for you on the fly. This is an other huge improvement over Varnish 2.1. In Varnish 2.1, this is solved using Vary: and storing different copies of the same object based on compression.

We can also do the same with the backend-data: If ESI needs to read the data, Varnish 3.0 can decompress it on the fly, parse it, then re-compress it before storing it.

And for you, the user, the complexity is fairly non-existent. Push the button, remove that nginx(no hard feelings)server you had doing compression, ????, profit!

VMODS!

VCL, the Varnish Configuration Language, is a flexible way of configuring Varnish. Since it is already translated from VCL to C, the compiled and linked in to the running Varnish instance, VCL has “always” had in-line C: Anywhere you want to, you can tell the VCL compiler that this is pure C code, and pass it directly to the compiler instead of trying to translate from VCL to C first. This was mainly provided because:

1. It didn’t add any complexity and was actually less work.
2. It provided an escape chute for features we didn’t want in Varnish-proper, but were valid for some users.

What features could that be? Syslog-support, geoip-integration, cryptographic verification of authorization headers, etc.

It turned out to be very useful, but impractical and difficult to re-use. Since it was all glued to your VCL, it meant that you had to stick C code in the middle of your regular configuration, and it made it very hard to combine two different features if you didn’t know C yourself. And linking towards external libraries required parameter changes.

Enter varnish modules.

Simply put, vmods are a way of letting you do the same thing you can do with in-line C, but in a more sustainable manner. A vmod will typically have a few functions exposed to VCL, and the VCL just has to declare that it imports the vmod to use the functions without a single line of C-code in your VCL.

This also means that the vmod has its own build system, own linking process, flexible development environment and are much easier to share.

In the time to come, I expect us to have a community-site for vmods. I also expect that you will see a lot of minor yet important changes to Varnish during the Varnish 3.0-cycle that exposes more of the internal varnish utilities so vmods can use them.

Small disclaimer, though: There is no API guarantee. We don’t wish to slow the pace of Varnish-development by restricting what we can change. That said, we wont tear things apart just to see our vmod-contributors bleed.

I plan to write a blog post on vmod-hacking in the near future, so expect more detail there.

Finishing words

Varnish 1.0 was, as any 1.0-release is, important. I was not involved with the project back then, but as I understand it, Varnish 1 was very much tailored to a specific site, or type of site.

With Varnish 2.0, Varnish became useful for anyone with a high-traffic site. The 2.0-series was a good release-series, adding a healthy mixture of bug fixes and non-intrusive features with each release. But constantly focused on real-world usage. We also saw the first Varnish User Group meeting during that time.

Varnish 2.1 has been a sort of intermittent release. Director-refactoring, ESI and to a certain degree persistent storage paved the way for architectural changes that had to be done. Meanwhile, the user-base really exploded.

Now, with Varnish 3.0.0 coming out, we are already seeing how useful vmods are for Varnish Software-customers like Softonic, who’re sponsoring a VMOD to allow header-modification targeted at the Set-Cookie header (which can’t be treated like the RFC2616 specifies, due to how this has been implemented historically). Doing this in a vmod allows the VCL to remain clean. It also lets us share the code that isn’t site-specific but widely useful, since it is all contained in a header vmod(Still under development). Gzip also means there are no drawbacks to using ESI with Varnish. You don’t need to add a second service to compress data.

It all boils down to Varnish being a more useful part of your architecture. It’s easy to get fast and maintainable C-code in there if you need it, you can even pay someone to write just that bit for you. Without being confined to Varnish’ own road map and release cycle. There are no longer any downsides to using ESI. It’s fast. It’s free software. There are professional service and support offerings available. Varnish follow standards – unless you tell it not to. And so forth.

I’m not usually one to pat myself too much on the back, but as I’m writing this, I feel proud to be part of the team that gives you Varnish 3.0.

If you’re in Oslo, I’ll see you next Thursday!

You will not find graphs. You will not find “this is best!”-quotes. I’m not even backing up my statements with httperf-output.

Disclaimer

This is not a comparison of G-Wan versus Varnish. It is not complete. It is not even a vague attempt at making either G-Wan or Varnish perform better or worse. It is not realistic. Not complete and in no way a reflection on the overall functionality, usability or performance of G-Wan.

Why not? Because I would be stupid to publicize such things without directly consulting the developers of G-Wan so that the comparison would be fair. I am a Varnish-developer.

This is a text about stress testing. Not the result of stress testing. Nothing more.

The basic idea

So G-Wan was supposedly much faster than Varnish. The feature-set is also very narrow, as it goes about things differently. The test showed that Varnish, Apache and nginx were almost comparable in performance, whereas G-Wan was ridiculously much faster. The test was also conducted on a local machine (so no networking) and using AB. As I know that it’s hard to get nginx, Apache and Varnish to perform within the same level, this indicated that G-Wan did something differently that affected the test to me.

I installed g-wan and Varnish on a virtual machine and started playing with httperf.

What to test

The easiest number to demonstrate in a test is the maximum request rate. It tells you what the server can do under maximum load. However, it is also the hardest test to do precisely and fairly across daemons of vastly different nature.

Other things I have rarely written about is the response time of Varnish for average requests. This is often much more interesting to the end user, as your server isn’t going to be running at full capacity anyway. The fairness and concurrency is also highly relevant. A user doing a large download shouldn’t adversely affect other users.

I wasn’t going to bother with all that.

First test

The first test I did was “max req/s”-like. It quickly showed that G-Wan was very fast, and in fact faster than Varnish. At first glance. The actual request-rate was faster. The CPU-usage was lower. However, Varnish is massively multi-threaded, which offsets the cpu measurements greatly and I wasn’t about to trust it.

Looking closer I realized that the real bottleneck was in fact httperf. With Varnish, it was able to keep more connections open and busy at the same time, and thus hit the upper limit of concurrency. This in turned gave subtle and easily ignored errors on the client which Varnish can do little about. It seemed G-Wan was dealing with fewer sessions at the same time, but faster, which gave httperf an easier time. This does not benefit G-Wan in the real world (nor does it necessarily detract from the performance), but it does create an unbalanced synthetic test.

I experimented with this quite a bit, and quickly concluded that the level of concurrency was much higher with varnish. But it was difficult to measure. Really difficult. Because I did not want to test httperf.

The hardware I used was my home-computer, which is ridiculously overpowered. The VM (KVM) was running with two CPU cores and I executed the clients from the host-OS instead of booting up physical test-servers. (… That 275k req/s that’s so much quoted? Spotify didn’t skip a beat while it was running (on the same machine). )

Conclusion

The more I tested this, the more I was able to produce any result I wanted by tweaking the level of concurrency, the degree of load, the amount of bandwidth required and so forth.

The response time of G-Wan seemed to deteriorate with load. But that might as well be the test environment. As the load went up, it took a long time to get a response. This is just not the case with Varnish at all. I ended up doing a little hoodwinking at the end to see how far this went, and the results varied extremely with tiny variations of test-parameters. The concurrency is a major factor. And the speed of Varnish at each individual connection played a huge part. At large amounts of parallel requests Varnish would be sufficiently fast with all the connections that httperf never ran into problems, while G-Wan would be more uneven and thus trigger failures (and look slower)…

My only conclusion is that it will take me several days to properly map out the performance patterns of Varnish compared to G-Wan. They treat concurrent connections vastly different and perform very different depending on the load-pattern you throw at them. Relating this to real traffic is very hard.

But this confirms my suspicion of the bogus-ness of the blog post that lead me to perform these tests. It’s not that I mind Varnish losing performance tests if we are actually slower, but it’s very hard to stomach when the nature of the test is so dubious. The art of measuring realistic performance with synthetic testing is not one that can be mastered in an afternoon.

Lessons learned

(I think conclusions are supposed to be last, but never mind)

First: Be skeptical of unbalanced results. And of even results.

Second: Measure more than one factor. I’ve mainly focused on request-rate in my posts because I do not compare Varnish to anything but itself. Without a comparison it doesn’t make that much sense to provide reply latency (though I suppose I should start supplying a measure of concurrency, since that’s one of the huge strong-points of Varnish.).

Third: Conclude carefully. This is an extension of the first lesson.

…

A funny detail: While I read the license for the non-free G-Wan, which I always do for proprietary software, I was happy to see that it didn’t have a benchmark-clause (Oracle, anyone?). But it does forbid removing or modifying the Server:-header. It also forces me to give the G-Wan-guys permission to use my using of G-Wan in their marketing… Hmm — maybe I should … — err, never mind.

(This must be my shortest blog-post by far)

In previous posts I’ve written I about hitting 27k requests per second on an aging Opteron (see http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/) and then again about reaching 143k requests per second on a more modern quad-core using a ton of test-clients. (see http://kristianlyng.wordpress.com/2010/01/13/pushing-varnish-even-further/).

Recently, we were going to do a stress test at a customer setup before putting it live. The setup consisted of two dual Xeon x5670 machines. The X5670 is a 2.93GHz six-core cpu with hyperthreading, giving these machines 12 cpu-cores and 24 cpu threads. Quite fast. During our tests, I discovered some httperf secrets (sigh…). And was able to push things quite far. This is what we learned.

The hardware and software

As described above, we only had two machines for the test. One is the target and one would be the originating machine. The network was gigabit.

Varnish 2.1.3 on 64bit Linux.

Httperf for client load.

The different setups to be tested

Our goal was not to reach the maximum limits of Varnish, but to ensure the site was ready for production. That’s quite tricky on many accounts.

The machines were originally configured with heartbeat and haproxy.

One test I’m quite fond of is site traversal while hitting a “hot” set at the same time. The intention is to test how your site fares if a ruthless search bot hits your site. Does your front page slow down? As far as Varnish goes, it tests the LRU-capabilities and how it deals with possibly overloaded backend servers.

We also switched out haproxy in favor of a dual-varnish setup. Why? Two reasons: 1. Our expertise is within the realm of Varnish. 2. Varnish is fast and does keep-alive.

When testing a product like Varnish we also have to take the balance between requests and connections into account. You’ll see shortly that this is very important.

During our tests, we also finally got httperf to stress the threading model of Varnish. With a tool like siege, concurrency is defined by the threading level. That’s not the case with httperf, and we were able to do several thousand _concurrent_ connections.

As the test progressed, we reduced the size of the content and it became more theoretical in nature.

As the specifics of the backends is not that relevant, I’ll keep to the Varnish-specific bits for now.

I ended up using a 301 redirect as a test this time. Mostly because it was there. Towards the end, I had to remove various varnish-headers to free up bandwidth.

Possible bottlenecks

The most obvious bottleneck during a test like this is bandwidth. That is the main reason for reducing the size of objects served during testing.

An other bottleneck is how fast your web servers are. A realistic test requires cache misses, and cache misses requires responsive web servers.

Slow clients are a problem too. Unfortunately testing that synthetically isn’t easy. Lack of test-clients has been an issue in the past, but we’ve solved this now.

CPU? Traditionally, the cpu-speed isn’t much of an issue with Varnish, but when you rule out slow backends, bandwidth and slow clients, the cpu is the next barrier.

One thing that’s important in this test is that the sheer amount of parallel execution threads is staggering. My last “big” test had 4 execution threads, this one has 24. This means we get to test contention points that only occur if you have  massive parallelization. The most obvious bottleneck is the acceptor-thread. The thread charged with accepting connections and delegating them to a thread. Even if multiple thread pools is designed to leverage this problem, the actual accept()-call is done in a single thread of execution.

Connections

As Artur Bergman of Wikia has already demonstrated, the amount of TCP connections Varnish is able to accept per second is currently our biggest bottleneck. Fortunately for most users, Artur’s work-load is very different from most other Varnish users. We (Varnish Software) typically see a 1:10 ration between connections and requests. Artur suggested he’s closer to 1:3 or 1:4.

During this round of tests I was easily able to reach about 40k connections/s. However, going much above that is hard. For a “normal” workload, that would allow 400k requests/second, which is more than enough. However, it should be noted that the accept-rate goes somewhat down as the general load increases.

It was interesting to note that this was largely unaffected by having two varnishes in front of each other. This essentially confirms that the acceptor is the bottleneck.

There wasn’t much we could do to affect this limit either. Increasing the listen_depth isn’t going to help you in a synthetic test. The listen_depth defines how many outstanding connections is allowed to queue up before the kernel starts dropping them. In the real world, the connection-rate will be sporadic and on an almost-overloaded system, it might help to increase the listen depth, but in a synthetic test the connection rate is close to constant. That means increasing the listen depth just means there’s a bigger queue to fill – and it will fill anyway.

The number of thread pools had little effect too. By the time the connection is delegated to a thread pool, it’s already past the accept() bottleneck.

Now, keep in mind that this is still a staggering number. But it’s also an obvious bottleneck for us.

Request rate

The raw request rate is essentially defined by how big the request is compared to bandwidth, how much CPU power is available and how fast you can get the requests into a thread.

As we have already established that the acceptor-thread is a bottleneck, we needed to up the number of requests per connection. I tested mostly with a 1:10 ratio. This is the result of one such test:

The above image shows 202832 requests per second while doing roughly 20 000 connections/s. Quite a number.

It proved difficult to exceed this.

At about 226k req/s the bandwidth limit of 1gbit was easily hit. To reach that, I had to reduce the connection-rate somewhat. The main reason for that, I suspect, is increased latency when the network is saturated.

At this point, Varnish was not saturating the CPU. It still had 30-50% idle CPU power.

Just for kicks and giggles, I wanted to see how far we could really get, so I threw in a local httperf, thereby ignoring large parts of the network issue. This is a screenshot of Varnish serving roughly 1gbit traffic over network and a few hundred mbit locally:

So that’s 275k requests/s. The connection rate at that point was lousy, so not very interesting. And because httperf was running locally, the load on the machine wasn’t very predictable. Still, the machine was snappy.

But what about the varnish+varnish setup?

The above numbers are for a single Varnish server. However, when we tested with varnish as a load balancer in front of Varnish, the results were pretty identical – except divided by two.

It was fairly easy to do 100k requests/second on both the load balancer and the varnish server behind it – even though both were running on the same machine.

The good thing about Varnish as load balancer is the keep alive-nature, speed and flexibility. The contention-point of Varnish is long before any balancing is actually done, so you can have a ton of logic in your “Varnish Load balancer” without worrying about load increasing with complexity.

We did, however, discover that the number of HTTP header overflows would spike on the second varnish server. We’re investigating this. The good news is that it was not visible on the user-side.

The next step

I am re-doing part of our internal test infrastructure (or rather: shifting it around a bit) to test the acceptor thread regularly.

I also discovered an assert issue during some sort of race at around 220k req/s, but that was only under certain very very specific situations. It was not possible to reproduce on anything that wasn’t massively parallel and almost saturated on CPU.

We’re also constantly improving our test routines both for customer setups and internal quality assurance on the Varnish code base. I’ve already written several load-generating scripts for httperf to allow us to test even more realistic work loads on a regular basis.

What YOU should care about

The only thing that made a real difference while tuning Varnish was the number of threads. And making sure it actually caches.

Beyond that, it really doesn’t matter much. Our defaults are good.

However, keep in mind that this does NOT address what happens when you start hitting disk. That’s a different matter entirely.

Some time during roughly the same night, PHK refactored all the director infrastructure and in the process implemented a client director and a hash director as a sort of side-job. PHK’s version obviously entered trunk and mine never saw the light of day (possibly because it was December and there isn’t much daylight to see here in December anyway). At least the duplicated effort wasn’t significant.

We’ve kept that somewhat hidden, mostly because the actual code for the hash and client director is about a screenfull of text once the VCL-bits are taken care of. And it’s such a small feature. Truth be told, they both live within the random director and are just special exceptions. The VCL syntax is the same, except now it’s called a client director instead of random.

The first up, the client director, will use the ascii-representation of the client IP to pick a “random” backend. (see why it’s the ascii representation here: http://varnish-cache.org/trac/browser/trunk/varnish-cache/bin/varnishd/cache_dir_random.c?rev=5304#L99)

There isn’t much to say about it. Your VCL will be the same as if it was a random director. If the “canonical backend” for the client is sick, the regular random algorithm will be used to pick a backend from the healthy ones.

In Varnish 2.1.4 you will also get a “client.identifier” VCL variable that you can use to tell varnish what identifies this as a unique client. In theory, you could pass a cookie to it and avoid wondering what happens if an IP changes. If it is set, the client director will use that instead of the IP. The only problem I have come up with using that approach is bootstrapping.

He very first request a client makes will have no cookies, so no data can be passed to client.identifier. The backend will presumably set a session-cookie of some sort so the next request could use that session cookie, but that means the first and second request is likely to go to different backends, even if the following requests will all go to the same backend.

Typical load balancers solve this by just setting the cookie them self. I suspect that is what it will come to, and I also suspect that it will be a lot nicer to do that in Varnish 3.0.0 when vmods are in place which could easily deal with these sort of things without messing up your regular VCL…

But until then, try out the client director for normal IP’s at least. The client director is available in 2.1.3 (I think it’s even in 2.1.0, though I can’t remember. Definitely not documented in 2.1.0, though).

http://www.varnish-cache.org/docs/trunk/reference/vcl.html#the-client-director

I’m very interested in your take on how to deal with sessions, if just basing it on IP is “close enough” and if you think the client director will be able to solve most session-stickyness scenarios.