Fastest server stack configuration for WordPress?

Achieving the "fastest" server stack involves optimizing multiple layers, not just the software modules (e.g. Ubuntu packages, etc) that make up the actual VPS. I tell clients to think about this sort of like a pyramid analogy, because the "top" means nothing without a good "base":

1. DNS. The best DNS technology these days is Anycast... use an ICANN-certified registrar and avoid reseller APIs to keep things clean and simple. Personally I put all my clients on CloudFlare because they continue to be the fastest DNS service in the world. And a side benefit of CloudFlare is that it helps tremendously in stabilizing/optimizing your website's SSL handshakes, too, among a few other nifty optional features.
2. Datacenter. Choose a well-managed rack in a major internet hub city to cut down on "hops" required to deliver your website content to users that may be far away. And whether or not you're using a proxy service like CloudFlare you can still still add-on a CDN service for your static assets to make sure they are loading from distributed edge locations.
3. Virtualization. The king of virtual containers in high-end datacenters is KVM currently, which has outpaced others like OpenVZ, XEM, etc. (Keep in mind this is different from e.g. Docker containers, a different technology.) Pretty much you should only be using KVM cloud servers for better resource allocation to your VPS machines.
4. OS. Most major Linux operating systems are decent, but a few are better tuned to web servers and reducing the amount of RAM/resources needed... Ubuntu Server is one of the best and user-friendly these days, for certain cases you might consider Debian or Red Hat, etc.
5. LEMP stack. Avoiding a major discussion here, LEMP stacks have lead the market in performance for several years already. This means Linux, Nginx, MySQL (MariaDB) and PHP-FPM.... and most sysadmins will also include Redis for object caching. (A few Nginx competitors like OpenLiteSpeed have claimed to be faster, but this is not accurate.)

Here's a specific example stack:

CloudFlare + DigitalOcean + Ubuntu LTS + Nginx + MySQL + PHP-FPM + Redis

But it's not that simple, because you also need to ensure every layer in said "stack" is well configured. This is the part of server management that most web developers don't know, which is why open source projects like my own SlickStack are trying to make it super easy to automate best practices and settings... including things such as the Linux kernel, etc.

There's a post here that's very good about load optimization and performance: Steps to Optimize WordPress in Regard to Server Load?

It might be a good idea to also utilize a CDN for a majority of your page requests.

If you want performance you'll have to minimize requests to your database and setup aggressive caching.

Working with Drupal i know this can be built into your drupal install. I'm not very familiar with Wordpress and if there are capabilities integrated into Wordpress to facilitate reverse proxy requests.

If you're going to use mem-cache you may need more Ram to serve up the page.

You may want to also setup varnish on your server. Setting up varnish alone will give you a big boost; as i've been told. Varnish http://www.varnish-cache.org/