Praxis

Happy 2nd Birthday, Mashery

Two years ago today, our friends at Ropes & Gray filed the incorporation for Mashery. What began as an idea that Rafer and I first mentioned a week before in a dinner with Josh and Chris has grown to a company with a phenomenal team of nearly 20 employees. We recently signed our 40th customer, and the "household names" among our newest (but as yet unannounced) customers are astonishing - as are all the amazing companies who have trusted a startup like us to provide critical API infrastructure to help their businesses grow.

We're also really fortunate to have so many great investors - Josh and Rob from First Round Capital, Clint from Formative, and an incredibly supportive group of angels.

To my wonderful co-founders and everyone who's been a part of this, thanks so much - and I can't wait to see what the next two years brings!

The Amazon EC2 outage no one noticed

"Everything fails all the time."

- Amazon's VP and CTO Werner Vogels at the NextWeb conference in Paris Amsterdam about two weeks ago

"Werner was right."

- Me, this past weekend

What happened:

This past Friday night around midnight, Mashery began to experience what appeared to be some intermittent connectivity issues with some of our Amazon Web Services (AWS) Elastic Computing Cloud (EC2) instances.

This gave us an opportunity to test our "fail over" infrastructure at Limelight. It worked.

It also gave us an opportunity to test AWS's brand spankin' new "Gold" support package. It worked too - we were put in immediate touch with people who knew what they were doing and could work collaboratively with us to isolate the cause and solve the problem. Anyone who uses AWS for critical infrastructure and is not on the Gold plan should sign up today.

But given Mashery's reliance on Amazon Web Services infrastructure for what we do, and the strong relationship we've been able to build with the AWS team, I thought I'd share a bit on what happened, why, and how everyone reacted to it. Everything fails. Minimizing the frequency, duration and impact of those failures, and learning from the failures that do happen, is of the utmost importance. After this weekend's problems, I remain a big fan of AWS, but I know that they, and we, learned some valuable lessons and gained an important data point that will help us minimize future outages.

Cloud computing - it's great, but it's evolving

For those of you not schooled in EC2 and "cloud computing", a brief primer:

Elastic Computing Cloud is just what it sounds like - a "cloud" of virtual servers that are available to us on an ad-hoc basis. With EC2, we are able to add and subtract virtual servers ("instances", in their parlance) as needed to meet demand in real time.

This can be a great advantage when load spikes, such as when our client Reuters was fortunate enough to get slashdotted - we brought ten additional servers on line in a few minutes just in case we needed more capacity, kept them online during the period of peak load, and returned them to the cloud when things had subsided.

Each piece of physical hardware (a server or, as AWS calls it, a "host") can be, in essence, divided into multiple virtual servers. So one computer with 8 gig of RAM can be made to function as 4 virtual machines, each of which a little less than one fourth of the processing power and a little less than 2 gig of RAM (since some of the memory and processor cycles are needed to run the software that makes the virtual machine magic work). Network infrastructure between physical (and therefore virtual) machines are shared, and everything costs less.

And, of course, we get to do all of this on a "pay as you go" model - our entire cost for the increased infrastructure during the Reuters load spike couldn't buy you a latte at Starbucks; our entire cumulative cost of servers, storage and bandwidth since we founded the company has been a small fraction of the salary we'd have paid one of the two or three sysadmins we'd have to have hired to run our server cage if we had a cage

We've been able to build and scale Mashery while taking in less than half of the capital we'd have needed if we were buying and operating our own infrastructure.

Sounds great in theory. It's pretty great in practice, too - many companies (Mashery included) are able to share infrastructure in a way that allows each of us to use the resources we need when we need them, and let others use them when we don't. Cheaper for us, better for the environment - all good.

One of the biggest challenges in virtualization, though, is inter-instance security and resource allocation. In theory, instances - even those on a shared host - are completely isolated from each other. One virtual instance should not be able to interfere with or impact the performance another - even if they are on the same hardware server, and even if one instance is getting hammered by traffic or other resource-intensive work. The closer your virtual server solution can get to perfection in this realm, the better and more secure your service. Solving this challenge needs to be one of the core competencies of any company (like Amazon) providing cloud computing infrastructure as a service.

It's like living in an apartment building - if you share the same kitchen vent, steps need to be taken to make sure that my pungent dinner doesn't leave your apartment smelling of garlic. And though we might share a lobby, the outside visitor should not really notice; to your visitor, the lobby is clearly yours, and to mine, it's mine. When I take a 30 minute shower in the morning, I shouldn't use up all of your hot water. And so on.

Given the complexity of data center security, the extent to which AWS is able to prevent inter-instance issues is impressive. Impressive, but, as it turns out, not perfect.

What caused the outage?

Well, as Amazon said in the forum where it provided some information on the outage:

This performance issue, affecting a small number of instances in a single availability zone, was the result of a customer applying a very large set of firewall rules while simultaneously launching a very large number of instances.  The high volume of firewall rule changes, combined with an usual rule configuration, exposed a performance degradation bug in the distributed firewall that lives on the physical hosts. The issue has been resolved. In addition, we are also increasing the density of our monitoring to detect and isolate issues in this area of our infrastructure more rapidly.

What does that really mean? Each "host" is a physical machine that hosts several instances. Since it's one physical machine, many resources, such as the network interface that connects the machine to the internal network and, ultimately, the Internet, are shared by all the instances on that machine. Since each instance is performing a different set of tasks for a different customer, each customer will configure that instance differently. One essential configuration of any server (physical or virtual) is its firewall - not a physical firewall appliance, but the software-defined rules that dictate what will and will not be allowed into that particular machine. Each of the instances on the host is allowed to set its own firewall rules, and the common network interface will implement those rules for the traffic coming to that instance.

So some other AWS customer with whom we were sharing one or more instances decided to do a load test of some sort that involved changing a bunch of firewall rules while starting up and configuring a bunch of new instances (probably to simulate how they would react to a spike in traffic that would need more instances to handle) and throwing a lot of traffic at the instances that were up exposed an issue that AWS had not tested for before, and the result was that the other instances on some of the hosts (several of which were being used by Mashery) found their network interface intermittently unable to connect to the network.

"Availability zones" is AWS-speak for "physically separate data centers", so they are saying that this particular test only impacted a portion of one of the several EC2 data centers.

How was the root cause diagnosed?

In response to many customer requests and a major EC2 outage a couple weeks ago, AWS recently rolled out a "Service Health Dashboard" which was intended to provide customers with a snapshot of current and recent service outages or degradation. The Red/Yellow/Green presentation is meant to be simple, which it is. But its simplicity severely limits its utility. Looking at whether an entire service is up, or even an entire availability zone is up, will miss instance-level issues like the one we experienced, and provides no visibility into root cause, estimated time to resolve, or steps one can take to work around the outage. So when we noticed that we were having issues, the Service Health Dashboard still showed "all green".

As it turns out, Amazon's internal monitoring did not detect the issue either. We reviewed recent configuration changes and releases, and determined that none had been made in the time leading up to the issue. We all wanted to be extra-careful - it is easy to point fingers during an outage, but the fact that no one had posted about the outage on the AWS forums or on any blog made it appear more likely to be a Mashery issue than an AWS issue. But as I learned many years ago, if you systematically rule out everything that is impossible, then whatever's left, however improbable, has to be the issue. Only after AWS and Mashery teams went through server log files (our instance-level logs and AWS's host-level and network logs),  and Mashery's own 24x7 monitoring powered by Webmetrics, was the problem discovered. Just as Clay and the Mashery team reached the conclusion that it had to be an AWS issue, the AWS team contacted us and let us know that they had isolated the cause and were applying a fix.

The major lesson learned

Werner's right - everything fails all the time.

Although the root cause of this particular issue was a resource contention issue between instances, things like that are going to continue to happen. There may now be a fix for this particular edge case, but there are undoubtedly others that will crop up over time.

The real failure here was a failure of monitoring, and a failure of transparency.

Amazon.com's own massive architecture is built on a lot of redundant hardware and software, with redundancies built into redundancies. Things break; wounds are routed around and ultimately heal themselves. This model is much easier to build when you have visibility into, and control of, network infrastructure. But when all you control are the individual servers and the DNS records that point to them, you are operating at a distinct disadvantage. AWS's new Elastic IP Addresses are a step in the right direction - a big improvement from what we had before, as I wrote previously - but it is a small step on the path toward a fully outsourced web-application-infrastructure-as-a-service. By definition, such a service needs to provide control over network infrastructure that is analagous to the "root access" we have to the EC2 instances themselves.

Until (and even after) we reach network control nirvana, AWS needs to add three elements to their monitoring:

• Amazon needs to pay attention to monitoring on the instance level, rather than on the host or network level, to ensure that even though a host might look "healthy", one of its instances may be having difficulty. Amazon appears to be recognizing this need - that would be the "increasing the density of our monitoring" mentioned in their statement.
• Amazon needs to recognize that even an anomaly like the one from this weekend, which only affected a few customers in one part of one availability zone, is an "outage". As commenter "mikklr" noted, "Why doesn't the Service Health Dashboard reflect these failures?"  Good question.
  
  
• Amazon needs to give their customers more visibility into what is happening in the network, and in fact in the hosts that are running our instances. It is not enough for them to say "well, if you're having a problem with an instance, shoot it, and spin up a new one" when we don't have the network control to achieve immediate and reliable cutover. And it's not enough for them to say "well, if you have a critical task being performed by a particular server, run that as an Extra Large Instance so it is not sharing hardware with other instances" when an extra-large instance carries an 8x price tag over a standard instance. For every instance we run, we need to not only be able to monitor the health and load of  our instance, but also monitor it for the host at large, so we can see if there is something happening on our physical host that might be causing an issue. It's simple - when diagnosing a problem, more data and more visibility is better.

On Mashery's side, we will to continue to work with all of our infrastructure vendors - AWS, Limelight, Webmetrics and Rightscale - to make sure that we're taking advantage of all the opportunities they can provide to make our service as redundant and fault-tolerant as possible. In other words, we leverage the power of virtualization and cloud computing to allow a small startup to provide the kind of reliability that previously required the infrastructure, in-house expertise and massive capital outlay that only large companies could afford.

All four of these companies have been very responsive. AWS mobilized top engineering, support and management resources to find and fix the problem, and then took the time to listen and react to our questions and concerns. Limelight has brought additional POPs online for us when we had some concerns over implementation speed. Webmetrics helped us reconfigure our monitoring in a way that will help isolate and identify issues similar to the one we had this weekend. And Rightscale, our newest partner, is helping us automate some of the failover, redundancy, load balancing and backup tasks that we used to either do manually or have to maintain our own scripts to do.

AWS and cloud computing allow us to  provide a scalable, economical service that meets our customers' ever-increasing load and need for mission-critical reliability with a solution that combines our API and domain expertise and 24x7 service with the best of the expertise, capitalization and sophistication of a company like Amazon. Everyone provides what they're best at, and the customers win. It's a great business.

iPhones, iPod Touch, and Connectivity

Two interesting data points today: First, the nationwide shortage of iPhones would seem to indicate that Apple is about to launch a new version - especially since it's timed with the CTIA show this week in Vegas.

And second, I learned today that Meebo gets 3x as many daily users on their service from iPod Touch models than from iPhones, even though there are many, many more iPhones in the world than Touches. Even though Meebo and similar services are the only messaging app available for the Touch, while the iPhone provides a lot more in the way of communication options, this still came as a surprise. I don't have the exact numbers on how many Touches have been sold vs how many iPhones, but I'm guessing that this means that a Touch user is, what, 10x as likely to use Meebo as an iPhone user? Something like that.

Interesting.

Two huge announcements from Amazon Web Services

Today saw two huge announcements from Amazon Web Services. The vast majority of Mashery runs on AWS infrastructure, and today they announced two features we've been asking for since they launched. Both will save us a ton of money and improve the reliability of the service we provide our customers.

The first new service is what Amazon calls "Elastic IP addresses". Amazon loves the term Elastic - it is the "E" in "EC2". Everyone at Mashery calls them Static IP Addresses, but either way, we're thrilled they exist. Why is this so huge? Well, we at Mashery have dozens of EC2 server instances up at any given time. And from time to time, we need to shut down one instance and replace it with another. It's one of the great things about EC2 - we don't have "maintenance" downtime for a server; we merely shoot the old one and  activate a new one.

The problem is that with a new instance comes a new IP address. That is easily remapped if it is only called internally, from other EC2 instances, but if it's being accessed from outside of EC2, the external URL needs to be remapped to the new IP address. This is done by changing your DNS settings, and these changes take considerable time to propagate to all the millions of nameservers around the world. Until the new address has propagated, any calls coming to that URL will try to hit the old (now dead) EC2 instance.

Before Elastic IP addresses, you had two choices:

• If you could plan the switchover in advance, you could either have both servers up and taking calls at the same time - one at the old IP, and one at the new IP - until all the nameservers were updated and the old server stopped seeing any traffic; at that point, you could shut it down. Of course, if the instance just died spontaneously, you didn't have that luxury. And like any server anywhere, sometimes EC2 instances do just die.
• You could have a very short TTL (Time to Live) on your DNS record. The TTL tells the nameservers around the world how often to check in with your DNS to see if the address has changed. In theory, if you have a 24 hour TTL on your DNS, each nameserver (there are over 11 million nameservers in the world) that handles a call to that URL will check with your DNS if the IP address it has on file is older than 24 hours. 24 hours is fine if your IP addresses don't ever change without warning. But if you have to replace an EC2 instance handling critical traffic, you would have an "outage" of up to 24 hours until all the DNS servers reach the 24 hour expiration.

So at Mashery, we set our TTLs to 2 minutes. That way, all the nameservers (well, those that actually obey the stated TTL) would never be more than 2 minutes out of date. The problem, of course, is that our DNS has to handle calls from millions of nameservers, and a lot of them were hitting us every two minutes. We pay for DNS services from an outsourced DNS provider, and they charge by how many DNS inquiries we receive each month. It got very expensive, but we had no choice.

But with Elastic IP addresses, we can keep the same (static) IP address, and map from an old EC2 instance to a new one. The DNS does not need to be updated at all, and we can increase our TTLs. Longer TTL = fewer DNS calls = we save money.

Their second announcement is simpler to explain, though no less important. Amazon calls it "Availability Zones". We call it "multiple datacenters". Until this announcement, Amazon would randomly assign newly activated EC2 instances to a particular physical location; we had no way of knowing where it was. Now, Amazon has explicitly said they have more than one data center in Virginia, and you can specify which of your instances should be in one datacenter, and which should be in the other. Those that frequently talk to each other should be together. And now it is finally possible to ensure redundant infrastructure in more than one physical facility, so if one goes down you still have access to the other. Both performance and reliability should improve.

Plus, if you read the fine print, Amazon says that EC2 "currently" only exposes availability zones in one region (a region being a particular geographic area or country). Which means that someday (hopefully soon), they will expose more regions. So we can quickly, easily and inexpensively provide a geographically distributed service through Amazon (currently, Mashery relies on Limelight Networks for its clients who need to have our services provided from somewhere other than Virginia).

Thanks for the new features, Amazon, and bring on those new regions!

Must be International Web Services Day today

Two great posts today on web services - probably having to do with YouTube's announcement of their new APIs. Both of them gave me great new quotes to put on Mashery's customer pitches:

First, we hear from Fred Wilson:

You cannot be a destination exclusively on the Internet anymore. If you are not a open web service, you won't get nearly as far these days...So if you are building a new web service today, forget about being a destination. Maybe it will happen and maybe it won't. Don't fuss about that. Focus on making your service available everywhere. If you do that, you'll build a much larger user base.

And then  Steve Rubel weighs in:

I believe most brand web sites may not matter in 2012 - unless they have satellites that make the mother ship stronger...everyone - including marketers - will need to think of their online brands not as sites but as portable services that can go anywhere and everywhere the consumer wants. Without such appendages, no brand will ever be able to break through the online clutter such unlimited choice offers.

Thanks to both of you for helping us spread the word about APIs! Anyone who wants to learn more should certainly join us at The Business of APIs this Monday (St. Patrick's Day) in New York. I've been sitting in on the pre-show conference calls with the participants, and it's clear that we're going to learn a lot from all the great speakers.

Hope to see you there - to make it that much easier, just use the code MASH1 for $100 off the regular $295 registration fee - a 33% savings!

Reuters launches Mashery - Powered API - with a $5k Bounty

Reuters announced yesterday * that their "Open Calais" API launched yesterday. We at Mashery are proud of the quick deployment of our Mashery Pro API infrastructure service for Reuters - check out the Mashery-powered API portal here.

One of the questions we're often asked is "once I launch my API, how do I attract developers to build cool things with it?" Well, Reuters has decided on a tried-and-true method - a $5,000 cash bounty for the first developer to build an OpenCalais-powered plugin for WordPress...and they have said that more such bounties will follow. Is it working? Well, there have been a lot of developer keys issued during the first 24 hours, and at least one developer has blogged that he's already working on an opencalais Ruby interface. Good luck to all!

[cross-posted from the Mashery blog]

[* I never thought I'd wind up linking to my piddling little hometown newspaper, the Centre Daily Times...but when I googled the Reuters press release, that was the first result that came up. Go figure]

Screenscaping, API Business Models, and the Value Layer

Having just returned from a brief but wonderful vacation, I decided to join Josh in resolving to blog more this year. Happily, four recent news items provided instant fodder for commentary...but as I began working on four blog posts, I realized that I was saying the same basic thing on all four which in and of itself was pretty interesting. Tying these articles together is like playing "six degrees of Kevin Bacon"...

The four stories:

• Josh McHugh's excellent piece in Wired on data ownership and screenscraping
• News that Rhapsody has shut down Yotta for using "non-public" APIs instead of the approved, "public" ones
• The discussion of Twitter's business model, or lack thereof.
• and, of course, the fallout from Scoble being banned from facebook for violating their terms of service by screenscraping email addresses (though, as of this writing, Facebook has decided to allow him back in if he promises not to do any more screenscraping).

Most web applications are built on a three-tier architecture:

• At the bottom is the Data Layer, which is a repository of (or a means of accessing) all the data necessary to run the app, whether it has been generated by the app, entered by the user, or pulled from a third-party source. 
• One step above is the Logic Layer, where the app performs calculations and otherwise "processes" the data to provide a useful service, aggregate the data in an interesting way, or perform other related tasks. 
• Finally, there is the presentation layer, where the computer-readable results of the data layer's work are rendered into human-readable (and browser-presentable) form, complete with navigation, images, widgets, ads, and whatever else constitutes the "user experience".

Now the business model/screenscraping/data ownership question that any provider of a web service (lower case W - any service provided over the web - as opposed to the upper case Web Service, which has particular meaning to those of us working in the "Web Services Space") is "which layer represents my core competency, the 'secret sauce' that I do particularly well, and which will drive the value I am creating with my business."

It is this "value layer" that needs to be protected from being co-opted by others who would try to go around it...but which also represents the biggest opportunity for rapidly increasing reach by, well, allowing others to co-opt it, but in a good way. Sounds contradictory, but it isn't. Let people provide other layers (presentation, data, or logic) to complement your value layer, but fight vigorously any attempt to circumvent your value layer for their own purposes.

Leaving aside the services that have no value at all, most services have the data layer, the logic layer, or some combination of the two as their value layer.

Twitter (business model notwithstanding) has, through their decision to allow any number of third party apps and services to offer UIs that compete with their own web interface, has explicitly acknowledged that they fit into this category, and whatever business model they develop will almost certainly involve building monetization into these layers. Twitter broadly distributes their value layer through others' presentation layers.

eBay, who has published the statistic that over half of their listings originate through their API (and therefore without any use of their presentation layer), has a similar orientation.

As do the financial institutions who allow for companies like Mint to screen-scrape users' data for inclusion in their vastly superior user interface.

Rhapsody blew it on this one big time, because clearly they have a compelling and defensible business model that people are paying real money for - just like eBay - and Yotta was providing a front-end that some people felt was superior. Rhapsody still made every cent on the subscription, but rather than embracing a service that got their "value layer" broader distribution, they shut it down. Clearly the "public API" does not provide adequate functionality for a service like Yotta; Rhapsody should have looked at how to make the "private" API into a public one, and seen how many other alternative UIs sprang up from their passionate community.

Very few services can legitimately claim that their presentation layer is their "value layer". Of course, most everyone wishes that the truth were otherwise, because so many new applications pin their revenue hopes on advertising, links, and other monetization schemes that can only work if the user visits their presentation layer. But wishes don't count - only actual value.

As for Scoble, I totally agree with Michael's take on the whole thing. Facebook is in the rare but lovely place of having two "layers", one of which is the presentation layer, and the other of which is the data layer. By going after both of them with a "ask forgiveness not permission" strategy, the result was inevitable. On the other hand, Facebook has been explicit that they care little about their logic layer, and as long as you don't try to abuse the other two, you have pretty much free reign to distribute your logic layer through their presentation layer, and leverage their data layer (also known as their user base).

And as for the whole "who owns the data" debate, Scoble owns data that he entered, but if he entered it, he probably has it somewhere offline as well. Here we are dealing with data that he was granted access to by facebook, and which they control under the terms of use. Given the lengths Facebook has gone to to make it hard to copy and paste email addresses or access them under the API, one could reasonably assume that something like Plaxo's new service would be met with resistance.

This is one of the early shots in what is likely to be a big debate over APIs vs. Screenscrapers. Services like Kapow and Dapper are hugely convenient for personal projects, but any large-scale (i.e., significant enough to be commercially relevant) use of scrapers will be easily blocked.

A well-managed open API program, with various tiers of access based on commercial relationship and demonstrated adherence to terms of use and privacy policies, is the best way to leverage your "value layer", and allow "good co-opting" while restricting "bad co-opting".

As McHugh says, such a relationship benefits both sides:

Some large Web companies don't relish the unregulated dispersal of their data and would love to find a way to monitor and control the information they dole out. That's why many of them have begun encouraging developers to access their data through sets of application protocol interfaces, or APIs. If scraping is similar to raiding someone's kitchen, using an API is like ordering food at a restaurant. Rather than create their own bots, developers use a piece of code provided by the data source. Then, all information requests are funneled through the API, which can tell who is tapping the data and can set parameters on how much of it can be accessed. The advantage for an outside developer is that with a formal relationship, a data source is less likely to suddenly turn off the taps.

Congrats to Compete for Passing Alexa!

Dave Cancel posts that Compete, the company he co-founded (but left to start Lookery with Rafer), has seen its compete.com site pass alexa.com in number of visitors.

How did this happen?  Well, they did a lot of things very well, but juxtaposing the inflection points on this graph with the milestones Dave showed in his slides at The Business of APIs Conference, their Mashery-powered API was a big contributing factor.

Mashery is thrilled to have played a part in this important milestone in Compete's growth, and thanks Compete for being an early customer and ardent supporter of ours - and for showing the world how one can gain immense value by offering a free, open API.

Congratulations to the entire Compete team!

Heading to Berlin and London

I'm heading to Berlin to speak at Web2, where I'll be from 4 Nov through 8 Nov 07, and then to London for 9 Nov. Let me know if you'll be around and want to grab coffee and talk APIs (or anything else, though APIs are of particular interest to me).

Join us for Dinner with Hugh MacLeod of Gapingvoid

One of the most interesting people I've met since I started hanging out with the denizens of the blogosphere has to be Hugh MacLeod. Hugh is best known for drawing cartoons on business cards, which he posts (and freely allows people to download and reproduce) on his wildly popular site, gapingvoid.

I've had the pleasure of hanging out with Hugh in both San Francisco and London, and each time we've talked I've learned a lot, been challenged, and had a lot of fun. Hugh understands marketing, people, social objects and community in a way that few people do - and even fewer can effectively articulate. So it's little wonder that he's created very successful viral/community/social marketing programs for brands as diverse as Stormhoek wine, Microsoft, and my personal favorite, English Cut. Sometimes, Hugh even gets his brands to co-market.

So when Hugh asked me to organize a dinner for him in San Francisco, I was honored to help out. The dinner is on Tuesday, October 30, at 7:30 pm at Osha Thai at 149 Second Street in SF. It's the usual format - click on this link to Acteva to register and pay $37 (plus Acteva's $2.70 processing fee) for dinner, tax and tip. Drinks are no-host, and Osha tells us that they are cash only.  Space is limited, so be sure to sign up soon if you want to come.

Hugh also has a lot of interesting friends from around the world, and I imagine several of them will wind up joining us. So if you're in town on Tuesday, by all means come have dinner with Hugh. 

(picture of Hugh from Laughing Squid)