Praxis

Screenscaping, API Business Models, and the Value Layer

Having just returned from a brief but wonderful vacation, I decided to join Josh in resolving to blog more this year. Happily, four recent news items provided instant fodder for commentary...but as I began working on four blog posts, I realized that I was saying the same basic thing on all four which in and of itself was pretty interesting. Tying these articles together is like playing "six degrees of Kevin Bacon"...

The four stories:

• Josh McHugh's excellent piece in Wired on data ownership and screenscraping
• News that Rhapsody has shut down Yotta for using "non-public" APIs instead of the approved, "public" ones
• The discussion of Twitter's business model, or lack thereof.
• and, of course, the fallout from Scoble being banned from facebook for violating their terms of service by screenscraping email addresses (though, as of this writing, Facebook has decided to allow him back in if he promises not to do any more screenscraping).

Most web applications are built on a three-tier architecture:

• At the bottom is the Data Layer, which is a repository of (or a means of accessing) all the data necessary to run the app, whether it has been generated by the app, entered by the user, or pulled from a third-party source. 
• One step above is the Logic Layer, where the app performs calculations and otherwise "processes" the data to provide a useful service, aggregate the data in an interesting way, or perform other related tasks. 
• Finally, there is the presentation layer, where the computer-readable results of the data layer's work are rendered into human-readable (and browser-presentable) form, complete with navigation, images, widgets, ads, and whatever else constitutes the "user experience".

Now the business model/screenscraping/data ownership question that any provider of a web service (lower case W - any service provided over the web - as opposed to the upper case Web Service, which has particular meaning to those of us working in the "Web Services Space") is "which layer represents my core competency, the 'secret sauce' that I do particularly well, and which will drive the value I am creating with my business."

It is this "value layer" that needs to be protected from being co-opted by others who would try to go around it...but which also represents the biggest opportunity for rapidly increasing reach by, well, allowing others to co-opt it, but in a good way. Sounds contradictory, but it isn't. Let people provide other layers (presentation, data, or logic) to complement your value layer, but fight vigorously any attempt to circumvent your value layer for their own purposes.

Leaving aside the services that have no value at all, most services have the data layer, the logic layer, or some combination of the two as their value layer.

Twitter (business model notwithstanding) has, through their decision to allow any number of third party apps and services to offer UIs that compete with their own web interface, has explicitly acknowledged that they fit into this category, and whatever business model they develop will almost certainly involve building monetization into these layers. Twitter broadly distributes their value layer through others' presentation layers.

eBay, who has published the statistic that over half of their listings originate through their API (and therefore without any use of their presentation layer), has a similar orientation.

As do the financial institutions who allow for companies like Mint to screen-scrape users' data for inclusion in their vastly superior user interface.

Rhapsody blew it on this one big time, because clearly they have a compelling and defensible business model that people are paying real money for - just like eBay - and Yotta was providing a front-end that some people felt was superior. Rhapsody still made every cent on the subscription, but rather than embracing a service that got their "value layer" broader distribution, they shut it down. Clearly the "public API" does not provide adequate functionality for a service like Yotta; Rhapsody should have looked at how to make the "private" API into a public one, and seen how many other alternative UIs sprang up from their passionate community.

Very few services can legitimately claim that their presentation layer is their "value layer". Of course, most everyone wishes that the truth were otherwise, because so many new applications pin their revenue hopes on advertising, links, and other monetization schemes that can only work if the user visits their presentation layer. But wishes don't count - only actual value.

As for Scoble, I totally agree with Michael's take on the whole thing. Facebook is in the rare but lovely place of having two "layers", one of which is the presentation layer, and the other of which is the data layer. By going after both of them with a "ask forgiveness not permission" strategy, the result was inevitable. On the other hand, Facebook has been explicit that they care little about their logic layer, and as long as you don't try to abuse the other two, you have pretty much free reign to distribute your logic layer through their presentation layer, and leverage their data layer (also known as their user base).

And as for the whole "who owns the data" debate, Scoble owns data that he entered, but if he entered it, he probably has it somewhere offline as well. Here we are dealing with data that he was granted access to by facebook, and which they control under the terms of use. Given the lengths Facebook has gone to to make it hard to copy and paste email addresses or access them under the API, one could reasonably assume that something like Plaxo's new service would be met with resistance.

This is one of the early shots in what is likely to be a big debate over APIs vs. Screenscrapers. Services like Kapow and Dapper are hugely convenient for personal projects, but any large-scale (i.e., significant enough to be commercially relevant) use of scrapers will be easily blocked.

A well-managed open API program, with various tiers of access based on commercial relationship and demonstrated adherence to terms of use and privacy policies, is the best way to leverage your "value layer", and allow "good co-opting" while restricting "bad co-opting".

As McHugh says, such a relationship benefits both sides:

Some large Web companies don't relish the unregulated dispersal of their data and would love to find a way to monitor and control the information they dole out. That's why many of them have begun encouraging developers to access their data through sets of application protocol interfaces, or APIs. If scraping is similar to raiding someone's kitchen, using an API is like ordering food at a restaurant. Rather than create their own bots, developers use a piece of code provided by the data source. Then, all information requests are funneled through the API, which can tell who is tapping the data and can set parameters on how much of it can be accessed. The advantage for an outside developer is that with a formal relationship, a data source is less likely to suddenly turn off the taps.

Congrats to Compete for Passing Alexa!

Dave Cancel posts that Compete, the company he co-founded (but left to start Lookery with Rafer), has seen its compete.com site pass alexa.com in number of visitors.

How did this happen?  Well, they did a lot of things very well, but juxtaposing the inflection points on this graph with the milestones Dave showed in his slides at The Business of APIs Conference, their Mashery-powered API was a big contributing factor.

Mashery is thrilled to have played a part in this important milestone in Compete's growth, and thanks Compete for being an early customer and ardent supporter of ours - and for showing the world how one can gain immense value by offering a free, open API.

Congratulations to the entire Compete team!

Heading to Berlin and London

I'm heading to Berlin to speak at Web2, where I'll be from 4 Nov through 8 Nov 07, and then to London for 9 Nov. Let me know if you'll be around and want to grab coffee and talk APIs (or anything else, though APIs are of particular interest to me).

Join us for Dinner with Hugh MacLeod of Gapingvoid

One of the most interesting people I've met since I started hanging out with the denizens of the blogosphere has to be Hugh MacLeod. Hugh is best known for drawing cartoons on business cards, which he posts (and freely allows people to download and reproduce) on his wildly popular site, gapingvoid.

I've had the pleasure of hanging out with Hugh in both San Francisco and London, and each time we've talked I've learned a lot, been challenged, and had a lot of fun. Hugh understands marketing, people, social objects and community in a way that few people do - and even fewer can effectively articulate. So it's little wonder that he's created very successful viral/community/social marketing programs for brands as diverse as Stormhoek wine, Microsoft, and my personal favorite, English Cut. Sometimes, Hugh even gets his brands to co-market.

So when Hugh asked me to organize a dinner for him in San Francisco, I was honored to help out. The dinner is on Tuesday, October 30, at 7:30 pm at Osha Thai at 149 Second Street in SF. It's the usual format - click on this link to Acteva to register and pay $37 (plus Acteva's $2.70 processing fee) for dinner, tax and tip. Drinks are no-host, and Osha tells us that they are cash only.  Space is limited, so be sure to sign up soon if you want to come.

Hugh also has a lot of interesting friends from around the world, and I imagine several of them will wind up joining us. So if you're in town on Tuesday, by all means come have dinner with Hugh. 

(picture of Hugh from Laughing Squid)

The Business of APIs conference is filling up!

We at Mashery are getting ready for the upcoming Business of APIs conference, which we're producing in association with Dealmaker Media. The  speaker list is very strong, and the agenda should be relevant to anyone with an interest in APIs. 

The first segment of the conference talks about the business case for launching an open API. We'll have case studies from companies that have launched successful API programs, including Pikeo , Compete and RockYou.  We'll also hear from executives at Microsoft, Yahoo! and AOL about what the "giants" are doing in APIs.

The second segment dives into the nuts and bolts of API infrastructure. Dave McClure, who launched one of the most successful developer/API programs around during his tenure at Paypal, will talk about the essential elements of a successful developer program. Then we'll present an overview of Mashery's on-demand API infrastructure.

The final segment will cover the implications of the Facebook Platform on APIs and business, and the effect that web services have had on investment and corporate strategy. This segment will be led by Quentin Hardy, Senior Editor and Silicon Valley Bureau Chief of Forbes.

Registration for this conference is already over 70% of capacity, so if you're interested in coming, sign up today. I look forward to seeing you there!

Is Mashery a brokerage?

Mark Gibbs at Network World just posted a nice piece about Mashery in which he talks about the nascent Web API market and how we might fit in - thanks, Mark!

I found it interesting that Mark sees Mashery as a potential "brokerage" for web services. We tend to use the term "marketplace" rather than "brokerage", but under either nomenclature, we do our best to discourage the notion that web services access will be "sold" in the conventional sense (such as on a CPM basis). The vast majority of the most successful APIs we see generate revenue indirectly - through additional traffic from links, through additional user signups, by selling more of the underlying service, through a revenue sharing arrangement, etc. It is the rare API that (a) has inherent value; (b) is unavailable in a free or indirectly monetized form; and (c) free of the risk that unexpectedly high usage could result in unpredictable costs for the developer paying the bill. In our view, all of those are necessary features of any API that will attract any significant direct revenue over the long term.

So we use the term "marketplace" to mean a place where developers can come and find the various building blocks they need to quickly and economically deploy Web applications, without the implication that they will use them on a "pay as you go" basis. The API providers will still generate a lot of revenue from their APIs, but Mashery's role will likely be more as a means to promote broad distribution, manage the chaos that would result from an API free-for-all, and provide the tools and reporting needed to determine the success (however each API vendor defines and measures it) of each API offered through our infrastructure.

Who actually runs a developer network?

Yesterday, I got one of those "can you recommend someone for this job" emails from LinkedIn...this time, for a job at LinkedIn itself. Lucian Beebe, LinkedIn's uber-product-manager, wants to find a Developer Network Manager. His job description reads like the bio of some of the best developer network manager/evangelist types I've met.

Anyone setting up a developer network should take a look at the job description Lucian wrote - it very succinctly defines the role of the developer network manager and the skills they should bring to the job (though I'll admit that it's the rare individual who can do everything Lucian is asking for).

Mashery can provide the tools and infrastructure, but tools are only as good as the people using them. Companies that explicitly devote resources to building, evangelizing, and constantly improving their developer networks will see the most success. Look at Amazon, eBay, and Salesforce.com - all companies that have built large and thriving developer communities by devoting resources to promoting them...and by hiring the kind of people who I'll bet Lucian was thinking of when he wrote his job description.

It's not a bug, it's a feature...

How many times have I heard that? How many more times have I said that to a customer?  Now, here's photographic proof, thanks to dratz...forwarded to me, of course, by Clay, whose features are never bugs...

Facebook apps or APIs? Why choose?

Charlie O'Donnell over at This Is Going To Be BIG has a great post today about facebook apps and conversions. Aptly titled "Fuck Facebook Conversion: Be platform agnostic and use your own APIs", he makes the very valid point that facebook apps are your site, and we need to reject the notion that they exist to refer traffic to your "main site".  He quickly encapsulates some of my favorite rants.

As Charlie puts it...

The whole idea that you have a "main site" is dead. Stick a fork in it.

Then he takes on widgets...

And while you're at it, stick a fork in the widgets, too--at least the way we're been creating them.  Right now, most of the widgets that are out there are an attempt to squeeze the elements of a service into a neat little sidebar rectangle--a bottleneck created by one-way APIs, limited space, and underwhelming goals.

If you use APIs, your service/data/"site" can be anywhere you (or your partners) want - on your site, in facebook, on an iPhone, or embedded in another company's site.

The last of my favorite rants he brings up is that a well-constructed site is built on its own APIs - you are "eating your own dogfood", as it were. This was a key direction I pushed for at my last company; it was inconvenient when our partners did not have the ability to build as feature-rich an experience as we could.

Facebook is encouraging this by offering a platform that rewards applications built on APIs that offer the richest possible experience. I called their effort a "Reverse API" - the platform consuming the API is in fact dictating the protocols...but not, of course, the functionality. Offer more in the API, and people can build more in Facebook, on the iPhone, or anywhere else.

At Mashery, we find that a lot of potential customers come to us with an architecture that is primed to be turned into APIs, but is not quite there yet...there are features or methods open to the "internal" app that provide shortcuts or bypass security in ways that would be inappropriate for open APIs. This, of course, presents Mashery with an opportunity to provide our API infrastructure services.

Until your API offers the ability to build an experience equal to your "main site", Charlie says we are doomed "to have dinky widgets and namby pamby Facebook apps."  Or, as one Alex de Jong says in the comments says, "Yes. And we're getting there the moment we let go of the concept of "a site".

We're also taking one more step - currently, we have a dashboard where people can add developer keys, set their limits, enable and disable access, etc. Just the kind of thing that should be an API for our clients. That is on our development roadmap near term, as our larger prospects are requesting...and we'll be rebuilding our own site and our dashboard around our APIs. It will take some time to get there. But when it's done, we'll launch our own developer program, Mashery-powered, of course.

365 Main - Second Annual Power Outage

What's been missing from the otherwise excellent coverage of yesterday's 365 Main power outage is that this is not the first major power outage that 365's customers have had to endure.  What makes their suddenly-missing press release even more inappropriate is that last February there was a total power outage in Colo 8...and that one could not be blamed on PG&E. One of the commenters on Techcrunch mentions it...I was actually in the colo when it happened.

I had just locked our cage and was heading out of the colo when all the lights went out for about half a second. They came back fast enough that at first I though I was imagining things - power doesn't go out in a colo, right? But the sudden beeping of alarms and the unmistakable sound of servers rebooting made me realize that, yes, the entire colo had lost power.

Two months later I left the company whose cage I'd been in; during that time, 365 had not provided any compensation for the downtime. Their excuse - that a single controller board on a Hitec power unit had failed due to some input frequency error - was hard to understand, given the things they told us when we toured the facility. The most expensive part of our data center package was the power - it was never supposed to go out, and in the rare case that something could go wrong, we were assured that our cage was fed from three different power infrastructures, so at worst we would only lose 1/3 of our servers.

That they could issue a press release on two years of uptime with Red Envelope when they knew full well that the only reason that they'd achieved two years of uptime was that Red Envelope had the good fortune to not be in Colo 8 is more than a bit disturbing.

When I asked the head of operations at 365 to sit down with us to map out which servers were getting power from where, so as to mitigate the damage from "the next such incident", I was assured that there would be no "next such incident", so no meeting would be necessary.

The team at 365 has a long way to go to restore credibility. They need to be less enamored of the supposed infallibility  of the hardware one sees on their impressive tours (and they are impressive...especially the generator room) and focus instead on a more reliable approach to mitigating outages. Until they can show that they have done so, I'll not be putting any servers anywhere near their facility.

Of course, At Mashery we're avoiding the need to own and operate servers in any colo - that's why we make use of services such as Amazon EC2 and S3.