Key takeaways
Documentation and description of Web APIs will form the foundation of developer understanding in the coming years. The dreamed-of future where machines browse the Web (and it's APIs) for us discovering content, executing tasks, and building out new systems of understanding for ourselves, remains an ideal. Discovery, which surrounds both documentation and definition formats, is a mostly untapped space which should see continued growth in the coming year. Documentation formats will remain largely human-focused and human-generated. Definition formats may become more "browsable" by machines and there may yet be hope for the machine-to-machine, discoverable, executable API. The vast majority of Web APIs are collection idiosyncratic endpoints and (typically, these days) JSON-based data formats that are as unique as the people who make them. Finding one's way through this ever expanding maze can be both daunting and discouraging. However, in the past few years there has been a related rise in both documentation and description formats for this constantly expanding space.
In this virtual panel we'll hear from four individuals deeply involved in the Web API space. Each of them has a unique take on the values, benefits, and costs of documentation and description formats in general, and provide their own unique perspective from their vantage points across the Web. At times, the distinction between documentation and description formats blur. At other points, it's clear to some of these panelists that they each have unique (and important) value to developers. Collectively they are confident of one thing: something must be done to help developers find their way through the world of Web APIs.
The panelists
Lorinda Brandon – Senior Product Manager at Capital One DevExchange Zdenek Nemec – Director for domain specific languages, product manager and research lead for API design at Apiary Ruben Verborgh - Researcher in semantic hypermedia at Ghent University – iMinds, Belgium and a postdoctoral fellow of the Research Foundation Flanders Mike Amundsen - Director of API Architecture, API Academy, CA Technologies InfoQ: Why are API description formats needed?
Lorinda Brandon: Honestly, I wouldn't say they are 'needed'. We made do with WSDL and WADL for a long time (maybe we should consider that a description format too) – and if you are unfortunate enough to be integrating with SOAP APIs, you can still get your work done without one of the modern description formats. So, rather than saying they're *needed*, I would say they're desirable. And that's because, first, they help communicate your API's capabilities and, second, they make it easy to adopt your API.
Zdenek Nemec: Description formats enable structured discussions about APIs. They make API design tangible, they provide blueprints engineers and contracts for stakeholders. Let me illustrate this on my "email" story:
About 4 years ago, my friend Paul and I were about to create a brand new iOS app backed by an API. I was working on the client and Paul on the server side of this project.
One day I've sent an email to Paul asking whether our API can do "AB" and "C". Paul replied "Sure Zdenek, but what about instead of doing C we would do D and while we are at it we can tweak B to work like this…". I liked what Paul suggested and immediately replied back with few more tweaks. Three days later we ended up with over 30 emails in the conversation. Sure, we were doing the API design over email. Sure, we have had there what were going to build. Sure those emails were our contract on what we agreed on, but good luck unwrapping the conversation! It was almost impossible to get a clear concise picture on what that API should be like.
That was the moment when I've realized there must be a better way to have this kind of discussion.
Ruben Verborgh: There exist different interpretations of "API description format", and each of them has their own justification. As two main categories, I see "descriptions targeting developers" and "descriptions targeting machines".
API descriptions for developers make client development easier and faster through features such as documentation, autocompletion, and auto-generation of boilerplate code. They go back all the way to SOAP APIs with the WSDL format, and are now continued for regular HTTP APIs by efforts such as OpenAPI (formerly Swagger) and API Blueprint. In essence, these provide a hard-coded contract between a client and a server, with as a main benefits programming language independence and developer friendliness.
In contrast, API descriptions targeting machines aim to facilitate the usage of APIs by automated clients. Rather than auto-generating boilerplate code at design time, they provide interpretation of an API at runtime. The Hydra Core Vocabulary is an example format, which allows a client such as the Hydra Console to discover at runtime what resources and operations an API supports. This is necessary because, unlike humans, machines are not good at figuring out how websites work.
Mike Amundsen: First, I'll say that I think there are a couple things going on in the API Description area — what i refer to as the "API metadata space" — that need to be separated. For example, I think there are distinct differences between API description formats like JSONHome, ALPS, and APIs.json and the various API definition formats like Swagger (OpenAPI), RAML, and API Blueprint. I think most people are thinking about Swagger/OpenAPI, RAML, and API Blueprint when they use the phrase "API Description" — that's fine, but I think it's important to take in a larger picture here.
Formats like Swagger, et al define things like URLs, resources, message formats, and other details of a single Web API. You can usually take these definitions and generate a working service interface; sometimes you can use the same definition to generate a working client app, too. That works because the specs are designed to define the details of an implementation. And that, of course, can cut down on the amount of up-front coding people need to do in order to get a working Web API up and running.
Formats like APIs.json, ALPS, and JSONHome do something else, though. They actually only describe the interface generally. It is not possible to use one of these formats (alone) to generate a working service or client app. It is possible, however, to use these formats to get an understanding of the protocols, media-types, and vocabularies supported by the services. I like to say that APIs.json and ALPS, etc. can be used to describe a class of Web APIs (e.g. shopping, user management, etc.), not just a single implementation (e.g. the Foxycart shopping API).
And, to get back to your question, I think both of these format types are needed on the Web. We need a way to define Web APIs in a specific way in order to create working implementations. We also need a way to describe Web APIs in a general way in order to support online discovery and interaction. In my mind, these two classes of metadata about APIs (definition and description) compliment each other.
InfoQ: Does having a description format obviate the need for hypermedia (or is it the other way around)?
Zdenek Nemec: I'd say neither. You need to have some way to discuss an API being build regardless of its architectural style. On the other hand, you want to capture the architectural style IN the API description. Understanding the selected architectural style of an API is one of the essentials for the API to succeed in a long term. Unfortunately most of nowadays API description formats are effectively prohibiting certain architectural style and pushing towards, often suboptimal, paradigms and patterns. This is especially true when it comes to the REST (hypermedia) architectural style (for example, see Benjamin's article on content negotiation).
We (at Apiary) have done some research on whether a certain API description can turn a non-hypermedia API to a hypermedia API. And while we've found out this would be–to some extent–possible, it isn't viable at the current course of the API industry.
Ruben Verborgh: We need both if we want clients to perform complex operations in a flexible manner and without hardcoding.
Hypermedia controls serve a double purpose: they allow people and machines to look ahead a single step at a time, and provide all details necessary to execute that next step. For example, when using a shopping website, the various links and forms will let you choose items to add to your shopping cart. The "checkout" button will contain the details about the HTTP request my browser needs to execute to proceed with ordering. So if we don't want to hardcode URLs and requests in clients, we also need to rely on hypermedia controls.
Descriptions instead allow clients to look ahead multiple steps, by listing the possible actions per resource in advance. We don't need this as humans, because we intuitively have expectations of what will come. For example, we know that a webshop will ask for our address and billing information at some point (even though we don't know the exact order). Automated client do not have such native expectations, so if we want them to plan multiple steps ahead without resorting to hardcoding, we must describe these steps explicitly.
Mike Amundsen: No. The use of Swagger or RAML, etc. in defining an API does not have a direct bearing on whether the actual runtime implementation of a Web API has (or does not have) any hypermedia aspects. The use of hypermedia is completely independent of definition or description formats.
I will say that, to date, none of the definition formats seems to do a very good job of supporting hypermedia-style implementations. I'd like to see more support hypermedia but, at least for now, the "big three" formats take a resource-centric approach to defining Web APIs — that happens to currently be the most popular approach.
I think there are some great opportunities in the ability to use definition formats that support an action-centric approach — one that starts with defining actions like add, edit, approve, share, etc. using links and forms and then allows developers to combine these action elements into sets that appear within resource responses. But that'd be quite different from the current collection of API definitions. Think of what it would be like to define HTML responses with <a href="..">…</a> and <form action="…" method="…">…</form> elements within the response. We don't have that level of definition yet.
InfoQ: RESTful Web Services have gone a long time without description formats - now the list is growing quickly - what's caused all the excitement?
Lorinda Brandon: I think you can largely attribute it to the API boom – with so many APIs out there, developers need to be able to choose the right one quickly but more importantly, so do non-devs. Often a Product Manager is choosing the API to integrate with and they don't want to wade through a WADL file. It's almost like technical marketing for your API – instead of a feature description, you put your API description out there and let it do the talking for you. And the tooling that surrounds the descriptions makes them even more powerful because outsiders can often try it out and see the response for themselves.
Zdenek Nemec: The answer is speed (and therefore the cost) of development. As my "email" story explains, you can do API design without API description formats. But now there are much more effective ways to design an API than long meetings, piles of emails and Word documents.
Furthermore using an API description format can give you a lot more than "just" a framework to talk about APIs. You can avoid waterfalls and dramatically improve your API flow with quick prototyping, testing, painless documentation and more.
Both developers and stakeholders realize this and that's why we see the adoption growing.
Ruben Verborgh: From a positive viewpoint, I'd say a growing interest in client development.
On a more pessimistic note, I'd say that the uncontrolled growth of APIs and lack of interface reuse makes them a necessity. We now have tons of APIs doing highly similar things, but their interfaces are wildly different, necessitating fresh development every time. For example, Facebook, Twitter, and Instagram all allow posting a status with a picture, yet their interfaces are completely different to the point that their clients are incompatible. In that sense, (developer-targeted) API descriptions only offer relief for symptoms. They do not cure a root problem of APIs, which is that every API designer reinvents the wheel over and over again, making client development without descriptions unnecessarily cumbersome in the first place.
Mike Amundsen: Yes, the list is quite long. And that list has been around for a long time, too. Back in 2013 I did a talk for RESTFest Greenville on the topic ("Describing the Possible") and was able to identify a dozen definition/description formats dating back for more than fifteen years. WSDL was released in 2001 and we've had many Web-oriented interface definition languages (IDLs) since then. I think we are seeing a burst of activity today because there is a perceived need — that's good news.
All these IDLs are children of their times, so to speak. The ones from the 2000s are primarily XML-based and the most recent crop of definition formats are JSON- or YAML-based. And, despite the change in the serialization formation, I think most of the recent work on Web IDLs is following in the footsteps of WSDL and WADL. They are defining a tightly-coupled, strongly-typed interface for making remote procedure calls over HTTP. I think there is another approach from the past typified by XMDP (2003), AtomSvc (2007), DCAP (2009) and others. This approach is most closer to describing the interface in a general way — something that I think is much more in keeping with the way the Web works.
So, while I see excitement in the API metadata space, I also see a duplication of past attempts — attempts that didn't fair so well in the long term. I hope our future direction will take us beyond the classic "early-binding" model we saw in SOAP-style WSDL and WADL and lead to more "late-binding" patterns from AtomSvc, JSONHome, and others. That late-binding approach can, I think, point to powerful future for interoperable, reusable services on the Web.
InfoQ: Do you see API Documentation and Description formats as a single thing? Or multiple things?
Lorinda Brandon: Oh I love this question. I think originally we all envisioned a description file as being the only thing you need. And in some cases, that may be true … but there's a lot more to implementing an API than just that. At Capital One, we're investing deeply in what we call "rich documentation", which includes all the information you need to use the API. Much of that information is not included in the description file – things like Hello World instructions, Terms of Use, branding guidelines where they exist, authentication model explanations, reference apps and tutorials, etc.
Zdenek Nemec: There are definitely two different things. But truth be told, the initial incentive for the use API description formats was definitely the vision of API documentation without much work. However, the tide is turning as more and more people are discovering the benefits of the upfront design, API contracts and quick prototyping.
Ruben Verborgh: They are different things. Documentation should barely be necessary, since APIs are just the machine equivalent of websites—and you don't need a manual for a website either. I see developer-targeted description formats as a transitionary measure, until APIs start learning from each other's design, just as websites do. In my opinion, machine-targeted descriptions are the way forward.
Mike Amundsen: I see "Documentation" as designed for human consumption and "Description" and "Definition" formats being designed for machine use. I don't think it is a good idea to conflate the two. And there is quite a bit of work needed to make documentation usable, reliable, and powerful. Documentation that focuses on defining a set HTTP URLs, methods, and return codes is a poor substitute for human-centric text that includes elements like "Getting Started", "Basic Concepts", and "Examples" that are often left out of human-readable documentation. Other topics like "Compliance", "Extensibility", and "Backward-Compatibility" are very important — especially to client-side developers who will be consuming the Web API and I don't often see them in API documentation.
So I think we need better human-readable documentation and I don't think auto-generating it is the way to go. One group that seems to be doing some great advocacy for documentation is the "Write The Docs Community". I like what they are talking about and hope to see more chapters of this community appear around the world.
I think description formats like the ones I mentioned earlier need to be optimized for machine use. One way that these formats can be used is to support runtime discovery on the Web. Developers should be able to create components that can "find and bind" to existing services on the Web at runtime, not just as design- or build-time. And that requires well-designed machine-to-machine formats.
InfoQ: Do we need indexing services for these formats? If so, what should developers hope for from them?
Lorinda Brandon: Ideally, it should be easy to discover an API using a standard search mechanism like Google. We already have easy ways to find content on the web and ways to mark content as "not indexed". It shouldn't be so difficult to find this information – right now, you have to go to one of the API directories with your fingers crossed that the provider registered their API there but none of the directories are vendor-agnostic so you end up with a bit of bias about what's listed there. Or you have to stumble over someone's API portal because you put just the right thing in a search box. A couple of years ago, Kin Lane and 3scale tried to solve this problem with APIs.json and I think people really embraced the concept.
The challenge is getting everyone to adopt the format. It would be ideal to have the community move forward a bit more on this conceptually so APIs can be more easily discovered by the consuming developer community.
Zdenek Nemec: We need a way for public resources to be discoverable and accessible. This is no different to Berners-Lee original motivation for which he has built the web. This is our ultimate DNA. The remix culture that makes us to take something somebody has build before, mutate it or mix it with something else and create something better.
Developers excel at this. Looking at existing code learning from it and then infuse it with new ideas and possibilities. When it comes to this API descriptions are no different to code. And if this is done right, making all the known and unknown resources available and interconnected could spark another "internet revolution" that has happened with the world wide web.
Ruben Verborgh: Indexes for machine-targeted API descriptions are useful, more or less like a Google for automated clients. We should expect clients to find services with a certain functionality there. For developers, Google seems largely sufficient.
Mike Amundsen: Yes! I think there is a huge opportunity to improve the reusability and reach of existing Web APIs when we start designing and building them to discover and interact with each other at runtime without extensive human intervention. This is, I think, where the real value of machine-based description formats can be unleashed.
I am especially encouraged by the work the APIs.json team has been doing here. They are collecting and exposing the properties of a Web API that will be important when attempting to create connections between components. The team has created a search engine called "APIs.io" that works well for human discovery and I see lots of good things coming from their work.
Mark Foster, Leonard Richardson, and I have been working on a similar project — the Application-Level Profile Semantics or ALPS specification — as a way to create high-level descriptions of Web APIs that are not tied to any single protocol or media type. I think the combination of APIs.json as a way to communicate the metadata for the service and ALPS as a way to communicate the service capabilities can be a powerful combination for supporting indexable runtime discovery on the Web.
InfoQ: Where should a curious developer begin their API description and documentation research?
Lorinda Brandon: People always ask me this and I find it hard to answer – not because there aren't places to start but because there are *so many* places to start. We're a chatty bunch in the API world. Here are some resources to begin with:
Zdenek Nemec: GitHub already recognizes API Blueprint as a language so you can search for open sourced API description written in this format directly. And even the other formats shouldn't be that hard to find as they usually use a certain file name. Of course quality of API description (and APIs!) differs. This is where the indexing and some sort of pagerank for APIs should come into the play.
Ruben Verborgh: It depends on what their goals are. In the short term, OpenAPI and others will be handy to speed up development. But I see these as solutions to a temporary problem. In the mid to long term, I hope to see more client-focused formats that will deprecate developer-focused boilerplate code generation and SDKs. Developers interested in a more intelligent generation of clients are warmly welcomed to the Hydra Community Group.
Mike Amundsen: I think a great starting point for those interested in getting a broad view of the API metadata space is the InfoQ series "Description, Discovery, and Profiles - The Next Level in Web APIs". There is quite a bit of good stuff in there from many points of view. Digging into the history of interface description languages (IDLs) in general and on the Web in particular can also be helpful to see where the industry has been in the last fifteen years and where new opportunities exist. Of course, learning the details of the currently-popular definition formats (Swagger, RAML, API Blueprint) is important as is exploring the newer description formats like APIs.json and ALPS.
There is quite a bit of working going on in the field of API metadata and I am optimistic that it will lead to improved runtime discovery and the ability to "find and bind" to services on the Web.
Conclusion
Documentation and description of Web APIs will form the foundation of at least developer understanding in the coming years. The dreamed-of future where machines browse the Web (and it's APIs) for us discovering content, executing tasks, and building out new systems of understanding for ourselves, remains as wonderful an ideal as it ever did in the late '90s. It remains to be seen if the architects and builders will collaborate closely enough to build it.
In the meantime, it seems prudent to expect some ebb and flow within this space. The shapes may change, but the objectives will remain the same: providing the shortest distance to success for all involved.
Discovery, which surrounds both documentation and definition formats, is a mostly untapped space which should see some continued growth in the coming year. Documentation formats will remain largely human-focused and human-generated. Definition formats may become more "browsable" by machines and there may yet be hope for the machine-to-machine, discoverable, executable API.
Until then, these panelists have given us a clear sampling of the thoughts behind these hopes, dreams, and data formats.
About the Panelists
Lorinda Brandon is Senior Product Manager at Capital One DevExchange. She has more than thirty years of experience in software development and has worked for companies such as SmartBear, RR Donnelley, EMC, Kayak Software and Intuit, among others. She has a particular passion for APIs and software testing, and has published extensively on InfoQ, Programmable Web, and Network World on the importance of both APIs and enabling the delivery of high-value software.
Zdenek Nemec currently works at Apiary as the Director for domain specific languages, product manager and research lead for API design. He is a computer systems researcher, creator and architect. Zdenek is the author of the innovative API description language API Blueprint and also the author of the data modeling language MSON. His background is in the distributed systems, client development and user experience design. Zdenek loves to interact with users and help them create things one couldn't even imagine before.
Ruben Verborgh is a researcher in semantic hypermedia at Ghent University – iMinds, Belgium and a postdoctoral fellow of the Research Foundation Flanders. He explores the connection between Semantic Web technologies and the Web's architectural properties, with the ultimate goal of building more intelligent clients. Along the way, he became fascinated by Linked Data, REST/hypermedia, Web APIs, and related technologies. He's a co-author of two books on Linked Data, and has contributed to more than 150 publications for international conferences and journals on Web-related topics.
Mike Amundsen is Director of API Architecture, API Academy, CA Technologies. In his role of Director of Architecture for the API Academy, Amundsen heads up the API Architecture and Design Practice in North America. He is responsible for working with companies to provide insight on how best to capitalize on the myriad opportunities APIs present to both consumers and the enterprise. Amundsen has authored numerous books and papers on programming over the last 15 years. His last book was a collaboration with Leonard Richardson titled "RESTful Web APIs" published in 2013. He is currently working to complete a new book - "RESTful Web clients" due out from O'Reilly in 2016.
Source:
Virtual Panel: Document and Description Formats for Web APIs
