We're all connected these days, and our giddy, sped-up world calls on computer programs to make these connections. More and more information sites boast, "We have an API" to permit consumption of data by remote computer programs. The sites’ owners understand that it's not enough to publish a document that humans can peruse. Sites must join their data with the vast computer processing infrastructure that spans Planet Earth—and beyond.
Nowadays, when somebody says, "We have an API," they're talking about REST. Microservices—one of the most popular architectures for new applications—also communicate using REST. Many sites are moving to GraphQL, which uses many of the same concepts and tools as REST. So technological innovation continues on the web, but the invention of REST in particular was a historic turning point.
The term REST was invented by Roy T. Fielding in a famous 2000 PhD dissertation, and stands for Representational State Transfer. In this article, I'll explain why previous attempts to create a universal data processing infrastructure failed, and what made today's REST-connected world possible.
Calls and Clusters
In the 1980s, as applications such as weather forecasting outgrew the capabilities of a single computer, organizations developed new layers of programming to enable distributed computing. To appreciate REST, it's worth examining some of these earlier efforts.
One of the first solutions to distributed computing consisted of designing expensive machines that held hundreds of cooperating processors, such as the Butterfly from Bolt, Beranek and Newman (BBN) and the Connection Machine from Thinking Computers.
But most scientists found it simpler to tie computers together with regular fabrics and networks. Some of the programming tools for these clusters were libraries that you would call, while others were preprocessors: you would stick special directives in your code that the preprocessor would turn into calls. Both types of programming would start processes on remote computers, pass them data, and wait for results to return. One particularly popular tool of this sort was the Message Passing Interface (MPI), about which I edited a book in the early 2000s. But, this programming paradigm required too much detail and planning, so it stayed mostly in research institutions.
Out in the business world, several organizations implemented an idea called remote procedure calls (RPC). Supposedly, it was a gentle step up from traditional programming. Unlike with the MPI model, you could just issue a function call and have it magically execute on another system.
But RPC was cumbersome to implement. You had to bind to the remote machine, a complex task in addressing. Besides installing special software to communicate between machines, you needed to create a small piece of runtime called a "stub" on the calling machine. If you made a call such as store_info(name, state, country), the stub would put the three arguments into a binary data stream that could cross the network, a process called marshaling (or marshalling). Corresponding software on the other side would receive the message, unmarshal the data, and pass it to the program to process.
In the 1980s and 1990s, object-oriented programming also gathered steam. RPC was extended into new object-oriented standards such as Simple Object Access Protocol (SOAP) and Common Object Request Broker Architecture (CORBA). They were all heavyweight and hard to get up and running.
The technologies I mentioned are putatively still in use, but they never became widespread. As we look over the flotsam of 1990s-era distributed computing, we can identify requirements that must be in place:
- Programs need to address remote computers and the programs running on them.
- The local and remote computers need a communications protocol to run over TCP/IP.
- Each side must marshal and unmarshal the call and its arguments.
Next we'll turn to the infrastructure that made REST a solution for all these needs.
A Protocol that Shows Promise
When you look over classic internet applications, such as File Transfer Protocol (FTP), Telnet, and Simple Mail Transfer Protocol (SMTP), nothing seems special about HTTP. Its commands (verbs), responses, and error messages fit the model created by all its internet predecessors. Nor was Tim Berners-Lee trying to design a distributed programming protocol. His proposal for the World Wide Web was obsessed with exposing relationships between things, a pursuit he later developed into Linked Data and that drives other ontological web efforts such as microdata and Schema.org. But along the way, Berners-Lee slipped in a few features that made HTTP suitable for distributed programming.
The database world is familiar with the appealing term CRUD, which represents the fundamental operations required for data storage and retrieval: CREATE, READ, UPDATE, and DELETE. Three of those terms are SQL commands, and READ is represented in SQL by the SELECT command. Virtually all databases, whether SQL or not, implement CRUD semantics.
It can't have been a coincidence that Berners-Lee also defined four main methods, and that these map neatly onto CRUD semantics:
GET—This is what we do all the time when we click on a link and retrieve a web page into our browser. GET corresponds to READ in the CRUD model.
POST—This adds something new to the server. When you fill out a form and click the Submit button, this method sends the information to the server. It was apparently designed to let people append something to a web page, such as a comment. POST corresponds to CREATE in the CRUD model.
PUT—This can either add a web page or replace an existing page. Because POST adds a new page, PUT is meant for replacing a page. The method had no clear purpose in the original Web. But it's very nice for updating a row of data. PUT corresponds to UPDATE in the CRUD model.
DELETE—This removes a resource. Like PUT, it had no use for web pages. But it rounds out the CRUD semantics.
Most of these methods hung back in the shadows in the first few years of the World Wide Web, while it became the most popular tool ever used on the Internet. But the web was solving problems 1 and 2 of distributed computing, listed earlier. The URL or URI contained all the information necessary to address remote computers and the programs running on them. HTTP provided a flexible protocol with the four basic CRUD commands required to exchange data.
Dynamic Web Pages
While the web was developing a culture of sharing cat pictures, researchers were looking for ways to make the web more useful. The result, called the Common Gateway Interface (CGI), changed the world forever. An early book on the topic by Shishir Gundavaram, edited by me, became an O'Reilly best-seller.
CGI let a browser or other web client pass data to the server. This technology enabled forms, and forms enabled e-commerce. CGI has been superseded by more efficient and powerful tools, but we can thank it for allowing us to order everything from our lunch to our next car during the COVID-19 pandemic.
CGI also solved problem 3 mentioned above in a simple way. Instead of creating a specialized binary format, destinations and their arguments were represented as a single stream of text in a URL or URI.
Let's take book ordering as an example. Start with a server name and a service:
https://www.example.com/order_book
The server www.example.com receives this as a GET request, but there is probably no web page at that URL. Instead, the server invokes a program with the name order_book (or another name chosen by the programmer), which will probably send you a generic form to fill out.
Suppose you fill out the "author" field of the form and enter Gundavaram. When you submit the form, your browser goes back to www.example.com with this URL:
https://www.example.com/order_book?author=Gundavaram
The question mark starts a string of arguments that are key/value pairs. All modern programs handle key/value pairs and generally store them in an associative array (also known as a dictionary or hash). If you want to request a topic as well as an author, the URL might look like:
https://www.example.com/order_book?author=Gundavaram&topic=CGI
What does a URL in CGI represent? It’s like RPC. This example is like calling a function named order_book and passing two arguments: author and topic. Tim Berners-Lee, interestingly, explored argument passing in URIs also. His suggestion, called Matrix URIs, was never implemented so far as I know.
If data in a CGI call gets more complicated (as it generally does when uploading information), you can use the POST request with JSON-formatted data. All this could be done manually; you don't need a form. Thus, a programmer could write simple code to generate and send the URL.
Like all the technologies in this article, CGI was a stepping stone. Modern web pages use more efficient methods for exchanging data, such as HttpRequest—also called Asynchronous JavaScript and XML (AJAX) in the early days—that allow the visitor to stay on the same page. The conventions for CGI remain central to sending and receiving information over the web, though.
Where Does REST Come In?
The web and CGI solved all three problems posed by earlier systems for distributed computing. The communications we call RESTful today rest on the ideas and syntax described earlier, not on Fielding's dissertation. RESTful URLs add another key idea that also appears nowhere in the dissertation: hierarchies.
Imagine a site that pulls together scientific data. It might create categories such as "earth" and "chemicals," and then create categories such as "ocean" under "earth," leading to URLs like:
https://www.example.com/data/earth/ocean
This brings a bit more order to data. With its combination of simple syntactic elements, REST is turning the world's computers into a unified processor. We are just starting to see the benefits and dangers of this processing.
Put these fairly simple ideas together, and you have the basic conventions of what today we call REST. Certainly there are complications, such as representing spaces and non-ASCII characters, but the fundamentals are astonishingly simple compared to the concoctions of pre-web systems. This is not to say that Fielding is irrelevant. One finds several bits of forward thinking with far-ranging consequences in his dissertation.
One is a requirement that each client request be stateless—that is, no request should depend on information sent in a previous request or reply. The web had moved away from stateless requests by developing the concept of a session based on cookies. (Like all the historic technologies covered in this article, the technology of tracking has evolved since then.) REST restored stateless requests, which facilitated independent, parallel processing, and thus the current architecture of virtual machines that can be freely created and retired.
Another forward-thinking concept was layered requests, which facilitate small modules working together. A client can send a shopping order to one server, which sends a list of items to another server to calculate the total cost, and that server can consult yet another server with a product database. This is how microserves work today.
One Unsolved Requirement
I have to admit that earlier distributed systems labored over one task that REST doesn't address: discovery. This is a problem many people face daily, whether a reader trying to locate an online bookstore or a scientist seeking a site that publishes ocean temperatures. And you tend to solve this problem manually: use a search engine, look in a journal, ask a colleague, or whatever. Systems such as CORBA were weighed down with search and cataloguing systems in a mostly vain attempt to implement discovery.
The reason discovery hasn't been solved is that it's much, much bigger in scope than the three problems I listed earlier. It requires ontologies and programs to crawl the internet. Linked Data could also play a role.
But it's not really a technical problem. Who will decide what's relevant to a request for ocean temperatures? Who will vet the quality of the sites? The goal is truly "boiling the ocean" (which we're coming closer and closer to doing, unfortunately). So, although very constrained versions of discoverability have been implemented for Java programs, local area networks, and other environments, a general solution to discoverability remains far off.
Meanwhile, the petabytes, exabytes, and zettabytes of data coming online are shaping a new world. Computers will do much more with this data than any team of researchers can do manually. REST unlocked the promise of distributed computing. In his review of this article, Simon St.Laurent observed that many sites are adopting GraphQL instead of REST, just as HttpRequest took over most tasks for which CGI was used. But the conventions and protocols of REST are seen throughout GraphQL. So the REST paradigm will live on for the foreseeable future.
Comments