Episode 49

Imagine you are developing a web application based on a typical modern technology stack. In essence, the goal is to create something that receives requests from the network, processes them, and responds to them with some kind of structured data. It can be a fancy HTML webpage to be displayed in client’s web browser or it can be a raw text object to be consumed by another application web API. Have you ever wondered what exactly is going on behind the scenes? What is happening between the moment when the code you wrote is executed and the moment when electrical impulses jumps the network cable sticking out of that metal box in the data center? Let me take you on the journey along numerous layers of abstraction in modern software stack that must be bypassed to make things happen.

There and Back Again: Abstractions

Programming is all about abstractions. We endlessly put one layer above the other in order to deal with tremendous complexity of software and hardware. Often, we ignore most of the layers to focus on solving the actual problem at hand. In principle, sending a JSON object over the network might seem like an easy task with modern tools and frameworks. You generate the project, tweak just a little bit here and there, write one method with few annotations and there you go, it works. But the engineering problem of making this possible in an easy, fast, secure, reliable, scalable and manageable way is gargantuan. Looking at the big picture, it’s probably an effort of hundreds of thousands of software developers, architects, electronic and electrical engineers, computer scientist and mathematicians, spanned over several decades of work. All that, to let you do the job in a single pomodoro.

Of course, to develop decent software, you don’t necessary have to understand exactly how all this works, it’s probably not even possible for a single human being to grasp all that in every detail. I believe however, that it’s good to have some level of understanding of each abstraction. Even if not out of necessity, at least out of curiosity.

The long-bearded wizard arrives carrying a mysterious message. You decide to accept the request and deliver the response to a place behind the mountains, rivers and forests. So, the journey begins.

An Unexpected Party: The Code

The journey begins in the code, the everyday clay we shape our digital reality in. Let’s focus on Bag End… hmm… I mean back end, and assume that we want to deliver a Hobbit to our client. Of course, Hobbits are complex creatures, not easy to instantiate, but let’s say we did a bit of research, ask here and there, and now we have reference to a healthy Hobbit at our disposal. We can now prepare him a bit for the journey, for example by adding an appropriate list of adventuring gear to his inventory to make life easier.

Ideally, as web application programmers, we would like to focus on business logic and great features in our code. We don’t want to care about all the complexity of networks, threading, consistency, reliability, distributed computing, efficiency of searching terabytes of data on hard drives etc. It would be best to leave all that fuss to others who already solved that problems. Just find the right tools, rule them, bring them and in the darkness bind them. You are the master and commander here, the smith of your own fate, but remember about your craft. Use the power of your language syntax, right programming idioms, design patterns, code cleanly and efficiently.

We can think of the source code as a plan before the actual journey. In the adventure, when things are happening, the source code is long gone, replaced by bytecode that is actually executing on a Java Virtual Machine. Basically, we are creating one syntax tree optimized for understanding and expressiveness that is being compiled into another syntax tree optimized for being mapped to real processor instructions and executed inside JVM. Your compiled code is bundled together with all libraries and frameworks and then… Then it’s another chapter of the story.

A Short Rest: The Web Framework

A wise adventurer should be in control of situation, but if he was to prepare everything by himself, he wouldn’t be even able to cross the threshold. You don’t sew your clothes, you don’t breed your horses, you don’t forge your weapons. Instead, you ask appropriate, specialized craftsmen to prepare them for you.

In software, it’s similar. In theory, you could write your own web framework, server and even operating system if you so desire. In practice it’s infeasible. Software components are so big and complex nowadays, that our job is to select and use proper existing ones. In our case, closest to our own code is the web framework, like Spring. Technically, the journey doesn’t even begin with our code, wherever we are talking about starting the web application or serving requests. Modern web frameworks promote inversion of control principle, where we don’t only call framework code, but the framework code calls ours. For example, when a request mapped to our controller is processed, the framework will call the method defined as this particular request assignee. Of course, we still call some other framework’s code, for example to create a HTTP response wrapper around actual data we would like to transmit. In our case, that would be our Hobbit object. We can set HTTP headers like encoding, status or cookies but after this, we hand the direct control over to the web framework, hoping it will do the job.

The living, well rounded Hobbit object existing in JVM memory will be serialized to a flat, suspended Hobbit in JSON, or another data transmit format we choose. Then, he will be in preparations for next chapters of the adventure, and finally sent over the network. In the end, whether we present a HTML page or serialized Hobbit in plain text, those are all encoded data in the final form of an array of bytes (remember, 1 Hobbyte is equal to 8 Hobbits). To manage this, the web framework gives us a layer of abstraction over Java Servlet interface. Servlet is implemented by our web framework, it provides a simple, low level contract to receive request object and produce response object. It’s a generic contract, and can be used for protocols different than HTTP. But who is taking care of our servlets then? If they are ready when required, fed the right data and managed efficiently? Well, that’s another chapter of our story.

Over Hill and Under Hill

If you are curious about our little Hobbit further whereabouts, come back next week. There will be riddles in the dark, flies and spiders and barrels out of bonds waiting on our path. Meanwhile, I wish you Merry Christmas!

image source