Episode 100

And here we are, dear reader, six years and one hundred episodes later. The idea and form of this blog did evolve a bit over the years, from being a diary of a project through a collection of book reviews, conference reports, introductions to various IT technologies and concepts, article series on Spring, AWS, and web API design. There was digging up in history, a little bit on hardware, a little bit on people, and a little bit on dragons. There is not so much Java here anymore, so in the retrospective, one might think that the blog name is not exactly adequate. We all have our roots and backgrounds though and I’m kind of a sentimental person.

Image by Sergey Vasnev

We are also in the middle of another journey here. In the first part of this article, we traced the early history of significant milestones in the evolution of software development, starting with early achievement in computer science and first hardware through Assembler, recognition of Software Engineering, Compilers, Unified Hardware Architectures, Object-Oriented Languages, Design Patterns, Personal Computers, IDEs and finally The Internet and WEB APIs. We assessed how all that created and shaped the IT industry through the decades. In this part, we will catch up with the timeline somewhere between the eighties and the nineties and traverse towards a new millennium. Software development as an engineering discipline has changed a lot over this time as well. Several key phenomena fuelled those changes, and we are going to explore them right now.

Gamers and Hackers

Perhaps one of the most overlooked aspects of stimulating the growth of software development is computer games. Many people are not aware, that the computer games industry nowadays is worth over twice the film and music industry combined (154B vs 41B + 19B in 2018) and is growing at a 10% yearly rate. This has several consequences. The chase of real-time, life-like computer graphics pushes the evolution of hardware. Modern AAA games are not only increasingly complex, requiring growing effort to develop, but also need entire online infrastructure with increasing scale behind the scenes craving for being put in the Cloud. And perhaps the most important point in our story – back in the eighties and nineties, when PCs were still new, video games were often a trigger for young people to get acquainted with computers. And in those times a path from playing games to trying to program games or do other tricks with computers was often quite short. I remember myself trying to write a simple number guessing game on my Commodore 64. Then followed a realization that If I spent so much time playing games, I might as well go to study computer science after high school.

Image by Doug Skelton

Playing computer games was one thing, but some people liked to play for real-world treasure. The treasure doesn’t have to be gold or fame, it might just be the satisfaction of solving a puzzle and finding a way to get to the castle. The rise of the Internet was followed by the rise of such digital castles – myriads of systems facing public network, with walls prone to infiltration. The term hacker is older than the public Internet and was used since the sixties to describe proficient computer programmers who enjoyed an intellectual challenge of creatively overcoming weakness in software systems to achieve interesting results. Computer security started to play an important role in the industry, especially since the nineties. The cost of cybercrime currently sets the global economy back a whopping 1.5 trillion USD each year. Companies have to fight back adequately with layers of defensive software, training, audits, hardening web applications, and often moving infrastructures from basement data centers to the Cloud. The computer virus idea dates back to 1949 lecture by John Von Neuman on self-replicating automata and the first practical examples were seen in the early seventies when the Creeper virus was detected in the ARPANET, infesting PDP-10 workstations running TENEX operating system.

Version Control and Open Source

As software systems started to grow along with the size of development teams and organizations, it became obvious that we need a tool to organize changes to the code base, as well as synchronize them between programmers. It became especially important with the rise of PC where work was done on individual machines, not on a single centralized mainframe. The oldest version control system dates back to the early seventies when SCCS was developed on OS/360 by Marc Rochkind. Version Control treats source code as a graph of changes where we can branch and merge new features as well as traverse between versions if needed.

The rise of the Internet and the modern Version Control System allowed software developers to share code much more effectively. One of the milestones in the Open Source world was the GNU project dating back to 1983 aiming at providing users freedom and control over their computers and software. It spawned an astonishing number of projects, many of them aimed at software developers themselves. Currently, most of the world’s computing runs on layers of open source from operating systems, through containers to software frameworks. Software companies often shift from charging for software to charging for services, support, and hosting. With free access and vibrant communities, the development of using free tools got easier.

Managed Runtime Environment and Virtual Machines

As software stacks grew into complexity new layers of abstraction were developed to handle the chaos. Two of those layers are two very distinct types of virtual machines. Both are old concepts but rose to popularity later in different periods. Let’s start with process virtual machines also known as application virtual machines or managed runtime environments. They are processes that execute code written in high-level programming language code indirectly via an interpreter or just-in-time compilation. Often there is an intermediate language in play between actual machine language that gets executed in the end and the original source code. Two canonical examples here are Java with its Java Virtual Machine and C# with its Common Language Runtime. The existence of process virtual machine with its ecosystems and intermediate languages followed the write once, run everywhere approach. It also allowed the bloom of various languages that used a common platform with all its benefits and resources. Although Java came out in 1995 and C# followed in 2002, the concept itself dates back to 1967 and O-code machine that was executing O-code emitted by BCPL compiler.

Image by KoTnoneKot

The second very distinct type of virtual machine is the system virtual machine, perhaps something that is commonly understood as just a “virtual machine” in a modern context. System virtual machine emulates an entire computer with its input, output, resources, and operating system. Virtual machines were an important step in the creation of applications that are not the only portable between environments themselves but have portable dependencies. If an application requires specific software installed to work, we can install everything once, build a virtual machine image and hand it over to anyone with all its required content immutably sealed inside. This approach simplified running growing infrastructure in data centers and ultimately became a must when the industry started migrating to the cloud. First system virtual machines date back to 1966 and IBM CP-40 operating system. A natural extension of that idea was operating-system-level virtualization with multiple isolated user spaces. An iconic example of leveraging this is Docker that runs containers providing similar benefits to virtual machines with significantly fewer resources required. The idea of modern containers dates back to 2005, while Docker was released in 2013. Let’s not jump too far ahead yet though.

Agile and DevOps

Having solved some of the technical problems with software development, the industry also had to solve organizational ones. Building a bridge is much different from building a software system yet early approach was somewhat similar. There was great planning, great building, and great testing. In the case of software, the following step was often a great disaster. Requirements for software systems tend to change rapidly and building a system is a constant learning path as each one is different. The classic waterfall model of software development proved to be inefficient thus the industry evolved an iterative approach where we produce a working piece of software at the end of each cycle, assess, observe adapt and repeat. Although attempts were made since the fifties, the idea gained widespread at the beginning of the 2000s with the rise of Scrum, Kanban, and the Agile Manifesto. Scrum provided a simple set of roles, artifacts, events, and rules to guide software development and became wildly popular.

The somewhat complementary idea that emerged later is DevOps. While Agile attempts to close the communication gap between development and customers, the DevOps attempt to close the similar gap between development and operations. The idea here is to empower developers with tools and environments that let them take care of their services along the whole lifecycle, including monitoring and troubleshooting in production. The focus on infrastructure automation aims to enable faster and more reliable software delivery. The first DevOps conference – devopsdays, was held in 2009 in Belgium. The adoption started to accelerate in the mid-2010s. DevOps term is connected with famous Google’s Site Reliability Engineering (SRE) set of practices. Technically speaking, DevOps is an interface and SRE is an implementation of it.

Continuous Integration and Automatic Tests 

Version Control System helps with keeping codebase changes organized, but it doesn’t make sure the code makes any sense after a commit is added to the repository. In order to do that, we need to be able to automatically run a set of actions on every commit. Those might be, for instance, a compiler pass that ensures every file that was meant to be included indeed was included. It can be an application build that assures developers that every dependency is in place. It can be a static code analysis tool that can find and flag potential problems in the code related to a higher level of abstraction that pure language correctness. And it can be an automated test suite invocation.

Image by 000Fesbra000

Automated Tests may work on many levels – smoke tests, unit tests, integration tests, stress tests, but the main idea is to make sure the change in software didn’t break anything in the closer or more distant vicinity. They also help to prove that code is working in an intended way in the first place and act as documentation. They require more effort initially, but pay off very soon and were widely adopted by the software development community. The idea emerged somewhere around the early nineties and became a must in modern times.

What’s next?

As with earlier part of the article, it’s not an easy task to arrange the topic we have covered so far in exact chronological order. Many ideas are very old but only managed to gain popularity much later. Many are co-dependent or synergized. This will be especially apparent in the next part of our journey. We are going to dive into Extreme Programming, Software Craftsmanship, Microservices, Continuous Deployment, Big Data, IoT, The Cloud, Serverless, Web 2.0, Artificial Intelligence, and the future. Stay tuned.