Wraz z rozwojem technologii i oprogramowania znacznie wzrosła złożoność środowisk, w których tworzy się aplikacje. Kiedyś, gdy aplikacje były proste, a ich wymagania techniczne ograniczone, programiści nie mieli większych problemów z konfiguracją środowiska pracy. Dziś, w świecie zaawansowanego oprogramowania, zgodność wersji bibliotek i narzędzi staje się powszechnym problemem. Weszło to do codzienności — być może już nawet nie irytuje, ale pochłania mnóstwo czasu i energii, które można by przeznaczyć na tworzenie czegoś użytecznego i rozwojowego.
Docker—A Remedy for the Complexity of the Programming Environment
As technology and software have developed, the complexity of the environments in which applications are created has increased substantially. In the past, when applications were simple and their technical requirements limited, programmers had no major problems configuring their working environments. Today, however, in the world of advanced software, the compatibility of libraries and software versions is becoming a common problem. It has become part of everyday life: it may no longer even be irritating, but it consumes a great deal of time and energy that could be spent creating something useful and developmental.
What is the environment-configuration problem?
A programming environment is a set of tools, libraries, dependencies and operating systems in which an application runs. Consider the Python language. It is a popular programming language that requires many libraries, such as NumPy, Pandas and TensorFlow.
Python is a programming language, while libraries are specialist programs. Some, such as the aforementioned Pandas, are used to group and organize data; others, such as TensorFlow and PyTorch, are used exclusively to create neural networks. A neural network can of course be built, and data organized, with Python alone. Specialist libraries, however, make this considerably easier and faster.
When I began my adventure with Python, I discovered libraries and eagerly began using them. Unfortunately, using different libraries simultaneously ended in conflicts and application stoppages. I was distraught: something that should have worked without difficulty caused paralysis and annihilated ready-made solutions. This is how I learned about the need to configure an environment—an exceptionally time-consuming and sometimes very irritating activity.
Each library has its own versions, created at different times. Over time, new versions may appear that are no longer compatible with earlier ones. For example, an application written in Python 3.6 may work correctly with NumPy 1.18 but cease to work with the newer NumPy 1.21. Problems of this kind, connected with mismatched versions, are a frequent headache for programmers. Python 3.6 works with Pandas version 0.25.3, but the new version 1.0.0 requires a newer Python version, such as 3.7. In such a case, programmers must either update their entire system, which may cause other conflicts, or find a way to run these libraries compatibly. The Anaconda platform is helpful in such situations, as it makes different versions of Python and its libraries easy to manage.
Anaconda as a solution
Anaconda is an environment that solves the problem of library-version compatibility. It makes it possible to create isolated environments in which each application has access to its specific library versions, eliminating conflicts among different versions. Although Anaconda handles the library problem in the Python ecosystem very well, the same problem also occurs in other programming languages and platforms.
Anaconda is simply a package of the most frequently used libraries, consistent with the program installed on the computer. One package of libraries, however, is rarely enough to complete a project. Something continually has to be added, and theoretically adding another library could disrupt the consistency of software inside the Anaconda ecosystem. The solution’s creators anticipated this scenario and created a solution called pip.
What is pip?
The abbreviation pip means “pip installs packages.” It is a recursive acronym, meaning that its expansion contains the name itself. Pip is a tool for installing and managing Python packages, downloading them mainly from the Python Package Index (PyPI).
Pip automatically handles the installation, updating and management of dependencies needed for a Python application or project to operate. It is the standard package manager, allowing the installation, updating and removal of libraries and dependencies required for work with Python. Pip downloads packages from the official Python Package Index (PyPI) repository, which contains an enormous number of libraries for tasks ranging from scientific calculations to web-application development. It enables packages to be installed, already installed libraries to be updated, and libraries to be removed. It is an exceptionally useful tool because the programmer does not have to take care of version and dependency compatibility.
The idea of enclosing the entire ecosystem in a “box”
Software consists not only of source code, but also of a set of libraries, tools and dependencies that must be configured appropriately so that they work correctly together. To solve this problem, someone came up with a brilliant idea: enclose an application’s entire ecosystem in an isolated environment. This approach makes it possible to package all dependencies and libraries into a single container that will operate identically everywhere, regardless of the environment in which it is run. Anaconda offers a similar solution, but Docker makes this way of solving problems possible on a broader scale.
Isolated environments are tools that make it possible to run applications separately in order to avoid dependency-related problems. The most popular tools for creating such environments include Virtualenv, the already mentioned Conda (Anaconda), and Docker.
What is Docker?
Docker is an open-source platform that enables applications to be created, deployed and run in isolated containers. The idea for Docker arose in 2013, when Solomon Hykes, founder of dotCloud, began working on a tool to simplify deploying applications in different environments. Docker’s history began with the need for a tool that would allow applications to run in a consistent environment on different machines—local, test or production—without each machine having to be configured manually.
One anecdote connected with Docker’s creation is that Hykes initially did not intend to make his project public. After several conversations and experiments at conferences, however, he decided in 2013 to release the first version of Docker on GitHub. Since then, Docker has become one of the most important tools in the IT world.
What does Docker consist of?
Docker consists of several key elements.
- Containers—lightweight, portable environments in which an application runs. Each container includes all dependencies required for the application to operate.
- Docker Engine—the Docker engine that manages containers and is responsible for running and isolating them.
- Docker images—saved application environments that can be run as containers. A Docker image is a “snapshot” of an application and its dependencies.
- Dockerfile—a configuration file containing instructions for building a Docker image.
Creating Docker in practice
Creating Docker is simple. To create a container for an application, a programmer writes a configuration file called a Dockerfile. Here is an example of a simple Dockerfile in several lines of text. A Dockerfile is a text file like a TXT file; it may consist, for example, of seven lines:
FROM python:3.8
WORKDIR /app
COPY . /app
RUN pip install -r requirements.txt
CMD ["python", "app.py"]This configuration file defines the base Python 3.8 image, sets the working directory, copies files into the container, installs dependencies from the requirements.txt file and launches the app.py application. Docker’s most important role is implementing prototype software in the cloud so that it can be used in production.
Cloud computing and its role
Clouds are external IT infrastructures that enable data to be stored and applications to run without the need to own servers. Clouds such as AWS, Google Cloud and Azure are used to run applications in a scalable manner accessible from different places around the world. Cloud applications allow resources to be allocated dynamically, increasing their flexibility.
Docker in the cloud
Programs written in Python—and not only Python—are often placed in clouds, but configuration problems may arise. A cloud is the same type of IT ecosystem as the one in our laptops and telephones. It is simply a server operating somewhere in the world and connected to us by a fast link. As on computers, different cloud servers may also have incompatible library versions, operating systems or dependencies. In such cases, Docker is an ideal solution because it makes it possible to package an application’s entire environment into a single image that can run on any server, regardless of its configuration.
Suppose someone has written a Python application that works locally but encounters problems when launched on a cloud server because of differences in library versions. Docker makes it possible to transfer that application together with its entire environment, causing it to work identically on every server and eliminating configuration-related problems.
Packaging in a container
Data engineers sometimes focus on creating models and analyses without fully understanding how the entire system on which their applications run operates. They work with advanced algorithms, use complicated machine-learning techniques and well-considered solutions, but encounter problems when it is time to deploy their work in a real environment. In such situations, instead of carefully analyzing the infrastructure or environment in which the application is to operate, they often resort to a simpler approach: let us package all of it in Docker and move it to the cloud.
Imagine a data engineer who has created an advanced machine-learning model in Python. The model depends on many libraries, such as Pandas, scikit-learn and TensorFlow, each of which has specific version and system-dependency requirements. Instead of analyzing in detail which library versions will work best in the production environment, the engineer simply packages everything into a single Docker container, including the whole of Python and all dependencies. The container is then moved to the cloud and launched there.
This approach often results from an incomplete understanding of the infrastructure or production environment. Docker has become a kind of magic box that “solves every problem.” Docker is, of course, a powerful tool, but this approach may waste resources by running unnecessarily large images with many dependencies that could be optimized or reduced.
Docker was not created to be a solution for unclear, poorly understood implementations. Its main purpose was to simplify application deployment by isolating environments, not to serve as a master key for every problem. Data engineers nevertheless often treat Docker precisely in this way: as a tool that helps them bypass configuration and compatibility problems. This is an additional function that Docker’s creators did not foresee—escaping a detailed understanding of the system in favor of “packaging everything” in a container and throwing it into the cloud.
Summary
Docker is a powerful tool that solves the problem of software compatibility across different environments. Instead of worrying about library versions and dependencies, programmers can enclose an application’s entire ecosystem in a container that will operate identically on every server. Docker not only makes programmers’ work easier, but also increases the efficiency of deploying applications in cloud computing because they do not have to understand everything. Their role is often reduced to that of an elevator operator sending everything “up in the elevator” to the cloud.
Wojciech Moszczyński
Wojciech Moszczyński — graduate of the Department of Econometrics and Statistics of Nicolaus Copernicus University in Toruń; specialist in econometrics, finance, data science and management accounting. He specializes in optimizing production and logistics processes. He conducts research into the development and application of artificial intelligence. For years, he has been engaged in popularizing machine learning and data science in business environments.