The existence of hundreds of different Linux distributions causes a challenge for software producers: How to provide a product for Linux as one single build only, but at the same time maximize the number of supported distributions? In this article, we first discuss a variety of approaches to tackle this portability issue and their applicability to one of our closed-source products. Then we explain our choice for a build and publication scheme that involves the Linux Standard Base (LSB), a collection of standards provided and maintained by the Linux Foundation. Finally, we share our experiences with this LSB approach in production, including insights on its practicability as well as on inherent problems.
As has been introduced in a previous blog article, mgm technology partners are developing the ELSTER Rich Client (ERiC), a library embedded in most German software products and online services that transmit tax-related information to fiscal authorities. ERiC offers functions to perform plausibility checks on a wide range of tax types (e.g. income or advance turnover tax returns) and to transmit the declared data to authority servers in a secure way. Every producer of software clients or online services that can file data to tax authorities is obliged to integrate and use our library.
While the government enforces the use of ERiC, it does not intend to constrain anyone in the choice of technologies utilized in the embedding software. This implies an important design goal for the development of the ERiC library and its API: It must cause as little constraints and obstacles as possible for tax software producers.
One aspect of this design goal is the interoperability with relevant programming languages: ERiC is developed in C++, but the resulting library is integrated into software packages written in all kinds of managed and non-managed programming languages, among them being C/C++, C#, Java and VB.NET. A second important aspect, on which we will focus in the following, is the support of operating systems.
Support of operating systems
We want to provide the ERiC library for all platforms relevant in the market. According to the installation distribution on German desktop computers (see e.g. netmarketshare.com for up-to-date statistics), the most commonly used operating systems are Microsoft Windows products, followed by Mac OS and a minor fraction of Linux installations. Consequently, the ERiC release from November 2015 supports current variants of Windows, Mac OS X and a well-defined spectrum of Linux distributions.
Both Microsoft and Apple publish information which makes it rather easy to track and predict which of their products are currently receiving support service – and which will do so in the near future. One aspect of such support schemes that is particularly essential for tax-related applications is the roll-out of security patches. Microsoft has an elaborate release cycle documentation for all of its products. It allows us to proclaim which Windows versions will be supported by an upcoming ERiC release months in advance. Naturally, such early announcements on including or removing the ERiC support of specific operating systems is appreciated by tax software producers, whose release planning is influenced by such decisions.
By means of date announcements, the OS X life cycle documentation is much less precise than that of Microsoft. Support for certain operating systems is coupled with the Mac hardware with or for which it was purchased. The question, if security patches are still provided for a certain product, cannot be answered directly from official statements. Nevertheless, a reasonably reliable periodicity of OS X releases allows for a support scheme that in practice is sufficient for the ERiC release management and its users.
Compared to the two commercial giants Microsoft and Apple, Linux is a much more complicated domain. Although Linux represents a minority among German Desktop installations, a significant number of tax data transmissions actually is performed via ERiC on variants of this open source operating system. The reason for that is the fact that some important solution and service providers for corporate customers run ERiC within web services on Linux servers. Due to the typically large numbers of customers they have, Linux cannot be neglected as a relevant platform.
How to cope with hundreds of Linux distributions
Figure 1 illustrates the evolution and diversification of Linux distributions over time. Since the first release of a Linux version under the GPL license in 1992, a vast number of variants and families has emerged. The portal DistroWatch.com currently hosts information about more than 250 variants, but there are clearly many more. The one thing all these distributions have in common is a kernel directly published by Linus Torvalds or in some way derived from the open source project he is still heading.
GNU/Linux Distribution Timeline, by A. Lundqvist, D. Rodic, published under the GNU Free Documentation Licence.
“Dependency hell”
Linux distributions for desktops and servers typically come with a large set of additional libraries, tools and applications. Most of these are open source community products as well, and one important representative in this context is the GNU Compiler Collection (GCC). This set of compilers can build and link binaries for various architectures from source code written in C, C++, Objective-C and many more. With this fundamental tool, every Linux user can not only read, but also change and recompile all of his open source Linux components, including the kernel itself.
Now why is the diversity of Linux distributions a problem for software producers?
Dynamically linking shared libraries into binaries is a strategy heavily used for most Linux applications. This is done mainly to reduce memory usage, obtain smaller application binaries and consequently achieve faster loading times. With their default configuration, compilers like those in GCC build binaries that are automatically linked against libraries present on the target system – if those libraries provide (or “export”) functions called within the compiled code.
Every Linux system offers a certain set of shared libraries, but from distribution to distribution this set can be composed of different libraries. And even if two Linux variants provide the same libraries, execute a binary on Linux distribution A that has been built for target distribution B, it is very likely that the program which tries to load the linked binary at run-time detects an unmet dependency. Trying to fix such unmet dependencies is often a frustrating process, since required libraries – if available at all – might cause new dependencies and so on. This often-experienced situation is also called “dependency hell”.
The role of the C standard library
If the portability of a Linux binary among multiple distributions is required, one way to at least minimize the probability of unmet dependencies is to avoid using shared libraries. Instead, compilers can be configured to statically link external function calls into the created binary wherever possible. If it is possible to create a monolithic binary that does not depend on a single shared library at run-time, this binary can be executed on every other Linux system which knows how to interpret the executable file format.
Unfortunately, creating a fully self-sustained binary without any external dependencies is nearly impossible. Every non-trivial program has to make use of so-called system calls at some point. These requests for kernel services (like opening a file) are usually called via functions offered by the C standard library (libc).
So why not statically link the libc into the target binary as well?
The system calls wrapped by a statically linked libc cannot be successfully executed on another Linux system, unless the libc on that system is exactly identical – the chances for which are small. Interesting articles which discuss this issue can be found e.g. in blogs by Sagar and Oracle.
Hence, going for fully statically linked monolithic binaries is a strategy not applicable for most software products – and clearly not for ERiC.
Strategies to increase and document portability
Obviously, we are not the first software producer hitting the mentioned issues. The ways this problem has been approached are surprisingly manifold, though. In the following we describe strategies that we found considerable.
Publishing source code
If software can be published under an open-source license, every potential customer is offered a possibility to obtain the source code and build the desired binaries directly on his or her target system. Hereby, the utilized compiler will link against libraries that are actually present on the target system. Unfortunately, for the ERiC library open source is not an option, since it contains security-relevant functions and cryptographic tokens.
Documenting the build and test systems
Among software providers who publish pre-compiled binaries for Linux, a common strategy is to describe the system on which the product was actually built. Such documentations usually include a subset of the following information pieces
the utilized compiler and its version (e.g. “GNU g++ 3.4″)
the version of the C standard library (e.g. “GLIBC 2.11.1″)
the Linux kernel version (e.g. “2.4 or 2.6 kernels”)
the Linux distribution name (e.g. “RedHat Linux 7.2″)
the hardware architecture (e.g. “x86_64″).
Most software producers repeatedly run some set of tests on the created binaries before they actually publish them. Furthermore, the tests are often performed on a certain set of different Linux distributions. Documenting details on these testing systems can provide useful information for the potential users.
This approach is in principle feasible for us. But since the ERiC QA team can maintain and operate only a certain number of different Linux systems for their tests, a published list of information in the described manner would always be rather limited. It would leave open questions and an uncertain risk to those users who need to run ERiC on a system not covered explicitly by the documentation.
Commercial certifications
Certain providers of Linux distributions like SUSE, Red Hat or Oracle offer certification schemes for applications (see e.g. the offers by PartnerNet). On the one hand, for the software vendor as well as for the Linux provider the certification is a way to advertise each other. Customers of the certified application, on the other hand, are assured that the program will function on that certain distribution.
For us, this approach has two major disadvantages: First, certification schemes are only available for a few distributions, but certainly not for all actually used with ERiC. Second, the periodic updates of tax laws in Germany result in the necessity of many minor ERiC releases per year, each of which is a new build and therefore would require a new certification. This process would create significant extra costs by means of time and money.
Creating self-sustained packages
Tools like CDE or Docker can create self-sustained packages which contain the application to be distributed plus everything this application needs to be successfully executed on any target system. Such containers can be considered micro-VMs tailored for one application only. On the target system they do not require specific system libraries, but only a single application that can execute the container. A related but less comprehensive approach is taken by rtldi, a so-called indirect runtime loader. The alternative ELF program interpreter enables programs to use their own runtime loader and shared libraries which can be shipped together with the program itself.
For many software providers such bring-what-you-need approaches perfectly eliminate the dependency hell. Unfortunately though, since the ERiC library is not an application by itself, but instead needs to be embedded into many different and to us unknown software projects, a self-sustained container approach just does not work.
Hiring experts
For producers of commercial software the portability among various Linux distributions can be of vital interest. Consequently, professionals offer their services to support their aim. For example, the IBM Chiphopper program helps companies to port their Linux applications to Red Hat or SUSE as well as to different hardware architectures.
For us, contracting a third party service provider to make the ERiC library portable among many Linux systems is no desirable option, and our customer permitted us to develop an own solution. Interestingly, the Chiphopper program utilizes the same LSB standard definitions like our final methodological choice, described further below.
Sticking to standards
When looking at the diversity of available Linux flavors and the problems in porting applications among them, one solution might seem as obvious as difficult to achieve: Standardization. The Unix and Linux world has seen multiple approaches to introduce standards (see e.g. the so-called „Unix Wars”). The ones relevant for Linux which have survived until today are the Single Unix Specification (SUS) and the Portable Operating System Interface (POSIX), where today SUS actually incorporates the latest POSIX standard. SUS and POSIX standardize system interfaces, system headers, a set of utility tools, a shell and more.
Officially, Linux aims towards compliance with SUS and POSIX. Although no official certification is available until today, any disagreement with these standards is considered a bug (see “Linux System Programming” by Robert Love, O’Reilly Media, Inc., 2007). For many practical tasks performed by Linux users every day, the boundary conditions defined by SUS and POSIX are still insufficient to easily port applications and workflows from one Linux flavor to another. Therefore, it was necessary to extend the set of SUS and POSIX standards, and this gap has been tried to be filled by the Linux Standard Base (LSB) project.
Linux Standard Base
LSB is the title of a workgroup within the Linux Foundation which publishes a set of standards that includes the already mentioned SUS and POSIX, but which, on top of these “core” definitions, covers more areas like “C++ language, libraries and interfaces”, “Desktop (X11, GL, PNG, JPG, fonts etc.)”, “Interpreter languages (currently Perl and Python)” as well as “Printing (CUPS library) and Imaging”. The full definition of all standards and their versioning scheme is available online.
With this collection of standards, a minimum set of APIs is defined which every LSB compliant distribution must support. Every provider of Linux systems can retrieve an LSB certification for a specific version of his distribution.
From the perspective of application developers, LSB-compliant Linux systems offer a comfortably well-defined set of available interfaces. Hence, an application that does not depend on any external interface beyond those defined in the specified LSB standard will most probably function on any LSB-compliant Linux.
LSB tool chain
In addition to providing the standard definitions, the LSB workgroup develops, provides and documents a tool set to actually build applications and libraries that are compliant with these standards. This so-called LSB Software Development Kit (LSB-SDK) consists of
lsbcc, a wrapper around the GNU compiler collection,
LSB-compliant header files,
stub libraries, which redirect interface calls exclusively to LSB-compliant interfaces that have to exist within real libraries on the build system.
If an application is built using lsbcc, including only the header files and linking only against the stub libraries within the LSB-SDK, it will be LSB standard-compliant. In order to successfully compile a project with this tool kit, the source code might have to be scrubbed free from sections which cannot be translated to purely LSB-compliant interface calls. For the whole process, i.e. installing and applying the LSB-SDK as well as for making the source code LSB-fit, many detailed tutorials exist both online (e.g. by the Linux Foundation itself or by Novell and as books (e.g. “Building Applications with the Linux Standard Base”, by Linux Standard Base Team, published Oct 29, 2004 by IBM Press).
The tool set also contains the so-called Linux Application Checker (App-Checker), which measures the support of applications and libraries for a certain standard. The same tool can access a database hosted by the LSB workgroup, containing detailed information about available interfaces for nearly 300 Linux distributions. With this information available, the App-Checker
analyzes all external interfaces that the inspected application depends on,
compares these dependencies with the lists of interfaces actually provided by every individual Linux distribution,
generates a list of those distributions which satisfy all interface dependencies of the application.
LSB certification
The LSB workgroup offers a certification scheme for both operating systems and applications. For the latter, certification of LSB-compliance is free. If this certificate will be used for the purpose of communicating supported Linux systems to customers, it needs to be renewed with every new release or version of the application or library.
LSB for the ERiC library
Reviewing all the options that are listed above in this article, we found the LSB tool set most promising. We installed the LSB-SDK and applied manageable efforts to adapt our source code (including some third library components) and build framework such that a successful build with the SDK was possible. After we learned that this way of building our product against the (at the time) newest LSB standard 4.1 was feasible, we had to decide for a documentation scheme: What is the best way to communicate to our customers the great fact that ERiC from now on depends on nothing but LSB-compliant external interfaces?
Getting our product LSB-certified would imply too much work load, because updates of ERiC are released at least twice per year (one reason for that being the dynamic German tax laws). Going through the certification process for each release would not provide sufficient benefit compared to a second option we eventually chose:
Instead of advertising our library as being certified for LSB-compliance, we decided to just use the LSB-SDK for building and the App-Checker for verifying the LSB-compliance of the result. These two facts we then communicate to our customers. In addition, we provide some of the App-Checker output in our documentation, i.e. an explicit list of Linux distributions which the LSB tool chain predicts to be supported by our product.
Benefits
By having chosen the tool set and methodology suggested and provided by the Linux Foundation, we have gained the following benefits:
We can provide a precise documentation of utilized tools and standards to our customers.
The used standards are in accordance with the recommendations of the Linux Foundation.
The documentation effort is minimal for us, since from the statement “Compliant with LSB-4.1″ our customers can look up all further details from the pages of the Linux Foundation.
We can publish an automatically generated list of distributions that are supported by ERiC.
Unintended deviations from the promised and documented interface contracts, possibly caused by ongoing code development, can be automatically detected by the QA team at an early stage within the release cycle.
Problems
The following list summarizes issues that we had not expected after the initial analysis. We came upon them only by working with the LSB-SDK for a while and therefore find them worth sharing.
The LSB-SDK is based on versions of GCC libraries that are very old and still contain bugs which have been removed in newer versions a long time ago. For example, we ran into some wrong behavior of our compiled product, found out that it was caused by faulty headers within the LSB-SDK and had to file a bug report to the LSB group. The buggy header still included in the LSB-SDK 4.1 has been removed from its original pattern (being part of gcc-4.0.0) many years ago already.
Some symbols and sections wrapped by the libraries of the LSB-SDK are actually not compliant with LSB themselves.
On some Linux distributions, special packages have to be installed on top of the default configuration in order to get full LSB support.
The project members of both the LSB standards as well as the LSB-SDK seem to have problems fixing known flaws as well as adapting to new technological developments in appropriate time. For example, the information in the database consulted by the App-Checker is often outdated. This results in the critical fact that some modern and relevant Linux distributions are missing in the database. As another example, there is no LSB standard available which complies the C++11 standard.
Including third party components like XML handlers results in the need to also adapt this third party source code such that it can be successfully compiled with the LSB-SDK. These efforts are often time consuming, and by changing the source code the risk of unintended side-effects grows. Furthermore, if these third party components require updates, e.g. by means of a security patch, every new version might bring in a change to the code that makes an LSB-compliance impossible. The risk is considered small, but it exists.
The continuation of the LSB project within the Linux Foundation seems to struggle from insufficient financing and hence manpower, as can be interpreted e.g. from unofficial statements in the LSB discussion mailing list (search e.g. for “C++ Language / Compiler Support Timeline”). This causes doubts that the need for an increased adaptation speed to technological developments is going to be met in the near future.
The number of LSB certifications for newly published Linux distributions is decreasing: LSB 3.x and LSB 4.x have both 27 certified Linux distributions each so far. LSB 5.0, published in June 2015, has only one certified distribution until today (see linuxbase.org for more details). This might indicate a decay of the distributors’ acceptance for this standard, which in the long run can result in distributions that do not offer all interfaces defined by LSB anymore.
Conclusion
A typical problem in the world of Linux is how to create and publish a closed-source application such that it works on a maximum number of relevant distributions. To overcome this for our own product, the ELSTER Rich Client, we inspected multiple common approaches. Various factors, many of them specific to our situation and constraints, made us finally going for LSB compliancy. By achieving and communicating this to our customers, we put our product on a strong fundament of precisely documented requirements. Since nearly two years now we build and publish the Linux-version of our product by means of the LSB tool set, and from that point we have not run into a single user complaint regarding unsupported distributions.
A minus to our choice is the remaining risk of future changes in our product requirements or within the embedded third-party packages, as these might cause conflicts with the LSB standards too expensive to fix. A further risk is the possibility that LSB might lose its relevance due to an insufficient standard development or – as a potential result from that – a continuously decreasing number of compliant distributions. Still, considering the lack of feasible alternatives, for us the use of LSB standards and the corresponding tool chain continues to be the best available strategy for providing precisely defined and widely installable Linux application software.
(c) 2017 mgm technology partners. This posting "Increasing the Portability of a Linux Application" is part of the mgm technology blog. The author of the posting is
Daniel Brüderle.
We are hiring! mgm technology partners is looking for good software engineers for all our offices. Check out www.mgm-tp.com/karriere.