User:Kr/Packaging

OpenOffice.org (OOo) and its derivatives are complex products. Many features, templates, configuration files, registry entries, binaries, localizations etc. need to be delivered and deployed in a reliable and platform compliant way, while giving the user broad choice regarding the particular features he wants to actually install.

OOos growing ecosystem brings the current approach to its limits, we are currently facing a set of problems, which the below proposed solution is going to address. OOo based products need to support different platforms (Operating System / Machine Architecture) as well as different deployment systems (e.g. RPM, Debian Packages, Ports, Solaris Packages, MS Windows Installer), localizations and feature sets, as well as they need to be easy to extend and maintain.

Overview

Some clarification is needed regarding the terms to describe how to deploy, update, maintain and configure a particular software program.

Products

A product consists of a particular set of features and localizations targeted to a particular set of platforms and deployment systems. A product is a set of bits provided by a medium such as a CD, a floppy disk or as a download. A product typically has a brand as well as predecessor and successor products and may require other products to be installed. Independent Products assembled of other (sub-) products may share these (sub-) products, even if these (sub-) products are implementation details only.

Deployment

As OOo is cross platform (Operating System / Machine Architecture), OOo installation sets need to integrate with different deployment systems.

These deployment systems can be categorized as to be

• package oriented (such as RPM) and / or
• product oriented (such as MS Windows Installer).

Package Support

Package oriented deployment systems typically lack product support, e.g. in a package oriented deployment system, the user typically only sees the deployed packages in the systems configuration and is neither able to directly see which products are represented by which packages, nor which additional features are available. To remove a particular product the user needs to find and to remove all belonging packages (sometimes even the indirectly installed packages) typically by searching the package database.

Package oriented deployment systems enable sharing of packages between products, allowing for re-usage of particular files or functionalities.

Packages can typically be set into relationship to one another. The following are typical relationships,

• one package may require on one or many another packages,
• one package may conflict with one or many other packages,
• one package may replace one or many other packages,
• one package may suggest one or many other packages.

Program updates may be deployed at least on a package granularity, while some deployment systems even support patch packages (packages which only contain the differences between a particular package and its successor).

Product Support

In a product oriented deployment system, the user sees the installed products in the systems configuration. Customization is typically supported by product, offering not yet installed features as well as allowing to remove the product as a whole.

Product based deployment systems may allow sharing of entities (e.g. files) on some level (e.g. "component", see below).

Product based deployment systems typically allow to set products into predecessor / successor relationship.

Product based deployment systems mostly only support product updates on a product level, e.g. as new products or as product patches, which may only be applied to one particular product.

Updates

Over time, new versions of products are released, providing bug fixes, additional features, usability improvements etc. During the deployment of product updates the older versions may be de-installed or altered respectively completed. Updates may be provided as dedicated products or as implicit downloads.

Requirements

Compatibility

As a product evolves, its interfaces may change in an incompatible fashion. For binary packages mostly interesting are

• ABI (Application Binary Interface) incompatible changes, as well as
• structural incompatible changes (removed / renamed files).

Some installation units try to stay compatible, expressing any change of compatibility in their version numbers, while others may change incompatible with every version.

Version numbers expressing compatibility are typically used as follows,

• a change in the micro version signals for bug fixes only,
• a change in a minor version signals additional features,
• a change in the major signals an incompatible (API / ABI etc.) change.

The OOo productizer needs to support compatibility changes, such that different versions and derivatives may be installed without conflict side by side.

Deployment Systems

Deployment systems to be supported by OOo are at least

• MS Windows Installer,
• Red Hat Package Manager,
• Debian Packages
• Solaris Packages and
• Mac OS X Packages.

Developers

Support for Changes:

• updated package(s)
• (automatically) remove package(s)
• add package(s)
• rename package(s)

• remove file(s)
• add file(s)
• move file(s)

Program Management

In a perfect world, program management would be able to create any kind of product, only depending on business needs, not restricted by technical constraints.

Program Management basically mediates between marketing / market requirements and the pool of available technologies.

Program management requires the following product relationships to be supported:

• No relationship -> self contained
• Update relationship
  • update only (e.g. add-ons, localizations)
  • self contained (e.g. OOo 2.1 updating OOo 2)
  • Multi-update relationship
    • Cross version (major / minor / micro)
    • Cross language
    • Cross brand
    • Cross variant
    • Cross platform

Wishful Thinking

What we need for OOo is partly already available, partly not. Ideally we would be able to represent OOo as well as derivatives by a Product View on all our target platforms. This Product View would than allow to update, extend and to (partly) de-install it. Getting any needed data from one or multiple media / repositories. This Product View would need to be accessible from inside the product as well as from the platforms standard location (e.g. "Add or Remove Program" on MS Windows), if any.

The "extension manager" (tools/Extension Manager) already offers some update / installation capabilities from inside OOo, restricted to extensions.

Some platforms also provide mechanisms for system update and maintenance, certainly an installed OOo, respectively derivative, needs to integrate with this as well.

Combining these things gets kind of complicated, remembering that parts of OOo may be installed system wide (e.g. "core" components), while others may be installed per user only (e.g. extensions).

Requirements

So, what actually is needed is

• product independent update, as supported by packages,
• Product View, to customize or complete a particular installation,
• dependency support,
• certainly versioning,
• patch support,
• system/user repository support, to enable product maintenance by standard maintenance actions (e.g. apt-get update; apt-get upgrade),
• system installation, to make the product available for multiple users,
• user installation, to make the product available for single users.

Non of OOos target deployment systems support all of these features / requirements. Some deployment systems provide APIs, hopefully enabling the implementation of missing desired features. Systems not providing a way to realize a particular feature may be left without it.

Product Independent Update

Product independent updates allow to share updates between different products. For example the Uno Runtime Environment (URE) is shared by OOo and its derivatives such as StarOffice and StarSuite. If products can only be updated product by product, than all derivatives have to offer updates for their particular products, even if the update only provides changes e.g. to the underlying Uno Runtime Environment, thus all updates being identical.

The package oriented systems typically treat any package as a product, while still allowing these packages (products) to require each other. Hence, package oriented systems allow product independent updates anyway.

Product View

The product view allows to see the parts of an installed product in their product context. Ideally allowing to customize these parts as well as to install other product parts, either from a medium such as a CD, or from any other repository.

Package oriented systems typically lack a product view, especially making it hard to de-install a particular product as well as to install completing parts.

Dependencies

Dependencies enable composition of products by other products. Required (sub-) products may even be installed and de-installed automatically (cascaded de-installation), satisfying other products dependencies.

Package oriented systems typically support dependencies, while product oriented mostly do not do so.

Versioning

Most concrete products are released over time as different versions of an abstract product. Mostly newer versions replacing older ones.

Versioning is supported by most deployment systems.

Patches

Deployment systems use some level granularity for deploying a products entities. This units are sometimes called packages or components. Patches allow to provide updates on a sub granularity level, e.g. enabling smaller downloads.

Repositories

Some systems use repositories to ease system maintenance and customization. These repositories typically list latest versions of software products, while enabling the user / administrator to easily update or customize the system.

Deployed products certainly need to be maintained in the same way as the underlying system, ideally being listed and available in these repositories.

In case user product installation is supported, a system may provide user installation repositories as well as system repositories.

User/System Installation

Software may be provided as system software, installed by an administrator, or as user dedicated software, installed by user.

Modelling

De-Composition

Looking at an installed OOo and its files, registry entries etc., we can see, that all these entities belong to one or multiple of the following categories, such that they provide or depend on it

1. brand
2. Operating System
3. Machine Architecture / interpreter
4. localization

The consequences of this observation are, that entities unrelated to one or multiple categories, but shared by products differing in these categories, are identical and thus may be re-used for creating, changing or updating products respectively installed products.

Additionally every entity exactly belongs to one

• feature

as otherwise it would not be needed for any feature, thus it would not be needed at all. Certainly features may be related such that they require one another, even if this is not noticeable in the Product View.

Composition

Model products by setting them into

• inheritance, respectively
• instantiation (template)

relationship.

Inheritance

Inheritance models a "is a" relationship. In practice that would mean, that a StarOffice 8 update 7 is an OOo 2.2.1 (respectively its basis) adding something.

Example

Template Product OOo-Standard {
  Features: writer, calc, impress, draw
}


Abstract Product OOo2.4 {
  Name: OOo 2.4
  Code-Base: SRC680m236
  Implements: OOo-Standard

}

Product OOo2.4-ISO : OOo2.4 {
  Name: OpenOffice 2.4
  Format: ISO-750
  Platform: Linux-x86, Windows-x86, Mac OS X x86
}

Product OOo2.4-download-linux-x86 : OOo2.4 {
  Name: OpenOffice 2.4
  Format: donwload
  Platform: Linux-x86
}

Product OOo2.4-download-windows-x86 : OOo2.4 {
  Name: OpenOffice 2.4
  Format: donwload
  Platform: windows-x86
}

Abstract Product SO8u9 : OOo2.4 {
  Name: StarOffice 8 update 9
  Features: so-templates, so-fonts, so-spellchecker, so-brand
  Updates: < StarOffice 8 u 9
}

Product SO8u9-ISO : SO8u9 {
  Name: StarOffice 8 update 9 ISO
  Format: ISO-750
  Platform: Linux-x86, Solaris-x86, Solaris-Sparc, Windows-x86, Mac OS X x86
}

Product SO8u9-donwload {
  Name: StarOffice 8 update 9 ISO
  Inherits: SO8u9
  Format: download
  Platform: Linux-x86
}


Abstract Product {
  Name: StarSuite 8 update 9
  Inherits: OOo 2.4
  Features: so-templates, so-fonts, so-spellchecker, ss-brand
  Updates: < StarSuite 8 u 9
}

Tooling

• Comparison of Installation Sets
• Check for conflicts
• Creation of Installation Sets
• Visualization

Product Pipeline

Source =compilation(switches)=> Binaries =packaging(version,...)=> Packages =productization(features)=> Products

.cxx => lib.so,lib.dll => lib.rpm,lib.msm => Our Product.sh, Our Product.msi

Implementations

Example Product creation for by target deployment system.

Capabilities

Name	Products	Packages	Patches	Dependencies	API	virtual Packages	customization	Installation on Demand	Advertisement	User Installation	Administrative Installation	Usage Metrics	Tagging	Repository	System Maintenance
MS Windows Installer	yes	no	yes	no	yes	no	yes	yes	yes	yes	yes	yes	no	no	system only
Red Hat Package Manager	no	yes	partly	yes: "provides", "requires"	?	yes	no	no	no	no (though workaround)	no (though workaround)	no	?	yes: yum, apt	yes: yum, apt
Solaris Packages	no	yes	yes	yes:	?	?	?	?	?	?	?	?	?	?	?
Debian Packages	no	yes	no	yes: "depends", "recommends", "conflicts", "suggests", "replaces", "pre-depends", "breaks"	yes: libapt-pkg	yes	no	no	no	no (though workaround)	no (though workaround)	no	yes: "priority", "section"	yes: apt	yes: apt
Mac OS X bundles	yes	yes	?	?	?	?	?	?	?	?	?	?	?	?	?

Product View

All of our package oriented target deployment system lack a Product View. This has been recognized already some while, when we switched to native installers, and lead to the Java Installer, basically offering a Product View similar to MS Windows "Add or Remove Programs" approach.

To establish a Product View, we need to now

• which parts of a particular are already installed, and
• which parts are additionally available.

Repository Support

To integrate into deployment systems using external repositories for updating / maintaining a system, we need to be able to

• add a repository (e.g. repository.openoffice.org) during installation, respectively to
• remove a repository during de-installation.

Debian Packages / APT

Product View

To query for the installed packages of a particular product (e.g. OOo):

~> dpkg -l "openoffice.org-*" | grep "^in"
in  openoffice.org-core01             <none>         (no description available)
in  openoffice.org-core02             <none>         (no description available)
in  openoffice.org-core03             <none>         (no description available)
in  openoffice.org-core03u            <none>         (no description available)
in  openoffice.org-core04             <none>         (no description available)
in  openoffice.org-core04u            <none>         (no description available)
...

To query for all available packages of a particular product (e.g. OOo):

~> apt-cache dumpavail | grep -i "^package.*openoffice"
Package: openoffice.org-filter-so52
Package: openoffice.org-gtk-gnome
Package: openoffice.org-help-en
Package: openoffice.org-hyphenation-hr
...

To find the files belonging to a particular package, including the Java Installer .xpd files:

~> dpkg -L openoffice.org-writer
/.
/usr
/usr/lib
/usr/lib/openoffice
/usr/lib/openoffice/program
/usr/lib/openoffice/program/libwpft680lx.so
/usr/lib/openoffice/program/liblwpft680lx.so
/usr/lib/openoffice/program/libmsworks680lx.so
/usr/lib/openoffice/program/libswui680lx.so
...

What we don't have yet is the .xpd information for not yet installed (downloaded) features. As a workaround, we could put the xpd description into a packages description, this would be enough information to start the Java Installer (the Product View) for any product.

Repository Support

To my knowledge there is no command line tool around, to modify the APT repository list (apt/sources.list) except apt-cdrom, which is used to add and register CDs only. Fortunately there is this "libapt-pkg" library, providing programmatic access to apt and hopefully supporting addition and removal of repositories.

Mac OS X

Links:

• Files & Software Installation
• Software Installation and Software Updates

MS Windows Installer Service

Experiments under Windows with WiX (Windows Installer XML) show,

• that any set of products may share any number of files,
• that any one product may update any number of other products (basically leading to a de-installation of these other products), and
• that any product sharing files with other products may update any number of these shared files.

This lets me think, that Windows is as dynamic and flexible as the classic package based platforms, while providing a simpler user interface.

Merge Modules

"In order to avoid versioning problems, you should use always merge modules for any component or file that will be shared by multiple applications." [1]