Build Environment Effort/Dependencies
Contents
The taming of the shrew
A current build contains roughly 200 "modules", top level directories in the repository. Our build system tracks build dependencies between them in the "build.lst" files in the "prj" directories of them. Not every tracked dependency is necessary or useful, some dependencies have been forgotten, some were never correct, some can be avoided by a few code or makefile changes. This is tracked in issue 103496.
If all dependencies are drawn in a module graph, the result Media:Complete.odg is hilarious and chaotic, due to the fact that at both ends of the graph some modules collect a lot of dependencies. At the lower end it's solenv as this is needed always and at the upper end it's postprocess as it "depends" on every module that without postprocess would be a top level module. The module instsetoo_native and smoketestoo_native depend on postprocess. By removing these modules as well as dmake and guw (as they are built outside the ooo build process) the graph Media:reduced.odg looks a bit better, but it's still too much as especially the indirect dependencies make it impossible to see the forest for the trees.
It should be possible to group modules, either by their "level" in our project or because they are somewhat related. Some of the mentioned unnecessary or bad dependencies prevent a proper separation between the groups, so fixing that by moving or changing source code is necessary. Here's a suggested group list, based on the DEV300_m57 build and the fixes from issue 103496 applied. I have created the groups so that their modules basically should have dependencies only on each other or to modules from a "lower" group. In some cases there are some dependencies that violate that principle (I will mention them below), but I'm confident that they can be fixed by changing code or module content. Issue 103496 or any follow-up issues will track that progress.
Build prerequisites
Some modules are needed before even the first "real" module can be built or they can be seen as part of the build environment: solenv, soltools, postprocess, instsetoo_native, dmake, guw. soltools still has a "dirty" dependency on stlport (that itself is part of the external modules), but must be built before any other module than stlport. This must be solved by either removing the stl usage, using the compiler stl in soltools or in the complete office. The last variant would make our UNO C++ language binding incompatible, so it needed some coordination. solenv mainly consists of scripts and makefile includes.
External modules
These comprises the following modules: afms, agg, apache-commons, apple_remote, beanshell, berkeleydb, bitstream_vera_fonts, boost, cairo, cppunit, curl, epm, expat, external, fondu, graphite, hsqldb, hunspell, hyphen, icc, icu, jfreereport, jpeg, libegg, libtextcat, libwpd, libxml2, libxmlsec, libxslt, lpsolve, lucene, MathMLDTD, moz, msfontextract, neon, np_sdk, nss, openssl, python, redland, rhino, sane, saxon, stax, stlport, tomcat, twain, unixODBC, vigra, x11_extensions, xpdf, xsltml, zlib
With a few exceptions all code modules are checked in as download + patch, "external" contains binary stuff that is not part of the repository. This group is "clean", means: it does not have any dependency on modules of any of the following groups.
URE - UNO Runtime Environment
In former times we even have built this as a separate product, but nowadays it's only a part of the ODK (OpenOffice.org Development Kit). All parts of the URE only depend on each other or on some external modules (so that a packaged URE could need some of them, e.g. stlport). A packaged URE would need a license file that currently would create a build dependency on readlicense_oo. Though no localized license files would be needed, the fact that readlicense_oo also does the localization would create a dependency on the localization tooling. An english build runs fine even without that (and packaging would be done in instsetoo_native anyway), so I have omitted that for the moment and so readlicense_oo is not a part of the URE list:
bridges, cli_ure, codemaker, cppu, cppuhelper, cpputools, idlc, io, javaunohelper, jurt, jvmaccess, jvmfwk, offapi, offuh, rdbmaker, registry, remotebridges, ridljar, sal, salhelper, stoc, store, testshl2, udkapi, unoil, ure, xml2cmp.
"pyuno" looks like a module that would fit in here nicely, but it is different to the "real" language bindings as it also cares for deployment and so explicitly works with the different layers (share/user) of the application. This requires some office code (pathutils from "tools"). This is comparable with OOo Basic that works in the same way.
In my current split build based on these groups I have to add the "testshl2" module here also, though it is not related to the URE. But the testshl2 module is needed to execute C++ unit tests and if we want to run them while building, we need to build this module as early as possible. It's very unfortunate that our C++ tests don't use "pure" CppUnit but unstead added some code that depends on sal. Another bad side effect of this is that the sal C++ tests currently can't be executed in the build. This should be fixed by removing testshl2 and using CppUnit directly.
The list contains offuh and offapi as currently the only way to generate UNO API headers is using offuh. As soon as we replace offuh by building headers inside of offapi and udkapi, we can remove offuh and offapi. The former would just go away, while the latter would become a part of the next group:
ODK - OpenOffice.org Development Kit
This group comprises odk, autodoc, udm, cosv, unodevtools. In the "odk" module all parts of an ODK installation set are collected in some zip files that will be used to create an installation set in instsetoo_native. In future this group also may contain offapi, as explained in the URE section. Autodoc and unodevtools are tools needed by the ODK or its users (udm and cosv are prerequisites of autodoc).
The ODK currently uses GNU make (and some tools from the URE) for builds, while the OOo build system uses build.pl and dmake. It would be nice to have both systems united, so that not only all extensions but also OOo could be built with an ODK. The future will show how and how fast we can achieve that.
The ODK group does not need to be built for an OOo build - nothing here is used to build or run OOo. OTOH it is a very small group and so perhaps it makes sense to unite it with the URE group.
Build and test tools
Beside solenv, dmake, soltools, cppunit/testshl2 and the URE/ODK build tools some other tools are needed to build and test OOo. This comprises the following modules:
building: idl, rsc
They should depend only on external or URE modules. Currently that isn't true, mainly due to a dependency on "tools". So currently in a real build these modules must stay in the "office" group.
testing: automation, smoketestoo_native, testautomation, testtools, qadevooo
"automation" is needed to use the OOo testtool for GUI tests that themselves are found in "testautomation". The OOo installation set contains the testtool and the automation library built in this module, so it must be part of the OOo build. We will see later where the best group of modules is.
"smoketestoo_native" is the first test after successful creation of an OOo installation set that can be executed in the build process. "qadevooo" is used by API and complex tests, partly inside the build. As some modules execute tests based on qadevooo in the build, we have to build qadevooo before that, so qadevooo could be moved into the URE group. "testtools" contains some other test tools that currently are not used inside of an OOo build.
Some special modules
Before we come to the "real thing", here are some modules that don't fit well into any other categorie. Some of them deliver parts of an OOo installation set:
crashrep, extras, helpcontent2, l10n, transex3, readlicense_oo, binfilter, javainstaller2, pyuno
None of these modules is a real extension, but an office can be built and run (but not packed) without them and OTOH they can be built separately (some require that the office code has been built before, some don't).
readlicense_oo is only needed for packaging, but not for building, with the exception of swext and sdext. Other extensions (dictionaries and reportbuilder) have their own license files).
Some other modules not even deliver parts of OOo:
migrationanalysis, soldep
I've put them aside for now.
Images
All images in OOo will be packed into images.zip. This requires the modules
default_images, external_images, ooo_custom_images, packimages
With the exception of packimages no other modules should get images directly from the images modules. Unfortunately the images can't be built and used separately as the rsc compiler checks images while it is used in the office modules and packimages needs the image list resources to know which images should be packed. So the image modules need to be present while OOo is building and the packimages step must be done after everything else has been built.
Extensions
Some modules are built as extensions:
sdext, swext, dictionaries, reportbuilder
Building extensions should not require anything than an ODK. But most of them still contain some "bad" dependencies:
- sdext (or more specifically: pdfimport) uses comphelper inlines and links statically against basegfx
- reportbuilder depends on wizards
It would be nice if "binfilter" could be added here, but unfortunately it still has dependencies on office libraries that can't be removed easily.
Office modules
Everything else can be seen as an "Office" module. It is hard to find good criteria to split them up, the best I can see now is going back to the "mothers of all modules". When we started with StarWriter and StarOffice more than 15 years ago, we based them on two class libraries called "tools" and "starview". The former still exists, the latter has mutated to "vcl". Over the years we added a lot of other libraries "below" these two and "above" them as well as "besides" them. I tried to create a "central" group around vcl and other groups "below" and "above" it.
A first attempt
This was just the start, a first structure that allowed a "split build": each of these groups or "modules" can be built separately if the deliverables of the modules “below” are provided either as a “solver” or as developer packages and they can deliver one or more packages. Some of the sub-modules are marked red because there are some issues with them that are explained here. Fixing them will enable us to improve the code architecture so that the modules will be more stable. This is very important if we want to store the modules in separate repositories.
Further improvements have been investigated to finally get a separation for a split build.
