OOo and Subversion
[ This page is under constrcution. I'll add more content over the next few days. HR]
- 1 OpenOffice.org migrates to Subversion
- 1.1 Trunk only migration
- 1.2 Required tools
- 1.3 Subversion documentation
- 1.4 Server access methods
- 1.5 Basic read/only SVN operations
- 1.6 SSH Setup
- 1.7 CWS tooling
- 1.8 Tips and tricks
- 1.9 Notes
OpenOffice.org migrates to SubversionAfter a long discussion and preparation time it finally happens, OpenOffice.org migrates with milestone DEV300 m31 to a new SCM (Software Configuration Management) system. The new SCM system will be Subversion (SVN) for now, which might disappoint the one or other who had hoped for a distributed SCM like Git, Bazaar or Mercurial. Please be assured that we'll keep our mind open and we'll reevaluate the fast moving SCM tooling situation periodically. We were very careful to make certain that our move now to Subversion does not preclude a move to a DSCM in the future.
Trunk only migration
Milestone DEV300 m31 is the last milestone which has been integrated via CVS and is the first milestone which is available in SVN. We did a trunk migration only. The trunk (DEV300) code line will be maintained via Subversion only starting with milestone DEV300 m32. Old code lines, including the OOo-3.0 release codeline(OOO300) will still be maintained via CVS.
Please let me repeat this: we did not migrate branches and tags to SVN. There are a number of advantages (and some disadvantages as well) to this approach. A discussion of the merits of different approaches to the actual repository conversion can be found here.
The most important reasons for choosing the "trunk only" approach were:
- No quiet time for developers
- The conversion of historical workspaces from CVS to SVN would be fragile at best due to some liberties we took in structuring our CVS repository. Having no historical branches and tags is preferable to having inaccurate branches and tags IMHO.
The "trunk only" approach enabled us to optimize the resulting SVN repository, it shrunk from 90 GBytes to about 6 GBytes overall size. The following rules have been applied during conversion
- Only modules which have active content in DEV300 m31 are included in the conversion.
- All files in "Attic" have been removed.
- All but the last revisions of binary files have been dropped - after all you can't diff them anyway
- All but the last revisions of "localize.sdf" files have been dropped - saves an incredible amount of repository space without loosing much
- subversion-1.5.1 or later; download...
- ssh for committers, preferably OpenSSH 2.4 or later
We make heavy use of the new SVN merge tracking feature. So please do not use a SVN client older than subversion-1.5.1 if you work on a child workspace (CWS).
The excellent Subversion documentation can be found here: http://svnbook.red-bean.com. Please use the nightly build documentation for 1.5 and later, it covers new features like merge tracking.
Server access methods
Read only SVN access
svn checkout svn://svn.services.openoffice.org/ooo/trunk
svn checkout http://svn.services.openoffice.org/ooo/trunk
Read only browser access
Point your browser to
Write access to the repository requires a ssh setup.
Basic read/only SVN operations
The OOo SVN repository is structured like this:
../trunk <= the main development line, currently master workspace DEV300 is on trunk ../cws <= child workspaces live here, names need to be unique ../cws/foo <= for example CWS foo ../cws/... ../branches <= future master workspaces which are not on trunk ../branches/OOO310 <= for example MWS OOO310 ../branches/... ../tags <= milestone tags, release tags ../tags/DEV300_m32 <= for example milestone DEV300 m32 ../tags/.. ../dist <= space for distribution specific stuff ../patches <= space for globally useful patches ../contrib <= do we need this?
Check out a milestone DEV300 m32 with:
Switch from milestone DEV300_m32 to DEV300_m35 (safes potentially a lot of checkout time):
cd DEV300_m32 svn switch svn://svn.services.openoffice.org/ooo/tags/DEV300_m35 cd .. mv DEV300_m32 DEV300_m35
Check out a child workspace foo:
svn checkout svn://svn.services.openoffice.org/ooo/cws/foo
Update working copy of foo:
cd foo svn update
Find out the base (milestone or cws) of working copy wc:
cd wc svn info
List all available milestone and release tags:
List all available child workspaces:
View commit log on a single file:
svn log file.cxx
Include all paths affected by change sets in log command:
svn log --verbose file.cxx
svn annotate file.cxx
SSH key generation
On Unix or Windows/cygwin a SSH identity (private/public key pair) is generated with:
ssh-keygen -t dsa
You'll be asked for pass phrase which protects your private key. Per default the public key is named $HOME/.ssh/id_dsa.pub.
Note that SSH keys created between September 2006 and May 2008 on Debian or Debian derived systems like Ubuntu may be insecure. Please use recent SSH tools.
If you want a dedicated identity for just OOo SVN usage, generate a specially named key pair
ssh-keygen -t dsa -f ~/.ssh/id_dsa_ooo_svn
and instruct Subversion to use this identity (key pair) with the following configuration entry in $HOME/.subversion/config:
[tunnels] ssh = ssh -i /home/jr93709/.ssh/id_dsa_ooo_svn
Initial Migration of Keys
Please query IssueZilla for your issue with the attachment holding your key and add it as a dependency to issue 94002.
We will then install those keys for DomainDevelopers for read/write access to the SVN repository.
Up to now only few code committers made use of the ssh key upload for the OpenOffice.org SVN repository. Please note that your key will not be migrated without your intervention.
We will continue to use IssueZilla for managing and tracking key additions and changes. Please see Submitting Your Certificate (Public Key).
The CWS tooling has been reworked to adapt to SVN. The basic CWS tool is now simply called cws and is invoked as cws <subcommand> in the style of the SVN client, there are no longer separate tools like cwsadd, cwsresync etc etc.
Note: the cws script will probably change over time as we learn how the usage patterns evolve. If you got a problem with the cws script, it might be worthwhile to rebase it to the latest version
$ cd <your_CWS>/ooo/solenv/bin $ svn merge svn+ssh://email@example.com/ooo/trunk/solenv/bin/cws.pl
The most important change in the way CWSs are handled now is that there is no longer a need nor possibility to add modules. Well, there is still a sw and config_office but all modules of old are now just top level directories, at least as far as SVN is concerned.
CWS config file
Please keep all your entries in your $HOME/.cwsrc for now, you'll need them if you want to work with CWSs for OOo-3.0. Just add the following line to refer to the OpenOffice.org SVN server:
Example of a complete .cwsrc file:
[CWS_CONFIG] # Network proxy for EIS connections. # Comment in if you need to access EIS via a proxy. # Note: This is just for accessing the EIS database. For SVN have a look here: # http://svnbook.red-bean.com/nightly/en/svn.advanced.confarea.html#svn.advanced.confarea.opts.servers # Example: PROXY=http://myproxy.company.com:8080 # PROXY= # CWS Database server (SOAP). It's possible specify to several backup server, # currently there are none. CWS_DB_SERVER_1=https://eis.services.openoffice.org/soap/servlet/rpcrouter # Your CVS login (for authentication with the CWS database). CVS_ID=<your_ooo_name> # Your scrambled CVS password (for authentication with the CWS database). # Take this one from .cvsrc CVS_PASSWORD=<your_current_cvs_scrambled_ooo_password> # OOo CVS tunnel # Example: CVS_SERVER_ROOT=pserver:firstname.lastname@example.org:/cvs CVS_SERVER_ROOT= # Path to the cvs binary (optional) # Example: CVS_BINARY=/usr/bin/cvs #CVS_BINARY= # SVN server for OOo-3.1 child workspaces # r/w access via svn+ssh pprotocol SVN_SERVER=svn+ssh://email@example.com/ooo # r/o access via svn protocol #SVN_SERVER=svn://svn.services.openoffice.org/ooo
Working with the new cws tool
Help can be invoked with:
cws -- version: 184.108.40.206 usage: cws <subcommand> [options] [args] Type 'cws help <subcommand>' for help on a specific subcommand. Available subcommands: help (h,?) create fetch (f) rebase (rb) analyze (an) query (q) task (t) integrate *** release engineers only *** eisclone *** release engineers only *** help (h, ?): Describe the usage of this script or its subcommands usage: help [subcommand]
The subcommands do more or less what their CVS predecessors did.
Create a CWS
Create a CWS with the cws create subcommand. Please note that this commands only creates a CWS branch on the OOo server and registers it with EIS. It changes nothing on your hard disk.
Example: create CWS foo on the latest published milestone of master DEV300
cws create DEV300 foo
Of course you can specify which milestone you want to use, here is the complete command reference:
cws -- version: 220.127.116.11 create: Create a new child workspace usage: create [-m milestone] <master workspace> <child workspace> -m milestone: Milestone to base the child workspace on. If ommitted the last published milestone will be used. --milestone milestone: Same as -m milestone.
Your CWS branch is then available as
for check out.
Check out a CWS
$ cws fetch -c foo
Currently this simply checks out your cws foo just as if you had issued the command
$ svn checkout svn+ssh://firstname.lastname@example.org/ooo/cws/foo
A milestone can be fetched with
$ cws fetch -m DEV300_m32
this is equivalent to
$ svn checkout svn+ssh://email@example.com/ooo/tags/DEV300_m32
Over time I'll add a number of convenience functions to cws fetch, like configurable partial checkouts, use of prebuild solvers, switching CWSs etc etc.
Please note that if you have already a OOo SVN working copy lying around, you can save considerable checkout time with the svn switch command.
cd <wc> svn switch svn+ssh://firstname.lastname@example.org/ooo/cws/foo
will fetch the differences between your working copy in <wc> and the CWS foo
Rebasing a CWS
The successor of the cwsresync tool is called cws rebase. rebasing is now a two step operation:
How to migrate still active CVS based child workspaces destined for DEV300 to SVN?
[Some detailed instructions will appear here soon. Basically resync your CWS to DEV300 m31 with the old tools, create a patch and apply it to a SVN version of the CWS. Release engineering will help you with this. Many nearly finished CWSs might even go in without a need to migrate to SVN]
Tips and tricks
Ignoring output trees
Platform output trees (unxlngi6, unxsols4, wntmsci12 etc) will show up as unversioned directories marked with a "?" in the output of svn status and other SVN commands. This is inconvenient and reduces the usefulness of these commands. In principle it's possible to deal with this in two ways:
- attach and commit a svn:ignore property to every top level directory listing all possible platforms
- add all the platforms which are usually build to the so called global ignore list
Actually the global ignore list is more of a kind of "personal" ignore list which is applied to all svn commandos which access your personal subversion configuration.
Since maintaing svn:ignore properties in about 200 directories for about 20 platforms is pretty clumsy, I suggest that we use the global ignore list for this purpose. Here is my personal ignore list, YMMV.
$ cat $HOME/.subversion/config ... [miscellany] ... global-ignores = common* unxsoli4* unxsols4* unxlngi6* unxlngx6* unxubti8* wntmsci12* unxlngi6* unxmacxi* ...
Detailed instructions on how to use ignore lists can be found here.
Speeding up svn+ssh: access by caching ssh connections
You might have notice that nearly all SVN operations are somewhat slower via svn+ssh: access than via plain (and read-only) svn: access. This is due to the ssh handshake, some SVN operations even suffer several times from the handshake overhead.
There is an easy and convenient way to get rid of this overhead by using a ssh connection cache.
The following description is taken more or less verbose from the the GCC wiki:
SSH connection caching works with OpenSSH 4.2 on all Unix look alikes, but not on Windows/cygwin. You can check your OpenSSH version with
$ ssh -V OpenSSH_4.6p1, OpenSSL 0.9.8e 23 Feb 2007
Create a ControlPath entry for svn.services.openoffice.org in $HOME/.ssh/config
$ cat $HOME/.ssh/config ... Host svn.services.openoffice.org ControlPath /tmp/ssh_ooo_control ...
Set the permissions 0600 on $HOME/.ssh/config, otherwise SSH refuses to work.
$ chmod 0600 $HOME/.ssh/config
Set up up a permanent connection to the ooo SVN server, for example right after login:
$ ssh -fMN email@example.com
Subsequent svn commands will use the already established connection, resulting in noticeably less lag for short commands. It's also an alternative to using ssh-agent if you have protected your private key with a pass phrase.
Create a OOo repository mirror
The repository can be replicated with the svnsync tool. No special server side setup is necessary (read-only access is sufficient, please use either the svn: or http: method), but you need to make certain that the target repository can't be modified by other means than svnsync.
- First create an empty target repository:
$ svnadmin create /absolute/path/to/rep
- Implement the pre-revprop-change and start-commit hooks
$ cat /absolute/path/to/rep/hooks/pre-revprop-change
#!/bin/sh USER="$3" if [ "$USER" = "syncuser" ]; then exit 0; fi echo "Only the syncuser user may change revision properties" >&2 exit 1
$ cat /absolute/path/to/rep/hooks/start-commit
#!/bin/sh USER="$2" if [ "$USER" = "syncuser" ]; then exit 0; fi echo "Only the syncuser user may commit new revisions" >&2 exit 1
- Initialize the target repository:
$ svnsync init --username syncuser file:///absolute/path/to/rep svn://svn.services.openoffice.org/ooo
- And finally synchronize the target repository with the source repository
$ svnsync synchronize file:///absolute/path/to/rep
The full details for replicating SVN repositories can be found [here].
Reporting Problems with the Repository
The list firstname.lastname@example.org is used to coordinate the initial migration and ongoing work on the repository and related tools. Please raise problems there first. Issues which need the attention of repository administrators should be filed in IssueZilla against the component www/openoffice.org SVN.
[More SVN Tips and tricks will appear here]