wiki:pm-svn-externals

Version 7 (modified by andreasw, 13 years ago) (diff)

added instructions to created backup copy of original external file when creating a release

Handling Subversion Externals

Externals allow subversion to download additional packages from other subversion repositories. For example, the COIN project Clp requires the COIN project CoinUtils to be compiled. Therefore, the Clp repository has externals defined so that CoinUtils is automatically downloaded when a user checks out Clp.

Basics

Externals are defined as a subversion property associated with a directory in a subversion repository (with the name svn:externals). If someone checks out such a directory, the corresponding externals (defined as URLs) are also checked out, into subdirectores of that directory. The names of those subdirectories, as well as the URL, are defined in the svn:externals property. It is possible to specify a specific revision number for an external. This helps us in COIN to make sure that people get a compatible version of the dependency; the latest development version in trunk of the dependency might not work with the code of the package one tries to download and compile, if the development of the dependency has proceeded. Also, we can ensure in this way that point releases will also always obtain the same version of their dependencies.

Externals are checked out recursively, i.e., if there is an svn:externals property defined in a directory downloaded for an external, it is also downloaded. One can suppress the checkout (or the action of other svn commands) of externals by specifying the --ignore-externals flag.

Externals in COIN

In COIN, we use externals mainly to make sure that COIN packages which require other COIN packages obtain the dependency code automatically.

Our policy for managing externals in COIN is that we put a file called Externals into the directory, from which the svn:externals property is set. For example, the Externals file in the base directory for the 1.3.3 point release of the Clp package looks like this:

MSVisualStudio -r83  https://projects.coin-or.org/svn/MSVisualStudio/trunk/ExternalsDirs/Clp
BuildTools    https://projects.coin-or.org/svn/BuildTools/releases/0.5.1
Data/Netlib   https://projects.coin-or.org/svn/Data/releases/1.0.0/Netlib
Data/Sample   https://projects.coin-or.org/svn/Data/releases/1.0.0/Sample
CoinUtils     https://projects.coin-or.org/svn/CoinUtils/releases/1.0.0/CoinUtils

The first entry in each row specifies the directory (relative to the current directory) into which the external is to be placed (this can specify several levels of subdirectories). After this, one can optionally specify the revision number of the dependency code that is to be obtained using the -rN flag, where N is the revision number. The last column is the URL that specifies the repository for the dependency.

Important Considerations For Externals

It is mandatory that the externals for a point release in a releases/ subdirectory specify dependencies that do not change at any later point in time, so that the original point release can always be recreated. We have no mechanism in place that enforces this, so we reply on the projects manager's discipline to adhere to this convention.

In the above example we see that all external have been set to things are not going to change at some later point in time: By convention, subdirectories in the releases/ directories for each COIN project are tags, i.e., they should never be changed once they have been created, so pointing to specific subdirectories there is Ok. If for some reason the required code is not available in a releases/ subdirectory, the externals definition must specify a particular subversion repostiroy revision number using the -r flag, such as for the MSVisualStudio entry above.

Manipulating Externals

To see the value of a property (such as svn:externals) one uses the svn propget command. For example, to see the current value set for the externals in the Clp base directory, you go into the Clp base directory and issue the command

svn propget svn:externals .

If you want to change the value of a property, you use the svn propset command. In COIN, we find it good practice to use the Externals file, so that users that don't have svn can also see the dependencies. You can edit the Externals file to add or remove external dependencies. Once you have edited the Externals file in a directory, you can use this file to update the svn:externals property with the command

svn propset svn:externals -F Externals .

The -F flag tells svn to take the content of the file as the value of the property to be set. Once the svn:externals property has been updated, new dependencies added to the Externals file will be downloaded at the next svn update command, and dependencies removed from the Externals file will be removed from subversion's records. (However, the directories must be removed by hand, after you've run svn update.)

Finally, if you decide to completely eliminate externals in a directory, you should delete the Externals file, and delete the svn:externals property with

svn propdel svn:externals .

Note: If you have configured your local copy with the --enable-maintainer-mode and have svn available on your system, the Makefiles will automatically do the svn propset command for you when you change the Externals file. However, you will need to run the svn update command by hand.

More information about externals can be obtained in the Externals Definitions chapter of the subversion book.

Preparing Externals For A Point Release

Typically, a COIN project that depends on other COIN projects will point to a stable branch of the dependencies, so that one will always receive the latest improvements and bugfixes of the dependencies. (By convention, the compilation should not break, since compatibility of the code should be ensure throughout a particular stable branch).

If you now want to create a new point release from the latest version in your stable branch, we suggest you follow these steps:

  1. Get a local copy of your stable branch (you probably have it already)
  2. Make a backup copy of your current Externals file
  3. For each dependency in your Externals file find the latest point release nuymber. You can do this for example using
    svn list https://projects.coin-or.org/svn/DepPrcjt/releases
    
    where DepPrcjt is the name of the dependency project.
  4. Change entries for all externals in the Externals file to point to the latest point releases.
  5. Update the actual subversion svn:externals property of the directory:
    svn pset svn:externals -F Externals .
    
  6. Update the local copies in your working copy by typing svn update. (Note: If you have local changes in your checked-out externals, you should get those committed into the dependecy's repository first. It is not a good idea in any case to have local modifications in your working copy of the stable branch, since you might forgot to commit them back, and then a user might have a non-working dependency.)
  7. Make sure your code now still works fine:
    1. clean everything (make clean)
    2. rebuild the autotools files (./BuildToolds/run_autotools)
    3. make sure (using svn status) that this did not change any files in the externals subdirectories. (If it did change, say, a configure file, it means that the dependency has not used the same version of BuildTools that you are using. This must be reconciled first.)
    4. rerun configure
    5. compile and install the code (make install)
    6. run your tests (make tests) and do whatever you do to convince yourself that it works.
  8. If that works fine, commit what is now in your local copy into back into the stable branch of your project (svn commit)
  9. As a sanity check, you might want to try if the code now in your stable branch compiles and runs fine on a different machine (svn update and make tests on the other machine)
  10. Create a copy of the current version of your stable branch as a new point release, as described here.
  11. Restore your original Externals file from your backup created in Step 2 above and restore the previous (non-release) externals:
    svn pset svn:externals -F Externals .
    
  12. Commmit the restored externals to your stable branch (svn commit)