wiki:pm-switch

Version 8 (modified by stefan, 9 years ago) (diff)

note on not using .pc file variables in xxx-uninstalled-pc.in files

Brief Tutorial on Switching to the New Buildtools

This tutorial attempts to explain how to make the switch to the new stable version of the build tools. The major changes with this release of build tools are:

  1. Support for the pkg-config command to build lists of dependencies and command line options. Configuration files associated with pkg-config are used when the pkg-config command is present.
  2. All dependent projects are treated more or less as "third-party", so that it is now easy to link against installed version of COIN-OR projects (using pkg-config if it is available). This also means that what were truly "third-party" projects are now treated more or less the same as COIN-OR projects. We provide build, pkg-config, and install support for these projects if source is available and otherwise provide a mechanism for linking to installed libraries, as before.

What Needs to be Edited

For the purposes of illustration, we shall assume that the project to be switched is Xxx. The project has been checked out with externals in the directory Root/ with project source code residing in the directory Root/Xxx. To make the switch, the following files need to be edited:

  • Root/configure.ac
  • Root/Makefile.am
  • Root/Xxx/configure.ac
  • Root/Xxx/Makefile.am
  • Root/Xxx/src/Makefile.am
  • Root/Xxx/test/Makefile.am
  • Root/Externals

the following files need to be created:

  • Root/Xxx/xxx.pc.in
  • Root/Xxx/xxx-uninstalled.pc.in
  • Root/Dependencies

and the following files need to be deleted:

  • Root/Xxx/xxx_addlibs.txt.in

There may also be changes needed in any examples that use installed versions of the COIN libraries, but these are a little less standardized, so it's difficult to say exactly what files would be affected.

Externals

At the moment, one must use only "trunk" versions for externals, since only the trunk version are using the new build tools. Here is the current set of externals for CoinAll:

BuildTools        https://projects.coin-or.org/svn/BuildTools/stable/0.6
ThirdParty/ASL    https://projects.coin-or.org/svn/BuildTools/ThirdParty/ASL/trunk
ThirdParty/Blas   https://projects.coin-or.org/svn/BuildTools/ThirdParty/Blas/trunk
ThirdParty/Lapack https://projects.coin-or.org/svn/BuildTools/ThirdParty/Lapack/trunk
ThirdParty/Glpk   https://projects.coin-or.org/svn/BuildTools/ThirdParty/Glpk/trunk
ThirdParty/FilterSQP https://projects.coin-or.org/svn/BuildTools/ThirdParty/FilterSQP/trunk
ThirdParty/HSL    https://projects.coin-or.org/svn/BuildTools/ThirdParty/HSL/trunk
ThirdParty/Mumps  https://projects.coin-or.org/svn/BuildTools/ThirdParty/Mumps/trunk
ThirdParty/Metis  https://projects.coin-or.org/svn/BuildTools/ThirdParty/Metis/trunk
Data/Sample       https://projects.coin-or.org/svn/Data/trunk/Sample
CoinUtils         https://projects.coin-or.org/svn/CoinUtils/trunk/CoinUtils
Osi               https://projects.coin-or.org/svn/Osi/trunk/Osi
Clp               https://projects.coin-or.org/svn/Clp/trunk/Clp
DyLP              https://projects.coin-or.org/svn/DyLP/trunk/DyLP
Vol               https://projects.coin-or.org/svn/Vol/trunk/Vol
Cgl               https://projects.coin-or.org/svn/Cgl/trunk/Cgl
SYMPHONY          https://projects.coin-or.org/svn/SYMPHONY/trunk/SYMPHONY
Cbc               https://projects.coin-or.org/svn/Cbc/trunk/Cbc
Ipopt             https://projects.coin-or.org/svn/Ipopt/trunk/Ipopt
Alps              https://projects.coin-or.org/svn/CHiPPS/Alps/trunk/Alps
Dip               https://projects.coin-or.org/svn/Dip/trunk/Dip

Another change with the new setup is that it is now highly recommended to use release versions in your externals (once they are there). To make this easy, there is a script called set_externals that is part of the build tools that will automatically set your externals. To use it, you maintain a "dependencies" file (usually called Root/Dependencies) as opposed to an "externals" file. The dependencies files will usually contain the stable versions on which a project depends in the same format as the current Externals file. When you run the command set_externals Dependencies, the dependencies files will be parsed and externals set to latest release versions automatically. Note that if your dependencies files contains trunk or specific release versions, these will be used instead, overriding the mechanism for using the latest release version. For example, here is the current Dependencies file for the trunk of Blis, which depends on the trunks of Alps and BiCePS.

BuildTools    https://projects.coin-or.org/svn/BuildTools/stable/0.5
ThirdParty/Blas https://projects.coin-or.org/svn/BuildTools/ThirdParty/Blas/stable/1.0
ThirdParty/Lapack https://projects.coin-or.org/svn/BuildTools/ThirdParty/Lapack/stable/1.0
CoinUtils     https://projects.coin-or.org/svn/CoinUtils/stable/2.6/CoinUtils
Clp           https://projects.coin-or.org/svn/Clp/stable/1.11/Clp
Osi           https://projects.coin-or.org/svn/Osi/stable/0.102/Osi
Cgl           https://projects.coin-or.org/svn/Cgl/stable/0.55/Cgl
Alps          https://projects.coin-or.org/svn/CHiPPS/Alps/trunk/Alps
Bcps          https://projects.coin-or.org/svn/CHiPPS/Bcps/trunk/Bcps
Data/Sample   https://projects.coin-or.org/svn/Data/stable/1.0/Sample

Root/configure.ac

In the Root/configure.ac file, the changes are as follows:

  1. Calls to AC_COIN_MAIN_SUBDIRS() should generally be changed to AC_COIN_MAIN_PACKAGEDIR(). For example, AC_COIN_MAIN_SUBDIRS(CoinUtils) => AC_COIN_MAIN_PACKAGEDIR(CoinUtils)
  2. Calls to AC_COIN_THIRDPARTY_SUBDIRS() should also be changed to AC_COIN_MAIN_PACKAGEDIR() (these packages are treated more or less the same way as COIN-OR packages now). One difference between COIN projects and third part projects is that the source is typically checked out into a subdirectory ThirdParty to make the separation clear. There is also a mechanism for checking to make sure that the project source is actually present (since this has to be downloaded separately by the user using the get.Yyy script. These capabilities are implemented using the optional second, third, and fourth arguments of AC_COIN_MAIN_PACKAGEDIR(), similar to the arguments of AC_COIN_THIRDPARTY_SUBDIRS() previously. The second argument denotes the subdirectory in which the project is to be found, the third argument is a file to check for to make sure that the source is present, and the fourth argument is the name of the pkg-config package associated with the installed library. So for BLAS, the call would be AC_COIN_MAIN_PACKAGEDIR(Blas,ThirdParty,[daxpy.f],coinblas).
  3. Note also that most calls to AC_COIN_HAS_USER_LIBRARY() for libraries like CPLEX are not needed in the configure.ac files in the root directory of higher-level projects, as these checks are made in the configure script of Osi and the list of libraries needed on the link line is automatically constructed and passed on to the higher-level project.

Root/Makefile.am

In the Root/Makefile.am file, the changes are as follows:

  1. The SUBDIRS variables does not need to be set to an explicit list of dependencies, the available subdirectories are determined automatically according to which projects are present in source form. So the command SUBDIRS = $(subdirs) replaces any command setting the value of SUBDIRS that was previously present.
  2. The command DISTCLEANFILES = coin_subdirs.txt should be set to clean up the file in which the list of subdirectories was written. Previously, this variable was set to the empty string.

Root/Xxx/configure.ac

  1. The function AC_COIN_PROJECTDIR_INIT now takes as an argument the name of the project, so the call would now be AC_COIN_PROJECTDIR_INIT(Xxx).
  2. Calls to AC_COIN_HAS_PROJECT() should be replaced with calls to AC_COIN_CHECK_PACKAGE(). There are also three arguments now.
    • The first argument is the name of the "package", which can consist of multiple COIN-OR projects. For example, one can defined a "package" consisting of both CoinUtils? and Clp, called CoinDepend and make the configuration bail out when any component of the package is not present. This mechanism is meant for cases in which a project is not useful by itself, but only in combination with others.
    • The second argument are the names the pkg-config files associated with the projects that make up the package, with the optional ability to specify specific required versions (see example below).
    • The third argument can be used to specify a particular library or binary to which the particular dependency applies. This is for cases where the project exports both a library and a binary executable and these have different dependencies. In this case, different command lines are built up for each of them separately.
    • An example of how this comes together is the call AC_COIN_CHECK_PACKAGE(CoinDepend, [coinutils = trunk osi = trunk alps = trunk], [DipLib]), which defines a package called CoinDepend that consists of the trunk versions of CoinUtils, Osi, and Alps. The dependency applies to the Dip library. We'll see below how this specification comes to play in the Makefiles.
  3. Calls to the function AC_COIN_HAS_USER_LIBRARY() can generally be replaced with calls to AC_COIN_CHECK_PACKAGE() if they apply to solvers to be used through Osi. This is because the OsiYyy packages now have pkg-config support. So the call to check for the presence of CPLEX, for example, is now AC_COIN_CHECK_PACKAGE(Cpx, [osi-cplex], [DipLib]). Other uses of the macros AC_COIN_HAS_USER_LIBRARY should be renamed to AC_COIN_CHECK_USER_LIBRARY.
  4. The list of files to be created by the configure script (listed as arguments to the AC_CONFIG_FILES command must now include Xxx.pc and Xxx-uninstalled.pc (see below for a description of the corresponding .in files).

Root/Xxx/Makefile.am

The main change here is that we are now replacing the old xxx_addlibs.txt file with xxx.pc and xxx-uninstalled.pc files (see below). If the pkg-config command is present, an addlibs file can be created in the old style from the .pc files. So we have the following changes:

addlibsfile = xxx_addlibs.txt 
addlibsdir = $(prefix)/share/doc/coin/$(PACKAGE_NAME) 

should be replaced with

pkgconfiglibdir = $(libdir)/pkgconfig 
pkgconfiglib_DATA = xxx.pc

The block

install-exec-local: install-doc 
        $(install_sh_DATA) $(addlibsfile) $(DESTDIR)$(addlibsdir)/$(addlibsfile) 

uninstall-local: uninstall-doc 
        rm -f $(addlibsdir)/$(addlibsfile) 

should be replaced with

if COIN_HAS_PKGCONFIG 
addlibsdir = $(DESTDIR)$(datadir)/coin/doc/Xxx

install-data-hook: 
       @$(mkdir_p) "$(addlibsdir)" 
       PKG_CONFIG_PATH=@COIN_PKG_CONFIG_PATH@ \ 
       $(PKG_CONFIG) --libs xxx > $(addlibsdir)/xxx_addlibs.txt 

uninstall-hook: 
        rm -f $(addlibsdir)/xxx_addlibs.txt 
endif 

Root/Xxx/xxx.pc.in

This is the template for the pkg-config file that will be produced at configure time. It has a very simple form something like the following.

prefix=@prefix@
exec_prefix=@exec_prefix@
libdir=@libdir@/coin
includedir=@includedir@/coin

Name: Xxx
Description: Xxx does nothing useful
Version: @PACKAGE_VERSION@
Libs: -L${libdir} -lXxx @XXXLIB_PCLIBS@
Cflags: -I${includedir}
Requires: @XXXLIB_PCREQUIRES@

Root/Xxx/xxx-uninstalled.pc.in

This the pkg-config file for finding uninstalled versions of a given library built from source. It's almost identical to the above with just a few path differences.

Also it is important to note that no .pc file variables should be used in the Libs and Cflags arguments, e.g., ${libdir} or ${includedir} as in the xxx.pc.in file. The reason is that if the case that pkg-config is not installed on a system, configure parsed the xxx-uninstalled.pc.in files itself, and this parsing code does not support variables in .pc files.

prefix=@prefix@

Name: Xxx
Description: Xxx does nothing useful
Version: @PACKAGE_VERSION@
Libs: -L@ABSBUILDDIR@/src -lXxx @XXXLIB_PCLIBS@
Cflags: -I@abs_source_dir@/src -I@ABSBUILDDIR@/inc
Requires: @XXXLIB_PCREQUIRES@

Root/Xxx/src/Makefile.am

The main difference in the construction of the various Makefile.am files is that not so much work has to be done now to build up the command lines and lists of dependent libraries. This is all done automatically. For each package Yyy defined in configure.ac, there are now two variables: YYY_CFLAGS and YYY_LIBS that hold the command line for compiling individual C and C++ files (i.e., the include paths, etc.) and the link line for linking to dependent libraries, respectively. The situation will be slightly different for each project, but for the most part, we can replace blocks like

AM_CPPFLAGS = \ 
        -I`$(CYGPATH_W) $(COINUTILSSRCDIR)/src` \ 
        -I`$(CYGPATH_W) $(COINUTILSOBJDIR)/inc` \ 
        -I`$(CYGPATH_W) $(OSISRCDIR)/src`       \ 
        -I`$(CYGPATH_W) $(OSISRCDIR)/inc`
        -I`$(CYGPATH_W) $(CLPSRCDIR)/src` \ 
        -I`$(CYGPATH_W) $(OSISRCDIR)/src/OsiClp`   \ 
        -I`$(CYGPATH_W) $(CLPOBJDIR)/inc` 
LDADD = $(COINUTILSOBJDIR)/src/libCoinUtils.la    \ 
        $(OSIOBJDIR)/src/libOsi.la    \ 
        $(OSIOBJDIR)/src/OsiClp/libOsiClp.la    \ 
        $(CLPOBJDIR)/src/libClp.la 

with something simpler like this:

AM_CPPFLAGS = $(COINDEPEND_CFLAGS)
LDADD = $(COINDEPEND_LIBS)

Note that the names of the variables correspond exactly to the names given to the libraries and binaries in the third argument AC_COIN_HAS_PACKAGE(), as described in the section on the Root/Xxx/configure.ac file above. Hence, the command AC_COIN_HAS_PACKAGE(CoinDepend, [coinutils = trunk osi = trunk alps = trunk], [XxxLib]) will result in the libraries and flags for each of those dependencies being put into variables called XXXLIB_LIBS and XXXLIB_CFLAGS respectively.

A final different is that there is no longer any need to use the ADDLIBS variable at all (it is not even defined), as the additional libraries are all included as dependencies when the lists of libraries are built up.

Root/Xxx/test/Makefile.am

The changes here are similar to those described above.

Root/Xxx/examples

Again, the main difference here is that one has to build from the new set of variables defined above in linking to the installed libraries. Alternatively, if you would like to check for or assume the presence of pkg-config on the system the examples will be built on, the most ideal thing would be to build your example Makefiles to show to use pkg-config, as this is the exact use case we are attempting to support.