Introduction to Zoltan2

Zoltan2 is a package for partitioning, load balancing and combinatorial scientific computing. It can be viewed as a complete redesign/refactoring of the well known Zoltan library into C++ in order to support template programming, to scale to larger problems, and to support architecture aware partitioning. This library is currently under active development. The capabilities in Zoltan2 roughly fall within the following areas (click on the link for a full description of an area and how Zoltan2 can be used in this context):

Building and Installing Zoltan2

Zoltan2 Build Basics

Zoltan2 is part of the Trilinos framework and uses the Trilinos CMake build system for configuration and builiding. In order to build Zoltan2, you need to use the Trilinos_ENABLE_Zoltan2 cmake directive. The most basic way to build Zoltan2 is to execute the following two commands in your build directory (typically distinct from the source directory):

% cmake \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    /path/to/Trilinos/src/directory 
% make

The first statement is the cmake configuration step that configures the build, creates Makefiles, etc. The second statement builds the Zoltan2 library and all required libraries. Zoltan2 requires several of the other Trilinos packages to be built. (Tpetra, Epetra, Teuchos, Xpetra and the packages that are required for those). The Trilinos framework should automatically enable these required packages during the configuration process if Zoltan2 is enabled.

Two important Trilinos configurations options of note relate to Fortran and MPI. If you do not have a fortran compiler installed, you should disable Fortran with the Trilinos_ENABLE_Fortran cmake directive. If you wish to have Zoltan run in a parallel (distributed memory) fashion, it is also important to enable MPI using the TPL_ENABLE_MPI directive. A very simple configure command for users with MPI but without Fortran is as follows:

% cmake \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D Trilinos_ENABLE_Fortran:BOOL=OFF \
    -D TPL_ENABLE_MPI:BOOL=ON \
    /path/to/Trilinos/src/directory

It is also important to note that by default the build system will enable all optional dependencies (Anasazi and Zoltan for Zoltan2). In order to reduce the compilation time of the Zoltan2 build, one may wish to disable these optional depencies using the Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES directive:

% cmake \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D Trilinos_ENABLE_Fortran:BOOL=OFF \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
    /path/to/Trilinos/src/directory

For more information on the Trilinos cmake build system see the Trilinos CMake Reference document (link can be found at http://trilinos.org/about/documentation/). In the next subsection, we discuss some of the Zoltan2 specific cmake directives and other Trilinos directives that directly impact the Zoltan2 functionality.

Third-Party Libraries (TPL)

Zoltan2 allows the optional use of several third-party libraries, which are not written nor maintained by the Zoltan2 team. To use a TPL, it must be explicitly enabled at configure time:

TPL_ENABLE_Scotch for graph partitioning/ordering using the Scotch package
TPL_ENABLE_ParMETIS for graph partitioning/ordering using the ParMETIS package
TPL_ENABLE_PaToH for hypergraph partitioning using the PaToH package
TPL_ENABLE_AMD for Approximate Minimum Degree ordering

One may also need to specify appropriate include and library paths, such as Scotch_INCLUDE_DIRS and Scotch_LIBRARY_DIRS. Example:

% cmake \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D TPL_ENABLE_Scotch:BOOL=ON \
-D Scotch_LIBRARY_DIRS:FILEPATH="/path/to/Scotch/lib" \
-D Scotch_INCLUDE_DIRS:FILEPATH="/path/to/Scotch/include" \
    /path/to/Trilinos/src/directory

Additional Zoltan2 Related Build Options

The following CMake configure directives can also impact Zoltan2 behavior:

Trilinos_ENABLE_OpenMP if enabled, use OpenMP for multithreaded execution. (Note only some parts of Zoltan2 can use OpenMP, and not all of Zoltan2 is thread-safe.)
Trilinos_ENABLE_EXPLICIT_INSTANTIATION if explicit instantiation is on, all Zoltan2 tests will be compiled with the instantiated types
Zoltan2_ENABLE_Experimental if enabled, code which is still under development will be available for use

These are some of the compilation flags used by Zoltan2:

Z2_OMIT_ALL_STATUS_MESSAGES The debug_level parameter controls the verbosity of status messages. When compiled with this option, the checks for debug_level are bypassed and all status output code is ignored.
Z2_OMIT_ALL_PROFILING Checks are done at runtime to determine whether any of the memory or timer parameters were set, prior to checking memory in use or to start or stop a ttimer. When compiled with this option, those checks are bypassed.
Z2_OMIT_ALL_ERROR_CHECKING The error_check_level parameter controls the amount of error checking done at runtime. When this flag is set, all error checking code is compiled out.

Zoltan2 Installation

To install Zoltan2 (and the other enabled Trilinos libraries) in a user specified location, the cmake directive CMAKE_INSTALL_PREFIX should be used as shown in the below installation example:

% cmake \
    -D Trilinos_ENABLE_Zoltan2:BOOL=ON \
    -D Trilinos_ENABLE_Fortran:BOOL=OFF \
    -D TPL_ENABLE_MPI:BOOL=ON \
    -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=OFF \
    -D CMAKE_INSTALL_PREFIX:PATH=/path/to/directory/of/installation \
    /path/to/Trilinos/src/directory 
% make
% make install

Once installed, many files important to Zoltan2 will be placed in this install directory:

What	Location
Header files	INSTALL_DIR/include
Libraries	INSTALL_DIR/lib
Makefile.export.Zoltan2	INSTALL_DIR/include
Zoltan2Config.cmake	INSTALL_DIR/lib/cmake/Zoltan2

If "make install" is executed without specifying the installation location, the build system will attempt to install the software in a common location (/usr/local at the time this was written).

Building Projects with Zoltan2

The above installation locations of header files and libraries are sufficient to be able to build an application using Zoltan2. However, for ease of use, there are two different methods that Trilinos (and thus Zoltan2) provide a listing of all the Trilinos libraries and third-party-libraries and other useful build information automatically (more fully described in the Trilinos documentation: http://trilinos.org/about/documentation/):

For cmake-aware projects, the cmake FIND_PACKAGE capability makes this information. Adding FIND_PACKAGE(Zoltan2) to your project's CMakeLists.txt file should fill in cmake variables Zoltan2_* that are needed for building with Zoltan2 (see http://trilinos.org/oldsite/Finding_Trilinos.txt for more details about usage). For instance,in the cmake variable Zoltan2_LIBRARIES will the the libraries that Zoltan depends upon and need to be linked by the applications. These libraries can be passed to the linker using the TARGET_LINK_LIBRARIES cmake directory.
The Makefile.export system is a way to build application code with Zoltan2 for applications that use simple Makefiles. Adding the line "include Makefile.export.Zoltan2" to your Makefile with import several variables Zoltan2_* that are needed for building with Zoltan2 (see http://trilinos.org/oldsite/Export_Makefile.txt for more details about usage). For instance,in the variable Zoltan2_LIBRARIES will the the libraries that Zoltan depends upon and need to be linked by the applications.

Overview of the library usage

Here we describe the typical user interaction with Zoltan2. We use partitioning as an example.

Create a Zoltan2::InputAdapter object for your data. This adapter provides a uniform interface to user data for the Zoltan2 library. Adapter interfaces exist for the following classes of data:
- Zoltan2::MatrixInput (a row oriented matrix with optional row weights)
- Zoltan2::GraphInput (an undirected weighted graph)
- Zoltan2::VectorInput (a single or multi-vector with optional weights, also used for geometric coordinates)
- Zoltan2::IdentifierInput (simple collection of identifiers)
- Zoltan2::MeshInput: available soon.
Create a Teuchos::ParameterList with your Zoltan2 parameters. If you are using a third party library (PT-Scotch, ParMetis) you can include a sublist of parameters for this library. See Zoltan2 Parameters for a detailed list of parameters.
Create a Zoltan2::PartitioningProblem. It is templated on your Zoltan2::InputAdapter type. The constructor arguments are typically your input adapter and your parameter list.
Call Zoltan2::PartitioningProblem::solve().
Obtain a Zoltan2::PartitioningSolution object from the Zoltan2::PartitioningProblem using the getSolution() function.

Contents