Base class for distributed Tpetra objects that support data redistribution. More...

#include <Tpetra_DistObject_decl.hpp>

Inheritance diagram for Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >:

Public Types
using	buffer_device_type = Kokkos::Device< typename device_type::execution_space, buffer_memory_space >
	Kokkos::Device specialization for communication buffers. More...

Typedefs
using	packet_type = typename::Kokkos::Details::ArithTraits< Packet >::val_type
	The type of each datum being sent or received in an Import or Export. More...

using	local_ordinal_type = LocalOrdinal
	The type of local indices. More...

using	global_ordinal_type = GlobalOrdinal
	The type of global indices. More...

using	node_type = Node
	The Node type. If you don't know what this is, don't use it. More...

using	device_type = typename Node::device_type
	The Kokkos Device type. More...

using	execution_space = typename device_type::execution_space
	The Kokkos execution space. More...

using	map_type = Map< local_ordinal_type, global_ordinal_type, node_type >
	The type of the Map specialization to use with this class. More...

Public Member Functions
Constructors and destructor
	DistObject (const Teuchos::RCP< const map_type > &map)
	Constructor. More...

	DistObject (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &)=default
	Copy constructor (default). More...

DistObject &	operator= (const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &)=default
	Assignment operator (default). More...

	DistObject (DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &&)=default
	Move constructor (default). More...

DistObject &	operator= (DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &&)=default
	Move assignment (default). More...

virtual	~DistObject ()=default
	Destructor (virtual for memory safety of derived classes). More...

Public methods for redistributing data
void	doImport (const SrcDistObject &source, const Import< LocalOrdinal, GlobalOrdinal, Node > &importer, const CombineMode CM, const bool restrictedMode=false)
	Import data into this object using an Import object ("forward mode"). More...

void	doExport (const SrcDistObject &source, const Export< LocalOrdinal, GlobalOrdinal, Node > &exporter, const CombineMode CM, const bool restrictedMode=false)
	Export data into this object using an Export object ("forward mode"). More...

void	doImport (const SrcDistObject &source, const Export< LocalOrdinal, GlobalOrdinal, Node > &exporter, const CombineMode CM, const bool restrictedMode=false)
	Import data into this object using an Export object ("reverse mode"). More...

void	doExport (const SrcDistObject &source, const Import< LocalOrdinal, GlobalOrdinal, Node > &importer, const CombineMode CM, const bool restrictedMode=false)
	Export data into this object using an Import object ("reverse mode"). More...

Attribute accessor methods
bool	isDistributed () const
	Whether this is a globally distributed object. More...

virtual Teuchos::RCP< const map_type >	getMap () const
	The Map describing the parallel distribution of this object. More...

I/O methods
void	print (std::ostream &os) const
	Print this object to the given output stream. More...

Implementation of Teuchos::Describable
virtual std::string	description () const
	One-line descriptiion of this object. More...

virtual void	describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const
	Print a descriptiion of this object to the given output stream. More...

Methods for use only by experts
virtual void	removeEmptyProcessesInPlace (const Teuchos::RCP< const map_type > &newMap)
	Remove processes which contain no entries in this object's Map. More...

Protected Types
enum	ReverseOption
	Whether the data transfer should be performed in forward or reverse mode. More...

using	buffer_memory_space = typename device_type::memory_space
	Kokkos memory space for communication buffers. More...

Protected Member Functions
virtual size_t	constantNumberOfPackets () const
	Whether the implementation's instance promises always to have a constant number of packets per LID (local index), and if so, how many packets per LID there are. More...

virtual void	doTransfer (const SrcDistObject &src, const ::Tpetra::Details::Transfer< local_ordinal_type, global_ordinal_type, node_type > &transfer, const char modeString[], const ReverseOption revOp, const CombineMode CM, const bool restrictedMode)
	Redistribute data across (MPI) processes. More...

virtual bool	reallocArraysForNumPacketsPerLid (const size_t numExportLIDs, const size_t numImportLIDs)
	Reallocate numExportPacketsPerLID_ and/or numImportPacketsPerLID_, if necessary. More...

virtual void	doTransferNew (const SrcDistObject &src, const CombineMode CM, const size_t numSameIDs, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &permuteToLIDs, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &permuteFromLIDs, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &remoteLIDs, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &exportLIDs, Distributor &distor, const ReverseOption revOp, const bool commOnHost, const bool restrictedMode)
	Implementation detail of doTransfer. More...

Methods implemented by subclasses and used by doTransfer().
The doTransfer() method uses the subclass' implementations of these methods to implement data transfer. Subclasses of DistObject must implement these methods. This is an instance of the Template Method Pattern. ("Template" here doesn't mean "C++ template"; it means "pattern with holes that are filled in by the subclass' method implementations.")
Teuchos::RCP< const map_type >	map_
	The Map over which this object is distributed. More...

Kokkos::DualView< packet_type *, buffer_device_type >	imports_
	Buffer into which packed data are imported (received from other processes). More...

Kokkos::DualView< size_t *, buffer_device_type >	numImportPacketsPerLID_
	Number of packets to receive for each receive operation. More...

Kokkos::DualView< packet_type *, buffer_device_type >	exports_
	Buffer from which packed data are exported (sent to other processes). More...

Kokkos::DualView< size_t *, buffer_device_type >	numExportPacketsPerLID_
	Number of packets to send for each send operation. More...

virtual bool	checkSizes (const SrcDistObject &source)=0
	Compare the source and target (this) objects for compatibility. More...

virtual void	copyAndPermute (const SrcDistObject &source, const size_t numSameIDs, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &permuteToLIDs, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &permuteFromLIDs)
	Perform copies and permutations that are local to the calling (MPI) process. More...

virtual void	packAndPrepare (const SrcDistObject &source, const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &exportLIDs, Kokkos::DualView< packet_type , buffer_device_type > &exports, Kokkos::DualView< size_t *, buffer_device_type > numPacketsPerLID, size_t &constantNumPackets, Distributor &distor)
	Pack data and metadata for communication (sends). More...

virtual void	unpackAndCombine (const Kokkos::DualView< const local_ordinal_type , buffer_device_type > &importLIDs, Kokkos::DualView< packet_type , buffer_device_type > imports, Kokkos::DualView< size_t *, buffer_device_type > numPacketsPerLID, const size_t constantNumPackets, Distributor &distor, const CombineMode combineMode)
	Perform any unpacking and combining after communication. More...

bool	reallocImportsIfNeeded (const size_t newSize, const bool verbose, const std::string *prefix)
	Reallocate imports_ if needed. More...

Detailed Description

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
class Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >

Base class for distributed Tpetra objects that support data redistribution.

DistObject is a base class for all Tpetra distributed global objects, including CrsGraph, CrsMatrix, MultiVector, and Vector. Classes that inherit from DistObject work as either the source or the target of an Export or Import (parallel redistribution) operation. If you want to write a class that can work as the source or target of an Export or Import operation, that class must inherit from DistObject.

Template Parameters

LocalOrdinal	The type of local indices. Same as Map's `LocalOrdinal` template parameter. See Map's documentation for a discussion of the types that are valid to use here.
GlobalOrdinal	The type of global indices. Same as Map's `GlobalOrdinal` template parameter. See Map's documentation for a discussion of the types that are valid to use here.
Node	Same as Map's `Node` template parameter. See Map's documentation for a discussion of the types that are valid to use here.

Summary

Most Tpetra users will only use this class' methods to perform data redistribution for subclasses such as CrsGraph, CrsMatrix, MultiVector, and Vector. DistObject provides four methods for redistributing data: two versions of doImport(), and two versions of doExport(). Import operations redistribute data from a nonoverlapping (one-to-one) distribution to a possibly overlapping distribution. Export operations redistribute data from a possibly overlapping distribution to a nonoverlapping (one-to-one) distribution. Once you have precomputed a data redistribution plan (an Import or Export object), you may use the plan to redistribute an input object's data into this object, by calling one of these methods. The input object of doImport() or doExport() is always the "source" of the redistribution operation, which sends the data. The *this object is the target, which receives and combines the data. It has the distribution given by this->getMap().

Forward or reverse redistribution modes

Both Import and Export operations occur in two modes: forward and reverse. Forward mode is the usual case, where you are calling a method with its matching plan type (doImport() for an Import plan, or doExport() for an Export plan). In that case, the input DistObject must have the same Map as the source Map of the plan, and the target DistObject must have the same Map as the target Map of the plan. Reverse mode is also possible, where you call a method with the opposite plan type (doImport() for an Export plan, or doExport() for an Import plan). In that case, the source DistObject's Map must be the same as the target Map of the plan, and the target DistObject's Map must be the same as the source Map of the plan. If you call doImport(), we still call this an Import operation, even if you are using an Export plan in reverse. Similarly, if you call doExport(), we call this an Export operation.

Most users will want to use forward mode. However, reverse mode is useful for some applications. For example, suppose you are solving a nonlinear partial differential equation using the finite element method, with Newton's method for the nonlinear equation. When assembling into a vector, it is convenient and efficient to do local assembly first into a vector with an overlapping distribution, then do global assembly via forward mode Export into a vector with a nonoverlapping distribution. After the linear solve, you may want to bring the resulting nonoverlapping distribution vector back to the overlapping distribution for another update phase. This would be a reverse mode Import, using the precomputed Export object.

Another use case for reverse mode is in CrsMatrix, for the transpose version of distributed sparse matrix-vector multiply ("mat-vec"). Non-transpose mat-vec (a function from the domain Map to the range Map) does an Import to bring in the source vector's data from the domain Map to the column Map of the sparse matrix, and an Export (if necessary) to bring the results from the row Map of the sparse matrix to the range Map. Transpose mat-vec (a function from the range Map to the domain Map) uses these precomputed Import and Export objects in reverse mode: first the Export in reverse mode to Import the source vector's data to the row Map, and then the Import in reverse mode to Export the results to the domain Map. Reverse mode lets us reuse the precomputed data redistribution plans for the transpose case.

How to implement a subclass

If you want to implement your own DistObject subclass, you must implement at least the following methods:

You may also implement constantNumberOfPackets(), if appropriate.

There is also an "old" interface, which is deprecated (see GitHub Issue #4853) and will be removed soon. Do not implement the "old" interface.

DistObject implements SrcDistObject, because we presume that if an object can be the target of an Import or Export, it can also be the source of an Import or Export.

Definition at line 326 of file Tpetra_DistObject_decl.hpp.

Member Typedef Documentation

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::packet_type = typename ::Kokkos::Details::ArithTraits<Packet>::val_type

The type of each datum being sent or received in an Import or Export.

Note that this type does not always correspond to the Scalar template parameter of subclasses.

Definition at line 338 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::local_ordinal_type = LocalOrdinal

The type of local indices.

Definition at line 340 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::global_ordinal_type = GlobalOrdinal

The type of global indices.

Definition at line 342 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::node_type = Node

The Node type. If you don't know what this is, don't use it.

Definition at line 344 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::device_type = typename Node::device_type

The Kokkos Device type.

Definition at line 347 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::execution_space = typename device_type::execution_space

The Kokkos execution space.

Definition at line 349 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

using Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::map_type = Map<local_ordinal_type, global_ordinal_type, node_type>

The type of the Map specialization to use with this class.

Definition at line 352 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::buffer_memory_space

protected

Kokkos memory space for communication buffers.

See #1088 for why this is not just device_type::memory_space.

Definition at line 733 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::buffer_device_type

Kokkos::Device specialization for communication buffers.

See #1088 for why this is not just device_type::device_type.

This needs to be public so that I can declare functions like packAndPrepareWithOwningPIDs.

Warning: This is an implementation detail. DO NOT DEPEND ON IT. It may disappear or change at any time.

Definition at line 749 of file Tpetra_DistObject_decl.hpp.

Member Enumeration Documentation

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

enum Tpetra::DistObject::ReverseOption

protected

Whether the data transfer should be performed in forward or reverse mode.

"Reverse mode" means calling doExport() with an Import object, or calling doImport() with an Export object. "Forward mode" means calling doExport() with an Export object, or calling doImport() with an Import object.

Definition at line 636 of file Tpetra_DistObject_decl.hpp.

Constructor & Destructor Documentation

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject ( const Teuchos::RCP< const map_type > & map )

explicit

Constructor.

Parameters

map	[in] Map over which the object is distributed.

Definition at line 96 of file Tpetra_DistObject_def.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & )

default

Copy constructor (default).

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::DistObject ( DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > && )

default

Move constructor (default).

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

virtual Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::~DistObject ( )

virtualdefault

Destructor (virtual for memory safety of derived classes).

Note: To Tpetra developers: See the C++ Core Guidelines C.21 ("If you define or <tt>=delete</tt> any default operation, define or <tt>=delete</tt> them all"), in particular the AbstractBase example, for why this destructor declaration implies that we need the above four =default declarations for copy construction, move construction, copy assignment, and move assignment.

Member Function Documentation

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

DistObject& Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::operator= ( const DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & )

default

Assignment operator (default).

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

DistObject& Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::operator= ( DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > && )

default

Move assignment (default).

template<class Packet , class LocalOrdinal, class GlobalOrdinal, class Node>

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doImport	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	source,
		const Import< LocalOrdinal, GlobalOrdinal, Node > &	importer,
		const CombineMode	CM,
		const bool	restrictedMode = `false`
	)

Import data into this object using an Import object ("forward mode").

The input DistObject is always the source of the data redistribution operation, and the *this object is always the target.

If you don't know the difference between forward and reverse mode, then you probably want forward mode. Use this method with your precomputed Import object if you want to do an Import, else use doExport() with a precomputed Export object.

"Restricted Mode" does two things:

Skips copyAndPermute
Allows the "target" Map of the transfer to be a subset of the Map of *this, in a "locallyFitted" sense.

This cannot be used if (2) is not true, OR there are permutes. The "source" maps still need to match.

Parameters

source	[in] The "source" object for redistribution.
importer	[in] Precomputed data redistribution plan. Its source Map must be the same as the input DistObject's Map, and its target Map must be the same as `this->getMap()`.
CM	[in] How to combine incoming data with the same global index.

Definition at line 274 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal, class GlobalOrdinal, class Node>

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doExport	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	source,
		const Export< LocalOrdinal, GlobalOrdinal, Node > &	exporter,
		const CombineMode	CM,
		const bool	restrictedMode = `false`
	)

Export data into this object using an Export object ("forward mode").

The input DistObject is always the source of the data redistribution operation, and the *this object is always the target.

If you don't know the difference between forward and reverse mode, then you probably want forward mode. Use this method with your precomputed Export object if you want to do an Export, else use doImport() with a precomputed Import object.

"Restricted Mode" does two things:

Skips copyAndPermute
Allows the "target" Map of the transfer to be a subset of the Map of *this, in a "locallyFitted" sense.

This cannot be used if (2) is not true, OR there are permutes. The "source" maps still need to match.

Parameters

source	[in] The "source" object for redistribution.
exporter	[in] Precomputed data redistribution plan. Its source Map must be the same as the input DistObject's Map, and its target Map must be the same as `this->getMap()`.
CM	[in] How to combine incoming data with the same global index.

Definition at line 316 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal, class GlobalOrdinal, class Node>

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doImport	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	source,
		const Export< LocalOrdinal, GlobalOrdinal, Node > &	exporter,
		const CombineMode	CM,
		const bool	restrictedMode = `false`
	)

Import data into this object using an Export object ("reverse mode").

The input DistObject is always the source of the data redistribution operation, and the *this object is always the target.

If you don't know the difference between forward and reverse mode, then you probably want forward mode. Use the version of doImport() that takes a precomputed Import object in that case.

"Restricted Mode" does two things:

Skips copyAndPermute
Allows the "target" Map of the transfer to be a subset of the Map of *this, in a "locallyFitted" sense.

This cannot be used if (2) is not true, OR there are permutes. The "source" maps still need to match.

Parameters

source	[in] The "source" object for redistribution.
exporter	[in] Precomputed data redistribution plan. Its target Map must be the same as the input DistObject's Map, and its source Map must be the same as `this->getMap()`. (Note the difference from forward mode.)
CM	[in] How to combine incoming data with the same global index.

Definition at line 359 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal, class GlobalOrdinal, class Node>

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doExport	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	source,
		const Import< LocalOrdinal, GlobalOrdinal, Node > &	importer,
		const CombineMode	CM,
		const bool	restrictedMode = `false`
	)

Export data into this object using an Import object ("reverse mode").

The input DistObject is always the source of the data redistribution operation, and the *this object is always the target.

If you don't know the difference between forward and reverse mode, then you probably want forward mode. Use the version of doExport() that takes a precomputed Export object in that case.

"Restricted Mode" does two things:

Skips copyAndPermute
Allows the "target" Map of the transfer to be a subset of the Map of *this, in a "locallyFitted" sense.

This cannot be used if (2) is not true, OR there are permutes. The "source" maps still need to match.

Parameters

source	[in] The "source" object for redistribution.
importer	[in] Precomputed data redistribution plan. Its target Map must be the same as the input DistObject's Map, and its source Map must be the same as `this->getMap()`. (Note the difference from forward mode.)
CM	[in] How to combine incoming data with the same global index.

Definition at line 402 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::isDistributed ( ) const

Whether this is a globally distributed object.

For a definition of "globally distributed" (and its opposite, "locally replicated"), see the documentation of Map's isDistributed() method.

Definition at line 445 of file Tpetra_DistObject_def.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

virtual Teuchos::RCP<const map_type> Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::getMap ( ) const

inlinevirtual

The Map describing the parallel distribution of this object.

Note that some Tpetra objects might be distributed using multiple Map objects. For example, CrsMatrix has both a row Map and a column Map. It is up to the subclass to decide which Map to use when invoking the DistObject constructor.

Definition at line 541 of file Tpetra_DistObject_decl.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::print ( std::ostream & os ) const

Print this object to the given output stream.

We generally assume that all MPI processes can print to the given stream.

Definition at line 1933 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

std::string Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::description ( ) const

virtual

One-line descriptiion of this object.

We declare this method virtual so that subclasses of DistObject may override it.

Reimplemented in Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LO, GO, Node >, Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsGraph< LO, GO, node_type >, Tpetra::Details::CooMatrix< SC, LO, GO, NT >, Tpetra::Vector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, and Tpetra::Experimental::BlockCrsMatrix< Scalar, LO, GO, Node >.

Definition at line 149 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::describe	(	Teuchos::FancyOStream &	out,
		const Teuchos::EVerbosityLevel	verbLevel = `Teuchos::Describable::verbLevel_default`
	)		const

virtual

Print a descriptiion of this object to the given output stream.

We declare this method virtual so that subclasses of Distobject may override it.

Reimplemented in Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LO, GO, Node >, Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsGraph< LO, GO, node_type >, Tpetra::Details::CooMatrix< SC, LO, GO, NT >, Tpetra::Vector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, and Tpetra::Experimental::BlockCrsMatrix< Scalar, LO, GO, Node >.

Definition at line 169 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::removeEmptyProcessesInPlace ( const Teuchos::RCP< const map_type > & newMap )

virtual

Remove processes which contain no entries in this object's Map.

Warning: This method is ONLY for use by experts. We highly recommend using the nonmember function of the same name defined in this file.; We make NO promises of backwards compatibility. This method may change or disappear at any time.

On input, this object is distributed over the Map returned by getMap() (the "original Map," with its communicator, the "original communicator"). The input newMap of this method must be the same as the result of calling getMap()->removeEmptyProcesses(). On processes in the original communicator which contain zero entries ("excluded processes," as opposed to "included processes"), the input newMap must be Teuchos::null (which is what getMap()->removeEmptyProcesses() returns anyway).

On included processes, reassign this object's Map (that would be returned by getMap()) to the input newMap, and do any work that needs to be done to restore correct semantics. On excluded processes, free any data that needs freeing, and do any other work that needs to be done to restore correct semantics.

This method has collective semantics over the original communicator. On exit, the only method of this object which is safe to call on excluded processes is the destructor. This implies that subclasses' destructors must not contain communication operations.

Returns: The object's new Map. Its communicator is a new communicator, distinct from the old Map's communicator, which contains a subset of the processes in the old communicator.

Note: The name differs from Map's method removeEmptyProcesses(), in order to emphasize that the operation on DistObject happens in place, modifying the input, whereas the operation removeEmptyProcess() on Map does not modify the input.; To implementers of DistObject subclasses: The default implementation of this class throws std::logic_error.

Reimplemented in Tpetra::MultiVector< Scalar, LO, GO, Node >.

Definition at line 236 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

size_t Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::constantNumberOfPackets ( ) const

protectedvirtual

Whether the implementation's instance promises always to have a constant number of packets per LID (local index), and if so, how many packets per LID there are.

If this method returns zero, the instance says that it might possibly have a different number of packets for each LID (local index) to send or receive. If it returns nonzero, the instance promises that the number of packets is the same for all LIDs, and that the return value is this number of packets per LID.

The default implementation of this method returns zero. This does not affect the behavior of doTransfer() in any way. If a nondefault implementation returns nonzero, doTransfer() will use this information to avoid unnecessary allocation and / or resizing of arrays.

Reimplemented in Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LO, GO, Node >, and Tpetra::Details::CooMatrix< SC, LO, GO, NT >.

Definition at line 452 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doTransfer	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	src,
		const ::Tpetra::Details::Transfer< local_ordinal_type, global_ordinal_type, node_type > &	transfer,
		const char	modeString[],
		const ReverseOption	revOp,
		const CombineMode	CM,
		const bool	restrictedMode
	)

protectedvirtual

Redistribute data across (MPI) processes.

Parameters

src	[in] The source object, to redistribute into the target object, which is `*this` object.
transfer	[in] The Export or Import object representing the communication pattern. (Details::Transfer is the common base class of these two objects.)
modeString	[in] Human-readable string, for verbose debugging output and error output, explaining what function called this method. Example: "doImport (forward)", "doExport (reverse)".
revOp	[in] Whether to do a forward or reverse mode redistribution.
CM	[in] The combine mode that describes how to combine values that map to the same global ID on the same process.

Definition at line 459 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::reallocArraysForNumPacketsPerLid	(	const size_t	numExportLIDs,
		const size_t	numImportLIDs
	)

protectedvirtual

Reallocate numExportPacketsPerLID_ and/or numImportPacketsPerLID_, if necessary.

Parameters

numExportLIDs	[in] Number of entries in the exportLIDs input array argument of doTransfer().
numImportLIDs	[in] Number of entries in the remoteLIDs input array argument of doTransfer().

Returns: Whether we actually reallocated either of the arrays.

Warning: This is an implementation detail of doTransferNew(). This needs to be protected, but that doesn't mean users should call this method.

Definition at line 662 of file Tpetra_DistObject_def.hpp.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::doTransferNew	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	src,
		const CombineMode	CM,
		const size_t	numSameIDs,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	permuteToLIDs,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	permuteFromLIDs,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	remoteLIDs,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	exportLIDs,
		Distributor &	distor,
		const ReverseOption	revOp,
		const bool	commOnHost,
		const bool	restrictedMode
	)

protectedvirtual

Implementation detail of doTransfer.

LID DualViews come from the Transfer object given to doTransfer. They are always sync'd on both host and device. Users must never attempt to modify or sync them.

Definition at line 743 of file Tpetra_DistObject_def.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

virtual bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::checkSizes ( const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > & source )

protectedpure virtual

Compare the source and target (this) objects for compatibility.

Returns: True if they are compatible, else false.

Implemented in Tpetra::CrsMatrix< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LocalOrdinal, GlobalOrdinal, Node >, Tpetra::MultiVector< Scalar, LO, GO, Node >, Tpetra::CrsGraph< LocalOrdinal, GlobalOrdinal, Node >, Tpetra::CrsGraph< LO, GO, node_type >, and Tpetra::Experimental::BlockMultiVector< Scalar, LO, GO, Node >.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::copyAndPermute	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	source,
		const size_t	numSameIDs,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	permuteToLIDs,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	permuteFromLIDs
	)

protectedvirtual

Perform copies and permutations that are local to the calling (MPI) process.

Subclasses must reimplement this function. Its default implementation does nothing. Note that the <t>target object of the Export or Import, namely *this, packs the source object's data.

Precondition: permuteToLIDs and permuteFromLIDs are sync'd to both host and device. That is, permuteToLIDs.need_sync_host(), permuteToLIDs.need_sync_device(), permuteFromLIDs.need_sync_host(), and permuteFromLIDs.need_sync_device() are all false.

Parameters

source	[in] On entry, the source object of the Export or Import operation.
numSameIDs	[in] The number of elements that are the same on the source and target objects. These elements live on the same process in both the source and target objects.
permuteToLIDs	[in] List of the elements that are permuted. They are listed by their local index (LID) in the destination object.
permuteFromLIDs	[in] List of the elements that are permuted. They are listed by their local index (LID) in the source object.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::packAndPrepare	(	const SrcDistObject< Packet, LocalOrdinal, GlobalOrdinal, Node > &	source,
		const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	exportLIDs,
		Kokkos::DualView< packet_type *, buffer_device_type > &	exports,
		Kokkos::DualView< size_t *, buffer_device_type >	numPacketsPerLID,
		size_t &	constantNumPackets,
		Distributor &	distor
	)

protectedvirtual

Pack data and metadata for communication (sends).

Subclasses must reimplement this function. Its default implementation does nothing. Note that the <t>target object of the Export or Import, namely *this, packs the source object's data.

Precondition: exportLIDs is sync'd to both host and device. That is, exportLIDs.need_sync_host () and exportLIDs.need_sync_device() are both false.

Parameters

source	[in] Source object for the redistribution.
exportLIDs	[in] List of the entries (as local IDs in the source object) that Tpetra will send to other processes.
exports	[out] On exit, the packed data to send. Implementations must reallocate this as needed (prefer reusing the existing allocation if possible), and may modify and/or sync this wherever they like.
numPacketsPerLID	[out] On exit, the implementation of this method must do one of two things: either set `numPacketsPerLID[i]` to the number of packets to be packed for `exportLIDs[i]` and set `constantNumPackets` to zero, or set `constantNumPackets` to a nonzero value. If the latter, the implementation must not modify the entries of `numPacketsPerLID`. If the former, the implementation may sync `numPacketsPerLID` this wherever it likes, either to host or to device. The allocation belongs to DistObject, not to subclasses; don't be tempted to change this to pass by reference.
constantNumPackets	[out] On exit, 0 if the number of packets per LID could differ, else (if nonzero) the number of packets per LID (which must be constant).
distor	[in] The Distributor object we are using. Most implementations will not use this.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

virtual void Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::unpackAndCombine	(	const Kokkos::DualView< const local_ordinal_type *, buffer_device_type > &	importLIDs,
		Kokkos::DualView< packet_type *, buffer_device_type >	imports,
		Kokkos::DualView< size_t *, buffer_device_type >	numPacketsPerLID,
		const size_t	constantNumPackets,
		Distributor &	distor,
		const CombineMode	combineMode
	)

protectedvirtual

Perform any unpacking and combining after communication.

Subclasses must reimplement this function. Its default implementation does nothing. Note that the <t>target object of the Export or Import, namely *this, unpacks the received data into itself, possibly modifying its entries.

Precondition: importLIDs is sync'd to both host and device. That is, importLIDs.need_sync_host () and importLIDs.need_sync_device() are both false.

Parameters

importLIDs	[in] List of the entries (as LIDs in the destination object) we received from other processes.
imports	[in/out] On input: Buffer of received data to unpack. DistObject promises nothing about where this is sync'd. Implementations may sync this wherever they like, either to host or to device. The allocation belongs to DistObject, not to subclasses; don't be tempted to change this to pass by reference.
numPacketsPerLID	[in/out] On input: If `constantNumPackets` is zero, then `numPacketsPerLID[i]` contains the number of packets imported for importLIDs[i]. DistObject promises nothing about where this is sync'd. Implementations may sync this wherever they like, either to host or to device. The allocation belongs to DistObject, not to subclasses; don't be tempted to change this to pass by reference.
constantNumPackets	[in] If nonzero, then the number of packets per LID is the same for all entries ("constant") and `constantNumPackets` is that number. If zero, then `numPacketsPerLID[i]` is the number of packets to unpack for LID `importLIDs[i]`.
distor	[in] The Distributor object we are using. Most implementations will not use this.
combineMode	[in] The CombineMode to use when combining the imported entries with existing entries.

template<class Packet , class LocalOrdinal , class GlobalOrdinal , class Node >

bool Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::reallocImportsIfNeeded	(	const size_t	newSize,
		const bool	verbose,
		const std::string *	prefix
	)

protected

Reallocate imports_ if needed.

This unfortunately must be declared protected, for the same reason that imports_ is declared protected.

Parameters

newSize	[in] New size of imports_.
verbose	[in] Whether to print verbose debugging output to stderr on every (MPI) process in the communicator.
prefix	[in] If `verbose` is `true`, then this is a nonnull prefix to print at the beginning of each line of verbose debugging output. Otherwise, not used.

Returns: Whether we actually reallocated.

We don't need a "reallocExportsIfNeeded" method, because exports_ always gets passed into packAndPrepare() by nonconst reference. Thus, that method can resize the DualView without needing to call other DistObject methods.

Definition at line 638 of file Tpetra_DistObject_def.hpp.

Member Data Documentation

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Teuchos::RCP<const map_type> Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::map_

protected

The Map over which this object is distributed.

Definition at line 1140 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Kokkos::DualView<packet_type*, buffer_device_type> Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::imports_

protected

Buffer into which packed data are imported (received from other processes).

Unfortunately, I had to declare these protected, because CrsMatrix uses them at one point. Please, nobody else use them.

Definition at line 1149 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Kokkos::DualView<size_t*, buffer_device_type> Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::numImportPacketsPerLID_

protected

Number of packets to receive for each receive operation.

This array is used in Distributor::doPosts() (and doReversePosts()) when starting the ireceive operation.

This may be ignored in doTransfer() if constantNumPackets is nonzero, indicating a constant number of packets per LID. (For example, MultiVector sets the constantNumPackets output argument of packAndPrepare() to the number of columns in the multivector.)

Unfortunately, I had to declare this protected, because CrsMatrix uses it at one point. Please, nobody else use it.

Definition at line 1187 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Kokkos::DualView<packet_type*, buffer_device_type> Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::exports_

protected

Buffer from which packed data are exported (sent to other processes).

Unfortunately, I had to declare this protected, because CrsMatrix uses it at one point. Please, nobody else use it.

Definition at line 1194 of file Tpetra_DistObject_decl.hpp.

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>

Kokkos::DualView<size_t*, buffer_device_type> Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >::numExportPacketsPerLID_

protected

Number of packets to send for each send operation.

This array is used in Distributor::doPosts() (and doReversePosts()) for preparing for the send operation.

This may be ignored in doTransfer() if constantNumPackets is nonzero, indicating a constant number of packets per LID. (For example, MultiVector sets the constantNumPackets output argument of packAndPrepare() to the number of columns in the multivector.)

Unfortunately, I had to declare this protected, because CrsMatrix uses them at one point. Please, nobody else use it.

Definition at line 1209 of file Tpetra_DistObject_decl.hpp.

The documentation for this class was generated from the following files:

Public Types

Public Member Functions

Protected Types

Protected Member Functions

Methods implemented by subclasses and used by doTransfer().

Detailed Description

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node> class Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >

Summary

Forward or reverse redistribution modes

How to implement a subclass

Member Typedef Documentation

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation

template<class Packet, class LocalOrdinal, class GlobalOrdinal, class Node>
class Tpetra::DistObject< Packet, LocalOrdinal, GlobalOrdinal, Node >