Sets up and executes a communication plan for a Tpetra DistObject. More...

#include <Tpetra_Distributor.hpp>

Inherits Describable, and ParameterListAcceptorDefaultBase.

Public Member Functions
Constructors and destructor
	Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm)
	Construct using the specified communicator and default parameters. More...

	Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::FancyOStream > &out)
	Construct using the specified communicator and default parameters, with an output stream. More...

	Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::ParameterList > &plist)
	Construct using the specified communicator and ParameterList. More...

	Distributor (const Teuchos::RCP< const Teuchos::Comm< int > > &comm, const Teuchos::RCP< Teuchos::FancyOStream > &out, const Teuchos::RCP< Teuchos::ParameterList > &plist)
	Construct using the specified communicator and ParameterList, with an output stream. More...

	Distributor (const Distributor &distributor)
	Copy constructor. More...

virtual	~Distributor ()=default
	Destructor (virtual for memory safety). More...

void	swap (Distributor &rhs)
	Swap the contents of rhs with those of *this. More...

Implementation of ParameterListAcceptorDefaultBase
void	setParameterList (const Teuchos::RCP< Teuchos::ParameterList > &plist)
	Set Distributor parameters. More...

Teuchos::RCP< const Teuchos::ParameterList >	getValidParameters () const
	List of valid Distributor parameters. More...

Gather / scatter "constructors"
size_t	createFromSends (const Teuchos::ArrayView< const int > &exportProcIDs)
	Set up Distributor using list of process ranks to which this process will send. More...

template<class Ordinal >
void	createFromRecvs (const Teuchos::ArrayView< const Ordinal > &remoteIDs, const Teuchos::ArrayView< const int > &remoteProcIDs, Teuchos::Array< Ordinal > &exportIDs, Teuchos::Array< int > &exportProcIDs)
	Set up Distributor using list of process ranks from which to receive. More...

void	createFromSendsAndRecvs (const Teuchos::ArrayView< const int > &exportProcIDs, const Teuchos::ArrayView< const int > &remoteProcIDs)
	Set up Distributor using list of process ranks to which to send, and list of process ranks from which to receive. More...

Attribute accessor methods
size_t	getNumReceives () const
	The number of processes from which we will receive data. More...

size_t	getNumSends () const
	The number of processes to which we will send data. More...

bool	hasSelfMessage () const
	Whether the calling process will send or receive messages to itself. More...

size_t	getMaxSendLength () const
	Maximum number of values this process will send to another single process. More...

size_t	getTotalReceiveLength () const
	Total number of values this process will receive from other processes. More...

Teuchos::ArrayView< const int >	getProcsFrom () const
	Ranks of the processes sending values to this process. More...

Teuchos::ArrayView< const int >	getProcsTo () const
	Ranks of the processes to which this process will send values. More...

Teuchos::ArrayView< const size_t >	getLengthsFrom () const
	Number of values this process will receive from each process. More...

Teuchos::ArrayView< const size_t >	getLengthsTo () const
	Number of values this process will send to each process. More...

Details::EDistributorHowInitialized	howInitialized () const
	Return an enum indicating whether and how a Distributor was initialized. More...

Reverse communication methods
Teuchos::RCP< Distributor >	getReverse () const
	A reverse communication plan Distributor. More...

Methods for executing a communication plan
template<class Packet >
void	doPostsAndWaits (const Teuchos::ArrayView< const Packet > &exports, size_t numPackets, const Teuchos::ArrayView< Packet > &imports)
	Execute the (forward) communication plan. More...

template<class Packet >
void	doPostsAndWaits (const Teuchos::ArrayView< const Packet > &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const Teuchos::ArrayView< Packet > &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Execute the (forward) communication plan. More...

template<class Packet >
void	doPosts (const Teuchos::ArrayRCP< const Packet > &exports, size_t numPackets, const Teuchos::ArrayRCP< Packet > &imports)
	Post the data for a forward plan, but do not execute the waits yet. More...

template<class Packet >
void	doPosts (const Teuchos::ArrayRCP< const Packet > &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const Teuchos::ArrayRCP< Packet > &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Post the data for a forward plan, but do not execute the waits yet. More...

void	doWaits ()

template<class Packet >
void	doReversePostsAndWaits (const Teuchos::ArrayView< const Packet > &exports, size_t numPackets, const Teuchos::ArrayView< Packet > &imports)
	Execute the reverse communication plan. More...

template<class Packet >
void	doReversePostsAndWaits (const Teuchos::ArrayView< const Packet > &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const Teuchos::ArrayView< Packet > &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Execute the reverse communication plan. More...

template<class Packet >
void	doReversePosts (const Teuchos::ArrayRCP< const Packet > &exports, size_t numPackets, const Teuchos::ArrayRCP< Packet > &imports)
	Post the data for a reverse plan, but do not execute the waits yet. More...

template<class Packet >
void	doReversePosts (const Teuchos::ArrayRCP< const Packet > &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const Teuchos::ArrayRCP< Packet > &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Post the data for a reverse plan, but do not execute the waits yet. More...

void	doReverseWaits ()

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doPostsAndWaits (const ExpView &exports, size_t numPackets, const ImpView &imports)
	Execute the (forward) communication plan. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doPostsAndWaits (const ExpView &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const ImpView &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Execute the (forward) communication plan. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doPosts (const ExpView &exports, size_t numPackets, const ImpView &imports)
	Post the data for a forward plan, but do not execute the waits yet. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doPosts (const ExpView &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const ImpView &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Post the data for a forward plan, but do not execute the waits yet. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doReversePostsAndWaits (const ExpView &exports, size_t numPackets, const ImpView &imports)
	Execute the reverse communication plan. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doReversePostsAndWaits (const ExpView &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const ImpView &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Execute the reverse communication plan. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doReversePosts (const ExpView &exports, size_t numPackets, const ImpView &imports)
	Post the data for a reverse plan, but do not execute the waits yet. More...

template<class ExpView , class ImpView >
std::enable_if <(Kokkos::Impl::is_view < ExpView >::value &&Kokkos::Impl::is_view < ImpView >::value)>::type	doReversePosts (const ExpView &exports, const Teuchos::ArrayView< const size_t > &numExportPacketsPerLID, const ImpView &imports, const Teuchos::ArrayView< const size_t > &numImportPacketsPerLID)
	Post the data for a reverse plan, but do not execute the waits yet. More...

void	getLastDoStatistics (size_t &bytes_sent, size_t &bytes_recvd) const
	Information on the last call to do/doReverse. More...

Implementation of Teuchos::Describable
std::string	description () const
	Return a one-line description of this object. More...

void	describe (Teuchos::FancyOStream &out, const Teuchos::EVerbosityLevel verbLevel=Teuchos::Describable::verbLevel_default) const
	Describe this object in a human-readable way to the given output stream. More...

Detailed Description

Sets up and executes a communication plan for a Tpetra DistObject.

Note: Most Tpetra users do not need to know about this class.

This class encapsulates the general information and communication services needed for subclasses of DistObject (such as CrsMatrix and MultiVector) to do data redistribution (Import and Export) operations. It is an implementation detail of Import and Export; in particular; it actually does the communication.

Here is the typical way to use this class:

Create a Distributor. (The constructor is inexpensive.)
Set up the Distributor once, using one of the two "plan creation" methods: either createFromSends(), or createFromRecvs(). This may be more expensive and communication-intensive than Step 3.
Communicate the data by calling doPostsAndWaits() (forward mode), or doReversePostsAndWaits() (reverse mode). You may do this multiple times with the same Distributor instance.

Step 2 is expensive, but you can amortize its cost over multiple uses of the Distributor for communication (Step 3). You may also separate out "posts" (invoking nonblocking communication) and "waits" (waiting for that communication to complete), by calling doPosts() (resp. doReversePosts()), then doWaits() (resp. doReverseWaits()). This is useful if you have local work to do between the posts and waits, because it may overlap communication with computation. Whether it actually does overlap, depends on both the MPI implementation and your choice of parameters for the Distributor.

Instances of Distributor take the following parameters that control communication and debug output:

"Barrier between receives and sends" (bool): Whether to execute a barrier between receives and sends in do[Reverse]Posts(). A barrier is required for correctness when the "Send type" parameter is "Rsend". Otherwise, a barrier is correct and may be useful for debugging, but not recommended, since it introduces useless synchronization.
"Send type" (std::string): When using MPI, the variant of MPI_Send to use in do[Reverse]Posts(). Valid values include "Isend", "Rsend", "Send", and "Ssend". The default is "Send". (The receive type is always MPI_Irecv, a nonblocking receive. Since we post receives first before sends, this prevents deadlock, even if MPI_Send blocks and does not buffer.)
"Debug" (bool): If true, print copious debugging output on all processes in the Distributor's communicator. This is useful only for debugging Distributor and other Tpetra classes that use it (like Import and Export). If the Distributor was created using one of the constructors that takes a Teuchos::FancyOStream, it will write debugging output to that stream. Otherwise, it will write debugging output to stderr. Currently, the "Debug" parameter overrides "VerboseObject" (see below).
"VerboseObject" (sublist): Optional sublist for controlling behavior of Distributor as a Teuchos::VerboseObject. This is currently useful only for debugging. This sublist takes optional parameters "Verbosity Level" (std::string) and "Output File" (std::string). "Verbosity Level" has six valid values: "VERB_DEFAULT", "VERB_NONE", "VERB_LOW", "VERB_MEDIUM", "VERB_HIGH", and "VERB_EXTREME", with increasing verbosity starting with "VERB_NONE". "Output File" is the name of a file to use for output; "none" means don't open a file, but write to the default output stream.

Definition at line 190 of file Tpetra_Distributor.hpp.

Constructor & Destructor Documentation

Tpetra::Distributor::Distributor ( const Teuchos::RCP< const Teuchos::Comm< int > > & comm )

explicit

Construct using the specified communicator and default parameters.

Parameters

comm	[in] Communicator used by the Distributor. MUST be nonnull.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 174 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor	(	const Teuchos::RCP< const Teuchos::Comm< int > > &	comm,
		const Teuchos::RCP< Teuchos::FancyOStream > &	out
	)

Construct using the specified communicator and default parameters, with an output stream.

Parameters

comm	[in] Communicator used by the Distributor. MUST be nonnull.
out	[in/out] Output stream (for debugging output). If null, the default is `std::cerr`.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 179 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor	(	const Teuchos::RCP< const Teuchos::Comm< int > > &	comm,
		const Teuchos::RCP< Teuchos::ParameterList > &	plist
	)

Construct using the specified communicator and ParameterList.

Parameters

comm	[in] Communicator used by the Distributor. MUST be nonnull.
plist	[in/out] List of parameters controlling how the Distributor performs communication, and specifying debug options. If null, all parameters take their default values. Please see the class documentation for a list of all accepted parameters and their default values.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 185 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor	(	const Teuchos::RCP< const Teuchos::Comm< int > > &	comm,
		const Teuchos::RCP< Teuchos::FancyOStream > &	out,
		const Teuchos::RCP< Teuchos::ParameterList > &	plist
	)

Construct using the specified communicator and ParameterList, with an output stream.

Parameters

comm	[in] Communicator used by the Distributor. MUST be nonnull.
out	[in/out] Output stream (for debugging output). If null, the default is `std::cerr`.
plist	[in/out] List of parameters controlling how the Distributor performs communication, and specifying debug options. If null, all parameters take their default values. Please see the class documentation for a list of all accepted parameters and their default values.

The constructor doesn't actually set up the distribution pattern. You need to call one of the "gather / scatter 'constructors'" to do that.

Definition at line 147 of file Tpetra_Distributor.cpp.

Tpetra::Distributor::Distributor ( const Distributor & distributor )

Copy constructor.

Definition at line 191 of file Tpetra_Distributor.cpp.

virtual Tpetra::Distributor::~Distributor ( )

virtualdefault

Destructor (virtual for memory safety).

Precondition: No outstanding communication requests. (We could check, but see GitHub Issue #1303.)

Member Function Documentation

void Tpetra::Distributor::swap ( Distributor & rhs )

Swap the contents of rhs with those of *this.

This is useful in Import's setUnion() method. It avoids the overhead of copying arrays, since it can use std::swap on the arrays.

Definition at line 232 of file Tpetra_Distributor.cpp.

void Tpetra::Distributor::setParameterList ( const Teuchos::RCP< Teuchos::ParameterList > & plist )

Set Distributor parameters.

Please see the class documentation for a list of all accepted parameters and their default values.

Definition at line 281 of file Tpetra_Distributor.cpp.

Teuchos::RCP< const Teuchos::ParameterList > Tpetra::Distributor::getValidParameters ( ) const

List of valid Distributor parameters.

Please see the class documentation for a list of all accepted parameters and their default values.

Definition at line 352 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::createFromSends ( const Teuchos::ArrayView< const int > & exportProcIDs )

Set up Distributor using list of process ranks to which this process will send.

Take a list of process ranks and construct a plan for efficiently scattering to those processes. Return the number of processes which will send me (the calling process) data.

Parameters

exportProcIDs [in] List of ranks of the processes that will get the exported data. If there is a process rank greater than or equal to the number of processes, all processes will throw an std::runtime_error exception. Process ranks less than zero are ignored; their placement corresponds to null sends in any future exports. That is, if exportProcIDs[0] == -1, then the corresponding position in the export array is ignored during a call to doPosts() or doPostsAndWaits(). For this reason, a negative entry is sufficient to break contiguity.

Returns: Number of imports this process will be receiving.

Definition at line 1041 of file Tpetra_Distributor.cpp.

template<class Ordinal >

void Tpetra::Distributor::createFromRecvs	(	const Teuchos::ArrayView< const Ordinal > &	remoteIDs,
		const Teuchos::ArrayView< const int > &	remoteProcIDs,
		Teuchos::Array< Ordinal > &	exportIDs,
		Teuchos::Array< int > &	exportProcIDs
	)

Set up Distributor using list of process ranks from which to receive.

Take a list of process ranks and construct a plan for efficiently scattering to those processes. Return the number and list of IDs being sent by me (the calling process).

Import invokes this method in order to create a Distributor from a list of receive neighbors and IDs. A common use case for this process is setting up sends and receives for the remote entries of the source vector in a distributed sparse matrix-vector multiply. The Mantevo HPCCG miniapp shows an annotated and simplified version of this process for that special case.

Parameters

remoteIDs	[in] List of remote IDs wanted.
remoteProcIDs	[in] The ranks of the process that will send the remote IDs listed in `remoteIDs`. Process ranks less than zero are ignored; their placement corresponds to null sends in any future exports. If there is a process rank greater than or equal to the number of processes, all processes will throw an `std::runtime_error` exception.
exportIDs	[out] List of IDs that need to be sent from this process.
exportProcIDs	[out] The ranks of the processes that will get the exported IDs in `exportIDs`.

The exportGIDs and exportProcIDs arrays are resized by the Distributor, which is why they are passed in as a nonconst Teuchos::Array reference.

void Tpetra::Distributor::createFromSendsAndRecvs	(	const Teuchos::ArrayView< const int > &	exportProcIDs,
		const Teuchos::ArrayView< const int > &	remoteProcIDs
	)

Set up Distributor using list of process ranks to which to send, and list of process ranks from which to receive.

Parameters

exportProcIDs	[in] List of process ranks to which this process must send a message.
remoteProcIDs	[in] List of process ranks from which this process must receive a message.

Definition at line 1377 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getNumReceives ( ) const

The number of processes from which we will receive data.

The count does not include the calling process.

Definition at line 407 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getNumSends ( ) const

The number of processes to which we will send data.

The count does not include the calling process.

Definition at line 413 of file Tpetra_Distributor.cpp.

bool Tpetra::Distributor::hasSelfMessage ( ) const

Whether the calling process will send or receive messages to itself.

Definition at line 410 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getMaxSendLength ( ) const

Maximum number of values this process will send to another single process.

Definition at line 416 of file Tpetra_Distributor.cpp.

size_t Tpetra::Distributor::getTotalReceiveLength ( ) const

Total number of values this process will receive from other processes.

Definition at line 404 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getProcsFrom ( ) const

Ranks of the processes sending values to this process.

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 419 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const int > Tpetra::Distributor::getProcsTo ( ) const

Ranks of the processes to which this process will send values.

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 425 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsFrom ( ) const

Number of values this process will receive from each process.

This process will receive getLengthsFrom[i] values from process getProcsFrom[i].

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 422 of file Tpetra_Distributor.cpp.

Teuchos::ArrayView< const size_t > Tpetra::Distributor::getLengthsTo ( ) const

Number of values this process will send to each process.

This process will send getLengthsTo[i] values to process getProcsTo[i].

This is a nonpersisting view. It will last only as long as this Distributor instance does.

Definition at line 428 of file Tpetra_Distributor.cpp.

Details::EDistributorHowInitialized Tpetra::Distributor::howInitialized ( ) const

inline

Return an enum indicating whether and how a Distributor was initialized.

This is an implementation detail of Tpetra. Please do not call this method or rely on it existing in your code.

Definition at line 422 of file Tpetra_Distributor.hpp.

Teuchos::RCP< Distributor > Tpetra::Distributor::getReverse ( ) const

A reverse communication plan Distributor.

The first time this method is called, it creates a Distributor with the reverse communication plan of *this. On subsequent calls, it returns the cached reverse Distributor.

Most users do not need to call this method. If you invoke doReversePosts() or doReversePostsAndWaits(), the reverse Distributor will be created automatically if it does not yet exist.

Definition at line 432 of file Tpetra_Distributor.cpp.

template<class Packet >

void Tpetra::Distributor::doPostsAndWaits	(	const Teuchos::ArrayView< const Packet > &	exports,
		size_t	numPackets,
		const Teuchos::ArrayView< Packet > &	imports
	)

Execute the (forward) communication plan.

Call this version of the method when you have the same number of Packets for each LID (local ID) to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numPackets	[in] The number of Packets per export / import. This version of the routine assumes that each LID has the same number of Packets associated with it. (`MultiVector` is an example of a DistObject subclass satisfying this property.)
imports	[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.

Definition at line 1037 of file Tpetra_Distributor.hpp.

template<class Packet >

void Tpetra::Distributor::doPostsAndWaits	(	const Teuchos::ArrayView< const Packet > &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const Teuchos::ArrayView< Packet > &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Execute the (forward) communication plan.

Call this version of the method when you have possibly different numbers of Packets for each LID (local ID) to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numExportPacketsPerLID	[in] The number of packets for each export LID (i.e., each LID to be sent).
imports	[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.
numImportPacketsPerLID	[in] The number of packets for each import LID (i.e., each LID to be received).

Definition at line 1087 of file Tpetra_Distributor.hpp.

template<class Packet >

void Tpetra::Distributor::doPosts	(	const Teuchos::ArrayRCP< const Packet > &	exports,
		size_t	numPackets,
		const Teuchos::ArrayRCP< Packet > &	imports
	)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have the same number of Packets for each LID to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Contains the values to be sent by this process. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until `doWaits()` has completed.
numPackets	[in] The number of Packets per export / import. (Same as the three-argument version of doPostsAndWaits().)
imports	[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until `doWaits()` has completed. Upon completion of `doWaits()`, this buffer contains the values exported to us.

Definition at line 1133 of file Tpetra_Distributor.hpp.

template<class Packet >

void Tpetra::Distributor::doPosts	(	const Teuchos::ArrayRCP< const Packet > &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const Teuchos::ArrayRCP< Packet > &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have possibly different numbers of Packets for each LID to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Same as in the three-argument version of doPosts().
numExportPacketsPerLID	[in] Same as in the four-argument version of doPostsAndWaits().
imports	[out] Same as in the three-argument version of doPosts().
numImportPacketsPerLID	[in] Same as in the four-argument version of doPostsAndWaits().

Definition at line 1508 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doWaits ( )

Wait on any outstanding nonblocking message requests to complete.

This method is for forward mode communication only, that is, after calling doPosts(). For reverse mode communication (after calling doReversePosts()), call doReverseWaits() instead.

Definition at line 518 of file Tpetra_Distributor.cpp.

template<class Packet >

void Tpetra::Distributor::doReversePostsAndWaits	(	const Teuchos::ArrayView< const Packet > &	exports,
		size_t	numPackets,
		const Teuchos::ArrayView< Packet > &	imports
	)

Execute the reverse communication plan.

This method takes the same arguments as the three-argument version of doPostsAndWaits().

Definition at line 1876 of file Tpetra_Distributor.hpp.

template<class Packet >

void Tpetra::Distributor::doReversePostsAndWaits	(	const Teuchos::ArrayView< const Packet > &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const Teuchos::ArrayView< Packet > &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Execute the reverse communication plan.

This method takes the same arguments as the four-argument version of doPostsAndWaits().

Definition at line 1912 of file Tpetra_Distributor.hpp.

template<class Packet >

void Tpetra::Distributor::doReversePosts	(	const Teuchos::ArrayRCP< const Packet > &	exports,
		size_t	numPackets,
		const Teuchos::ArrayRCP< Packet > &	imports
	)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the three-argument version of doPosts().

Definition at line 1953 of file Tpetra_Distributor.hpp.

template<class Packet >

void Tpetra::Distributor::doReversePosts	(	const Teuchos::ArrayRCP< const Packet > &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const Teuchos::ArrayRCP< Packet > &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the four-argument version of doPosts().

Definition at line 1970 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::doReverseWaits ( )

Wait on any outstanding nonblocking message requests to complete.

This method is for reverse mode communication only, that is, after calling doReversePosts(). For forward mode communication (after calling doPosts()), call doWaits() instead.

Definition at line 585 of file Tpetra_Distributor.cpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doPostsAndWaits	(	const ExpView &	exports,
		size_t	numPackets,
		const ImpView &	imports
	)

Execute the (forward) communication plan.

Call this version of the method when you have the same number of Packets for each LID (local ID) to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numPackets	[in] The number of Packets per export / import. This version of the routine assumes that each LID has the same number of Packets associated with it. (`MultiVector` is an example of a DistObject subclass satisfying this property.)
imports	[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.

Definition at line 1990 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doPostsAndWaits	(	const ExpView &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const ImpView &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Execute the (forward) communication plan.

Call this version of the method when you have possibly different numbers of Packets for each LID (local ID) to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Contains the values to be sent by this process. On exit from this method, it's OK to modify the entries of this buffer.
numExportPacketsPerLID	[in] The number of packets for each export LID (i.e., each LID to be sent).
imports	[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. On exit, contains the values exported to us.
numImportPacketsPerLID	[in] The number of packets for each import LID (i.e., each LID to be received).

Definition at line 2038 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doPosts	(	const ExpView &	exports,
		size_t	numPackets,
		const ImpView &	imports
	)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have the same number of Packets for each LID to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Contains the values to be sent by this process. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until `doWaits()` has completed.
numPackets	[in] The number of Packets per export / import. (Same as the three-argument version of doPostsAndWaits().)
imports	[out] On entry, buffer must be large enough to accomodate the data exported (sent) to us. This is an ArrayRCP and not an ArrayView so that we have the freedom to use nonblocking sends if we wish. Do not modify the data in this array until `doWaits()` has completed. Upon completion of `doWaits()`, this buffer contains the values exported to us.

Definition at line 2057 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doPosts	(	const ExpView &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const ImpView &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Post the data for a forward plan, but do not execute the waits yet.

Call this overload when you have possibly different numbers of Packets for each LID to send or receive.

Template Parameters

Packet The type of data to send and receive.

Parameters

exports	[in] Same as in the three-argument version of doPosts().
numExportPacketsPerLID	[in] Same as in the four-argument version of doPostsAndWaits().
imports	[out] Same as in the three-argument version of doPosts().
numImportPacketsPerLID	[in] Same as in the four-argument version of doPostsAndWaits().

Definition at line 2503 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doReversePostsAndWaits	(	const ExpView &	exports,
		size_t	numPackets,
		const ImpView &	imports
	)

Execute the reverse communication plan.

This method takes the same arguments as the three-argument version of doPostsAndWaits().

Definition at line 2877 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doReversePostsAndWaits	(	const ExpView &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const ImpView &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Execute the reverse communication plan.

This method takes the same arguments as the four-argument version of doPostsAndWaits().

Definition at line 2888 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doReversePosts	(	const ExpView &	exports,
		size_t	numPackets,
		const ImpView &	imports
	)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the three-argument version of doPosts().

Definition at line 2906 of file Tpetra_Distributor.hpp.

template<class ExpView , class ImpView >

std::enable_if<(Kokkos::Impl::is_view< ExpView >::value &&Kokkos::Impl::is_view< ImpView >::value)>::type Tpetra::Distributor::doReversePosts	(	const ExpView &	exports,
		const Teuchos::ArrayView< const size_t > &	numExportPacketsPerLID,
		const ImpView &	imports,
		const Teuchos::ArrayView< const size_t > &	numImportPacketsPerLID
	)

Post the data for a reverse plan, but do not execute the waits yet.

This method takes the same arguments as the four-argument version of doPosts().

Definition at line 2924 of file Tpetra_Distributor.hpp.

void Tpetra::Distributor::getLastDoStatistics	(	size_t &	bytes_sent,
		size_t &	bytes_recvd
	)		const

inline

Information on the last call to do/doReverse.

Returns the amount of data sent & recv'd on this processor in bytes

Definition at line 768 of file Tpetra_Distributor.hpp.

std::string Tpetra::Distributor::description ( ) const

Return a one-line description of this object.

Definition at line 592 of file Tpetra_Distributor.cpp.

void Tpetra::Distributor::describe	(	Teuchos::FancyOStream &	out,
		const Teuchos::EVerbosityLevel	verbLevel = `Teuchos::Describable::verbLevel_default`
	)		const

Describe this object in a human-readable way to the given output stream.

You must call this method as a collective over all processes in this object's communicator.

Parameters

out	[out] Output stream to which to write. Only Process 0 in this object's communicator may write to the output stream.
verbLevel	[in] Verbosity level. This also controls whether this method does any communication. At verbosity levels higher (greater) than Teuchos::VERB_LOW, this method behaves as a collective over the object's communicator.

Teuchos::FancyOStream wraps std::ostream. It adds features like tab levels. If you just want to wrap std::cout, try this:

auto out = Teuchos::getFancyOStream (Teuchos::rcpFromRef (std::out));

Definition at line 662 of file Tpetra_Distributor.cpp.

The documentation for this class was generated from the following files:

Public Member Functions

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation