# @HEADER
#
########################################################################
#
#  Zoltan Toolkit for Load-balancing, Partitioning, Ordering and Coloring
#                  Copyright 2012 Sandia Corporation
#
# Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
# the U.S. Government retains certain rights in this software.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
#
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# 3. Neither the name of the Corporation nor the names of the
# contributors may be used to endorse or promote products derived from
# this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY SANDIA CORPORATION "AS IS" AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SANDIA CORPORATION OR THE
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# Questions? Contact Karen Devine	kddevin@sandia.gov
#                    Erik Boman	        egboman@sandia.gov
#
########################################################################
#
# @HEADER

Utilities to measure memory usage.

Included is a C++ function get_heap_usage() that measure current memory 
heap usage.  To use it, include "get_heap_usage.h" in your file, call
heap_size = get_heap_usage(), and compile with mpic++.  The get_heap_usage
function is defined for linux systems (including tbird) and mac systems.
Code exists for suns, but I couldn't make it work on seamus.

Programs commdup.c and commsplit.c perform repeated MPI_Comm_dup and 
MPI_Comm_split operations, followed by MPI_Comm_free, 
all surrounded by calls to get_heap_usage().  They print stats showing 
the heap usage before and after each MPI call.  They are used to determine
whether there are memory leaks resulting from MPI_Comm_dup and MPI_Comm_split.

In general, the first call to MPI_Comm_dup and MPI_Comm_split causes
memory leaks on every platform.  On macs and linux boxes running LAM MPI,
subsequent calls to MPI_Comm_dup and MPI_Comm_split do not cause leaks.

On tbird, however, with mpich and openmpi, subsequent calls to 
MPI_Comm_dup and MPI_Comm_split continue to cause memory leaks.  This
problem is significant for our RCB and PHG codes.  For RCB, using parameter
TFLOPS_SPECIAL=1 circumvents the problem.  For PHG, we have no work-around.

tbird environments tested:
  Default environment on log-in:
    > module list
      Currently Loaded Modulefiles:
        1) /opt/modules/oscar-modulefiles/default-manpath/1.0.1
        2) compilers/intel-9.1
        3) misc/env-openmpi
        4) mpi/openmpi-1.1.2-ofed-intel-9.1
        5) libraries/intel-mkl

  Mike Glass's environment:
    > module list
      Currently Loaded Modulefiles:
        1) compilers/gcc-3.4.6                
        2) misc/env-mvapich-devel
        3) mpi/mvapich-0.9.8-ofed-gcc-3.4.6



Program rcblike.c performs MPI_Comm_dup and MPI_Comm_split in a manner
similar to that in RCB (i.e., recursively splitting the communicator).
It shows similar results.
