Memo‎ > ‎

Install mpi4py on Amazon EC2

posted May 29, 2014, 8:00 AM by Teng-Yok Lee   [ updated May 29, 2014, 8:36 AM ]
It sounds very simple

# REF: http://mpi4py.scipy.org/docs/usrman/install.html
$ sudo pip install mpi4py

Nevertheless, as my EC2 instance use openmpi-devel installed via yum, it cannot find mpi.h. Later I searched for the installation path and found it:

# REF: http://stackoverflow.com/questions/1766380/determining-the-path-that-a-yum-package-installed-to
$ rpm -ql openmpi-devel | grep mpi.h
/usr/include/openmpi-x86_64/mpi.h

Then how to run pip with the specified mpi.h?

NOTE: mpi4py will look for pyconfig.h and Python.h, which are only available in python2.7.

As a result, build from the source is needed. The following steps are my procedure:
$ wget http://mpi4py.googlecode.com/files/mpi4py-1.3.1.tar.gz
$ tar zxvf mpi4py-1.3.1.tar.gz
# Edit the mpi section in mpi4py-1.3.1/mpi.cfg as follows

## define_macros        =
## undef_macros         =
## include_dirs         = %(mpi_dir)s/include
include_dirs         = /usr/include/openmpi-x86_64:/usr/include/python2.7
## libraries            = mpi
libraries            = mpi
## library_dirs         = %(mpi_dir)s/lib
library_dirs         = /usr/lib64/openmpi/lib
## runtime_library_dirs = %(mpi_dir)s/lib
runtime_library_dirs = /usr/lib64/openmpi/lib
$ python2.7 setup.py build
$ sudo python2.7 setup.py install

# Edit ~/.bashrc by adding the following line:
# export PYTHONPATH=$PYTHONPATH:/usr/lib64/python2.7/site-packages/

Meanwhile, I also use mpirun+NFS to install mpi4py to all nodes.

$ cd /mnt/share/mpi4py-1.3.1
$ mpirun -hostfile ~/hostfile -n 2 -npernode 1 sudo python2.7 setup.py install

NOTE: To make this work, root need the permission to write to the NFS (here /mnt/share). Then in /etc/exports, the permission for each drive should be (rw,no_root_squash) (REF: http://superuser.com/questions/605464/unable-to-write-as-root-but-can-as-user)

Comments