Encyclopaedia Index

Installation of PHOENICS 2010

6 Parallel PHOENICS with MS-MPI

6.1 Introduction

This chapter has been re-written to describe the installation of parallel PHOENICS specifically on Microsoft Windows HPC Server 2008 using MS-MPI as the message passing interface (MPI).

It will be assumed in what follows that PHOENICS has been successfully installed on the head node of the HPC cluster in accordance with the instructions in Chapter 3 of this document. A full PHOENICS installation need only be made on this head node (the one from which jobs will be submitted).

6.2 Preliminary Considerations

When PHOENICS is installed only on the head node it is necessary to share the \phoenics directory so that all compute nodes may see it. To do this open Windows Explorer, navigate so that you can see the phoenics folder and use a right mouse click over the name phoenics to bring up the short cut menu and choose the item 'Share...'.' This will bring up a dialog where you can 'Choose people to share with'. Who it is you share access with and what permission level to choose will depend on who will be using parallel PHOENICS on your cluster and on where your case files and results will be situated.

Traditionally, when working on a stand-alone workstation the case files have been located within the phoenics folder. If you continue this approach and use sub-folders within the phoenics folder on the head node for the working directory when running the parallel solver, then users will need to have the share permission level set to 'contributor'. This should allow them read access to the installation files and to add/modify/delete new files.'

However, when multiple users are running on the same system, it may be preferable to keep the user files out of the phoenics area and set the share permission level at 'reader'. Then set up a separate share, say 'phoenicsdata', with permissions set to 'contributor' in which to place the user data/results.

6.3 Installing configuration files onto the Compute nodes

Each compute node in the cluster will also be required to have a valid local licence file phoenics.lic and other local configuration files. A setup type 'Compute node only' has been added to the PHOENICS installation to simplify doing this (see below):

This option will just install the minimum configuration files needed to run the PHOENICS solver via the HPC job submission process. The get_hostid utility has also been added should the user require the host ID for unlocking purposes.  This compute node installation will also set the appropriate PHOENICS environment variable.

6.4 Installation of MS-MPI

It is assumed that the server software and cluster have already been set up and configured. The MS-MPI software, needed for the message passing interface (MPI) for parallel running of the PHOENICS solver, is provided within the HPC Pack 2008 SDK. If this is not currently installed on the HPC servers, then it is available as a free download from the Microsoft website. The HPC Pack 2008 SDK needs to be installed on the head node and all of the compute nodes in the cluster.

A search of the Microsoft website with the keywords 'HPC Pack 2008 SDK' will quickly locate the latest version.  When downloading from the Microsoft website, ensure that you choose the 64 bit (x64) version of the SDK software.  Alternatively, the PHOENICS installation includes the SDK install file as /phoenics/d_allpro/d_libs/d_windf/mpi/sdk_x64.msi, this was the version used to build the PHOENICS solver.

Installation instructions are as follows:

  1. In order to run the MS-MPI installation program you must first be logged onto the PC using an account that has Administrator privileges.
  2. Locate the file
    \phoenics\d_allpro\d_libs\d_windf\MPI\sdk_x64.msi
    In Windows Explorer, using the mouse right-click over the filename then use the option 'Install' from the short cut menu to begin the installation.  It should be sufficient to choose all the default options while installing the software.  Repeat this on all nodes in the HPC cluster.
  3. The PHOENICS parallel run scripts assume that mpiexec.exe is located on the users PATH on the head and compute nodes; this should be added for you by the SDK installation process in step 2.
  4. In contrast to MPICH2 installations, it is not necessary to have the SMPD process running before starting a parallel run. With MS-MPI, SMPD is launched automatically when the parallel run is started.

Note: It is possible to install HPC Pack 2008 SDK on a 64 bit Windows Workstation to run parallel PHOENICS on a single multi-core machine. It cannot be used though across a Workstation cluster.  If one wishes to run across a Workstation cluster is recommended to use that version of PHOENICS built with MPICH2 libraries.

6.5 Windows Firewall settings

If you have the Windows Firewall active, then running the PHOENICS solver, earexe, may generate a Windows Firewall Security Alert. When running in parallel mode it is essential that earexe is Unblocked, even if you are only using the processors of the head node. If mpiexec is run with an MPI configuration file instead of the executable program as the argument, then there will be an additional security alert for the mpiexec program. Again, it is essential that this program be unblocked.

With the Windows Firewall, the user may choose to unblock the earexe executable from the security alert dialog above (one will require Administrator permissions to unblock). However, if you are operating across a cluster, this will not be sufficient to enable Parallel Phoenics to run. There are additional settings needed on both the head and compute nodes.

6.6 Running Parallel PHOENICS

6.6.1 Running the standard parallel solver from the GUI

The simplest way to launch the parallel solver is from the VR-Editor, although with MS-MPI this is currently only suitable for running the solver on a single computer.

If a parallel PHOENICS licence has been purchased, an additional sub-menu, 'Parallel Solver', will appear under the 'Run' menu option in the VR-Editor. Once the parallel solver is chosen, a dialog box will appear on the screen where the user can either specify the number of processes to use or to specify a MPI configuration file.

The pulldown combo box provides the user with an option to select up to thirty-two processes.

6.6.2 Running parallel solver through HPC Job Manager

To run the parallel solver across the HPC cluster it is best to submit the job via the Job Manager. In the following scenario it will be assumed that the head node computer is called PHOE-HEAD1. It will be further assumed that PHOENICS has been installed in C:\phoenics (shared as \phoenics) and that the user case files will be in the directory C:\phoenicsData\myCase1 (where c:\phoenicsData is shared as \phoenicsData).

  1. Set up the case using the VR Editor on the head node, Save Working files and/or Exit making sure to 'Save changes to EARDAT file for Solver'.
  2. In HPC Job Manager, in the Actions pane, click New Job.
  3. In the left pane of the New Job dialog box, click Edit Tasks.
  4. Point to the Add button, click the down arrow, then click Basic Task.
  5. In the task dialog box, type a name for your task.
  6. Type the task command in the Command line entry box.
    mpiexec //phoe-head1/phoenics/d_earth/d_windf/earexe.exe
  7. Specify the Working directory for your task. In general, a working directory should be indicated with a Universal Naming Convention (UNC) path, not a relative or a local path. In our example it will be
    //phoe-head1/phoenicsData/myCase1
  8. Specify the Standard input, Standard output, and Standard error file names relative to the working directory.
  9. Set the minimum and maximum number of cores necessary for your job.
  10. Click OK to add the task to your job. The new task now appears in the list of tasks for your job.

6.6.3 Monitoring the solution of jobs running on the cluster

Parallel runs submitted onto the cluster run in batch mode, and this mode does not permit the interactive viewing of the solver GUI. However, it is still possible to view the progress of the solution through the use of the file snapshot, best used in combination with the file monicurr. The file monicurr is an optional file written by the solver to provide the current monitoring settings.  It is turned on by the following setting in the file cham.ini

[Earth]
Monicurr  = on

This file contains the current monitoring cell and relaxation parameters for the solved variables and number of sweeps. To obtain a snapshot of the current state of the solution one need just copy (or rename) the file monicurr to a file called snapshot, the solver will then read the snapshot file and generate a GXMONI monitor plot. The file snapshot will then be deleted and a new monicurr file generated. Here is an example of the contents of a monicurr file:

  Start of monitor location
IX =    112
IY =     67
IZ =      5
NX =    200
NY =    200
NZ =     47
  End of monitor location
LSTEP =          1
LSWEEP =       3000
  Start of relaxations
SARAH =   0.000    
P1   =  -1.000    
U1   =  0.1000    
V1   =  0.1000    
W1   =  0.1000    
KE   = -0.3000    
EP   = -0.3000    
PRPS =  -1.000    
EPKE =  -1.000    
DEN1 =  -1.000    
EL1  = -0.2500    
ENUT = -0.2500    
  End of relaxations

If changes are made to the contents of monicurr (eg a change in monitor location or number of sweeps) before being copied to snapshot then these values will be updated in the solver and used for subsequent computations.

It is also possible to cause the current solution to terminate early by creating a file called stopjob. This will signal the solver to stop after the current sweep, write the results to file and terminate cleanly.

6.6.4 Automatic domain decomposition

When using the default automatic domain decomposition, parallel PHOENICS only differs from sequential when Earth is run: problem set-up and post-processing of results can be done in exactly the same way as for the sequential version.  A case that has been run in sequential mode can be run in parallel without any changes being made. The output from a parallel PHOENICS simulation will be result and phi files, having the same format as for sequential simulations.

6.6.5 User-specified sub-domains

It is also possible to by-pass the automatic domain decomposition algorithm, and to specify how you want to decompose the calculation domain into sub-domains. This can be done by setting the appropriate date-for-solver arrays in the Q1 file.

For example, to split the domain into 8 sub-domains (2 in each direction), the following arrays must be set in the Q1 file:

LG(2)=T
IG(1)=2
IG(2)=2
IG(3)=2

The logical LG(2) will instruct the splitter to by-pass the automatic domain decomposition, and split the domain according to the settings defined in the IG array as follows.

IG(1) specifies the number of sub-domains in the x-direction;
IG(2) specifies the number of sub-domains in the y-direction;
IG(3) specifies the number of sub-domains in the z-direction;

In this case, the domain has been divided into sub-domains according to the settings made in the Q1 file.

6.6.6 Command mode operation

In a Command Prompt window, if the EARTH executable is launched directly, then the sequential solver will be used; to run the parallel solver, the program name earexe is used as an argument to mpiexec.

A script RUNPAR.BAT [nnodes] is provided.  The optional argument [nnodes] indicates the number of processes to be launched on the current PC.  The default is to launch two processes.

For example, RUNPAR 2 will execute the MPI command:

mpiexec -np 2 \phoenics\d_earth\d_windf\earexe

6.6.7 Testing Parallel PHOENICS

The parallel installation should be tested by loading a library case.

The different solver used for parallel operation requires a slight modification to the numerical controls. For example, the user may use main 'Menu' in the VR-Editor, and select 'Numerics' and then 'Iteration control': change the number of iterations for TEM1 (temperature) from 20 to 300. (Increasing the relaxation for the velocity components, U1 and W1, from 1.0 to 10.0 will also improve performance.)

For parallel operation it is recommended that velocities should be solved whole-field (rather than slab-by-slab); this can be achieved from the VR Editor (under 'Models', 'Solution control/extra variables') or by direct editing of the q1 file (by setting 'Y' as the third logical in a SOLUTN command).

6.7 Further Information

Further information on job submission for the Windows HPC Server 2008 can be found here,
http://technet.microsoft.com/en-us/library/ff919394(WS.10).aspx