This chapter has been re-written to describe the installation of parallel PHOENICS specifically on Microsoft Windows HPC Server 2008 using MS-MPI as the message passing interface (MPI).
It will be assumed in what follows that PHOENICS has been successfully installed on the head node of the HPC cluster in accordance with the instructions in Chapter 3 of this document. A full PHOENICS installation need only be made on this head node (the one from which jobs will be submitted).
When PHOENICS is installed only on the head node it is necessary to share the \phoenics directory so that all compute nodes may see it. To do this open Windows Explorer, navigate so that you can see the phoenics folder and use a right mouse click over the name phoenics to bring up the short cut menu and choose the item 'Share...'.' This will bring up a dialog where you can 'Choose people to share with'. Who it is you share access with and what permission level to choose will depend on who will be using parallel PHOENICS on your cluster and on where your case files and results will be situated.
Traditionally, when working on a stand-alone workstation the case files have been located within the phoenics folder. If you continue this approach and use sub-folders within the phoenics folder on the head node for the working directory when running the parallel solver, then users will need to have the share permission level set to 'contributor'. This should allow them read access to the installation files and to add/modify/delete new files.'
However, when multiple users are running on the same system, it may be preferable to keep the user files out of the phoenics area and set the share permission level at 'reader'. Then set up a separate share, say 'phoenicsdata', with permissions set to 'contributor' in which to place the user data/results.
Each compute node in the cluster will also be required to have a valid local licence file phoenics.lic and other local configuration files. A setup type 'Compute node only' has been added to the PHOENICS installation to simplify doing this (see below):
This option will just install the minimum configuration files needed to run the PHOENICS solver via the HPC job submission process. The get_hostid utility has also been added should the user require the host ID for unlocking purposes. This compute node installation will also set the appropriate PHOENICS environment variable.
It is assumed that the server software and cluster have already been set up and configured. The MS-MPI software, needed for the message passing interface (MPI) for parallel running of the PHOENICS solver, is provided within the HPC Pack 2008 SDK. If this is not currently installed on the HPC servers, then it is available as a free download from the Microsoft website. The HPC Pack 2008 SDK needs to be installed on the head node and all of the compute nodes in the cluster.
A search of the Microsoft website with the keywords 'HPC Pack 2008 SDK' will quickly locate the latest version. When downloading from the Microsoft website, ensure that you choose the 64 bit (x64) version of the SDK software. Alternatively, the PHOENICS installation includes the SDK install file as /phoenics/d_allpro/d_libs/d_windf/mpi/sdk_x64.msi, this was the version used to build the PHOENICS solver.
Installation instructions are as follows:
Note: It is possible to install HPC Pack 2008 SDK on a 64 bit Windows Workstation to run parallel PHOENICS on a single multi-core machine. It cannot be used though across a Workstation cluster. If one wishes to run across a Workstation cluster is recommended to use that version of PHOENICS built with MPICH2 libraries.
If you have the Windows Firewall active, then running the PHOENICS solver, earexe, may generate a Windows Firewall Security Alert. When running in parallel mode it is essential that earexe is Unblocked, even if you are only using the processors of the head node. If mpiexec is run with an MPI configuration file instead of the executable program as the argument, then there will be an additional security alert for the mpiexec program. Again, it is essential that this program be unblocked.
With the Windows Firewall, the user may choose to unblock the earexe executable from the security alert dialog above (one will require Administrator permissions to unblock). However, if you are operating across a cluster, this will not be sufficient to enable Parallel Phoenics to run. There are additional settings needed on both the head and compute nodes.
The simplest way to launch the parallel solver is from the VR-Editor, although with MS-MPI this is currently only suitable for running the solver on a single computer.
If a parallel PHOENICS licence has been purchased, an additional sub-menu, 'Parallel Solver', will appear under the 'Run' menu option in the VR-Editor. Once the parallel solver is chosen, a dialog box will appear on the screen where the user can either specify the number of processes to use or to specify a MPI configuration file.
The pulldown combo box provides the user with an option to select up to thirty-two processes.
To run the parallel solver across the HPC cluster it is best to submit the job via the Job Manager. In the following scenario it will be assumed that the head node computer is called PHOE-HEAD1. It will be further assumed that PHOENICS has been installed in C:\phoenics (shared as \phoenics) and that the user case files will be in the directory C:\phoenicsData\myCase1 (where c:\phoenicsData is shared as \phoenicsData).
Parallel runs submitted onto the cluster run in batch mode, and this mode does not permit the interactive viewing of the solver GUI. However, it is still possible to view the progress of the solution through the use of the file snapshot, best used in combination with the file monicurr. The file monicurr is an optional file written by the solver to provide the current monitoring settings. It is turned on by the following setting in the file cham.ini
[Earth] Monicurr = on
This file contains the current monitoring cell and relaxation parameters for the solved variables and number of sweeps. To obtain a snapshot of the current state of the solution one need just copy (or rename) the file monicurr to a file called snapshot, the solver will then read the snapshot file and generate a GXMONI monitor plot. The file snapshot will then be deleted and a new monicurr file generated. Here is an example of the contents of a monicurr file:
Start of monitor location IX = 112 IY = 67 IZ = 5 NX = 200 NY = 200 NZ = 47 End of monitor location LSTEP = 1 LSWEEP = 3000 Start of relaxations SARAH = 0.000 P1 = -1.000 U1 = 0.1000 V1 = 0.1000 W1 = 0.1000 KE = -0.3000 EP = -0.3000 PRPS = -1.000 EPKE = -1.000 DEN1 = -1.000 EL1 = -0.2500 ENUT = -0.2500 End of relaxations
If changes are made to the contents of monicurr (eg a change in monitor location or number of sweeps) before being copied to snapshot then these values will be updated in the solver and used for subsequent computations.
It is also possible to cause the current solution to terminate early by creating a file called stopjob. This will signal the solver to stop after the current sweep, write the results to file and terminate cleanly.
When using the default automatic domain decomposition, parallel PHOENICS only differs from sequential when Earth is run: problem set-up and post-processing of results can be done in exactly the same way as for the sequential version. A case that has been run in sequential mode can be run in parallel without any changes being made. The output from a parallel PHOENICS simulation will be result and phi files, having the same format as for sequential simulations.
It is also possible to by-pass the automatic domain decomposition algorithm, and to specify how you want to decompose the calculation domain into sub-domains. This can be done by setting the appropriate date-for-solver arrays in the Q1 file.
For example, to split the domain into 8 sub-domains (2 in each direction), the following arrays must be set in the Q1 file:
LG(2)=T IG(1)=2 IG(2)=2 IG(3)=2
The logical LG(2) will instruct the splitter to by-pass the automatic domain decomposition, and split the domain according to the settings defined in the IG array as follows.
IG(1) specifies the number of sub-domains in the x-direction;
IG(2) specifies the number of sub-domains in the y-direction;
IG(3) specifies the number of sub-domains in the z-direction;
In this case, the domain has been divided into sub-domains according to the settings made in the Q1 file.
In a Command Prompt window, if the EARTH executable is launched directly, then the sequential solver will be used; to run the parallel solver, the program name earexe is used as an argument to mpiexec.
A script RUNPAR.BAT [nnodes] is provided. The optional argument [nnodes] indicates the number of processes to be launched on the current PC. The default is to launch two processes.
For example, RUNPAR 2 will execute the MPI command:
mpiexec -np 2 \phoenics\d_earth\d_windf\earexe
The parallel installation should be tested by loading a library case.
The different solver used for parallel operation requires a slight modification to the numerical controls. For example, the user may use main 'Menu' in the VR-Editor, and select 'Numerics' and then 'Iteration control': change the number of iterations for TEM1 (temperature) from 20 to 300. (Increasing the relaxation for the velocity components, U1 and W1, from 1.0 to 10.0 will also improve performance.)
For parallel operation it is recommended that velocities should be solved whole-field (rather than slab-by-slab); this can be achieved from the VR Editor (under 'Models', 'Solution control/extra variables') or by direct editing of the q1 file (by setting 'Y' as the third logical in a SOLUTN command).
Further information on job submission for the Windows HPC Server 2008 can be found here,
http://technet.microsoft.com/en-us/library/ff919394(WS.10).aspx