This chapter describes the installation of MPICH2 for the user who purchased a licence for running Parallel PHOENICS on machines that are running the 64-bit version of Windows XP Professional, Vista Business, Windows 7 or Server 2003. It applies to multi-processor machines and clusters of single or multi-processor machines. The minimum recommended configuration for running 64-bit parallel PHOENICS is a multi-core PC with 2GB of RAM per processor. The PCs in the cluster should be joined by a switch or hub running at 100Mbps or more. For efficient operation the processors in a cluster should be of equal performance: if not, the loads will be unbalanced and the slowest processor will determine the speed of the system. MPICH2 also requires the ability to make TCP/IP socket connections between all hosts.
The installation process for MPICH2 on a PC with multiple CPUs (Dual-core, quad-core and so on) is simpler than the installation process for a cluster. MPICH2 should be installed on the single PC as outlined in section 6.3 (Items 1, 2, 3 and 5 only), and the PATH set appropriately so that mpiexec.exe is visible. SMPD should also be installed on this PC.
Before starting the installation make sure that each PC in the cluster belongs to the same Workgroup or Domain. To check this, go to the Control Panel and open up the Systems Properties dialog. Then look at the 'Computer Name' (see figure below):
Here the computer name is CHAM-CFD1 and it belongs to the Workgroup PARALL. You may choose the name of the workgroup, but all the PCs which are to be in the cluster must belong to the same one. Use the 'Change…' button to reset the Workgroup of any PCs where necessary.
It is recommended to install MPICH2 on each PC in the cluster. However, before you can install MPICH2 you must first either install Microsoft’s Visual Studio 2005/2008 or two Redistributable Packages (available as free downloads from the Microsoft website). For a 64-bit PC these packages are:
Microsoft .NET Framework version 2.0 Redistributable Package (x64)
Microsoft Visual C++ 2005 SP1 Redistributable Package (x64)
These links were correct October 2010, if you are unable to find them, try a search of the Microsoft site using the package name.
If you choose not to install MPICH2 on each PC, you will still need to install SMPD as a service on each PC. To install MPICH2 or to install the SMPD service, the user will need to be logged in using an account with Administrator permissions. This need not be the account on which you run PHOENICS, although it is preferable when running over a cluster to use an administrator account.
Since the advent of Service Pack 2 in Windows XP, Windows has had the personal Firewall switched on by default. Experienced users may configure the firewall settings to allow parallel PHOENICS to run successfully, but while installing and until the user is sure that MPICH has been configured correctly, it is recommended that any personal firewalls are turned off. For XP, open the Windows Firewall icon on the Control Panel, and then choose the 'Off (not recommended)' option, then click OK. For Vista and Windows 7, the Firewall settings are found under 'System and Security'.
After the user has established that MPICH2 is working correctly then the Firewall should be switched back on.
If the master PC has difficulty seeing one or more of the slave PCs and you are using static IP addresses in your cluster then you should consider defining the IP addresses in the host file on each of the PCs. The host file is normally located as
C:\Windows\system32\drivers\etc\hosts
An example 'hosts' file is:
# Copyright (c) 1993-1999 Microsoft Corp. # # This is a sample HOSTS file used by Microsoft TCP/IP for Windows. # # This file contains the mappings of IP addresses to host names. Each # entry should be kept on an individual line. The IP address should # be placed in the first column followed by the corresponding host name. # The IP address and the host name should be separated by at least one # space. # # Additionally, comments (such as these) may be inserted on individual # lines or following the machine name denoted by a '#' symbol. # # For example: # # 102.54.94.97 rhino.acme.com # source server # 38.25.63.10 x.acme.com # x client host 127.0.0.1 localhost 192.168.11.1 cham-cfd1 192.168.11.2 cham-cfd2 192.168.11.3 cham-cfd3 192.168.11.4 cham-cfd4
In the following sections, the PC from which parallel PHOENICS is launched is known as the master (process) and the other PCs/processes in the cluster will be termed slaves.
It will be assumed in what follows that PHOENICS has been successfully installed in accordance with the instructions in the earlier chapters of this document. A full PHOENICS installation need only be made on one machine in a parallel cluster (the one from which jobs will be run); however it is recommended that an installation is made on each PC. If PHOENICS is installed on a single host then it will be necessary to share the \phoenics directory so that all PCs may see it. Each machine in the cluster will also be required to have a valid local licence file 'phoenics.lic' accessible.
On Windows platforms parallel PHOENICS uses MPICH2 as the message passing interface (MPI) for the communication between the different processors. MPICH2 is freely available on the Internet but for compatibility it is recommended that the installation is made from the MPI provided with PHOENICS package. Installation instructions are as follows:
> mpiexec.exe -register account: cfd1 password: ******** confirm: ******** Do you want this action to be persistent (y/n)? yIf the cluster has been set up within a network Domain (rather than a Workgroup), then in the above, you should also specify the domain as part of the account name. For example, if 'phoenics' is the domain and 'cfd1' is the user account within that domain, enter phoenics\cfd1. The response 'y' ensures that this action is persistent, i.e. this registration process does not have to be repeated for each session on this PC.
> smpd.exe -installIf Phoenics is being run across a cluster (as opposed to running on a single multi-core PC) then it will be necessary to identify the PCs within the cluster. This may be done in a Command Prompt window, using smpd.exe. For example, to include the four PCs CHAM-CFD1, CHAM-CFD2, CHAM-CFD3 and CHAM-CFD4 in a cluster we type,
> smpd.exe -sethosts CHAM-CFD1 CHAM-CFD2 CHAM-CFD3 CHAM-CFD4To check which hosts have been added to the cluster one can type,
> smpd.exe -hostsThe response will be a list of the PCs in the cluster.
Since the introduction of XP Service Pack 2 the personal firewall has been activated by default. When the firewall is activated, then running the PHOENICS solver, earexe, may generate a Windows Firewall Security Alert. When running in parallel mode it is essential that earexe is Unblocked, even if you are only using the processors of the host PC. If mpiexec is run with an MPI configuration file instead of the executable program as the argument, then there will be an additional security alert for the mpiexec program. Again, it is essential that this program be unblocked.
With the Windows Firewall, the user may choose to unblock the Earexe executable from the security alert dialog above. However, if you are operating across a cluster, this will not be sufficient to enable Parallel Phoenics to run. There are additional settings needed on both the master and slave PCs.
For those using the Windows XP open the Windows Firewall icon from the Control Panel, then go to the Exceptions Page. On the Master PC you will need to use the 'Add Program…' button to add the following programs:
C:\Phoenics\d_earth\d_windf\earexe.exe
C:\Program Files\MPICH2\binsmpd.exe
C:\Program Files\MPICH2\bin\mpiexec.exe
You may also use the 'Change scope..' button to restrict access to My network (subnet) only.
On each of the slave PCs, you will need to add the programs
C:\Phoenics\d_earth\d_windf\earexe.exe
C:\Program Files\MPICH2\bin\smpd.exe
On Windows 7, open the Control Panel then select the option 'System and Security' (or 'Security' on Vista). Here there is a option to 'Allow a program through Windows Firewall', click on this option and add the program. You will need Administrator permissions though to make these changes.
Users of other personal firewall will need to unblock the above programs in a manner suitable for their firewall software.
The simplest way to launch parallel EARTH is from the VR-Editor, although it can be run from a Command Prompt window.
If a parallel PHOENICS licence has been purchased, an additional sub-menu, 'Parallel Solver', will appear under the 'Run' menu option in the VR-Editor. Once the parallel solver is chosen, a dialog box will appear on the screen where the user can either specify the number of processes to use or to specify a MPI configuration file.
The pulldown combo box provides the user with an option to select up to thirty-two processes. Those users who have more than thirty-two processors on their PC cluster may type the appropriate number into the box. This method does have its limitations though, it does require that:
The MPI configuration file option gives a more flexible way of launching the parallel solver. Assuming we have PHOENICS installed on each PC in the cluster, the following config file will use the public earexe.exe to run a single process on each of the four PCs.
-localroot -n 1 -host cham-cfd1 c:\phoenics\d_earth\d_windf\earexe.exe -n 1 -host cham-cfd2 c:\phoenics\d_earth\d_windf\earexe.exe -n 1 -host cham-cfd3 c:\phoenics\d_earth\d_windf\earexe.exe -n 1 -host cham-cfd4 c:\phoenics\d_earth\d_windf\earexe.exe
The following example launches two processes on each of two PCs where PHOENICS is installed only on the master PC:
-localroot -n 2 -host cham-cfd1 \\cham-cfd1\phoenics\d_earth\d_windf\earexe.exe -n 2 -host cham-cfd2 \\cham-cfd1\phoenics\d_earth\d_windf\earexe.exe
Users should create their own configuration and 'run' files, based on the examples provided, tailored to their own installation. These can either be located in \phoenics\d_utils\d_windf or the local working directory.
All Nodes in the cluster should belong to the same Workgroup or Domain, and the user should be logged into each Node on the Cluster using the same Workgroup/Domain User account.
PHOENICS must be installed on the Master PC, but installation on the other Nodes (Slave PCs) is optional.
When using the default automatic domain decomposition, parallel PHOENICS only differs from sequential when Earth is run: problem set-up and post-processing of results can be done in exactly the same way as for the sequential version. A case that has been run in sequential mode can be run in parallel without any changes being made. The output from a parallel PHOENICS simulation will be result and phi files, having the same format as for sequential simulations.
It is also possible to by-pass the automatic domain decomposition algorithm, and to specify how you want to decompose the calculation domain into sub-domains. This can be done by setting the appropriate date-for-solver arrays in the Q1 file.
For example, to split the domain into 8 sub-domains (2 in each direction), the following arrays must be set in the Q1 file:
LG(2)=T
IG(1)=2
IG(2)=2
IG(3)=2
The logical LG(2) will instruct the splitter to by-pass the automatic domain decomposition, and split the domain according to the settings defined in the IG array as follows.
IG(1) specifies the number of sub-domains in the x-direction;
IG(2) specifies the number of sub-domains in the y-direction;
IG(3) specifies the number of sub-domains in the z-direction;
In this case, the domain has been divided into sub-domains according to the settings made in the Q1 file.
In a Command Prompt window, if the EARTH executable is launched directly, then the sequential solver will be used; to run the parallel solver, the program name ‘earexe’ is used as an argument to mpirun.
A script RUNPAR.BAT [nnodes] is provided. The optional argument [nnodes] indicates the number of processes to be launched on the current PC. The default is to launch two processes.
For example, RUNPAR 2 will execute the MPI command:
mpiexec –localroot -np 2 \phoenics\d_earth\d_windf\earexe
If a cluster has been defined by SMPD then, the command will execute on two processors in the cluster, otherwise it will launch multiple processes on the local machine.
A MPICH2 user guide is installed as part of the installation in PDF format; it is accessible from the MPICH menu item on the Start menu. Online documentation is available at