How to make GROUND coding compatible with Parallel PHOENICS

Discussion

The domain decomposition in Parallel PHOENICS is usually handled automatically. The directions chosen will depend on the grid size, solver settings and number of cells used. The decomposition will be made in such a way as to minimise the data exchange between processes.

Each subdomain is assigned to a different parallel process, and each subdomain is enlarged at each adjoining end by the addition of two columns, rows or slabs (planes) of cells. These additional 'overlap' or 'halo' cells are used to store field values from the adjacent subdomains so as to facilitate data exchange between subdomains during the parallel computation.

Loops in which indices are restricted to the current cell and its' immediate neighbours will work without any change.

The ground coding of loops will require special attention if:

  1. summations are made, whether over the domain or over patches;
  2. references are made to non-neighbour indices.

The reason is that Parallel PHOENICS interprets each loop as referring to the current subdomain, including 'halo' cells. Such coding is likely to occur in Group 19 or Group 11.


Example 1

This is an example of a GROUND file which includes coding in Group 19 Section 6 which sums up the gas volume over the whole field within the main IZ loop of PHOENICS. Ground coding in other groups and sections is not shown because it did not require modification for use with Parallel PHOENICS.

Head of GROUND file

      SUBROUTINE GROUND
      INCLUDE '/phoenics/d_includ/farray'
      INCLUDE '/phoenics/d_includ/satear'
      INCLUDE '/phoenics/d_includ/grdloc'
      INCLUDE '/phoenics/d_includ/satgrd'
      INCLUDE '/phoenics/d_includ/grdear'
      INCLUDE '/phoenics/d_includ/grdbfc'
      INCLUDE '/phoenics/d_includ/parear' 
      LOGICAL INPARDOM

Group 19 Section 6 Coding

Parallel-specific code is in red.

   196 CONTINUE
C   * ------------------- SECTION 6 ---- Finish of iz slab.
C
C... Calculate overall 'free' volume
      L0VL=L0F(VOL)
      IF(IZ.EQ.1) VOLSUM=0. ! Initialise on first Z slab
      DO IX=1,NX  ! loop over slab
        DO IY=1,NY
          I=(IX-1)*NY+IY
          IF(.NOT.SLD(I)) THEN ! skip any solid cells
            IF(NPROC.GT.1) THEN
              IF(INPARDOM(IX,IY,IZ)) VOLSUM=VOLSUM+F(L0VL+I)
            ELSE
              VOLSUM=VOLSUM+F(L0VL+I)
            ENDIF
          ENDIF
        ENDDO
      ENDDO
      IF(NPROC.GT.1) CALL GLSUM(VOLSUM)
C      
      IF(IZ.EQ.NZ.AND.MYID.EQ.0) THEN ! print at last slab from master
        CALL WRIT40('The total free volume in the domain is (m^3)')
        CALL WRIT1R('VOLUME  ',VOLSUM)
      ENDIF
      RETURN

The foregoing coding includes the use of various parallel utilities such as

  • INPARDOM(ix,iy,iz) - this returns .FALSE. if the cell (ix,iy,iz) is one of the 'halo' cells, and .true. otherwise.
  • GLSUM(real) - sums a single real value over all processors and sends the global sum back to all processors.
  • NPROC - is the number of processors - always 1 for sequential.
  • MYID - is the number of the current processor. The master processor has the ID number 0. In sequential runs this is always 0.

Example 2

This example shows how to sum up the sources, in this case of mass, over all patches to find the total mass inflow.

The problem lies in that on each process the sequence of patches may be different, as not all patches will exist on all processes. However, when summing across processes the loop indices must be in step, otherwise the summation will fail and the run will stall.

In the example code, the following variables and routines are used:

  • NPROC - the number of processes. Will be > 0 for a parallel run.
  • GD_NUMPAT - this is the total number of patches in the global domain. NUMPAT is the number of patches on the current process, or in a sequential run.
  • GD_INDPAT(iglob,1) - this returns the local patch index for the global patch iglob. If the value returned is < 0, the patch does not exist on the current process.
  • PGETCV(iglob,ivar,coef,val) - this returns the COefficient and VALue for variable ivar for global patch iglob. If the coefficient is returned as -999.0, no COVAL exists for this variable at this patch.
  • GLSUM(real) - which sums a single real value over all processes and sends the global sum back to all processes.
  • MYID - is the number of the current process. The master process has the ID number 0. In sequential runs this is always 0.

Head of GROUND file

      SUBROUTINE GROUND
      INCLUDE '/phoenics/d_includ/farray'
      INCLUDE '/phoeclos/d_includ/d_earth/parvar'
      INCLUDE '/phoenics/d_includ/satear'
      INCLUDE '/phoenics/d_includ/grdloc'
      INCLUDE '/phoenics/d_includ/satgrd'
      INCLUDE '/phoenics/d_includ/grdear'
      INCLUDE '/phoenics/d_includ/grdbfc'
      INCLUDE '/phoenics/d_includ/parear'
Group 19 Section 7 Coding

Parallel-specific code is in red.

  197 CONTINUE
C   * ------------------- SECTION 7 ---- Finish of sweep.

C... Calculate overall mass-inflow
      IF(NPROC.GT.1) THEN
        ILIM=GD_NUMPAT
      ELSE
        ILIM=NUMPAT
      ENDIF  
      FMASIN=0.0
      DO I=1,ILIM ! loop over global or local patches
        IF(NPROC.GT.1) THEN
          IR=GD_INDPAT(I,1) ! get local index IR for global patch no I
        ELSE
          IR=I ! in sequential, local and global are the same!
        ENDIF
        IF(NPROC.GT.1) THEN
          CALL PGETCV(I,R1,GCO,GVAL) ! get GO and VAL for Mass 
        ELSE 
          CALL GETCOV(NAMPAT(IR),R1,GCO,GVAL) ! get GO and VAL for Mass
        ENDIF  
        IF(QEQ(GVAL,-999.)) CYCLE ! no COVAL for mass so skip to next patch
        IF(IR.LT.0) THEN
          SORCE=0.0 ! patch does not exist on this processor
        ELSE  
          CALL GETSO(IR,R1,SORCE) ! get mass source for local patch IR
        ENDIF
        IF(NPROC.GT.1) CALL GLSUM(SORCE) ! sum over all processors
        IF(SORCE.GT.0.0) THEN ! mass inflow
          FMASIN=FMASIN+SORCE  ! sum mass
        ENDIF
      ENDDO
      IF(MYID.EQ.0) THEN ! print on master
        CALL WRIT40('The total mass inflow is (kg/s)')
        CALL WRIT1R('MASSIN  ',FMASIN)
      ENDIF

Utility Subroutines for use by Parallel PHOENICS Interface

A selection of the utility subroutines employed in the PHOENICS Parallel Interface is listed below.

Routines that sum variables across all the processes

SUBROUTINE LGSUM_AND(VAR) - It gives the Global AND of LOGICAL variable VAR among Working Processes.

SUBROUTINE LGSUM_OR(VAR) - It gives the Global OR of LOGICAL variable VAR among Working Processes.

SUBROUTINE DGSUM2(VAR1,VAR2) - It calculates the Global Sum Of VAR1 & VAR2 among Working Processes. Data type: REAL*8.

SUBROUTINE DGSUM3(VAR1,VAR2,VAR3) - It calculates the Global Sum Of VAR1, VAR2 & VAR3 among Working Processes. Data type: REAL*8.

SUBROUTINE DGSUMN(DATA, N) - It calculates the Global Sum Of n REAL*8 values across Processes. Data type: REAL*8. Max n is 256 !!

SUBROUTINE RGSUMN(DATA, N) - It calculates the Global Sum Of n REAL*4 values across all Working Processes. Max n is 256 !!

SUBROUTINE GLSUM(VAR) - It calculates the Global Sum Of VAR among Working Processes. Data type REAL*4.

SUBROUTINE GLSUM2(VAR1,VAR2) - It calculates The Global Sum Of VAR1 & VAR2 among Working Processes. Data type: REAL*4.

SUBROUTINE GLSUM4(VAR1,VAR2,VAR3,VAR4) - It calculates The Global Sum Of VAR1 & VAR2 & VAR3 & VAR4 among Working Processes. Data type: REAL*4.

SUBROUTINE GLNXYZ(NIX,NIY,NIZ) - It returns the global size of the whole-domain grid when each domain is of size NIX, NIY and NIZ.

Routines for distributing data from the master process to all the slave processes, often after having been read at the start of the run.

SUBROUTINE RTFSII(data) - Process 0 broadcasts INTEGER 'data' to the other processes.

SUBROUTINE RTFAII(data,size) - Process 0 broadcasts INTEGER 'data', size 'size' to the other processes.

SUBROUTINE RTFARI(data,size) - Process 0 broadcasts REAL 'data', size 'size' to the other processes.

SUBROUTINE RTFALI(data,size) - Process 0 broadcasts Logical 'data', size 'size' to the other processes.

SUBROUTINE RTFACI(data,size) - Process 0 broadcasts character 'data', size 'size' to the other processes.

SUBROUTINE RTVACI(data,size,nchar) - Process 0 broadcasts an array of CHARACTER*nchar 'data', size 'size' to the other processes.

Routines to broadcast data from a specified process to all other processes

SUBROUTINE BCASTI_FOR(iarry,length,idsndr) - Process 'idsndr' broadcasts INTEGER array 'iarry', size 'length' to the other processes.

SUBROUTINE BCASTR_FOR(rarry,length,idsndr) - Process 'idsndr' broadcasts REAL array 'rarry', size 'length' to the other processes.

SUBROUTINE BCASTL_FOR(larry,length,idsndr) - Process 'idsndr' broadcasts LOGICAL array 'larry', size 'length' to the other processes.

SUBROUTINE BCASTD_FOR(darry,length,idsndr) - Process 'idsndr' broadcasts REAL*8 array 'darry', size 'length' to the other processes.

SUBROUTINE BCASTC_FOR(cdata,length,idsndr) - Processor 'idsndr' broadcasts CHARACTER string 'cdata', size 'length' to the other processes.

Routines for sending data to a specific process

SUBROUTINE SENDI_FOR(iarry,length,itag,idrecvr) - Current process sends INTEGER array 'iarry', size 'length' to the process 'idrecvr'.

SUBROUTINE SENDR_FOR(rarry,length,itag,idrecvr) - Current process sends REAL array 'rarry', size 'length' to the process 'idrecvr'.

SUBROUTINE SENDD_FOR(darry,length,itag,idrecvr) - Current process sends REAL*8 array 'darry', size 'length' to the process 'idrecvr'.

SUBROUTINE SENDC_FOR(cdata,length,itag,idrecvr) - Current process sends CHARACTER string 'cdata', size 'length' to the process 'idrecvr'.

Routines for receiving data from a specific process

SUBROUTINE RECVI_FOR(iarry,length,itag,idsndr) - Current process receives INTEGER array 'iarry', size 'length' from the process 'idsndr'.

SUBROUTINE RECVR_FOR(rarry,length,itag,idsndr) - Current process receives REAL array 'rarry', size 'length' from the process 'idsndr'.

SUBROUTINE RECVD_FOR(darry,length,itag,idsndr) - Current process receives REAL*8 array 'darry', size 'length' from the process 'idsndr'.

SUBROUTINE RECVC_FOR(cdata,length,itag,idsndr) - Current process receives CHARACTER string 'cdata', size 'length' from the process 'idsndr'.

Barrier routine used to help syncronise processes

SUBROUTINE BARRIER - causes each process to pause until every process has called it.

Other useful PHOENICS related routines for use in parallel

LOGICAL FUNCTION INPARDOM(IX,IY,IZ) - returns .FALSE. if the cell (IX,IY,IZ) is a 'halo' cell, and .TRUE. otherwise.

SUBROUTINE PGETCV(iglob,ivar,coef,val) - this returns the COefficient and VALue for variable ivar for global patch iglob. If the coefficient is returned as -999.0, no COVAL exists for this variable at this patch.


For detailed advice on how to ensure specific GROUND routines are compatible with parallel operation, please contact CHAM User Support.