!===============================================================================
!
! cg/README
!
!-------------------------------------------------------------------------------
!
! Copyright (C) 2017 Hinnerk Stueben
!
! This file is part of BQCD -- Berlin Quantum ChromoDynamics program
!
! BQCD is free software: you can redistribute it and/or modify
! it under the terms of the GNU General Public License as published by
! the Free Software Foundation, either version 3 of the License, or
! (at your option) any later version.
!
! BQCD is distributed in the hope that it will be useful,
! but WITHOUT ANY WARRANTY; without even the implied warranty of
! MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
! GNU General Public License for more details.
!
! You should have received a copy of the GNU General Public License
! along with BQCD.  If not, see <http://www.gnu.org/licenses/>.
!
!-------------------------------------------------------------------------------

This directory contains a new implementation of the cg solver.

Goals of the new implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- Enable SIMD vectorisation with SIMD intrinsics.
  . Introduction of suitable array layout. 
  . Portabilility.

- Enable runtime choice of implementation variants.
  . Use function pointers macros for array access.

- One should always be able to study the code that the compiler sees.


Implementation details
~~~~~~~~~~~~~~~~~~~~~~

*** Array layout

The original array layout is:

    complex(8) :: x(dim1, dim2, volh_tot)

This was changed in such away, that the base storage unit is a SIMD
vector:

    real(8) :: x(simd_width, re:im, dim1, dim2, volh_tot / simd_width)

Another optimisation is a new ordering of the dimensions of the
spin-colour field

    old (variant '1'):  complex(8) :: x(2, 2, 3, volh_tot)

    new (variant '22'): complex(8) :: x(2, 3, volh_tot, 2)

(correspondingly for SIMD layout) which is mainly an MPI optimisation.
In the new layout of the spin-colour field all MPI buffers are
consecutive.


*** SIMD portability

Portablility is achieved by using a common set of intrinsics, which are
coded by the following macros (see simd_*.h):

    Load, Store, Add, Sub, Mul, Fmadd, Fmsub

Unfortunately the computation of 'D-slash' in 'x-direction' needs more
functionality which is not common.

    For x64 unaligned loads and stores are employed: LoadU, StoreU

    For PowerPC (BlueGene/Q) the vec_sldw() intrinsic is employed. The
    macro is called: ShiftLeft2 (see simd_qpx.h)


In orer to introduce another set of SIMD intrinsics the following steps
are needed:

- define macros in simd_NEW.h

- define 'simd_width' and 'alignment_value' accordingly in
  ../Makefile.var

- define SIMD_WIDTH_R8 in 'simd_generic' (this is currently set to 4,
  which is only correct for AVX, AVX2, QPX)


*** Domain decomposition

Due to the more complicatied situation in the x-direction that direction
cannot be decomposed when using SIMD, ie the domain decomposition can
only be three-dimensional while it can be four-domensional otherwise.


*** Alignment

Arrays are allocated from Fortran (see cg_init_modules.F90).  Hence
alignment can only be ensured if the Fortran compiler has an alignment
directive (arrays are 'allocatable' rather than 'pointer's).  Alignment
directives are implemented for IBM and Intel compilers (see
alignment.h). (When compiling with GNU unaligned loads and stores are
called throughout.)

If arrays are aligned, all MPI buffers are aligend as well.


*** Toolkit

The idea was to create a toolkit in which additional tuning variants can
be selected.  These additional tuning variants are single vs. double
precision and loop blocking.  Both are prepared but not fully usable
yet.


'Generic' SIMD
~~~~~~~~~~~~~~

SIMD-intrinsics are only employed in routines that implement
multiplications with small matrices.  In other routines that are needed
to handle SIMD array layout Fortran TYPEs are used and SIMD is
implemented implicitely in a 'generic' way.


Code generation
~~~~~~~~~~~~~~~

Code generation needs a lot of preprocessing.  Various methods were
tried (and probably the template of C++ were re-invented).  In
particular, two kinds of loops are needed in the code generation steps:
loop unrolling and loops over array layouts.  Loops are implemented:

- by a loop preprocessor (see ../tool/loopp)
- in shell scripts
- in the Makefile

Fortran files that the compiler sees have .f90 extensions (and possibly
tmp- prefices).  C files that the compiler sees are named tmp-*.c . 


Tuning parameters
~~~~~~~~~~~~~~~~~

There are the following tuning parameters:

     tuning_cg_version
     tuning_cg_precision_r4
     tuning_cg_loop_blocking
     tuning_cg_block_length
     tuning_cg_spincol
     tuning_cg_clover
     tuning_cg_d
     tuning_cg_precision
     tuning_cg_simd

Explanations:

- tuning_cg_version (values: 1, 2)

  1: use old/original cg implementation
  2: use new cg implementation (includes SIMD and choice of data
     structures and d-slash implementations at runtime)

- tuning_cg_precision_r4 (values: 0, 1)

  0: 64-bit calculation
  1: 32-bit calculation

  This is used by 'tuning_cg_version 1' and is mainly used for
  benchmarking. ('tuning_cg_version 2' uses tuning_cg_precision for the
  same purpose.)

- tuning_cg_loop_blocking (values: 0, 1)

  0: use loop blocking
  1: do not use loop blocking

  Is not implemented yet.

- tuning_cg_spincol (values: 1, 22)

  Data layout of the spin-colour arrays.

  1: original layout

     complex(8) :: a(4, 3, volume)

     which is equivalent to

     complex(8) :: a(2, 2, 3, volume)

  22: new layout (the second dimension '2' is shifted to the last position)

     complex(8) :: a(2, 3, volume, 2)

     With this layout the boundaries in t-direction are consecutive in
     memory (optimal layout for MPI in t-direction).

- tuning_cg_clover (values: 1, 2)

     Layout of the clover arrays.

     1: original layout (a packed format)
     2: new layout (packed differently, better for SIMD)

- tuning_cg_d (values: 2, 21, 25, 35)

     Implementation of d-slash.

     2:  original version: MPI copies 4 component spinors
         (is also implemented in 'tuning_cg_version 1')
     21: 'BlueGene' version (was created first for BGL): MPI copies
         2-component spinors, 2-component spinors are projected for all
         directions and all sites (including t-direction, where
         projection reduces to copy into a consecutive array)
         (is also implemented in 'tuning_cg_version 1')
     25: MPI copies 2-component spinors, only the minimal number of
         projections is calculated
     35: '25' plus overlapping communication and computation.
         '35' needs at least 2 OpenMP threads per MPI process.

- tuning_cg_precision (values: 32, 64)

     32: single precision
     64: double precision

- tuning_cg_simd (values: 0, 1)

    0: standard Fortran
    1: use SIMD-vectorised code (C code)


Tuning parameters are reported in the output:

 >BeginTuningInfo
 tuning_cg_version                2
 --more flexible implementation--
 tuning_cg_d                      25
 tuning_cg_simd                   1
 tuning_cg_precision              64
 tuning_cg_loop_blocking          0
 tuning_cg_block_length           256
 tuning_cg_spincol                22
 tuning_cg_clover                 2
 >EndTuningInfo

Tuning suggestions
~~~~~~~~~~~~~~~~~~

The following set of tuning parameters is good starting point:
 
   tuning_cg_version                2
   tuning_cg_d                      25 or 35
   tuning_cg_simd                   1
   tuning_cg_precision              64
   tuning_cg_spincol                22
   tuning_cg_clover                 2

!===============================================================================
