package style args
kokkos args = keyword value ... zero or more keyword/value pairs may be appended keywords = comm or reduction comm value = threaded or classic threaded = perform pack/unpack using KOKKOS (e.g. on GPU or using OpenMP) (default) classic = perform communication pack/unpack in non-KOKKOS mode reduction = parallel/reduce or atomic parallel/reduce = use parallel reduction for statistics (default) atomic = use atomic reduction for statistics
package kokkos comm classic package kokkos comm threaded reduction atomic
This command invokes package-specific settings for the KOKKOS accelerator package available in SPARTA.
If this command is specified in an input script, it must be near the top of the script, before the simulation box has been created. This is because it specifies settings that the accelerator package used in its initialization, before a simulation is defined.
This command can also be specified from the command-line when launching SPARTA, using the "-pk" command-line switch. The syntax is exactly the same as when used in an input script.
Note that the KOKKOS accelerator package requires the package command to be specified, if the package is to be used in a simulation (SPARTA can be built with the accelerator package without using it in a particular simulation). However, a default version of the command is typically invoked by other accelerator settings. For example, the KOKKOS package requires a "-k on" command-line switch respectively, which invokes a "package kokkos" command with default settings.
NOTE: A package command for a particular style can be invoked multiple times when a simulation is setup, e.g. by the "-k on", "-sf", and "-pk" command-line switches, and by using this command in an input script. Each time it is used all of the style options are set, either to default values or to specified settings. I.e. settings from previous invocations do not persist across multiple invocations.
See the the Accelerating SPARTA section of the manual for more details about using the various accelerator packages for speeding up SPARTA simulations.
The kokkos style invokes settings associated with the use of the KOKKOS package.
All of the settings are optional keyword/value pairs. Each has a default value as listed below.
The reduction keyword sets the type of reduction used to gather statistics. The parallel/reduce option uses a parallel reduction and is typically the preferred method when running on CPUs and Xeon Phis. The atomic option uses thread atomics and is typically faster when running on GPUs.
The comm keyword determines whether the host or device performs the packing and unpacking of data when communicating per-atom data between processors. The value options are threaded or classic.
The optimal choice for this keyword depends on the hardware used. When running on CPUs or Xeon Phi, the classic option is typically fastest. When using GPUs, the threaded value will typically be optimal. In this case data can stay on the GPU for many timesteps without being moved between the host and GPU. This requires that your MPI is able to access GPU memory directly. Currently that is true for OpenMPI 1.8 (or later versions), Mvapich2 1.9 (or later), and CrayMPI.
This command cannot be used after the simulation box is defined by a create_box command.
The kk style of this command can only be invoked if SPARTA was built with the KOKKOS package. See the Making SPARTA section for more info.
suffix, "-pk" command-line setting
For the KOKKOS package, the option defaults are comm = threaded, and reduction = parallel/reduce. These settings are made automatically by the required "-k on" command-line switch. You can change them by using the package kokkos command in your input script or via the "-pk kokkos" command-line switch.