Chapter 07
Chapter 7: Domain decomposition
Context: Parallelization in molecular dynamics typically occurs through domain decomposition. The simulation domain is divided into subdomains, each of which runs within an MPI process. This distributes the workload among different compute units. Communications occurs only at the interface of the subdomain, either to exchange atoms between subdomains or to communicate ghost atoms that are required for the computation of correct forces in short-range interatomic potentials.
7.1 Simulation domain
Our atomic system has so far lived in an infinite space consisting of vaccum. We have made no reference to a simulation domain and the code developed up to Milestone 07 makes not reference to such a domain. We now introduce domain decomposition and for this need a simulation domain, i.e. a region of space () in which our atoms can reside. This domain can be periodic, which we will discuss in more detail in the next chapter.
We will assume that the simulation has its origin at ((0,0,0)) and is spanned by three linearly independent vectors (_1), (_2) and (3). Any atomic position can then be expressed as with (s(#Xbruge_concurrent_1990) (1990); Liem et al. (1991); Chynoweth et al. (1991); Pinches et al. (1991); Brown et al. (1993); Plimpton (1995).
7.3 Ghost atoms
https://uni-freiburg.cloud.panopto.eu/Panopto/Pages/Embed.aspx?id=03a83e2f-54d9-4423-acb8-ad6b007de308
The atoms within each subdomain are not sufficient to compute the forces upon these atoms. In order to compute forces for atoms near the domain boundary, we need to transfer atoms that sit outside of the subdomain from the neighboring subdomains. These atoms are called ghost atoms. All atoms up to a distance (r_) from the subdomain boundary are transferred. For a Lennard-Jones potential, (r_=r_c) but for the EAM potential discussed here (r_=2 r_c). This is because a force in the EAM potential is affected by an atom that sits twice the cutoff radius (r_c) away.
7.4 Communication pattern
https://uni-freiburg.cloud.panopto.eu/Panopto/Pages/Embed.aspx?id=2ff96a92-b994-4cee-84b6-ad6b008fc0d3
The basic communication pattern involves two MPI_Sendrecv commands per Cartesian direction. The atoms that are send (either exchanged or as ghost atoms) must be serialized into a send buffer. Given that serialization has occured into the buffers send_left and send_right, the communication pattern looks as follows:
1MPI_Sendrecv(&send_left, left_, &recv_right, right_, comm_)}; 2MPI_Sendrecv(&send_right, right_, &recv_left, left_, comm_)};
Here comm_ contains the MPI communicator and left_ and right_ the MPI ranks of the processes that host the subdomain to the left and the right, respectively, of the current subdomain. The buffers recv_left and recv_right hold the serialized atomic information received from the left and right, respectively. This information needs to be deserialized into the respective atom data type.
Bibliography
D. Brown, J. H. R. Clarke, M. Okuda, and T. Yamazaki. A domain decomposition parallelization strategy for molecular dynamics simulations on distributed memory machines. Comput. Phys. Comm., 74(1):67–80, 1993. URL https://doi.org/10.1016/0010-4655(93)90107-N.
F. Brugè and S. L. Fornili. Concurrent molecular dynamics simulation of spinodal phase transition on transputer arrays. Comput. Phys. Comm., 60(1):31–38, 1990. URL https://doi.org/10.1016/0010-4655(90)90076-D.
S. Chynoweth, U. C. Klomp, and L. E. Scales. Simulation of organic liquids using pseudo-pairwise interatomic forces on a toroidal transputer array. Comput. Phys. Comm., 62(2):297–306, 1991. URL https://doi.org/10.1016/0010-4655(91)90102-Q.
S. Y. Liem, D. Brown, and J. H. R. Clarke. Molecular dynamics simulations on distributed memory machines. Comput. Phys. Comm., 67(2):261–267, 1991. URL https://doi.org/10.1016/0010-4655(91)90021-C.
M. R. S. Pinches, D. J. Tildesley, and W. Smith. Large scale molecular dynamics on parallel computers using the link-cell algorithm. Molecular Simulation, 6(1-3):51–87, 1991. URL https://doi.org/10.1080/08927029108022139.
S. Plimpton. Fast parallel algorithms for short-range molecular dynamics. J. Comput. Phys., 117(1):1–19, 1995. URL https://doi.org/10.1006/jcph.1995.1039.