This is a Triton implementation of the Flash Attention v2 algorithm from Tri Dao (https://tridao.me/publications/flash2/flash2.pdf) ...

For example, the parents of variables change from one regime to the other, and/or the strength and function of their dependencies. A straight-forward way to design a model that follows this hypothesis ...