SCCA: Shifted Cross Chunk Attention for long contextual semantic
  expansion
Yuxiang Guo
Sparse attention as a efficient method can significantly decrease the
computation cost, but current sparse attention tend to rely on window self
attention which block the global information flow. For this problem, we present
Shifted Cross Chunk Attention (SCCA), using different KV shifting strategy to
extend respective field in each attention layer. Except, we combine Dilated
Attention(DA) and Dilated Neighborhood Attention(DNA) to present Shifted
Dilated Attention(SDA). Both SCCA and SDA can accumulate attention results in
multi head attention to obtain approximate respective field in full attention.
In this paper, we conduct language modeling experiments using different pattern
of SCCA and combination of SCCA and SDA. The proposed shifted cross chunk
attention (SCCA) can effectively extend large language models (LLMs) to longer
context combined with Positional interpolation(PI) and LoRA than current sparse
attention. Notably, SCCA adopts LLaMA2 7B from 4k context to 8k in single V100.
This attention pattern can provide a Plug-and-play fine-tuning method to extend
model context while retaining their original architectures, and is compatible
with most existing techniques.