llama : support sliding window attention

[ggerganov]

For more info, see: https://github.com/mistralai/mistral-src and references there in.

Also: https://arxiv.org/pdf/2310.06825v1.pdf

With #3228 it should be relatively easy to support this.

[CoruNethron]

Trying to figure out this. I found this reference to be helpful, as it seems to emplace all necessary code in one place:
https://github.com/mistralai/mistral-src/blob/main/one_file_ref.py

Looks like Attention.forward() Py function have something in common with build_llama() // self-attention C++ part

Also, as KQ_mask already exists, it looks like, there isn't many changes needed indeed.

Probably I can manage it in a week or two, but not sure about it.

[stygmate]

Any news on that one ? 🥹
The best low memory usage models are actually based on mistral and being locked with 4096 window is very limiting for different task like document analysis.

[h3ndrik]

I think KoboldCPP introduced a similar feature with their recent rewrite of the "smartcontext" feature.

[HiroseKoichi]

I think KoboldCPP introduced a similar feature with their recent rewrite of the "smartcontext" feature.

Nope, they added cache shifting; sliding window attention is a different attention mechanism; they do two very different things.

[stygmate]

Maybe it's not appropriate to insist on it ( I apologize if this bothers you ) but this feature seems to be one with the most thumbs up on the dashboard. 😢

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.