Article
How to Reduce Memory Use in Reasoning Models
By
and 1 other
•
•
8Ksenia Se
Kseniase
AI & ML interests
None yet
Recent Activity
replied to
their
post
about 1 hour ago
15 types of attention mechanisms
Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.
Here is a list of 15 types of attention mechanisms used in AI models:
1. Soft attention (Deterministic attention) -> https://huggingface.co/papers/1409.0473
Assigns a continuous weight distribution over all parts of the input. It produces a weighted sum of the input using attention weights that sum to 1.
2. Hard attention (Stochastic attention) -> https://huggingface.co/papers/1508.04025
Makes a discrete selection of some part of the input to focus on at each step, rather than attending to everything.
3. Self-attention -> https://huggingface.co/papers/1706.03762
Each element in the sequence "looks" at other elements and "decides" how much to borrow from each of them for its new representation.
4. Cross-Attention (Encoder-Decoder attention) -> https://huggingface.co/papers/2104.08771
The queries come from one sequence and the keys/values come from another sequence. It allows a model to combine information from two different sources.
5. Multi-Head Attention (MHA) -> https://huggingface.co/papers/1706.03762
Multiple attention “heads” are run in parallel. The model computes several attention distributions (heads), each with its own set of learned projections of queries, keys, and values.
6. Multi-Head Latent Attention (MLA) -> https://huggingface.co/papers/2405.04434
Extends MHA by incorporating a latent space where attention heads can dynamically learn different latent factors or representations.
7. Memory-Based attention -> https://huggingface.co/papers/1503.08895
Involves an external memory and uses attention to read from and write to this memory.
See other types in the comments 👇
posted
an
update
about 1 hour ago
15 types of attention mechanisms
Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.
Here is a list of 15 types of attention mechanisms used in AI models:
1. Soft attention (Deterministic attention) -> https://huggingface.co/papers/1409.0473
Assigns a continuous weight distribution over all parts of the input. It produces a weighted sum of the input using attention weights that sum to 1.
2. Hard attention (Stochastic attention) -> https://huggingface.co/papers/1508.04025
Makes a discrete selection of some part of the input to focus on at each step, rather than attending to everything.
3. Self-attention -> https://huggingface.co/papers/1706.03762
Each element in the sequence "looks" at other elements and "decides" how much to borrow from each of them for its new representation.
4. Cross-Attention (Encoder-Decoder attention) -> https://huggingface.co/papers/2104.08771
The queries come from one sequence and the keys/values come from another sequence. It allows a model to combine information from two different sources.
5. Multi-Head Attention (MHA) -> https://huggingface.co/papers/1706.03762
Multiple attention “heads” are run in parallel. The model computes several attention distributions (heads), each with its own set of learned projections of queries, keys, and values.
6. Multi-Head Latent Attention (MLA) -> https://huggingface.co/papers/2405.04434
Extends MHA by incorporating a latent space where attention heads can dynamically learn different latent factors or representations.
7. Memory-Based attention -> https://huggingface.co/papers/1503.08895
Involves an external memory and uses attention to read from and write to this memory.
See other types in the comments 👇
upvoted
an
article
3 days ago
How to Reduce Memory Use in Reasoning Models
Organizations
Kseniase's activity
upvoted
an
article
3 days ago
upvoted
an
article
5 days ago
Article
🌁#91: We are failing in AI literacy
By
and 1 other
•
•
3
upvoted
an
article
6 days ago
Article
🦸🏻#13: Action! How AI Agents Execute Tasks with UI and API Tools
By
•
•
4
upvoted
an
article
7 days ago
Article
🦸🏻#12: How Do Agents Learn from Their Own Mistakes? The Role of Reflection in AI
By
•
•
5
upvoted
an
article
9 days ago
Article
Everything You Need to Know about Knowledge Distillation
By
and 1 other
•
•
18
upvoted
an
article
13 days ago
Article
🌁#90: Why AI’s Reasoning Tests Keep Failing Us
By
•
•
9
upvoted
5
papers
13 days ago
Guardians of the Agentic System: Preventing Many Shots Jailbreak with Agentic System
Paper
•
2502.16750
•
Published
•
10
Towards an AI co-scientist
Paper
•
2502.18864
•
Published
•
42
SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Paper
•
2502.18449
•
Published
•
69
Beyond Release: Access Considerations for Generative AI Systems
Paper
•
2502.16701
•
Published
•
12
Slamming: Training a Speech Language Model on One GPU in a Day
Paper
•
2502.15814
•
Published
•
66
upvoted
5
papers
15 days ago
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Paper
•
2410.04444
•
Published
•
2
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Paper
•
2305.10601
•
Published
•
12
Chain of Hindsight Aligns Language Models with Feedback
Paper
•
2302.02676
•
Published
•
1
ReAct: Synergizing Reasoning and Acting in Language Models
Paper
•
2210.03629
•
Published
•
24
Reflexion: Language Agents with Verbal Reinforcement Learning
Paper
•
2303.11366
•
Published
•
5
upvoted
an
article
17 days ago
Article
Inside the family of Smol models
By
and 1 other
•
•
7
upvoted
an
article
19 days ago
Article
🌁#89: AI in Action: How AI Engineers, Self-Optimizing Models, and Humanoid Robots Are Reshaping 2025
By
•
•
4
upvoted
a
paper
20 days ago
upvoted
an
article
20 days ago
Article
🦸🏻#11: How Do Agents Plan and Reason?
By
•
•
10