Article
How to Reduce Memory Use in Reasoning Models
By
and 1 other
•
•
8Ksenia Se
Kseniase
AI & ML interests
None yet
Recent Activity
replied to
their
post
11 minutes ago
15 types of attention mechanisms
Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.
Here is a list of 15 types of attention mechanisms used in AI models:
1. Soft attention (Deterministic attention) -> https://huggingface.co/papers/1409.0473
Assigns a continuous weight distribution over all parts of the input. It produces a weighted sum of the input using attention weights that sum to 1.
2. Hard attention (Stochastic attention) -> https://huggingface.co/papers/1508.04025
Makes a discrete selection of some part of the input to focus on at each step, rather than attending to everything.
3. Self-attention -> https://huggingface.co/papers/1706.03762
Each element in the sequence "looks" at other elements and "decides" how much to borrow from each of them for its new representation.
4. Cross-Attention (Encoder-Decoder attention) -> https://huggingface.co/papers/2104.08771
The queries come from one sequence and the keys/values come from another sequence. It allows a model to combine information from two different sources.
5. Multi-Head Attention (MHA) -> https://huggingface.co/papers/1706.03762
Multiple attention “heads” are run in parallel. The model computes several attention distributions (heads), each with its own set of learned projections of queries, keys, and values.
6. Multi-Head Latent Attention (MLA) -> https://huggingface.co/papers/2405.04434
Extends MHA by incorporating a latent space where attention heads can dynamically learn different latent factors or representations.
7. Memory-Based attention -> https://huggingface.co/papers/1503.08895
Involves an external memory and uses attention to read from and write to this memory.
See other types in the comments 👇
posted
an
update
11 minutes ago
15 types of attention mechanisms
Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.
Here is a list of 15 types of attention mechanisms used in AI models:
1. Soft attention (Deterministic attention) -> https://huggingface.co/papers/1409.0473
Assigns a continuous weight distribution over all parts of the input. It produces a weighted sum of the input using attention weights that sum to 1.
2. Hard attention (Stochastic attention) -> https://huggingface.co/papers/1508.04025
Makes a discrete selection of some part of the input to focus on at each step, rather than attending to everything.
3. Self-attention -> https://huggingface.co/papers/1706.03762
Each element in the sequence "looks" at other elements and "decides" how much to borrow from each of them for its new representation.
4. Cross-Attention (Encoder-Decoder attention) -> https://huggingface.co/papers/2104.08771
The queries come from one sequence and the keys/values come from another sequence. It allows a model to combine information from two different sources.
5. Multi-Head Attention (MHA) -> https://huggingface.co/papers/1706.03762
Multiple attention “heads” are run in parallel. The model computes several attention distributions (heads), each with its own set of learned projections of queries, keys, and values.
6. Multi-Head Latent Attention (MLA) -> https://huggingface.co/papers/2405.04434
Extends MHA by incorporating a latent space where attention heads can dynamically learn different latent factors or representations.
7. Memory-Based attention -> https://huggingface.co/papers/1503.08895
Involves an external memory and uses attention to read from and write to this memory.
See other types in the comments 👇
upvoted
an
article
3 days ago
How to Reduce Memory Use in Reasoning Models
Organizations
Kseniase's activity
upvoted
an
article
3 days ago
upvoted
an
article
5 days ago
Article
🌁#91: We are failing in AI literacy
By
and 1 other
•
•
3
upvoted
an
article
6 days ago
Article
🦸🏻#13: Action! How AI Agents Execute Tasks with UI and API Tools
By
•
•
4
upvoted
an
article
7 days ago
Article
🦸🏻#12: How Do Agents Learn from Their Own Mistakes? The Role of Reflection in AI
By
•
•
5
upvoted
an
article
9 days ago
Article
Everything You Need to Know about Knowledge Distillation
By
and 1 other
•
•
18
upvoted
an
article
13 days ago
Article
🌁#90: Why AI’s Reasoning Tests Keep Failing Us
By
•
•
9
upvoted
5
papers
13 days ago
Guardians of the Agentic System: Preventing Many Shots Jailbreak with Agentic System
Paper
•
2502.16750
•
Published
•
10
Towards an AI co-scientist
Paper
•
2502.18864
•
Published
•
42
SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
Paper
•
2502.18449
•
Published
•
69
Beyond Release: Access Considerations for Generative AI Systems
Paper
•
2502.16701
•
Published
•
12
Slamming: Training a Speech Language Model on One GPU in a Day
Paper
•
2502.15814
•
Published
•
66
upvoted
5
papers
15 days ago
Gödel Agent: A Self-Referential Agent Framework for Recursive Self-Improvement
Paper
•
2410.04444
•
Published
•
2
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Paper
•
2305.10601
•
Published
•
12
Chain of Hindsight Aligns Language Models with Feedback
Paper
•
2302.02676
•
Published
•
1
ReAct: Synergizing Reasoning and Acting in Language Models
Paper
•
2210.03629
•
Published
•
24
Reflexion: Language Agents with Verbal Reinforcement Learning
Paper
•
2303.11366
•
Published
•
5
upvoted
an
article
17 days ago
Article
Inside the family of Smol models
By
and 1 other
•
•
7
upvoted
an
article
19 days ago
Article
🌁#89: AI in Action: How AI Engineers, Self-Optimizing Models, and Humanoid Robots Are Reshaping 2025
By
•
•
4
upvoted
a
paper
20 days ago
upvoted
an
article
20 days ago
Article
🦸🏻#11: How Do Agents Plan and Reason?
By
•
•
10