This is also a good paper on the subject: What Algorithms can Transformers Learn... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		hansonw on April 27, 2024 \| parent \| context \| favorite \| on: What can LLMs never do? This is also a good paper on the subject: What Algorithms can Transformers Learn? A Study in Length Generalization https://arxiv.org/abs/2310.16028

shawntan on April 27, 2024 [–]

Yes this is a good empirical study on the types of tasks that's been shown to be impossible for transformers to generalise on.

With both empirical and theoretical support I find it's pretty clear this is an obvious limitation.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact