{"id":"arxiv:2510.26622","title":"Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model","date":"2025-10-30","section":"LLMs: architectures, context, training","quality":0.064707,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/","source_url":"https://arxiv.org/abs/2510.26622","text_available":true,"authors":["Biao Zhang","Yong Cheng","Siamak Shakeri","Xinyi Wang"],"outgoing":[{"source":"arxiv:2510.26622","target":"arxiv:2001.08361","evidence":"id","id":"arxiv:2001.08361","title":"Scaling Laws for Neural Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/82302458a6d03bc051b2e4fdb8277893a81d4bd6a029969045df6a8380904eac/"},{"source":"arxiv:2510.26622","target":"arxiv:2002.05202","evidence":"id","id":"arxiv:2002.05202","title":"GLU Variants Improve Transformer","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/e052a2974593d29355ab45c301766ac6f5545ac1bc81d605e78d3adc71e9e2e4/"},{"source":"arxiv:2510.26622","target":"arxiv:2101.02235","evidence":"title","id":"arxiv:2101.02235","title":"Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5b44455f95aaaeefc838faceeaf20aaf1008889ca2e479d95291e245d5365a36/"},{"source":"arxiv:2510.26622","target":"arxiv:2206.04615","evidence":"title","id":"arxiv:2206.04615","title":"Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/70882486c1f8c46cb7b26cf100847f770eb7b39b69dee56b8a21141141ef814d/"}],"incoming":[{"source":"arxiv:2512.14856","target":"arxiv:2510.26622","evidence":"id","id":"arxiv:2512.14856","title":"T5Gemma 2: Seeing, Reading, and Understanding Longer","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/e982edc00d6b6d9e20d11b62004bb8e91131e2d7f717eca944a9914897f273b6/"}],"text_parts":[{"start":0,"end":15271,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html"},{"start":15271,"end":28925,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html"}],"contents":[{"title":"Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"Abstract","offset":84,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"1 Introduction","offset":1443,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"2 Background","offset":5862,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"3 RedLLM: Revisiting Encoder-Decoder LLM","offset":10364,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"4 Setup","offset":12886,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"Pretraining","offset":13132,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"Finetuning","offset":13964,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"Evaluation","offset":14495,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-1.html#text"},{"title":"5 Pretraining Results","offset":16320,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"RedLLM and DecLLM scale at similar rates when increasing the compute (training Flops) and model parameters.","offset":16346,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"RedLLM lags behind DecLLM for compute-optimal training.","offset":17502,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"RedLLM scales slightly for few-shot learning but is poor at zero-shot; both largely underperform DecLLM.","offset":18546,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"RedLLM shows comparable and even better length extrapolation capability than DecLLM along model scaling.","offset":19726,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"The decoder self- and cross-attention in RedLLM show intriguing patterns under long context.","offset":21169,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"6 Finetuning Results","offset":22890,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"RedLLM shows high adaptability: matching and even surpassing DecLLM across scales after finetuning.","offset":22915,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"Bidirectional input attention improves DecLLM greatly but doesn’t change the quality-compute frontier.","offset":24782,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"RedLLM has clear advantage over DecLLM on training and inference efficiency.","offset":25778,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"7 Conclusion and Future Work","offset":26425,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"Acknowledgments","offset":27720,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"Appendix A Appendix","offset":27937,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"},{"title":"Instructions for reporting errors","offset":27961,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/part-2.html#text"}],"text_code_points":28925,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/dc73f87f798d016e2d83acd1d362b4b3525af826e68b055bb5196f120d23e080/source.md"}