Conference Report: COLM 2026`\footnote{Published as blog post at \url{https://www.romanklinger.de/blog/2026-10-23-colm/}}`{=latex}

Introduction

It’s been a while that I wrote a conference report. My last one was about ACL 2025, but I went to two more conferences afterwards: EACL 2026, which was an interesting experience because of it’s comparably small size (in contrast to ACL or EMNLP these days), but also because of the place (Rabat, Morocco), which I did personally not like for reasons of personal safety and censorship by the hosting country, that a couple of people I talked to experienced. Further, I’ve been to LREC 2026 in Palma de Mallorca, a conference is not considered a flagship venue by some people in the CL/NLP community, but the place to be to report on data sets and resources. It’s also a very welcoming venue for less fashionable topics and good discussions.

Nevertheless, I didn’t find the motivation to write a conference report about these two conferences, because in the end, they were not very surprising: A lot of LLM work, many interesting papers, both well organized, but nothing I felt that needed to be pointed out in particularly. Still, maybe I’ll still summarize my thoughts about the “smaller” venues in CL/NLP at some point in a blog post. Another reason is that I am constantly overwhelmed by the amount of work I am faced with as a still comparably new full professor.

That situation didn’t change too much, but the COLM conference I attened is a comparably new, doesn’t live in the ACL/CL/LREC world as much, and has a different focus. Therefore, I felt it might be more helpful for some people to read my impressions about the conference to decide if they might want to consider submitting to COLM in addition to their regular ACL Rolling Review submission cycle, and/or COLING and LREC, or TACL and the Computational Linguistics Journal.

Let’s get started…

The Third Annual Conference on Language Modeling

The COLM 2026 took place beginning of October in San Francisco. It’s only the third iteration of the venue, and I remember 2024, when the organizers reached out to the community about it. It’s an interesting venue – language models are around since a very long time. They are generally models that estimate probabilities of language – given a word sequence, what is the likelyhood that it would be found like that in the real world (or a subsample of it which is used to estimate the model). Another definition could be more focused on prediction and generation, following formal grammars @chomsky_three_1956, but I want to focus on the statistical perspective here. Such probabilities for a sequence of words can be estimated for instance by counting the frequency of particular n-grams (groups of consecutive words of length n), nicely described in @jurafsky2026ngram. While these models are around for decades, language models entered the world of many people only with the visibility of neural language models and their use in chat systems which are considered to be helpful, such as Gemini or ChatGPT. These systems are not language models, but they use language models as part of their system.

As these models are sometimes considered general problem solvers (though this idea is much older), they also and particularly solve language tasks such as generation and understanding. Therefore, they dominate the CL/NLP conferences these days; and these conferences increase in size, and my impression is that non-LM topics are pushed to the side, at least to some degree. This is not a problem in COLM, of course: COLM is designed to focus on language models, and to be honest, this gave me quite some piece of mind. I didn’t need to search for other methods I like around CL/NLP (like probabilistic graphical models, for instance), and there was no need to be disappointed. I first thought that this conference will feel narrow. It did not. Not at all.

Review Process and Statistics

The conference was still different from *ACL/LREC/COLING conferences. The review process took place via [OpenReview](https://openreview.net/], the platform that ACL moved to after softconf went down a couple of times on the paper submission deadline day, presumably due to large submission numbers. OpenReview does, however offer more possibilities than just reviewing – it has the idea of being open, and also offers to publish the papers afterwards. Anyway, the differences between ARR (the rolling review process happining for ACL and COLING these days) felt minor. The program chairs mentioned that hierarchies in COLM are more shallow than in ACL (reviewers+AC+PC, but no senior area chairs in COLM), but as a reviewer and author, I didn’t perceive this too much. The reviews we got for our papers were of very good quality.

The conference received 2939 submissions and accepted 856 papers. These 29% are similar to the acceptances rates of main papers in ACL conferences together with Findings papers, which are often presented as well. I did not perceive a qualitative difference in the program of COLM to ACL or EMNLP. In fact, I have only seen one paper where I wondered how it slipped through in the review process - I typically think that a bit more often (but that could also be because I am more knowledgeable in ACL topics than language model topics).

Interesting was the distribution of countries. While ACL in Vienna had 19% authors from the US and 50% authors from China (Germany had 3%), the COLM distribution was different:

• USA: 62%
• China: 10%
• Canada: 3.9%
• Great Britain: 3.3%
• South Korea: 3.2%
• Germany: 2.6%
• Switzerland: 2%

That is of course not a surprise. Chinese scholars have more often issues to travel to the US than the Europe, and of course there are practical reasons that San Francisco is easier to reach from other places in the US.

It was, however, the first conference which I attended which actually sold out: authors needed to buy their tickets earlier, but then, only 3000 people were admitted altogether. Another difference was that (I only heard that from somebody, so I am not sure if it’s accurace) 1/3 of the participants were not from Academia, a number which was claimed to be higher in other CL/NLP venues (again, not sure if true).

Organization

The conference was organized very differently than other (big) conferences I attended:

• It was a single track conference, except for the workshop days, there were never multiple oral sessions taking place in parallel.
• The conference was structured in 6 slots (3 days, morning/afternoon), where each slot started with a keynote or a panel, followed by 4 spotlight presentations (which were also available in the poster session), followed by the poster session.

That was easy to follow, but there was no change to choose between oral presentations, when one session wasn’t too much in the own interest. Nevertheless, it felt less stressful than other conferences, because no opportunity/change/need to run between sessions.

The poster sessions were not thematically structured, as much as I could see. That was also not the case at ACL in Vienna, and I find that a bit unfortunate. I pick the posters that I want to see from the program, but I also like to walk around the aisles and talk to authors of posters that just seem interesting. At ACL, this was problematic because of incredibly huge poster sessions which were incredibly crowded. It was also very loud.

The last aspect was much better at COLM (because of the venue with lower ceilings and carpet in the rooms). The sessions were also not that huge. They were still too crowded, and there was too little space around each poster, but: I actually perceived it different than in Vienna: At COLM, I was happy that the posters were not thematically clustered, because I could just talk to the authors of 3-6 posters in each session that I found interesting. If they were clustered, I would have had very stressful and very boring sessions. Just one wish: put posters that are thematically similar closer to each other, such that one doesn’t need to walk that far. But as the venue wasn’t huge, that was also not such a big deal.

But generally, I still wish for:

• more space around each poster in poster sessions
• thematically similar posters should be close to each other

Altogether, this was not a small conference at all, but it felt much more manageable than the last ACL that I attended in Vienna. (EACL in Morocco and LREC in Mallorca where smaller and more managable anyway)

Another interesting aspect I found was that there was no social event anywhere. Companies invited to socials, but there was no evening dinner or welcome reception. I really, really missed that. I didn’t know as many people at this conference as a do know in the CL/NLP world, and I would have really liked to get to know more people who I didn’t know yet, but that was more difficult than expected. In this regard, COLING is my favorate place, by the way, who typically offer trips in the middle of the conference during which you definitely start to talk to random people. I would wish COLM to adapt that concept.

Conference Program

I need to start with our own contributions in this blog post, because.

We had just one paper, and it was the first time (if I recall correctly) that we submitted to COLM. It’s the work by Sean Papay (and me) @papay2026making, in which we propose a decoding algorithm that can consider lexical constraints; words that need to appear in the output. Current methods are either computational efficient, or prefer constraint words that are more common on a language to appear first in the output. Our method is fair regarding this aspect and computationally efficient.

Now, let’s talk more about the program, which is, particularly for this conference which is different than others I attended so far, interesting. The papers are all available at OpenReview. To get an overview, I downloaded all paper titles and ran a topic model (non-negative matrix factorization on one-hot vectors):

1. Synthetic data generation (from LMs and for LM training)
2. Long context architectures
3. RAG systems
4. Code systems
5. Self evolution, including agentic skills
6. Efficient inference
7. Preference optimization and alignment, including safety
8. Multimodal reasoning

This topic list also reflects my own impression of what dominated the conference. Nevertheless, I will not follow this categorization in my own paper overview, because that topic list does not reflect too well my own interests. Instead, I will follow these categories (which do NOT reflect what has been popular at the conference, but purely my own research interests):

We had a set of contributions to this conference, which I would like to briefly summarize in the following.

• @bamnlp2025 reports on our experiments with socio-demographic prompting for offensive language detection with instruction-tuned models. We tested if socio-demographic prompting (make prediction from the perspective of a person with a particular age or gender) has an effect stronger than pseudo-demographic prompts (the house number of a person). Further, we tested to which demographics a prediction is most similar if no demographics are provided in the prompt. We found, as expected, that some particular demographics seem to be better represented in large language models. This paper was the result of a joint effort from the whole group – writing one paper in one week during our first retreat. We are very happy that this paper made it into the main conference; but we also agree that the stress level of writing a paper in this short amount of time was too high. By the way, thanks to Steffen Eger for the idea to do such type of retreat. It worked really well to get to know each other and that was clearly a success in the retreat.
• @bagdon-etal-2025-donate studied various ways to get emotion and appraisal annotated data. In our project ITEM (with Carina Silberer from Stuttgart), we investigate how and why social media users express their emotions, particularly implicitly with text and images. In the paper we published at the main conference of ACL 2025, we wanted to understand if asking crowdworkers to create a post for a given emotion and including an image that would realistically use from an image data base works as a reasonable approximation for realistic data. The advantage would be that the data has less data privacy issues and copyright issues than real data. We compared these data to “donation” (we paid for them though) of real posts from users. We find that the experimentally elicited data is fine as training data, but to study the phenomena one needs real test data.
• @greschner-etal-2025-qolas presented her work on the question if we can automatically detect aspects that influence the perceived quality of life of people with mental disorders from social media, which are not yet known. To do so, she annotated data, built classifiers, and did topic modeling and found a set of aspects that were not yet represented in standardized test instruments.
• @papay-etal-2025-regular proposed a method to consider long-distance relations in text on the output level – with a conditional random field (joint work with Sebastian Pado from Stuttgart). This CRF could be put on top of a neural network and is therefore a relevant option for an output layer. Most importantly, Sean found a way to decode in linear runtime, which is not generally possible for loopy probabilistic structures.
• @li-klinger-2025-iprop published the first paper from our INPROMPT project, in which we develop prompt optimization and engineering methods that involve a human user whenever automatic optimization is not sufficiently successful. Therefore, the proposed methods support human prompt developers. The paper in the student research workshop summarizes the project plans and discusses the upcoming research questions and tasks.
• @hofmann-etal-2025-prompt has been presented by me, but the work has been conducted mainly by Jan Hofmann (in collaboration with Cornelia Sindermann from Stuttgart and Ulm). In this work, we studied language model based agents which learn which posts in a social media profile are helpful for personality profiling. The data is only annotated on the profile level, so we use a reward function for reinforcement learning to learn to distinguish relevant and irrelevant posts. The method could readibly be transferred to any other long-text analysis and shows substantial runtime and cost savings, because the prompt that makes the personality prediction can work without a lot of provided context.

My Favorite Contributions

In the following, I want to highlight some papers that have been presented (or at least published) at this ACL. As I said - I found this conference particularly difficult to navigate; and if I don’t mention a paper that you would expect me to like it doesn’t mean that I didn’t like it. I probably just missed it (and I’d appreciate if you told me about that paper that I should read.).

Emotion analysis

• @palma-etal-2025-llamas aim at understanding where emotion and sentiment information is represented in large language models. They then train small models on the local emotion/sentiment representation which works better than fine-tuning the whole model (and it is cheaper).
• @du-hoste-2025-another propose to calculate annotator disagreement not based on categorical values but instead map them to a valence and arousal space in which the continuous values are used for an error estimation. They show that such disagreement calculation is a more realistic estimate.
• @barz-etal-2025-understanding do also focus on inter-annotator agreement, but more on understanding (the reasons for) disagreement. The authors annotate a corpus on environmental aspects and analyze it for topics and emotion distributions. Understanding disagreement was mostly analyzed based on qualitative interviews and less on statistical analyses. One main reason for disagreement were different perspectives, another reason to build personalized models and include contextuali information in corpora (like we did for instance in @troiano-etal-2023-dimensional, but the idea of qualititative interviews in this ACL2025 paper are a good idea that I really like).
• @lee-etal-2025-large detect neurons that are particularly relevant for particular emotions and show that removing them comes with a drop in emotion classification performance.
• @li-etal-2025-third reproduce prior work that shows that readers have limited ability to reproduce writer’s emotions; and LLMs are better than humans. The particular novelty is that the authors distinguish ingroup and outgroup annotations. The related work section is unfortunately a bit limited in this paper - there has been work that failed to show such influence of demographic factors in natural language processing (while its known across other modalities). I still would like to understand which factors influence if in/outgroup context matters or not.
• @lee-etal-2025-emotional is an interesting study, because the authors use entirely automatically generated data, and then study how language models analyze this artificial data. Currently, I do not have a good understanding what the findings in the paper mean - because neither the data nor the annotations are human-made or naturally occurring. I admit that the data is generated based on human data though, but it is not clear to me if the findings therefore generalize to data as it occurs in the wild naturally. A similar criticism also applies to a lot of studies we do, in which we elicit data from humans in non-natural experimental environments. I think the question how much such analysis allow interesting insights is still an open research question.
• @muhammad-etal-2025-brighter is not just another emotion data set. It is a corpus for many languages, and many of which did not receive enough attention yet. The corpus is manually annotated, contains many domains and various genres. It contains intensity and categorical labels.
• @duong-etal-2025-cheer is the first work that I am aware of that annotates emotion expressions for bodily reactions. We did also find in @casel-etal-2021-emotion that a substantial number of emotion expressions use body descriptions, so it is really nice to see this work. The authors also rely on automatic annotation with best-worst scaling, as proposed by @bagdon-etal-2024-expert.

Appraisals in Emotion Analysis

• @tak-etal-2025-mechanistic builds on top of our Crowd-enVent corpus to study the cognitive evaluation process taking place in emotion event processing. While our corpus only provided emotion and appraisal annotation and predictions (@troiano-etal-2023-dimensional) the authors of this paper really focus on understanding how LLMs process emotions and if that process is aligned with human processing. To do so, they build on top of the idea of mechanistic interpretability, by probing the model. A very impressive idea in this paper is to make use of the model understanding to then intervene on the cognitive evaluation process to study the relation to the emotion category. I like appraisals and the authors use our data, so I am biased, but this paper goes the extra mile to bring together LLM introspection methods with psychological concepts.
• @yeo-jaidka-2025-beyond build on top of appraisals, which they consider to be a fundament for the interpretation of implicitly expressed emotions, to curate a data set focused on the Theory of Mind. They focus therefore not so much on the analysis of emotions from one particular perspective, but on the interpretation of an emotion in a person as a private state. I think this is also quite related to various work on empathy. While I really like the idea, this paper suffers a bit from the lack of a related work section (due to it being a short paper, but still, the context of this work is a bit opaque for me).
• @debnath-etal-2025-appraisal train an appraisal predictor on our appraisal data set Crowd-enVent and automatically label dialogue data to study the information flow in dialogues. The paper therefore brings together event-centered emotion analysis (@klinger-2023-event) and emotion recognition in conversations (@pereira_deep_2024). They do so in a multi-task learning setup, which may also benefit from the emotion labels in the conversation data.

Personality

• @wei-etal-2025-mecot ensure that a dialogue, guided by an LLM is consistent regarding the emotion and the personality. They do so by modeling the emotion and personality transitions with a Markov chain. What is not clear for me in this paper is if personality and emotions are handled differently according to the fact that one is a state and the other are traits.
• @lim-etal-2025-persona show how agents in text-based games change their behaviour based on different personality traits.
• @hartley-etal-2025-personality study how LLMs change their risk-taking behaviour based on differing personality traits given as conditions. This work is related to our work on measuring regulatory focus theory (RFT, promotion or prevention orientation), but we did only build classifiers (@velutharambath-etal-2023-prevention). The authors here do use personality conditions for guiding the behaviour of an agent. Bringing RFT and such studies together could be an interesting step in future work.

Other

• @wu-etal-2025-clamp-3 may be the first paper on music information retrieval I have seen at ACL conferences. They authors align sheet music, audio recordings, performance data and multilingual text for an improved retrieval process.
• @quensel-etal-2025-investigating study subjective factors of argument strengths. Their work aggregates various aspects such as emotions, hedging and storytelling in a joint analysis. The emotion labels stem from a domain transfer of a predefined corpus. Next to our work ((@greschner-klinger-2025-fearful)[https://aclanthology.org/2025.nlp4dh-1.52/]) this is one of the few studies that do not consider binary emotionality but distinguish various emotion categories.
• @menis-mastromichalakis-etal-2025-dont advcocate for not removing harmful information from historic sources; but instead automatically contextualize the information, such that it is better understood. I find this is an interesting perspective on offensive language processing.
• @pramanick-etal-2025-nature is a meta-study on the research field of NLP. The authors show empirically that the focus on language shifts towards more computational methods, people care more about human-centric studies, and that there is a steady increase in methods and data sets.
• @russell-etal-2025-people probably has the best title in this conference, because it makes it very easy to summarize the main result: “People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text”
• @sicilia-alikhani-2025-evaluating also study theory of mind (as mentioned further above), but with a focus on uncertainty prediction. The authors propose a benchmark to evaluate the uncertainty in participants in a dialogue. Therefore, the prediction is really not about the language model, but of a second order. Very interesting idea and a new twist to uncertainty prediction!
• @corso-etal-2025-conspiracy propose data and methods to find conspiracy theories on TikTok. An interesting task and setup. What remains is a study what the properties of these conspiracy theories are, and if also novel instances can be found. Otherwise, the task might not focus on properties of the instances, but only on similarities.
• @chen-etal-2025-empathy ask people to judge the own perceived empathy in a story, without clearly defining the task for the annotators. This is an interesting idea, because it leaves the decision what “empathy” actual means to the annotators. Maybe it is related to stance or opinion in this setup.
• @jin-etal-2025-multi is the first work I have seen that does study argument quality with a clear perspectivism angle - people with different backgrounds assess arguments differently. Unfortunately, the persona descriptions are automatically generated, and the assessment and rational generation also seems to only be automatic. It is not clear to me if there is human annotation from these various personas is involved.
• @yang-jin-2025-matters perform book-long evaluations automatically, but with the help of human assigned scores. The setup is quite interesting: the authors automatically structure book reviews into a structured representation; and then, they develop methods to automatically assess these scores from the book alone. This is challenging because of the text length, and the authors propose various approaches for aggregation into a shorter representation.
• @cahyawijaya-etal-2025-crowdsource is a paper that is not exactly related to my main interests - it’s about a data set to develop vision-language models. The interesting aspect for me here is that the authors evaluate different ways to collect the data: they crowdsource, crawl or generate. Therefore, this paper is quite related to the corpus we publish at the same conference. Our paper is called “Donate or Create”. While both terms in our case refer to crowdsourcing, there is an interesting overlap in methodology (@bagdon-etal-2025-donate). The authors of this paper, however, do also evaluate automatic data generation, which is something we did not do (yet). By the way, it’s also the first paper which has enough authors such that the abstract continues on the second page ;-).
• @bavaresco-etal-2025-llms is a very nice exception from the many papers that ask “can LLMs do X” by studying the same question in a systematic manner, across many tasks. I think this is a very natural but very well carried out study that consolidates various ideas that came up in recent work. I assume this will be one of the mostly highly cited papers in this conference.
• @chen-eger-2025-emotions describes results that come from the same project as @greschner-klinger-2025-fearful. The authors of this paper do automatically generate non-emotional arguments and emotional arguments with language models, to setup a human annotation study in a controlled manner.

The whole proceedings are available in the ACL Anthology.

Venue and Place

The conference took place in Vienna - a city I recently visited for KONVENS, so my pressure to do sightseeing was not too strong. The conference was north of the Danube, in an area I have not seen so far, and it was mostly a modern concrete building with more high concrete buildings around. What was really nice is that I could cycle every day from the hotel over a bridge to the venue. Further, there was a beach/river promenade-like area with some restaurants around; where one could also go swimming. This was quite nice.

The social event took place in the conference venue, probably the only possible decision with a conference of this size. I still think that such conference dinners should not serve meat, given how many animals they alone are responsible then to kill, but with this opinion I seem to be quite alone. The vegetarian food quality was good, though.

Next to the unavoidable (and probably expected-by-many) Waltz session, the DJ had a sax player and two singers; and they were playing Electro Swing. I did unfortunately not learn who that was, but if any of you knows, please tell me. I like this type of music quite a lot and was very happy about this; such parties do not take place in areas in which I live. Dancing was, however, not possible for me - the floor was moving so strongly that I couldn’t stay in this area without fear ;-). (nothing happened though)

Bibliography

[Download this post as PDF]