AI Model Outputs Inappropriate Language During Reasoning Process, Cause Unknown

It has been revealed that Thinking Machines' AI model 'Inkling 2 Small' repeatedly output inappropriate language during its internal reasoning process while running benchmark tests. According to an investigation by Vals AI, which specializes in AI evaluation and analysis, a total of 316 instances of inappropriate expressions were identified across 2,655 benchmark executions.

This phenomenon occurred in the part corresponding to the 'thought process' that the model performs internally to arrive at an answer. Recent reasoning-specialized AI models build up thinking steps like a draft before producing a final answer. The inappropriate expressions in this case originated from that internal process and were separate from the responses directly returned to users.

According to Vals AI's analysis, inappropriate expressions most often appeared concentrated in the later stages of cases where processing took longer and ultimately ended in failure. In other words, there was a pattern where such expressions were more likely to occur when the model reached a deadlock without arriving at the correct answer. However, only a small fraction of failed executions contained inappropriate expressions, with 96% of all failed executions containing no such output whatsoever.

The key point this figure demonstrates is that inappropriate expressions do not function as a 'predictor of failure.' If a particular output pattern consistently appeared as a sign of failure, it could potentially be utilized for filtering or early termination decisions. However, in this case, since failures without inappropriate expressions represent the vast majority, the practical value of using such expressions as a warning signal is virtually nonexistent.

While AI models unintentionally generating expressions is not uncommon, the repeated observation of such outputs in the internal reasoning process constitutes a notable event from the perspective of model quality control. In particular, for use cases such as enterprise or educational applications where output content constraints are required, the necessity for monitoring that includes the model's internal thought process has been brought into sharp focus once again.

This case exemplifies how evaluating AI models based solely on final accuracy rates is insufficient. Methods that visualize the reasoning process itself and track in detail what outputs are generated under what circumstances are likely to increase in importance in future AI evaluation. It is worth noting the trend in which the role of third-party evaluation agencies like Vals AI is becoming increasingly vital in ensuring the trustworthiness of the industry as a whole.