Is AI Reliable?

I ran my news summary AI shortcut twice yesterday, only one minute apart. I had received no new emails in the minute between the two runs, but it returned different results. The shortcut gathers my news emails from a folder, sends them to Claude with the instructions to find the top 5 stories across the emails. Across the two runs, the top topics were the same, but not their descriptions, and the fifth was different. 

The focus on different points for each headline is an important reminder that LLMs are nondeterministic. That means each time you ask it to complete a task with the same inputs, you will get different results. This is not inherently “good” or “bad”, but it is a very important factor when deciding how to deploy AI. Here’s what Claude has to say about it: 

Susan: I'm writing an article on how LLMs are nondeterministic. Can you share an explanation, in 5 sentences or less, about why I get different results when I ask you for the same task more than once?

Claude: When generating text, I predict the next token (roughly a word or word-piece) one at a time by producing a probability distribution over my vocabulary. Rather than always picking the single most likely token, I sample from that distribution — introducing randomness at every step. A parameter called temperature controls how much variation is allowed: higher values flatten the distribution and let less-probable tokens through more often. Because each token choice shifts the context for the next prediction, small differences early in a response compound into meaningfully different outputs by the end. This is largely intentional — it's what makes language models generative rather than deterministic lookup tables.

Wow, that last sentence is hitting me hard: “it’s what makes language models generative rather than deterministic lookup tables.” It says a lot about what LLMs can do, and cannot do.

Companies I’ve worked with or talked to want reliably reproducible results from their systems. An example from my recent past is building an LLM-based translation system, for which the model needed to follow our brand guidelines per country and language, including a dictionary of terms that must be used. When we first built the system, it was AI alone and, while it followed the brand guidelines overall, we weren’t reliably seeing our dictionary of terms used. As Claude says above, it is not a lookup table. Our solution was to pair the AI translation with Retrieval Augmented Generation (RAG). It allows the AI model to retrieve (i.e., lookup) information from your knowledge base before generating the response.  After adding RAG, our AI translations received significantly higher acceptance rates from the translation users. 

I recommend focusing AI on non-deterministic tasks, like translations, that do not have or need a “right answer”. If your company has a low-risk profile, turn that temperature parameter down to 0. For all other tasks, let deterministic systems manage the rule-based automation. AI is too expensive to use when you don’t need it, and a waste of resources if it isn’t generating results you can use. I’ll keep using AI for my news summaries, but I’m going to check in on my temperature parameter, and maybe ask for more than the top 5 topics.