Prompt Engineering for Engineers: Roles, Few-Shot & Chain-of-Thought

Prompts Are Structured Messages, Not Just Text

Every LLM API call is really a list of role-tagged messages, not one blob of text. The system message sets persistent behavior for the whole conversation — tone, persona, constraints. The user message is the actual request. The assistant message is the model’s reply — and in a multi-turn conversation, earlier assistant replies get fed back in as context for the next turn. Change only the system message and the entire tone of the response changes, even with an identical user question:

How Your Prompt Talks to Itself

Here’s the actual mechanism behind everything above. The self-attention formula from the “How LLMs Work” Advanced page — softmax(QK^T / √d_k) — doesn’t just run on tokens inside the model’s training data. It runs on your prompt’s own words, every time. Every word in the prompt below computes an attention weight toward every other word in it, including itself. Click a word and watch which other words it’s actually “listening to” before the model decides what comes next:

Click “Explain” and notice it isn’t just attending to itself — “photosynthesis”, “simply.”, and “teacher.” all pull real weight. That’s not a coincidence built into this demo — it’s the entire reason role and format instructions change the output at all. They aren’t suggestions the model reads once and discards; they’re words the model keeps attending back to at every single generation step.

The softmax Piece, in Code

The part of that formula actually doing the “how much attention” work is softmax. Given any list of raw scores, it turns them into a probability distribution — every value between 0 and 1, all of them summing to exactly 1 — which is exactly what the bars above are showing for whichever word you clicked:

import numpy as np

def softmax(scores):
    # Subtract the max before exponentiating — purely for numerical
    # stability (avoids overflow on large scores). Mathematically it
    # doesn't change the result at all.
    scores = np.array(scores, dtype=float)
    shifted = scores - np.max(scores)
    exp_scores = np.exp(shifted)
    return exp_scores / np.sum(exp_scores)      # every value in [0,1], sums to 1

# "Explain"'s raw affinity scores toward the 8 words in the demo above
raw_scores = [0.3, 0.3, 0.3, 1.4, 1.5, 3.0, 2.3, 2.0]
weights = softmax(raw_scores)

print(np.round(weights, 3))
# [0.027 0.027 0.027 0.081 0.09  0.401 0.199 0.148]
# same numbers as the "Explain" bars in the widget above

Few-Shot Prompting: Show, Don’t Just Tell

Instead of describing the format or judgment call you want, show a few worked examples right in the prompt — this is few-shot (or “multishot”) prompting. It’s especially effective for judgment calls that are hard to fully specify in words, like borderline sentiment or tone matching. Toggle examples on below and watch a genuinely ambiguous test case get both more accurate and more confident:

Structuring Output for Parsing

If downstream code has to consume the model’s response, free-text output is a liability — extracting “the age” from a sentence means writing fragile string-parsing logic. Asking for XML tags or a JSON schema instead turns the response into something a parser can trust every time:

Chain-of-Thought: Asking the Model to Show Its Work

Multi-step problems are where models most often skip a step when asked to jump straight to an answer. Adding an instruction like “let’s think step by step” pushes the model to generate intermediate reasoning tokens first — and each of those tokens becomes extra context the model can condition on before committing to a final answer:

Worth knowing: this gap is shrinking. A 2025 Wharton study found chain-of-thought still helps older, non-reasoning models, but gives only marginal benefit — for a real time cost — on models already built to reason by default.

What the Paper Says

“

Prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

Fun Fact

A June 2025 Wharton study tested chain-of-thought prompting across both older and dedicated reasoning models. For non-reasoning models it still gave a real, useful boost. For models already built to reason by default, the same prompt trick added 20–80% more processing time for only a marginal accuracy gain — the technique that started as a universal upgrade is quietly becoming optional.

Source: Meincke, Mollick, Mollick & Shapiro, “The Decreasing Value of Chain of Thought in Prompting,” Wharton Generative AI Labs (2025)

Test Yourself

Ready for prompt caching, constrained decoding, self-consistency, and prompt injection? Continue to the Advanced level →