Claude team's guide to word engineering, a roundtable video worth watching ten times

What exactly is the prompt word project?

Why is it called "engineering" and not just "writing"?

In this internal roundtable discussion, members of Anthropic's Claude team dive deep into the core elements, best practices, and future trends of prompt engineering.

https://www.youtube.com/watch?v=T9aRN5JkmL8

As Zach Witten, Anthropic Prompt Engineer, puts it: "I think of prompting engineering as trying to get the model to do its job, to get the model to do its best, to get it to do things that we otherwise wouldn't be able to do. A lot of it is the art of clear communication. In essence, talking to the model is a lot like talking to a person, and requires a deep understanding of the model's 'psyche'. "

When asked why it’s called “engineering,” Zach explains, “ The engineering part comes from the trial and error process. One of the nice features of communicating with the model is that you have a reset button, a big ‘back to square one’ button that allows you to start from scratch. That gives you the ability to really start from scratch and try different approaches, so one thing doesn’t interfere with another. Once you have that ability to experiment and design different approaches, the engineering part comes into play.”

Qualities of a Good Tips Engineer

Amanda Askell, head of the Fine-Tuning team, shares what she believes are the qualities that make a good prompt engineer:

"First is the ability to communicate clearly, to be able to state things clearly, to understand tasks clearly, to think and describe concepts well. That's the writing component . But interestingly, I think being a good writer is less correlated to being a good prompt engineer than people think."

"The second is the ability and willingness to iterate . When I sit down with a model, in 15 minutes I might send hundreds of prompts to the model, testing it back and forth. That willingness to iterate, to think, 'What's being misunderstood here?' and then fix it is really important."

"The third point is to think about the ways that the prompt could go wrong . If you have a prompt that you're going to apply to 400 cases, it's easy to just focus on the typical case and make sure it works fine there, and then call it a day. That's the typical mistake people make. What you really should be doing is looking for unusual cases. For example, you have a prompt that says, 'I'm sending you a bunch of data, and I want you to extract all the rows where the first name starts with G.' Then you think, 'What if there's no such name in the dataset I sent? What if I didn't send a dataset? What if I just sent an empty string?' These are the cases you need to try so that you can understand what the model will do in those cases, and then give it more instructions on how to handle them."

David added: "I often work with clients who are engineers and are building something, and they have a section of their prompt where their customers will write content. They all assume that the user will enter perfectly formatted content, but in reality, user input is often full of typos, lacks punctuation, and lacks structure. "

Key Tip: Reading Model Responses

Zach stressed the importance of reading model outputs: “It’s almost become a cliché in a machine learning context that you should look at the data: ‘Look at your data.’ I feel like the equivalent in hint engineering is ‘Look at the model outputs.’ Read a lot of outputs and pay close attention to them.”

He gives the example: “ People often put instructions like ‘think about it step by step’ in their prompts, but they don’t check whether the model is actually thinking about it step by step , because the model may understand this instruction in a more abstract or general way, rather than literally ‘you must write down your thought process in these specific labels.’ If you don’t read the model output, you may not even notice that it made this mistake.”

Understanding Model Mind: The Importance of Theoretical Thinking

Alex points out the "theory of mind" component of prompt engineering: "As a prompt engineer, you need to think about how the model sees your instructions. If you're writing prompts for an enterprise use case, you also need to think about how the user talks to the model, and you're a third party in that relationship ."

David added: "Writing down instructions for a mission is very difficult. It's hard to unpack in your own head all the information that you know that Claude doesn't know and write it down. It's an extremely challenging task , stripping away all your assumptions and being able to communicate the complete set of information very clearly. "

“This is another factor that separates good and bad prompt engineers. A lot of people will just write down what they know, but they don’t really take the time to systematically break down the full set of information needed to understand the task . One obvious problem I see a lot is people writing prompts that rely too heavily on their prior understanding of the task, and when they show it to me I say, ‘That doesn’t make sense, the words you wrote don’t make any sense because I don’t know anything about your interesting use case.’”

"A good way to think about prompt engineering and the skills associated with it is, can you step back from what you know and communicate to this weird system that knows a lot but not everything what it needs to know to accomplish its task?"

Prompt clarification: Let the model help identify the problem

Amanda shared a helpful tip: “The first thing I do with the initial prompt is I’ll give it the prompt and say, ‘I don’t want you to follow these instructions, I just want you to tell me where they’re unclear, where there’s ambiguity, or where you don’t understand.’ It won’t always understand perfectly, but it’s a fun approach.”

“Also, when people see the model make a mistake, one thing they don’t usually do is ask the model directly. They can say to the model: ‘You made that mistake, can you think about why? Can you write a revised version of my instructions so you don’t make the same mistake again?’ A lot of times the model will give the correct answer and say: ‘Oh, yes, there was something unclear here, here are the fixes to the instructions,’ and then you put those fixes in and it works fine.”

The Temporary Worker Analogy: A Powerful Mental Model

Alex shared a thought experiment he often uses: “ Imagine you have this task and you hire a temporary worker to do this task. This person arrives and they’re fairly competent and understand your industry and so on, but they don’t know the name of your company . They just show up and say, ‘Hey, I was told you guys have a job for me to do, tell me what it is.’ And what would you say to that person then?”

"Sometimes I'm like, yeah, imagine this person has just been without a lot of context, but they're fairly competent, they understand a lot of things about the world . Try a first version that actually assumes that they might know things about the world, and if that doesn't work, you can make some adjustments. But usually the first thing I try is this, and then I realize 'this is what works.'"

Providing a "way out": handling edge cases

Amanda stressed the importance of giving models options for handling uncertain situations: “That’s one thing people often forget in tips. For edge cases, think about what you want the model to do, because by default it will do its best to follow your instructions, like a temp .”

" ' Give it a way out, a fallback mechanism. Give it something to do if there's a really unexpected input. 'And you also improve the quality of your data because you find all the examples where it went wrong."

Prompt historical changes in engineering

Team members had the following insights on how prompt engineering has changed over time:

Zach believes, “Whenever we get a really good hint engineering trick or technique, the next thing is how to train it into the model. So the best tricks are always short-lived . For example, for math problems, it used to be that you had to tell the model to think in steps, and that gave a huge boost. Then we thought, ‘What if we make the model naturally want to think in steps when it sees a math problem?’ So now you don’t need to do that for math problems anymore. I think those tricks are gone, or to the extent that they haven’t gone, we’re busy training them out.”

Amanda shared an interesting tip: “ When I want the model to learn a hinting technique, I give it the paper. I say: ‘Here’s the paper on hinting techniques. I just want you to write down 17 examples of this technique.’ And then it does it because it read the paper. People don’t have that intuition, they think about explaining the technique themselves, whereas I think: ‘The paper is there, the model can read it and do what I did.’”

Hints for the future of engineering

The final part of the discussion focused on the future of prompt engineering. Team members raised several interesting points:

David argues that from an information theoretic perspective: "You need to provide enough information that one thing is specified, that what you want the model to do is specified. To that extent, I think prompt engineering will always be around. It will always be important to be able to clearly state what the goal should be.

"For myself, I find myself using the model more to write prompts. One thing I've been doing is generating examples by giving the model some realistic inputs, the model writes some answers, I tweak those answers a little bit, and that's much easier than me writing the perfect answer from scratch, and then I can generate a lot of those examples. "

The core of Cue Engineering: Externalize your brain

Amanda offered a profound insight, likening prompt engineering to philosophy writing: "There is a style in philosophy writing, which is the way I was taught how to write philosophy. It's basically an anti-bullshit device, that your paper and what you write should be understandable to an educated layperson. Someone just finds your paper, picks it up and starts reading it, and they're able to understand everything."

" So I'm very used to thinking about the educated layperson when I'm writing, who's really smart, but who doesn't know anything about the subject. And that's just from years and years of writing texts in that form. And I think that's very helpful for prompt engineering, because I'm so used to that: I have the educated layperson who doesn't know anything about the subject."

"What I need to do is I need to take extremely complex ideas and make them understand it. I'm not going to be condescending to them, I'm not going to be inaccurate, but I need to express things in a way that's extremely clear to them, and Cue Engineering feels very similar. "

"So it's at least similar in that you take things out of your head and analyze them enough that you feel like you understand them. You can take any random person, a rational person, and externalize your brain to them. I think that's the core of the prompting project ."

Conclusion: Continue to learn and experiment

Through this in-depth discussion, we can see that prompt engineering is not only a technical skill, but also a way of thinking. As AI models continue to develop, the methods of prompt engineering are also evolving, but its core is always the mindset of clear communication and understanding the model.

Whether you are a researcher, developer or enterprise user, these practical experiences from the Claude team can help you better communicate with AI models and fully realize their potential. The key is to continue learning, experimenting and adapting, and constantly adjust your prompting strategy as the model's capabilities improve.

As Amanda said, the essence of final prompt engineering is the ability to convert complex thoughts in your brain into clear instructions so that AI can understand and implement your intentions . This ability to "externalize your brain" will be a key skill for effective collaboration with AI in the future.