Open in Playground
\nSYSTEM
\nYou will be provided with a thesis abstract and a suggested title for it. The thesis title should give the reader a good idea of the topic of the thesis but should also be eye-catching. If the title does not meet these criteria, suggest 5 alternatives.
\nUSER
\nAbstract: insert abstract here<\/p>\n
Title: insert title here
\nOpen in Playground
\nFor straightforward tasks such as these, using delimiters might not make a difference in the output quality. However, the more complex a task is the more important it is to disambiguate task details. Don\u2019t make GPTs work to understand exactly what you are asking of them.<\/p>\n
Tactic: Specify the steps required to complete a task
\nSome tasks are best specified as a sequence of steps. Writing the steps out explicitly can make it easier for the model to follow them.<\/p>\n
SYSTEM
\nUse the following step-by-step instructions to respond to user inputs.<\/p>\n
Step 1 – The user will provide you with text in triple quotes. Summarize this text in one sentence with a prefix that says “Summary: “.<\/p>\n
Step 2 – Translate the summary from Step 1 into Spanish, with a prefix that says “Translation: “.
\nUSER
\n“””insert text here”””
\nOpen in Playground
\nTactic: Provide examples
\nProviding general instructions that apply to all examples is generally more efficient than demonstrating all permutations of a task by example, but in some cases providing examples may be easier. For example, if you intend for the model to copy a particular style of responding to user queries which is difficult to describe explicitly. This is known as “few-shot” prompting.<\/p>\n
SYSTEM
\nAnswer in a consistent style.
\nUSER
\nTeach me about patience.
\nASSISTANT
\nThe river that carves the deepest valley flows from a modest spring; the grandest symphony originates from a single note; the most intricate tapestry begins with a solitary thread.
\nUSER
\nTeach me about the ocean.
\nOpen in Playground
\nTactic: Specify the desired length of the output
\nYou can ask the model to produce outputs that are of a given target length. The targeted output length can be specified in terms of the count of words, sentences, paragraphs, bullet points, etc. Note however that instructing the model to generate a specific number of words does not work with high precision. The model can more reliably generate outputs with a specific number of paragraphs or bullet points.<\/p>\n
USER
\nSummarize the text delimited by triple quotes in about 50 words.<\/p>\n
“””insert text here”””
\nOpen in Playground
\nUSER
\nSummarize the text delimited by triple quotes in 2 paragraphs.<\/p>\n
“””insert text here”””
\nOpen in Playground
\nUSER
\nSummarize the text delimited by triple quotes in 3 bullet points.<\/p>\n
“””insert text here”””
\nOpen in Playground
\nStrategy: Provide reference text
\nTactic: Instruct the model to answer using a reference text
\nIf we can provide a model with trusted information that is relevant to the current query, then we can instruct the model to use the provided information to compose its answer.<\/p>\n
SYSTEM Question: Tactic: Instruct the model to answer with citations from a reference text SYSTEM Question: Suppose for example that for a customer service application, queries could be usefully classified as follows:<\/p>\n SYSTEM Primary categories: Billing, Technical Support, Account Management, or General Inquiry.<\/p>\n Billing secondary categories: Technical Support secondary categories: Account Management secondary categories: General Inquiry secondary categories: USER SYSTEM – Ask them to check that all cables to\/from the router are connected. Note that it is common for cables to come loose over time. USER Tactic: For dialogue applications that require very long conversations, summarize or filter previous dialogue There are various workarounds to this problem, one of which is to summarize previous turns in the conversation. Once the size of the input reaches a predetermined threshold length, this could trigger a query that summarizes part of the conversation and the summary of the prior conversation could be included as part of the system message. Alternatively, prior conversation could be summarized asynchronously in the background throughout the entire conversation.<\/p>\n An alternative solution is to dynamically select previous parts of the conversation that are most relevant to the current query. See the tactic “Use embeddings-based search to implement efficient knowledge retrieval”.<\/p>\n Tactic: Summarize long documents piecewise and construct a full summary recursively To summarize a very long document such as a book we can use a sequence of queries to summarize each section of the document. Section summaries can be concatenated and summarized producing summaries of summaries. This process can proceed recursively until an entire document is summarized. If it\u2019s necessary to use information about earlier sections in order to make sense of later sections, then a further trick that can be useful is to include a running summary of the text that precedes any given point in the book while summarizing content at that point. The effectiveness of this procedure for summarizing books has been studied in previous research by OpenAI using variants of GPT-3.<\/p>\n Strategy: Give GPTs time to “think” SYSTEM Student’s Solution: Let x be the size of the installation in square feet. SYSTEM Student’s Solution: Let x be the size of the installation in square feet. The student’s solution is incorrect. They made an error in the maintenance cost calculation, using 100x instead of 10x. The correct total cost for the first year of operations is 360x + 100,000. Inner monologue is a tactic that can be used to mitigate this. The idea of inner monologue is to instruct the model to put parts of the output that are meant to be hidden from the user into a structured format that makes parsing them easy. Then before presenting the output to the user, the output is parsed and only part of the output is made visible.<\/p>\n SYSTEM Step 1 – First work out your own solution to the problem. Don’t rely on the student’s solution since it may be incorrect. Enclose all your work for this step within triple quotes (“””).<\/p>\n Step 2 – Compare your solution to the student’s solution and evaluate if the student’s solution is correct or not. Enclose all your work for this step within triple quotes (“””).<\/p>\n Step 3 – If the student made a mistake, determine what hint you could give the student without giving away the answer. Enclose all your work for this step within triple quotes (“””).<\/p>\n Step 4 – If the student made a mistake, provide the hint from the previous step to the student (outside of triple quotes). Instead of writing “Step 4 – …” write “Hint:”. Student Solution: First, we can ask the model to solve the problem on its own. Since this initial query doesn’t require the student\u2019s solution, it can be omitted. This provides the additional advantage that there is no chance that the model\u2019s solution will be biased by the student\u2019s attempted solution.<\/p>\n USER SYSTEM Your solution: “”” Student\u2019s solution: “”” SYSTEM Your solution: “”” Student\u2019s solution: “”” Analysis: “”” SYSTEM Ensure that excerpts contain all relevant context needed to interpret them – in other words don’t extract small snippets that are missing important context. Provide output in JSON format as follows:<\/p>\n [{“excerpt”: “…”}, USER A text embedding is a vector that can measure the relatedness between text strings. Similar or relevant strings will be closer together than unrelated strings. This fact, along with the existence of fast vector search algorithms means that embeddings can be used to implement efficient knowledge retrieval. In particular, a text corpus can be split up into chunks, and each chunk can be embedded and stored. Then a given query can be embedded and vector search can be performed to find the embedded chunks of text from the corpus that are most related to the query (i.e. closest together in the embedding space).<\/p>\n Example implementations can be found in the OpenAI Cookbook. See the tactic \u201cInstruct the model to use retrieved knowledge to answer queries\u201d for an example of how to use knowledge retrieval to minimize the likelihood that a model will make up incorrect facts.<\/p>\n Tactic: Use code execution to perform more accurate calculations or call external APIs SYSTEM SYSTEM “`python Strategy: Test changes systematically Evaluation procedures (or “evals”) are useful for optimizing system designs. Good evals are:<\/p>\n Representative of real-world usage (or at least diverse) DIFFERENCE TO DETECT\tSAMPLE SIZE NEEDED FOR 95% CONFIDENCE Evaluation of outputs can be done by computers, humans, or a mix. Computers can automate evals with objective criteria (e.g., questions with single correct answers) as well as some subjective or fuzzy criteria, in which model outputs are evaluated by other model queries. OpenAI Evals is an open-source software framework that provides tools for creating automated evals.<\/p>\n Model-based evals can be useful when there exists a range of possible outputs that would be considered equally high in quality (e.g. for questions with long answers). The boundary between what can be realistically evaluated with a model-based eval and what requires a human to evaluate is fuzzy and is constantly shifting as models become more capable. We encourage experimentation to figure out how well model-based evals can work for your use case.<\/p>\n Tactic: Evaluate model outputs with reference to gold-standard answers For example, using the following system message:<\/p>\n SYSTEM – Neil Armstrong was the first person to walk on the moon. For each of these points perform the following steps:<\/p>\n 1 – Restate the point. Finally, provide a count of how many “yes” answers there are. Provide this count as {“count”: SYSTEM SYSTEM SYSTEM SYSTEM Step 1: Reason step-by-step about whether the information in the submitted answer compared to the expert answer is either: disjoint, equal, a subset, a superset, or overlapping (i.e. some intersection but not subset\/superset).<\/p>\n Step 2: Reason step-by-step about whether the submitted answer contradicts any aspect of the expert answer.<\/p>\n Step 3: Output a JSON object structured like: {“type_of_overlap”: “disjoint” or “equal” or “subset” or “superset” or “overlapping”, “contradiction”: true or false} SYSTEM Submitted Answer: “””Didn’t he walk on the moon or something?”””<\/p>\n Expert Answer: “””Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.””” SYSTEM Submitted Answer: “””On the 21st of July 1969, Neil Armstrong became the second person to walk on the moon, following after Buzz Aldrin.”””<\/p>\n Expert Answer: “””Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.””” SYSTEM Submitted Answer: “””At approximately 02:56 UTC on July 21st 1969, Neil Armstrong became the first human to set foot on the lunar surface, marking a monumental achievement in human history.”””<\/p>\n Expert Answer: “””Neil Armstrong is most famous for being the first person to walk on the moon. This historic event occurred on July 21, 1969.””” Prompting libraries & tools *These tend to be very unstable so it’s nice to have them recorded as ascii some place. https:\/\/platform.openai.com\/docs\/guides\/gpt-best-practices\/strategy-test-changes-systematically GPT best practices This guide shares strategies and tactics for getting better results from GPTs. The methods described here can sometimes be deployed in combination for greater effect. We encourage experimentation to find the methods that work […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-3508","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"yoast_head":"\n
\nUse the provided articles delimited by triple quotes to answer questions. If the answer cannot be found in the articles, write “I could not find an answer.”
\nUSER
\n
\nOpen in Playground
\nGiven that GPTs have limited context windows, in order to apply this tactic we need some way to dynamically lookup information that is relevant to the question being asked. Embeddings can be used to implement efficient knowledge retrieval. See the tactic “Use embeddings-based search to implement efficient knowledge retrieval” for more details on how to implement this.<\/p>\n
\nIf the input has been supplemented with relevant knowledge, it’s straightforward to request that the model add citations to its answers by referencing passages from provided documents. Note that citations in the output can then be verified programmatically by string matching within the provided documents.<\/p>\n
\nYou will be provided with a document delimited by triple quotes and a question. Your task is to answer the question using only the provided document and to cite the passage(s) of the document used to answer the question. If the document does not contain the information needed to answer this question then simply write: “Insufficient information.” If an answer to the question is provided, it must be annotated with a citation. Use the following format for to cite relevant passages ({“citation”: \u2026}).
\nUSER
\n“””
\nOpen in Playground
\nStrategy: Split complex tasks into simpler subtasks
\nTactic: Use intent classification to identify the most relevant instructions for a user query
\nFor tasks in which lots of independent sets of instructions are needed to handle different cases, it can be beneficial to first classify the type of query and to use that classification to determine which instructions are needed. This can be achieved by defining fixed categories and hardcoding instructions that are relevant for handling tasks in a given category. This process can also be applied recursively to decompose a task into a sequence of stages. The advantage of this approach is that each query will contain only those instructions that are required to perform the next stage of a task which can result in lower error rates compared to using a single query to perform the whole task. This can also result in lower costs since larger prompts cost more to run (see pricing information).<\/p>\n
\nYou will be provided with customer service queries. Classify each query into a primary category and a secondary category. Provide your output in json format with the keys: primary and secondary.<\/p>\n
\n– Unsubscribe or upgrade
\n– Add a payment method
\n– Explanation for charge
\n– Dispute a charge<\/p>\n
\n– Troubleshooting
\n– Device compatibility
\n– Software updates<\/p>\n
\n– Password reset
\n– Update personal information
\n– Close account
\n– Account security<\/p>\n
\n– Product information
\n– Pricing
\n– Feedback
\n– Speak to a human<\/p>\n
\nI need to get my internet working again.
\nOpen in Playground
\nBased on the classification of the customer query, a set of more specific instructions can be provided to a GPT model to handle next steps. For example, suppose the customer requires help with “troubleshooting”.<\/p>\n
\nYou will be provided with customer service inquiries that require troubleshooting in a technical support context. Help the user by:<\/p>\n
\n– If all cables are connected and the issue persists, ask them which router model they are using
\n– Now you will advise them how to restart their device:
\n— If the model number is MTD-327J, advise them to push the red button and hold it for 5 seconds, then wait 5 minutes before testing the connection.
\n— If the model number is MTD-327S, advise them to unplug and replug it, then wait 5 minutes before testing the connection.
\n– If the customer’s issue persists after restarting the device and waiting 5 minutes, connect them to IT support by outputting {“IT support requested”}.
\n– If the user starts asking questions that are unrelated to this topic then confirm if they would like to end the current chat about troubleshooting and classify their request according to the following scheme:<\/p>\n
\nI need to get my internet working again.
\nOpen in Playground
\nNotice that the model has been instructed to emit special strings to indicate when the state of the conversation changes. This enables us to turn our system into a state machine where the state determines which instructions are injected. By keeping track of state, what instructions are relevant at that state, and also optionally what state transitions are allowed from that state, we can put guardrails around the user experience that would be hard to achieve with a less structured approach.<\/p>\n
\nSince GPTs have a fixed context length, dialogue between a user and an assistant in which the entire conversation is included in the context window cannot continue indefinitely.<\/p>\n
\nSince GPTs have a fixed context length, they cannot be used to summarize a text longer than the context length minus the length of the generated summary in a single query.<\/p>\n
\nTactic: Instruct the model to work out its own solution before rushing to a conclusion
\nSometimes we get better results when we explicitly instruct the model to reason from first principles before coming to a conclusion. Suppose for example we want a model to evaluate a student\u2019s solution to a math problem. The most obvious way to approach this is to simply ask the model if the student’s solution is correct or not.<\/p>\n
\nDetermine if the student’s solution is correct or not.
\nUSER
\nProblem Statement: I’m building a solar power installation and I need help working out the financials.
\n– Land costs $100 \/ square foot
\n– I can buy solar panels for $250 \/ square foot
\n– I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 \/ square foot
\nWhat is the total cost for the first year of operations as a function of the number of square feet.<\/p>\n
\n1. Land cost: 100x
\n2. Solar panel cost: 250x
\n3. Maintenance cost: 100,000 + 100x
\nTotal cost: 100x + 250x + 100,000 + 100x = 450x + 100,000
\nASSISTANT
\nThe student’s solution is correct.
\nOpen in Playground
\nBut the student’s solution is actually not correct! We can get the model to successfully notice this by prompting the model to generate its own solution first.<\/p>\n
\nFirst work out your own solution to the problem. Then compare your solution to the student’s solution and evaluate if the student’s solution is correct or not. Don’t decide if the student’s solution is correct until you have done the problem yourself.
\nUSER
\nProblem Statement: I’m building a solar power installation and I need help working out the financials.
\n– Land costs $100 \/ square foot
\n– I can buy solar panels for $250 \/ square foot
\n– I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 \/ square foot
\nWhat is the total cost for the first year of operations as a function of the number of square feet.<\/p>\n
\n1. Land cost: 100x
\n2. Solar panel cost: 250x
\n3. Maintenance cost: 100,000 + 100x
\nTotal cost: 100x + 250x + 100,000 + 100x = 450x + 100,000
\nASSISTANT
\nLet x be the size of the installation in square feet.
\n1. Land cost: 100x
\n2. Solar panel cost: 250x
\n3. Maintenance cost: 100,000 + 10x
\nTotal cost: 100x + 250x + 100,000 + 10x = 360x + 100,000<\/p>\n
\nOpen in Playground
\nTactic: Use inner monologue or a sequence of queries to hide the model’s reasoning process
\nThe previous tactic demonstrates that it is sometimes important for the model to reason in detail about a problem before answering a specific question. For some applications, the reasoning process that a model uses to arrive at a final answer would be inappropriate to share with the user. For example, in tutoring applications we may want to encourage students to work out their own answers, but a model\u2019s reasoning process about the student\u2019s solution could reveal the answer to the student.<\/p>\n
\nFollow these steps to answer the user queries.<\/p>\n
\nUSER
\nProblem Statement:
\nOpen in Playground
\nAlternatively, this can be achieved with a sequence of queries in which all except the last have their output hidden from the end user.<\/p>\n
\n
\nOpen in Playground
\nNext, we can have the model use all available information to assess the correctness of the student\u2019s solution.<\/p>\n
\nCompare your solution to the student’s solution and evaluate if the student’s solution is correct or not.
\nUSER
\nProblem statement: “””
\nOpen in Playground
\nFinally, we can let the model use its own analysis to construct a reply in the persona of a helpful tutor.<\/p>\n
\nYou are a math tutor. If the student made an error, offer a hint to the student in a way that does not reveal the answer. If the student did not make an error, simply offer them an encouraging comment.
\nUSER
\nProblem statement: “””
\nOpen in Playground
\nTactic: Ask the model if it missed anything on previous passes
\nSuppose that we are using a model to list excerpts from a source which are relevant to a particular question. After listing each excerpt the model needs to determine if it should start writing another or if it should stop. If the source document is large, it is common for a model to stop too early and fail to list all relevant excerpts. In that case, better performance can often be obtained by prompting the model with followup queries to find any excerpts it missed on previous passes.<\/p>\n
\nYou will be provided with a document delimited by triple quotes. Your task is to select excerpts which pertain to the following question: “What significant paradigm shifts have occurred in the history of artificial intelligence.”<\/p>\n
\n…
\n{“excerpt”: “…”}]
\nUSER
\n“””
\nASSISTANT
\n[{“excerpt”: “the model writes an excerpt here”},
\n…
\n{“excerpt”: “the model writes another excerpt here”}]<\/p>\n
\nAre there more relevant excerpts? Take care not to repeat excerpts. Also ensure that excerpts contain all relevant context needed to interpret them – in other words don’t extract small snippets that are missing important context.
\nOpen in Playground
\nStrategy: Use external tools
\nTactic: Use embeddings-based search to implement efficient knowledge retrieval
\nA model can leverage external sources of information if provided as part of its input. This can help the model to generate more informed and up-to-date responses. For example, if a user asks a question about a specific movie, it may be useful to add high quality information about the movie (e.g. actors, director, etc\u2026) to the model\u2019s input. Embeddings can be used to implement efficient knowledge retrieval, so that relevant information can be added to the model input dynamically at run-time.<\/p>\n
\nGPTs cannot be relied upon to perform arithmetic or long calculations accurately on their own. In cases where this is needed, a model can be instructed to write and run code instead of making its own calculations. In particular, a model can be instructed to put code that is meant to be run into a designated format such as triple backtics. After an output is produced, the code can be extracted and run. Finally, if necessary, the output from the code execution engine (i.e. Python interpreter) can be provided as an input to the model for the next query.<\/p>\n
\nYou can write and execute Python code by enclosing it in triple backticks, e.g. “`code goes here“`. Use this to perform calculations.
\nUSER
\nFind all real-valued roots of the following polynomial: 3*x**5 – 5*x**4 – 3*x**3 – 7*x – 10.
\nOpen in Playground
\nAnother good use case for code execution is calling external APIs. If a model is instructed in the proper use of an API, it can write code that makes use of it. A model can be instructed in how to use an API by providing it with documentation and\/or code samples showing how to use the API.<\/p>\n
\nYou can write and execute Python code by enclosing it in triple backticks. Also note that you have access to the following module to help users send messages to their friends:<\/p>\n
\nimport message
\nmessage.write(to=”John”, message=”Hey, want to meetup after work?”)“`
\nOpen in Playground
\nWARNING: Executing code produced by a model is not inherently safe and precautions should be taken in any application that seeks to do this. In particular, a sandboxed code execution environment is needed to limit the harm that untrusted code could cause.<\/p>\n
\nSometimes it can be hard to tell whether a change \u2014 e.g., a new instruction or a new design \u2014 makes your system better or worse. Looking at a few examples may hint at which is better, but with small sample sizes it can be hard to distinguish between a true improvement or random luck. Maybe the change helps performance on some inputs, but hurts performance on others.<\/p>\n
\nContain many test cases for greater statistical power (see table below for guidelines)
\nEasy to automate or repeat<\/p>\n
\n30%\t~10
\n10%\t~100
\n3%\t~1,000
\n1%\t~10,000<\/p>\n
\nSuppose it is known that the correct answer to a question should make reference to a specific set of known facts. Then we can use a model query to count how many of the required facts are included in the answer.<\/p>\n
\nYou will be provided with text delimited by triple quotes that is supposed to be the answer to a question. Check if the following pieces of information are directly contained in the answer:<\/p>\n
\n– The date Neil Armstrong first walked on the moon was July 21, 1969.<\/p>\n
\n2 – Provide a citation from the answer which is closest to this point.
\n3 – Consider if someone reading the citation who doesn’t know the topic could directly infer the point. Explain why or why not before making up your mind.
\n4 – Write “yes” if the answer to 3 was yes, otherwise write “no”.<\/p>\n
\nHere’s an example input where both points are satisfied:<\/p>\n
\n
\nUSER
\n“””Neil Armstrong is famous for being the first human to set foot on the Moon. This historic event took place on July 21, 1969, during the Apollo 11 mission.”””
\nOpen in Playground
\nHere’s an example input where only one point is satisfied:<\/p>\n
\n
\nUSER
\n“””Neil Armstrong made history when he stepped off the lunar module, becoming the first person to walk on the moon.”””
\nOpen in Playground
\nHere’s an example input where none are satisfied:<\/p>\n
\n
\nUSER
\n“””In the summer of ’69, a voyage grand,
\nApollo 11, bold as legend’s hand.
\nArmstrong took a step, history unfurled,
\n“One small step,” he said, for a new world.”””
\nOpen in Playground
\nThere are many possible variants on this type of model-based eval. Consider the following variation which tracks the kind of overlap between the candidate answer and the gold-standard answer, and also tracks whether the candidate answer contradicts any part of the gold-standard answer.<\/p>\n
\nUse the following steps to respond to user inputs. Fully restate each step before proceeding. i.e. “Step 1: Reason…”.<\/p>\n
\nHere’s an example input with a substandard answer which nonetheless does not contradict the expert answer:<\/p>\n
\n
\nUSER
\nQuestion: “””What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.”””<\/p>\n
\nOpen in Playground
\nHere’s an example input with answer that directly contradicts the expert answer:<\/p>\n
\n
\nUSER
\nQuestion: “””What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.”””<\/p>\n
\nOpen in Playground
\nHere’s an example input with a correct answer that also provides a bit more detail than is necessary:<\/p>\n
\n
\nUSER
\nQuestion: “””What event is Neil Armstrong most famous for and on what date did it occur? Assume UTC time.”””<\/p>\n
\nOpen in Playground
\nOther resources
\nFor more inspiration, visit the OpenAI Cookbook, which contains example code and also links to third-party resources such as:<\/p>\n
\nPrompting guides
\nVideo courses
\nPapers on advanced prompting to improve reasoning<\/p>\n","protected":false},"excerpt":{"rendered":"