Confluence How To: Using Rating Scales to Get Better Output
Using rating scales as a prompt structure can improve output on the first attempt and simplify the process of making adjustments.
As you use large language models (LLMs) like ChatGPT or Bing over time you start to get a good sense of their tendencies and how to prompt well given those tendencies. As a result, you get better at producing output on the first try that’s close to what you’re looking for. Quite often, though, the output isn’t quite right, and it can be frustrating to create increasingly specific prompts in an attempt to dial the output in on what you’re looking for. It can be tiresome to keep prompting, “a bit more X” or “a bit less Y.” And sometimes, because the way these models work is opaque, you’re not quite sure what to prompt in hopes of tailoring the output in the first place.
One of the techniques we’ve developed to help with this is using rating scales as a prompt structure. Here’s an example of a typical prompt (in this case, to Bing):
Describe the inverted yield curve.
An inverted yield curve is a situation where the interest rates on short-term bonds are higher than the interest rates on long-term bonds. This means that investors expect lower returns in the future, which could indicate a weak economy or a recession. An inverted yield curve is rare and often seen as a warning sign for the financial markets12
A good answer, at a readable length. Let’s say, though, that we know from the start that we want a lengthy answer. So instead we prompt:
Describe the inverted yield curve and give me a long answer.
An inverted yield curve is a situation where the interest rates on short-term debt instruments are higher than the interest rates on long-term debt instruments of the same credit quality. This means that investors are willing to accept lower returns for lending money for longer periods of time, which implies that they expect lower economic growth and inflation in the future. An inverted yield curve is often seen as a sign of an impending recession, as it has preceded every U.S. recession since 19551.
There are different ways to measure the yield curve, such as using the difference between two maturities (e.g., 10-year and 2-year Treasury yields) or using a single maturity (e.g., 3-month Treasury bill rate). The choice of measurement depends on the preference of analysts and market participants, as well as the availability of data. However, some studies have suggested that using the difference between two maturities is more reliable than using a single maturity, as it captures more information about the expectations and preferences of investors2.
An inverted yield curve can have various impacts on consumers and investors. For consumers, an inverted yield curve can mean higher borrowing costs for mortgages, car loans, student loans, and other types of debt. This can reduce their purchasing power and increase their financial stress. For investors, an inverted yield curve can mean lower returns on long-term bonds, which can affect their portfolio allocation and risk tolerance. An inverted yield curve can also signal a shift in market sentiment from optimism to pessimism, which can affect consumer confidence and spending behavior3.
To adjust your portfolio to account for an inverted yield curve, you may want to consider some of the following strategies:
Diversify your portfolio across different asset classes, sectors, regions, and maturities. This can help reduce your exposure to any single source of risk or return.
Increase your duration exposure if you are a long-term investor who expects interest rates to rise in the future. Duration measures how sensitive a bond’s price is to changes in interest rates. A higher duration means a higher price volatility and potential capital gain or loss.
Reduce your duration exposure if you are a short-term investor who expects interest rates to fall in the future. A lower duration means a lower price volatility and potential capital gain or loss.
Seek out opportunities in other markets that may benefit from an inverted yield curve environment, such as emerging markets, commodities, real estate investment trusts (REITs), or dividend-paying stocks4.
I hope this answer helps you understand what an inverted yield curve is and how it affects you as a consumer or investor. If you have any further questions or comments, please feel free to ask me 😊.
Longer, yes. But as much detail as we want? It’s hard to know what general terms like “detailed” or “lengthy” are going to get you with an LLM. So instead of using general adjectives, consider giving the LLM a rating scale to work from. In this case we’ll use “verbosity.”
Describe the inverted yield curve. On a verbosity scale from 1 to 5 where 1 is “extremely terse” and 5 is “highly detailed with great texture and context,” reply at level 5.
An inverted yield curve is a rare and unusual phenomenon in the financial markets, where the interest rates on long-term bonds are lower than the interest rates on short-term bonds. This means that investors are willing to accept lower returns for lending money over a longer period of time, which implies that they expect the economy to perform poorly in the near future. An inverted yield curve is often considered a reliable predictor of an impending recession, as it has preceded every U.S. recession since 19551.
To understand why an inverted yield curve is so significant, we need to first understand what a normal yield curve looks like and what it represents. A yield curve is a graphical representation of the relationship between the interest rates (or yields) and the maturity dates of different types of bonds. Typically, bonds with longer maturities have higher interest rates than bonds with shorter maturities, because investors demand a higher return for locking up their money for a longer period of time. This is also known as the term premium, which reflects the risk and uncertainty associated with future inflation and interest rate movements. A normal yield curve is upward-sloping, meaning that the yield increases as the maturity date gets farther away. A normal yield curve reflects a healthy and growing economy, where investors expect higher inflation and interest rates in the future.
However, sometimes the yield curve can become flat or inverted, meaning that the yield decreases as the maturity date gets farther away. A flat yield curve occurs when the interest rates on short-term and long-term bonds are roughly equal, indicating that investors have no clear expectations about the future direction of the economy. A flat yield curve is often seen as a transition stage between a normal and an inverted yield curve.
An inverted yield curve occurs when the interest rates on short-term bonds are higher than the interest rates on long-term bonds, indicating that investors are pessimistic about the future performance of the economy. An inverted yield curve suggests that investors expect lower inflation and interest rates in the future, which could be caused by a slowdown in economic growth, a decline in consumer spending, or a rise in unemployment. An inverted yield curve also implies that investors are shifting their money from risky assets, such as stocks, to safer assets, such as bonds, which reduces the demand and increases the supply of short-term bonds, driving up their yields. Conversely, the increased demand and reduced supply of long-term bonds lowers their yields. An inverted yield curve is often interpreted as a sign of an upcoming recession, as it reflects a lack of confidence and optimism in the economy.
There are different ways to measure and define an inverted yield curve, depending on which types of bonds are compared. One common method is to look at the spread between the yields on the 10-year and the 2-year U.S. Treasury bonds, which are considered the benchmark for long-term and short-term interest rates, respectively. When the 10-year yield is lower than the 2-year yield, the yield curve is said to be inverted. Another common method is to look at the spread between the yields on the 10-year and the 3-month U.S. Treasury bonds, which are also widely used as indicators of long-term and short-term interest rates, respectively. When the 10-year yield is lower than the 3-month yield, the yield curve is also said to be inverted. Both methods have been shown to have a strong historical correlation with recessions, although they may not always occur at the same time or with the same magnitude.
The impact of an inverted yield curve on consumers and investors can vary depending on the severity and duration of the inversion, as well as the underlying causes and effects of the economic downturn. Generally speaking, an inverted yield curve can have negative consequences for both borrowers and lenders, as well as for the overall economy. Some of the possible impacts are:
Lower profits for banks and financial institutions, as they borrow money at short-term rates and lend money at long-term rates. When the short-term rates are higher than the long-term rates, their net interest margin (the difference between the interest income and the interest expense) shrinks, reducing their profitability and ability to lend.
Higher borrowing costs for consumers and businesses, as banks and lenders pass on the higher short-term rates to their customers. This can discourage borrowing and spending, which can further weaken the economic activity and demand.
Lower returns for savers and investors, as they receive lower interest income from their long-term bonds and other fixed-income securities. This can reduce their purchasing power and wealth, as well as their incentive to save and invest.
Higher volatility and uncertainty in the financial markets, as investors react to the changing expectations and sentiments about the future state of the economy. This can lead to increased risk aversion and flight to quality, which can amplify the movements in asset prices and exchange rates.
Lower economic growth and output, as the reduced lending, borrowing, spending, and investing lead to a contraction in the aggregate demand and supply of goods and services. This can result in lower income, employment, and tax revenue, as well as higher deficits and debt.
An inverted yield curve is not a guarantee that a recession will occur, nor does it indicate when or how severe the recession will be. There may be other factors that influence the shape and movement of the yield curve, such as the monetary and fiscal policies of the government and the central bank, the global economic and geopolitical conditions, and the supply and demand dynamics of the bond market. Moreover, there may be a lag between the time when the yield curve inverts and the time when the recession actually begins, which can range from a few months to a few years. Therefore, an inverted yield curve should be seen as a warning signal, rather than a definitive forecast, of a potential economic downturn.
Much more detail. And it works both ways:
Describe the inverted yield curve. On a verbosity scale from 1 to 5 where 1 is "extremely terse" and 5 is "highly detailed with great texture and context," reply at level 1.
An inverted yield curve is when short-term interest rates are higher than long-term ones1. It often signals a recession2.
Verbosity is one example. You can use almost any dimension of the desired output — length, tone, structure, etc. — as a rating structure in prompts. Doing so can help direct the first-attempt output, but it also — and this is very helpful — can serve as a structure for revisions. For example:
Please write a year-end toast for our cocktail party. On an “optimism” scale of 1 (deeply cynical) to 10 (completely assured of future success) make it a 9.
Ladies and Gentlemen,
As we gather here tonight, on the cusp of a new year, let us raise our glasses to a future that shines bright with promise and potential. This past year has been a testament to our resilience, our ingenuity, and our unyielding spirit.
As we stand at this crossroads, let us embrace the unwavering belief that the best is yet to come. Our journey together has been marked by challenges, yes, but more importantly, by triumphs that speak to our collective strength and determination.
The year ahead holds uncharted waters, but let us navigate them with confidence and optimism. Let us be inspired by the possibilities, not daunted by the unknown. For in every challenge lies an opportunity, and in every setback, a chance to rise even higher.
We are architects of our own destiny, fortified by our shared experiences and driven by a relentless optimism. So, here’s to a year of boundless opportunities, to growth, to prosperity, and to the unwavering belief that together, we can achieve greatness.
To a brilliant future, to us!
Cheers! 🥂
Too much. How about a 6.
Ladies and Gentlemen,
As we gather this evening, let’s raise our glasses to the close of one chapter and the beginning of another. This past year has had its share of ups and downs, but through it all, we’ve shown resilience and adaptability.
Looking ahead, there’s reason for cautious optimism. The future is not without its challenges, but we’re equipped with the experience and wisdom to meet them head-on. Let’s approach the new year with a balanced mix of hope and pragmatism, ready to seize the opportunities that come our way.
Here’s to a year where we strive, learn, and grow. To a future where success is not guaranteed, but definitely within our reach if we work for it. Let’s toast to progress, to the journey, and to each other.
To a hopeful and rewarding year ahead!
Cheers! 🥂
Another strategy is to make a prompt and then ask the model to rate its own output on a scale you provide. This gives you a set point to work from in adjusting the output, as we do in this example:
I have to email some feedback to someone about their recent meeting facilitation. The feedback is that they were clearly well prepared, but they need to listen better to the participants in the room, and compile more accurate notes of the proceedings. Please write this email for me.
Subject: Feedback on Your Recent Meeting Facilitation
Dear [Recipient's Name],
I hope this message finds you well. I wanted to take a moment to share some thoughts regarding the meeting you facilitated recently.
Firstly, I want to commend you on your evident preparation for the meeting. It was clear that you had invested significant effort in organizing the agenda and ensuring that all necessary topics were covered. This level of dedication is truly appreciated and contributed substantially to the structure and flow of the discussion.
However, I would like to offer some constructive feedback that I believe could further enhance the effectiveness of your facilitation. It was observed that there were moments during the meeting where participant input could have been more thoroughly acknowledged and integrated. Active listening is a critical aspect of effective meeting management, and I encourage you to focus more on this in future sessions. This practice not only helps in validating the contributions of all participants but also ensures that diverse perspectives are accurately captured and considered.
Additionally, it would be beneficial to refine the process of note-taking during the meetings. Ensuring that the notes accurately reflect the discussions and decisions made is paramount. This not only aids in maintaining a clear record but also supports effective follow-up and accountability. Perhaps a more structured approach to documenting key points and action items could be implemented to achieve this.
I am confident that with these adjustments, your already strong facilitation skills will be further enhanced, leading to even more productive and engaging meetings.
Please feel free to reach out if you would like to discuss these points in more detail or explore strategies for implementing these suggestions. I am here to support you in this process.
Thank you for your attention to this feedback and for your ongoing commitment to excellence in your role.
Warm regards.
If you were to rate the warmth of that email from 1 to 10 where 1 is "extremely distant and cold" and 10 is "enormously warm and affectionate," what would you give it?
I would rate the warmth of the email at about a 7. The tone is professional and respectful, acknowledging the positive aspects of the recipient's performance while also offering constructive criticism. The use of phrases like “I hope this message finds you well,” “truly appreciated,” and “I am here to support you” adds a personal and supportive touch, indicating a considerate and encouraging approach. The closing, “Warm regards,” further contributes to this warm tone.
Please revise so it’s an 8.5.
Subject: Feedback and Appreciation for Your Recent Meeting Facilitation
Dear [Recipient's Name],
I hope you're doing wonderfully. I wanted to take a moment to share some feedback from the recent meeting you facilitated, which I found quite impressive.
First off, let me express my sincere admiration for the level of preparation you put into the meeting. Your thorough organization and attention to detail did not go unnoticed. Your dedication to ensuring that every important topic was addressed made a significant impact on the structure and effectiveness of our discussion. It's this kind of commitment that truly makes a difference, and I want you to know how much it is appreciated.
On a constructive note, I believe there is an opportunity for growth in the area of participant engagement. Your expertise in leading the meeting is evident, yet amplifying your listening skills could add even more depth to our sessions. Actively listening and absorbing the insights from all attendees can enrich the discussion and lead to a more comprehensive understanding of the topics at hand.
Another area for potential enhancement is the note-taking process. Precise and reflective notes are crucial as they form the backbone of our action plans and follow-up strategies. Perhaps adopting a more detailed method of recording key points and decisions could help in capturing the essence of our discussions more accurately.
I am truly excited to see how these small adjustments can elevate the already high standard of your facilitation skills. I believe in your capabilities and am here to offer my support and guidance as you continue to develop and refine your skills.
Please feel free to reach out if you'd like to discuss these suggestions further or explore additional strategies. Your growth and success in this role are important to me, and I'm eager to assist in any way I can.
Thank you once again for your hard work and dedication. It's a pleasure collaborating with you.
Finally, this approach works for visual output as well:
Create a logo representing the confluence of two ideas.
Here is a logo representing the confluence of two abstract ideas. This design symbolizes unity and harmony through the dynamic interplay of shapes and colors, integrating the concepts in a modern and professional manner.
If you were to rate that design on a scale from 1 to 10 where 1 is “utmost minimalism” and 10 is “utmost maximalism,” what score would you give it?
Based on the design elements present in the logo, I would rate it around a 4 on the scale from “utmost minimalism” (1) to “utmost maximalism” (10). The design maintains a level of simplicity and elegance, avoiding overly complex elements, yet incorporates enough detail and interplay of shapes to move it beyond a purely minimalist approach.
Please give me a 7.
Here is a revised logo design, created to reflect a more complex and vibrant aesthetic.
And finally, you can use multiple scales at once (although for this to work well the scales should not be in opposition to each other). Example:
Create a logo representing the confluence of two ideas. If you were to rate the design on a scale from 1 to 10 where 1 is “utmost minimalism” and 10 is “utmost maximalism,” make it a 1. On a scale where 1 is “black and white” and 10 is “an explosion of colors” make it a 7. On a scale where 1 is “gothic” and 10 is “Bauhaus,” make it a 10.
Here is the logo created based on your specifications. It represents the confluence of two abstract ideas, with a minimalist design and a moderately vibrant color scheme, in a modern Bauhaus style.
We use this strategy all the time to improve our output, and it works in all the major models (ChatGPT, Bing, Claude). Whenever output seems a bit off, ask the model to rate its output on a dimension meaningful to you and then ask for a revision against the scale. If trying to create output with a certain quality from the start, give the model a rating scale for that quality with a desired number, see what you get, and adjust from there. We find it’s far more effective than trying to hone the output through more general prompts and adjectives.
AI Disclosure: We used generative AI in creating imagery for this post. We also used it selectively as a creator and summarizer of content and as an editor and proofreader.