How to Ensure We Have the Most Accurate Data Possible
Before we get into Rater Reliability, let’s review how Truvelop’s Evaluation came to be. Truvelop’s Evaluation was created as part of a research study conducted by the University of Baltimore’s Industrial Psychology Graduate Studies Department. In their study, they set out to create a question set that reliably measures employee performance. Today, the algorithm only considers the answers of the 15 questions. It does not analyze any of the comments left.
Today, Truvelop’s Evaluation has over 95% accuracy, meaning that more than 95% of the time managers will complete the Evaluation, receive the output, and agree that it is an accurate depiction of their interactions with that team member. The Truvelop Evaluation accuracy rating has never dipped below 95%, so managers and employees can feel confident that when they take the time to complete the Evaluation as honestly and accurately as possible, they will receive a summary with accurate, relevant actionable next steps.
WHAT IS RATER RELIABILITY?
Rater reliability is the concept that when something is being assessed, it would get the same accurate score, no matter who is doing the rating. If you were evaluating one of your Team Members, a sign of high rater reliability would be if another Manager gave the same scores in the evaluation of that same Team Member. Essentially, if you had the same information in front of you each time, you would give the same score. Ideally, we want to have high rater reliability as it means that we have accurate data and a clear picture of employee performance.
Example of high rater reliability:
- John Manager and Jane Manager both gave their Team Member Bobbi Blast from Sales an A, indicating that they objectively saw Bobbi as a leading contributor.
WHAT IS RATER BIAS?
So, what happens when rater reliability is absent? When objectivity is removed, rater bias takes place. This means that the rater gives a score that would not be completely accurate because they allowed emotions, fatigue, low focus, etc. to influence their evaluation process. If there is unchecked rater bias occurring during the evaluation process, then the data is skewed and does not present an accurate picture of employee performance.
Examples of rater bias:
- John Manager just discovered that he won the lottery. He is so overjoyed and is giving each of his Team Members perfect scores in their evaluations.
- Jane Manager is exhausted and hungry after working overtime. She can barely focus on who she is doing the evaluations for and finds herself just going through the motions and filling in the bubbles closest to her hand.
WHAT ARE THE TYPES OF RATER BIAS?
Everyone is guilty of practicing bias at one point or another. However, we can try to mitigate the presence of bias in our evaluations as much as possible. Below are several different types of Rater Bias to watch out for:
If we can become aware of the different types of bias that may be affecting our evaluations, then we can actively check our thinking as we are rating individuals. We aren’t just going through the motions and clicking the circle that we feel is right, but asking ourselves why we believe this is an accurate reflection of the employee.
HOW CAN WE PRACTICE RATER RELIABILITY IN TRUVELOP?
As you can tell from the examples, completing the evaluations in a neutral state is really important if you are going to gain accurate data. Truvelop is only a tool, it’s what you do with it as a Manager that will impact your results. With that in mind, the Truvelop team would like to share some questions to ask yourself so that you may maximize the effectiveness of the tool, fully utilizing the evaluation process every time.
BEFORE STARTING AN EVALUATION, ASK YOURSELF:
- What is my current mood?
- If you are in a more emotional state, try taking a couple of deep breaths until you feel yourself calmed down. We don’t want anger to produce a horns bias in our rating, or for good news to create a halo bias in our rating.
- What is my current energy level?
- If you notice you are tired or are overly energized, this may not be the time to fill out the observation! Maybe you need that morning coffee to wake up or to go for a walk to get some of those jitters out before you start the evaluation.
- Am I able to give this evaluation my full attention right now?
- If there is something distracting you, let’s address that first. Inattentiveness can drastically affect the accuracy of an evaluation. When we aren’t paying attention, bias can sneak into our thoughts without challenge.
- Am I able to fill out this evaluation objectively?
- As we learned in “What are the types of Rater Bias?”, sometimes we need to separate the actions from the person so that we can give an accurate and objective assessment. Ask yourself, if another employee had shown the same behaviors, would I be giving the same rating? Go back to your comments and see if those are in alignment with the overall rating.
- Can I think of an example that would exemplify this rating?
- We want these ratings to be as objective as possible. If we are able to think of concrete documented behaviors, actions, instances, etc., where the employee earned this rating, then we will likely have more accurate data. If we can provide evidence for a rating, then we are mitigating rater bias.
ONE LAST PIECE TO KEEP IN MIND: THE MORE DATA WE COLLECT, THE MORE ACCURATE IT WILL BE. WHEN WE FREQUENTLY COMPLETE OBJECTIVE EVALUATIONS, THEN WE WILL BE ABLE TO HAVE A CLEARER PICTURE OF EACH INDIVIDUAL EMPLOYEE.
Even thought Truvelop’s Evaluation has a 95% accuracy rating, there are the 5% of Evaluations that are not accurate. In these instances, there are a few steps that we can take to ensure that you’re able to collect accurate data.
- Mark “No” in the Accuracy Check pop-up after the Evaluation. This ensures that you’re able to share feedback with Truvelop about what went wrong.
- Reflect on any questions that you might have been on the bubble for. These questions might have yielded a different score had they been answered differently. Be sure to review your Spark history as well to remind yourself of recently shared feedback with the Team Member. When you’re ready, reach out to firstname.lastname@example.org. We’re here to help! If redoing the Evaluation is what you want to do, then we can remove the previous Evaluation.
- Reflect on the output and how comfortable you are with sharing the feedback. Is the summary actually inaccurate, or are you uncomfortable with sharing the feedback? If it’s the latter, the Truvelop team is here to help! Reach out to email@example.com and we can share resources and best practices around having difficult conversations.
For many of our customers, completing an Evaluation can be a brand new practice. If you ever want to hop on a call to walk-through the Evaluation practice together, just let us know! We are more than happy to support you on your journey towards becoming a more effective leader for your team.