The metaggR
package implements the knowledge-weighted estimate proposed in Palley and Satopää (2021). This procedure aggregates judges’ estimates of a continuous outcome. To do so, each judge is asked to provide two responses:
Aggregation is done with the function knowledge_weighted_estimate(E,P)
that inputs J judges’ estimates of the outcome (E
) and predictions of others (P
), and outputs the knowledge-weighted estimate. In the following two sections we will illustrate this aggregator on a simple example and on experimental data included in the package.
This section illustrates metaggR
on the Three Gorges Dam example in Palley and Satopää (2021). First, we will upload the library:
library(metaggR)
Next, we can aggregate the judges’ estimates:
# Judges' estimates:
= c(50, 134, 206, 290, 326, 374)
E1 # Judges' predictions of others:
= c(26, 92, 116, 218, 218, 206)
P1 # Knowledge-weighted estimate:
knowledge_weighted_estimate(E1,P1)
#> [1] 329.305
Therefore the final knowledge-weighted estimate is 329.305.
This section illustrates metaggR
on real-world data collected in an experiment in Palley and Satopää (2021). In this experiment participants were presented with 36 different pictures of food from different restaurants and were asked to estimate the total number of calories in these dishes. Each response involves three steps:
The metaggR
package includes the data in the calories
dataset. This dataset is a list with 4 elements:
true_calories
: A vector of true calorie counts of each 36 meals.estimates_initial
: A list of the judges’ initial estimates of the calorie counts in each of the 36 meals.estimates_final
: A list of the judges’ final estimates of the calorie counts in each of the 36 meals.predictions_of_others
: A list of the judges’ predictions of the others’ average estimate of the calorie counts in each of the 36 meals.The elements of each member of calories
correspond to the same meal. Specifically, the jth elements of true_calories
, estimates_initial
, estimates_final
, and predictions_of_others
represent the true calories, initial estimates, final estimates, and predictions of others of the jth meal.
To illustrate, we will consider the responses given for the first meal. First, we will load the package and the calories
dataset:
library(metaggR)
data("E_CALORIES_INITIAL")
data("E_CALORIES_FINAL")
data("P_CALORIES")
data("THETA_CALORIES")
Next, we will pick out the responses given for the first meal:
= 1
meal # True number of calories in the first meal:
theta = THETA_CALORIES[meal])
(#> [1] 990
# Judges' initial estimates of the number of calories in the first meal:
E_initial = E_CALORIES_INITIAL[[meal]])
(#> [1] 140 351 450 20 800 1200 50 475 850 330 1000 600 340 950 600
#> [16] 300 1250 750 1200 670 500 1200 520 75 300 900 25 800 1000 500
#> [31] 900 825 1200 300 600 5000 800 600 800 780 1000 900 630 2000 950
#> [46] 600 800 550 1200 900 950 250 900 800 900 2000 800 1200 40 760
#> [61] 350 1200 1800 725 650 1200 800 500 700 450 1000 400 200
# Judges' final estimates of the number of calories in the first meal:
E_final = E_CALORIES_FINAL[[meal]])
(#> [1] 122 654 400 10 850 1200 35 475 850 375 900 550 2000 950 600
#> [16] 300 1250 750 1200 670 700 1200 540 95 310 950 25 850 1200 500
#> [31] 900 850 1250 300 600 5000 800 600 800 810 500 900 625 1800 950
#> [46] 750 800 500 1150 900 950 250 950 800 900 2150 800 1150 60 760
#> [61] 350 1100 1800 725 600 1200 800 500 600 475 1000 400 200
# Judges' predictions of others' average estimate of the number of calories in the first meal:
P = P_CALORIES[[meal]])
(#> [1] 112 654 300 19 900 1200 40 545 800 250 800 500 300 1200 600
#> [16] 300 1250 599 1000 700 600 1000 580 80 360 1000 30 900 1800 700
#> [31] 900 650 1600 300 650 5000 750 600 500 860 500 900 600 1500 950
#> [46] 700 800 500 1000 1000 900 250 1000 800 700 2200 600 1000 50 980
#> [61] 400 1000 1700 750 450 1200 600 500 500 400 800 300 200
A total of 73 judges provided responses for this meal. The true calorie count is 990 calories. The first judge under-estimated the calorie count and provided an initial and final estimates of 140 and 122 calories, respectively. This judge predicted that the others’ average estimate is 112 calories. The root-mean-squared-errors (RMSE) of the initial and final estimates, and their knowledge-weighted estimates are:
# RMSE of the initial estimates:
sqrt(mean((E_initial-theta)^2))
#> [1] 670.6013
# RMSE of the final estimates:
sqrt(mean((E_final-theta)^2))
#> [1] 674.0261
# RMSE of the knowledge-weighted estimate based on judges' initial estimates:
KWE1 = knowledge_weighted_estimate(E_initial, P, no_inf_check = TRUE))
(#> [1] 814.6392
sqrt((KWE1 - theta)^2)
#> [1] 175.3608
# RMSE of the knowledge-weighted estimate based on judges' final estimates:
KWE2 = knowledge_weighted_estimate(E_final, P, no_inf_check = TRUE))
(#> [1] 865.0712
sqrt((KWE2 - theta)^2)
#> [1] 124.9288
This shows that the knowledge-weighted estimate can improve the average accuracy of an individual judge. Specifically, based on the judges’ initial and final estimates, the knowledge-weighted estimates are 814.6392 and 865.0712 calories, respectively. Given that the true calorie count is 990 calories, both aggregate estimates are too low but the knowledge-weighted estimate based on the judges’ final estimates is more accurate. In this example, it improves the individual judges’ RMSE from around 670 calories to around 125 calories.