The opinions of expert judges and lay audiences can diverge significantly – we can see that every year, for example in the Eurovision Song Contest, various literature prizes, and other artist competitions. But do experienced experts really do a better job at judging the quality of artistic performances than the public? Or do audience ratings provide a better and more neutral yardstick of artistic quality?

Roberto Asmat from WU Vienna’s Department of Economics has taken a closer look at this question. “In many areas, from stock portfolios to wine tastings, research has cast doubts on the value of expert assessments,” Asmat points out. “And there’s a rich body of literature on the ‘wisdom of crowds,’ showing that large groups often make better assessments.”

Together with Karol J. Borowiecki from the University of Southern Denmark and Marc T. Law from the University of Vermont, Roberto Asmat has set out to explore this question from a new perspective. The researchers analyzed data from an area that has so far received only very little attention in academia: the results of classical music competitions.

Expert judges and audiences rarely agree

For their study, the researchers analyzed 370 music competitions, held in 22 different countries in the period between 1979 and 2021, where prizes for the best performances were awarded both by panels of expert judges and by the audiences. “Our analysis shows that jury and audience preferences matched only 38% of the time,” Roberto Asmat explains. Even though this percentage varied significantly depending on the instrument category and the country, there was rarely ever full agreement between the expert judges and the audience.

Surprisingly, the researchers found clear indications that experts are actually not very neutral in their judgments: According to their analysis, jury ratings tend to be more negative if the performers are women and also if they are from the country where the competition is held. But does this mean that the experts are biased against women and locals? “Rating an artistic performance is ultimately a question of taste, and it’s always subjective, so the word ‘bias’ must be used with caution here,” says Roberto Asmat.

However, the data also show that there is an objective criterion for putting the subjective judgments of the juries to the test: How likely are the top-rated performers to win further competitions in the future? Roberto Asmat’s analysis showed that for the winners of the audience votes, the chances of winning further competitions were higher. However, the same did not apply to the winners of jury prizes. In other words: When it comes to predicting future success, audience ratings are more accurate than the decisions of expert juries. “In this context, it is therefore justified to speak of gender and nationality bias in the expert juries.”

There is accounting for taste

One possible factor to explain this bias is the composition of the juries: Traditionally, they tend to be male dominated, which may put female performers at a disadvantage. However, this explanation does not account for the full picture. “The question of who’s represented in a jury may play a certain role, but it’s clearly not a very big one,” says Roberto Asmat. “We haven’t found any statistically significant correlations between the genders, nationalities, or native languages of the jury members and the winners.”

As far as the composition of the audience is concerned, the researchers have no comparable data. However, they observed a surprising degree of neutrality in the audiences anyway: Compared to the juries, the audience members present at the competitions that the researchers analyzed showed no gender or nationality bias. There are other types of events, however, where audience ratings do show a clear bias when it comes to the nationality of the performers, as Roberto Asmat notes – once again, the Eurovision Song Contest with its blocks of nations is a typical case in point. But at least in the realm of classical music, this is obviously not the case.

So what do these results tell us about how artistic performances can be rated and evaluated? “The discussions about the quality of art are as old as art itself,” says Roberto Asmat. “In any case, our work shows that the voices of laypeople are at least as valuable as those of specialists in discussions about what makes good art.”

Detailed study results and further information

Roberto Asmat, Karol J. Borowiecki, Marc T. Law: Do experts and laypersons differ? Some evidence from international classical music competitions. Journal of Economic Behavior & Organization, Volume 214, 2023.
Link to the paper