ChatGPT gives better advice, but we’d rather hear it from someone with a pulse, study shows



summary
Summary

A recent study shows that ChatGPT advice is perceived as more balanced, comprehensive, empathetic, and helpful than advice column responses.

The study, conducted by researchers at the University of Melbourne and the University of Western Australia and published in Frontiers of Psychology, compared ChatGPT and human responses to 50 social dilemma questions randomly selected from ten popular advice columns.

People prefer AI when they don’t know

For the study, the researchers used the paid version of ChatGPT with GPT-4, currently the most capable LLM on the market.

They showed 404 subjects a question along with the corresponding answer from a columnist and ChatGPT. The participants were asked to rate which answer was more balanced, comprehensive, empathetic, helpful, and overall better.

Ad

Ad

The researchers found that ChatGPT “significantly outperformed” the human advisors on each of the five randomly asked questions and in all categories queried, with preference rates ranging from about 70 to 85 percent in favor of the AI.

The study also showed that ChatGPT participants’ answers were longer than those of the advice columnists. In a second study, the researchers shortened the ChatGPT participants’ responses to about the same length as the advice columnists’ responses. This second study confirmed the first, albeit at a slightly lower level, and showed that the advantage of ChatGPT was not solely due to more detailed responses.

Beide Umfragen zeigen, dass viele Personen die Beratung durch ChatGPT als ausgewogener, vollständiger, einfühlsamer, hilfreicher und insgesamt besser empfanden als die Beratung durch einen professionellen Berater.Von den 404 Teilnehmern in Umfrage 1 und den 401 Teilnehmern in Umfrage 2 stimmte eine Mehrheit für ChatGPT in allen Punkten.
Both surveys show that many people found ChatGPT advice to be more balanced, complete, empathetic, helpful, and overall better than advice from a professional counselor. Of the 404 participants in Survey 1 and the 401 participants in Survey 2, a majority voted in favor of ChatGPT on all counts. | Image: Howe et al.

People prefer people when asked

Despite the perceived quality of ChatGPT advice, the majority (77%) of study participants still preferred a human response to their social conflict questions. This preference for human responses is consistent with previous research.

However, participants could not reliably distinguish which responses were written by ChatGPT and which were written by humans. In other words, the preference for human responses is not directly related to the quality of the responses, but rather appears to be a social or cultural phenomenon.

The researchers suggest that future research should explore this phenomenon in more detail, for example by informing participants in advance which answers were written by the AI and which were written by humans. This could increase the willingness to seek advice from the AI.

Recommendation

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top