This study aims to conceptually replicate the findings of Schreiber et al. (2024) using an updated selection of topical (as of June 2024) large language models (LLMs): Qwen 2 72B Instruct (Alibaba Cloud, 2024), Llama 3 70B Instruct (Meta, 2024), and GPT-4 (OpenAI, 2023a). Although music-specific AI applications (e.g., UDIO or SUNO) may produce better results, they currently lack open input formats (e.g., based on Python code) required for the application of a standardized melody continuation task. Therefore, these systems are outside the scope of our research. In line with the definition by Wooldridge and Jennings (1995), the selected LLMs are termed “AI systems” as they do not fulfill the criteria of proactive, flexible, or cooperative behavior with other computing systems, which would characterize “AI agent” systems. The AI systems were accessed via the platform AcademicCloud (https://academiccloud.de/), which is available to all university members in Lower Saxony.

To extend the stimulus basis of the original study, we used a different melody from the domain of popular music as a starting point for the standardized continuation task. This replication also encompassed the implication of a discrimination task: Based on the signal detection theory (SDT), we were interested in the respective discriminative performances of humans and AI systems vis-a-vis creative products.

The following research questions (RQ) were formulated: (RQ₁) Can the significantly poorer subjective assessments of AI systems compared to human compositions in Schreiber et al. (2024) be replicated with another melody task based on a similar highly expert group of participants (music students)? (RQ₂) Do any significant differences emerge in the subjectively perceived quality of the creative products among the three different LLMs used? (RQ₃) Is the overall discrimination performance in the given task (humans vs. AI composer) better than would otherwise happen by chance?

Based on the results by Schreiber et al. (2024), our first two hypotheses are as follows:

Based on the results by Cope (1996), our third hypothesis is as follows:

We measured four dependent variables, each involving a single item, to determine subjective ratings for the aesthetic quality of human and AI-based melody continuations. These four scales were adopted from the study conducted by Schreiber et al. (2024): convincing, logical and meaningful, interesting, and liking. The condition composer_specific served as an independent variable with four gradations: Qwen 2, Llama 3, GPT-4, and human. This resulted in a repeated measures design with four independent variables and the composer as within-subject factor. Given our interest in establishing a distinction between AI systems and humans, we aggregated the three AI models in portions of the statistical analysis, resulting in a dichotomous differentiation with two remaining gradations for the independent variable (AI vs. human).

An a priori power analysis was calculated using G*Power (Faul et al., 2009) based on the (large) effect sizes reported by Schreiber et al. (2024). Regarding the relevant Pillai’s trace value of 0.857 (the value of the main effect composer), the number of participants required to reach the desired power of 1−β = .95 at a standard probability of .05 α-error was a minimum of N ≥ 12. The study was pre-registered (see Meier et al., 2024).

We used a melody continuation paradigm in line with Schreiber et al. (2024) and set it against the lack of tools for the standardized evaluation of performance in AI and music research, as diagnosed by Mycka and Mańdziuk (2025). As a starting point, we simplified the chorus melody of a German pop ballad (Durch die schweren Zeiten [Through the Hard Times] by Udo Lindenberg; see Figure A1). The musical stimuli were created as follows: (a) music students from various study programs (mainly music education) at the Hanover University of Music, Drama and Media composed continuations of the given melody following a standardized instruction (see Appendix 1); (b) The three selected AI systems also continued the melody following the same standardized instruction (see Appendix 2 for the given text prompt). During the next step, which involved converting musical products from all three AI systems into audio files for experimental use, we transcribed the melody output in Python syntax using the module SCAMP (Evanstein, 2023) and instructed the systems to return their continuations in the same format.

The resulting melody continuations from the music students and AI systems were then converted into MIDI format using MuseScore (MuseScore Ltd, 2024) and SCAMP (Evanstein, 2023) respectively. In the final step, we imported these MIDI files into Reaper (Cockos, 2024), before exporting them as MP3 audio files using the BBC Symphony Orchestra Discover (Spitfire Audio, 2023) sample library with timbre clarinet. Sound files were normalized to a loudness of -20 LUFS-I. A total of N = 98 melodies were generated (Qwen 2: n = 25, Llama 3: n = 25, GPT-4: n = 25, and human n = 23; see Supplementary Materials section for details, Meier et al., 2025).

The study was conducted as an online experiment: 55 participants completed a questionnaire on the SoSci Survey platform (https://www.soscisurvey.de/en/index). Participants were informed that they would listen to some melodies, which started in the same way but then continued with either AI- or human-composed portions. After giving their informed consent via a checkbox (see Statement of Ethics), participants had the chance to hear example audio of the given melody, from which the AI systems and music students had established a musical continuation. Participants were presented with a random selection of 20 melodies (five for each condition: Qwen 2, Llama 3, GPT-4, and human) in a randomized, blinded trial, resulting in an incomplete study design. The melodies were rated on a 5-point scale (1 = not at all [gar nicht] to 5 = very much [sehr]) using the criteria convincing, logical and meaningful, interesting, and liking. There was no trial run. In addition to the rating, participants used an eight-point scale (1 = clearly human [eindeutig Mensch], 8 = clearly machine [eindeutig Maschine], see Figure A2) to indicate who they thought had composed the musical continuation, human or AI and how confident they were (with endpoints of the scale indicating higher confidence).

After listening to and rating the melodies, participants specified their gender, age, and musical identity on a single item (Zhang & Schubert, 2019). They were also informed of their number of correct responses in the detection task at the end of the experiment. Given time constraints and the presence of a homogenous group of highly expert participants, we decided against using an additional inventory to control musical sophistication and the use of different melodic probe positions (given melodic lengths), since neither of these variables was found to have any significant impact on the study by Schreiber et al. (2024). The experiment took about 20 minutes to complete, and participants were not remunerated for their involvement.

Of the 55 initial participants, one was excluded due to spurious responses, resulting in a final sample size of N = 54. Participants were recruited via mailing lists at different German-speaking Universities of Music. From the sample, 30 participants (55.6%) were male, 23 (42.6%) female, and one (1.9%) non-binary. The participants were aged between 19 and 61 years (Mdn = 25, IQR = 9). Sporadic outliers from the average age can be explained by the participation of some professors completing the questionnaire in addition to their students. The fact that we actively targeted music university students should have ensured an above-average level of musical expertise for the entire sample, with 45 participants (83.3%) reporting at least 10 years of formal music lessons.

As Figure 1 shows, the human-composed melodies were rated higher for all four dependent variables. Comparing the three LLMs, we can see that Qwen 2 always obtained the highest average ratings, followed closely by or tied with GPT-4. Llama 3 always showed the lowest scores.

Given our key focus on comparing AI systems and humans, we aggregated the melody ratings for the three LLMs into a single independent variable composer. Figure 2 thus shows the rating curve across all four dependent variables grouped by AI vs. human.

Accordingly, we conclude that Hypothesis 1 (improved aesthetic verdict on human-based melody continuations compared to three different AI systems) could be confirmed. Results also confirm Hypothesis 2 (significant differences between the three AI systems in the aesthetic quality of the melody continuations).

To test for the main effect of the independent variable composer, we calculated a repeated measures MANOVA in R (R Core Team, 2024) using the stats package. This analysis revealed that the composer factor impacted significantly on the rating of the four dependent variables, F(1, 53) = 94.07, p < .001, Pillai’s trace = 0.88, η² = 0.64. We also conducted t-tests for groupwise comparisons for all four dependent variables from Figure 2. Effect sizes show differences in the vicinity of large effects (d_Z > 0.8; for benchmarks see Ellis, 2010) in favor of the human-based compositions (see Table 1). Differences between the four evaluation items and both sources of melodic origin for the original study and its replication are very similar.

To round off this part of the data analysis, we calculated a correlation matrix (Pearson) for all four dependent variables to assess the strength of inter-variable correlations between the different rating scales (see Table 2). The target variables convincing and liking showed the strongest inter-correlations.

The second step of data analysis aimed to reveal the underlying discrimination performance of listeners as measured by Signal Detection Theory (SDT). Since the melody continuation falls into two distinct categories (composed either by AI or by humans), with only one at a time presented for evaluation, our discrimination task falls into the family of so-called A-Not A or Yes-No experiments (Bi & Ennis, 2001; Hautus et al., 2021). In our study, a melody continuation composed by an AI system is designated as “A”, “Yes”, or the presence of the signal, while a human-composed continuation is assigned to the category “Not A”, “No”, or the absence of the signal—in short, an AI-Not AI design. Following the classification of A-Not A sub-designs provided by Düvel and Kopiez (2022, Table 1), we have a replicated mixed A-Not A (but not paired) design—replicated because participants were presented with multiple stimuli; mixed because they were presented with stimuli from both categories (A and Not A).

Participants’ responses on the eight-point rating scale were dichotomized (AI and human), with responses ranging from one to four being counted as human and five to eight being counted as AI. They were then classified as follows: If an AI-composed melody continuation was correctly identified, the response was coded as a “hit”, but if a human-composed melody continuation was misidentified as AI-composed, the response was coded as “false alarm“. Conversely, if a human-composed melody continuation was correctly identified as human-composed, the response was coded as “correct rejection”, but identifying an AI-composed melody continuation as human-composed triggered a “miss” response. Table 3 shows the respective frequencies of hits, misses, correct rejections, and false alarms. The independence of the stimulus type and the participants’ responses can be tested based on this 2 × 2 table. Following Bi (2015) and Brier (1980), the test statistic is a conventional Pearson χ² test with a single degree of freedom if the data are transformed with a correction factor. The test results in χ² = 111.62, p < .001, and with Yates’ continuity correction in χ² = 109.82, p < .001. Accordingly, the null hypothesis that participants’ responses are independent of the stimulus type has to be rejected.

To quantify a participant’s discrimination performance, we calculated the sensitivity d prime (d’), which was based on the participant’s hit rate and false-alarm rate (Hautus et al., 2021, p. 7). These rates were then converted to z scores with the inverse of the normal cumulative distribution function (Φ⁻¹, see Equation 1).

A d’ value of 0 designates an occurrence that would otherwise happen by chance. Positive values indicate scope for participants to discriminate AI-composed melody continuations from those by humans.

We also calculate the response bias or criterion c (see Equation 2).

The response bias reflects the tendency toward one of the two response categories (Hautus et al., 2021). In our case, c < 0 indicates a tendency to decide that the melody was continued by an AI system. For both, sensitivity and response bias, hit or false-alarm rates of 0 or 1 had to be corrected (otherwise d’ or c equal ±∞). Following Hautus et al. (2021, p. 7), a rate of 0 was corrected to 1/(2N), where N is the number of trials on which the rate is based; a rate of 1 was corrected to 1−1/(2N).

As per Figure 3A, most sensitivity values and their mean of d’ = 1.09 exceeded 0. A one-tailed one-sample t-test also revealed a significant difference from 0, t(53) = 8.88, p < .001, Cohen’s d = 1.21, 95% CI [0.61, 1.80]. Based on the benchmarks for d’ provided by Bi (2015, Table 3.1), a mean sensitivity of d’ = 1.09 constitutes a meaningful discrimination sensitivity (0.74 ≤ d’ ≤ 1.81) between AI- and human-composed melody continuations.

Considering Figure 3B, the response bias scores seem to be distributed around 0. Indeed, their mean value is almost 0. A two-tailed one-sample t-test indicated no difference from 0, t(53) = −0.02, p = .986, Cohen’s d = −0.00, 95% CI [−0.55, 0.54].

Table A1 (see Appendix) shows the frequencies of hits and misses for each AI system. The melody continuations by Llama 3 were assigned as AI more often than those by ChatGPT 4 which, in turn, were detected as AI-based more often than those generated by Qwen 2. Accordingly, the frequencies of misses are in the same ranking order as the AI system ratings.

Using the musical identity item (years of formal instrumental lessons), the sample was split into two groups (Group 1: > 10 years, n_{> 10 years} = 45; Group 2: 6–10 years, n_{6–10 years} = 9). As shown by Figure A3 (see Appendix), the sensitivity values covered a similar range in both groups. However, their means of d’_{> 10 years} = 1.18 and d’_{6–10 years} = 0.68 were compared using a Welch’s t-test, leading to a non-significant result of t(9.86) = −1.24, p = .242.

Although Schreiber et al. (2024) did not report any results on discrimination performance, the authors had used the same discrimination paradigm in their study. For this reason, after obtaining the raw data from their study, responses were coded identically to conduct equivalent discrimination analyses. Nine participants were removed from the data set (one due to missing values and spurious responses, and eight because they had been presented with one less stimulus in the detection task) resulting in a valid sample of N = 62.

Table A2 (see Appendix) displays the absolute and relative frequencies of the SDT response types. The statistical test on this 2 × 2 table resulted in χ² = 227.11, p < .001, and with Yates’ continuity correction in χ² = 225.58, p < .001. This means the null hypothesis that participants’ responses in the study by Schreiber et al. (2024) are independent of the stimulus type also has to be rejected.

Sensitivity and response bias spawn mean values of d’ = 1.12 and c = −0.03, respectively. As shown by Figure 4, the distributions of sensitivity and response bias broadly resemble those from the replication study (see Figure 3). Based on one-sample t-tests, only the sensitivity differs from 0, one-tailed t(61) = 11.21, p < .001, Cohen’s d = 1.42, 95% CI [0.86, 1.99], but not the response bias, two-tailed t(61) = −0.85, p = .401, Cohen’s d = −0.11, 95% CI [−0.62, 0.40]. As in the data set of our replication study, the mean sensitivity of d’ = 1.12 in the original study is in the benchmark interval for a meaningful difference between AI- and human-composed melody continuations (0.74 ≤ d’ ≤ 1.81; Bi, 2015, Table 3.1).

Similar to the results based on the ratings reported by Schreiber et al. (2024), ChatGPT 3.5 recorded a higher number of misses (i.e., its melody continuations were more often misassigned as of “human” origin) than Magenta (see Table A3). The data by Schreiber et al. (2024) encompassed all three levels of musical identity. As can be seen in Figure A4 (see Appendix), the sensitivity values are similarly distributed in the groups. Neither a one-way analysis of variance, F(2, 59) = 0.41, p = .666, η_generalized = .014, nor a Kruskal-Wallis test, χ²(2) = 1.58, p = .453, revealed any significant difference in sensitivity values between the three groups.

To ensure the composition task would be as fair and the results as comparable as possible across all four conditions of the independent variable composer_specific (Qwen 2, Llama 3, GPT-4, and human), we opted to use the same standardized melody continuation task as in Schreiber et al. (2024). Similarly, our use of available AI models was considerably restricted by the existence of an interface for the input of a prompt while referencing the original melody.

Currently, AI systems specialized in music production such as Suno AI (Suno, 2024) or AIVA (Aiva Technologies, 2016) do not offer this option of prompt input. Thus, we decided to focus on the comparison of three selected, currently competing LLMs that represent a generic AI approach. We cannot exclude that a more openly phrased prompt or the use of AI models specializing more strongly in the domain of music would yield better results in favor of AI systems. We agree with Schreiber et al. (2024), that