Case Study on ChatGPT’s Performance in Assisting Students with Physics Tests

Innal Mafudi, Heru Kuswanto, Jumadi Jumadi, Intan Fatmawati

Abstract


The rapid development of artificial intelligence (AI), particularly ChatGPT, has sparked interest in its application in education. This study aims to investigate the potential of ChatGPT in helping students understand and solve physics problems, focusing on the Test of Understanding Graphs in Kinematics and the Determining and Interpreting Resistive Electric Circuit Concepts Test. The study involved 25 physics education students who completed these tests independently and with ChatGPT's assistance. The results revealed that students with a strong foundational understanding and reflective abilities interacted more effectively with ChatGPT, leading to improved answers and deeper conceptual understanding. In contrast, students with weaker prior knowledge tended to accept ChatGPT’s answers without critical reflection, perpetuating errors. Furthermore, ChatGPT showed limitations in interpreting image-based questions, reading scales, and providing consistent responses to concept-specific queries. These findings suggest that while ChatGPT has the potential to enhance learning, it requires thoughtful integration, particularly in helping students develop critical thinking and problem-solving skills. Teachers are encouraged to use ChatGPT’s limitations to design assessments that minimize the risk of cheating and foster deeper understanding. In conclusion, this study underscores the importance of combining AI tools with strong conceptual foundations and active reflection to optimize learning outcomes in physics education. Future research should focus on refining strategies for using AI in education to address its current limitations and enhance its effectiveness in complex learning scenarios.

Keywords


artificial intelligence; chatGPT; physics concept understanding

Full Text:

PDF

References


Abbas, N., Ali, I., Manzoor, R., Hussain, T., & Hussain, M. H. L. (2023). Role of artificial intelligence tools in enhancing students’ educational performance at higher levels. Journal of Artificial Intelligence Machine Learning and Neural Network, 3(5), 36–49. https://doi.org/10.55529/jaimlnn.35.36.49

Akavova, A., Temirkhanova, Z., & Lorsanova, Z. (2023). Adaptive learning and artificial intelligence in the educational space. E3s Web of Conferences, 451, 06011, 1-4. https://doi.org/10.1051/e3sconf/202345106011

Amin, B. D., Sahib, E. P., Harianto, Y. I., Patandean, A. J., Herman, & Sujiono, E. H. (2020). The interpreting ability on science kinematics graphs of senior high school students in South Sulawesi, Indonesia. Jurnal Pendidikan IPA Indonesia, 9(2), 179–186. https://doi.org/10.15294/jpii.v9i2.23349

Beichner, R. (1994). Testing student interpretation of kinematic graphs. American Journal of Physics, 62, 750-762. https://doi.org/10.1119/1.17449

Chiu, T. K. F., Moorhouse, B. L., Chai, C. S., & Ismailov, M. (2023). Teacher support and student motivation to learn with Artificial Intelligence (AI) based chatbot. Interactive Learning Environments, 32(7), 3240–3256. https://doi.org/10.1080/10494820.2023.2172044

Cooper, G. (2023). Examining science education in ChatGPT: an exploratory study of generative artificial intelligence. Journal of Science Education and Technology, 32, 444–452. https://doi.org/10.1007/s10956-023-10039-y

Eaton, P., & Willoughby, S. D. (2018). Confirmatory factor analysis applied to the force concept inventory. Physical Review Physics Education Research, 14, 1-11. https://doi.org/10.1103/physrevphyseducres.14.010124

Engelhardt, P. V., & Beichner, R. J. (2004). Students’ understanding of direct current resistive electrical circuits. American Journal of Physics, 72, 98–115. https://doi.org/10.1119/1.1614813

Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846

Gregorcic, B., & Pendrill, A. M. (2023). ChatGPT and the frustrated socrates. Physics Education, 58,1-9. https://doi.org/10.1088/1361-6552/acc299

Hashem, R., Ali, N., Zein, F. E., Fidalgo, P., Khurma, O. A. (2023). AI to the rescue: Exploring the potential of chatgpt as a teacher ally for workload relief and burnout prevention. Research and Practice in Technology Enhanced Learning, 19, 1-26. https://doi.org/10.58459/rptel.2024.19023

Henderson, R., & Stewart, J. (2018). Racial and ethnic bias in the force concept inventory. Conference Proceedings, 172–175. https://doi.org/10.1119/perc.2017.pr.038

Hidaayatullaah, H. N. (2022). The science literacy profile based on madrasah students’ misconceptions on science concepts. Proceedings of the International Conference on Madrasah Reform, 111-116. https://doi.org/10.2991/assehr.k.220104.017

Hikmatiar, H., Sya’bania, N., Jayadin., Kasman, R. A., Imranah., Sahlan., & Saputra, S. (2024). The effectiveness of Chatgpt in completing astronomy lectures: Building awareness of its use. Jurnal Pendidikan Fisika, 12(2), 121-130. https://doi.org/10.26618/jpf.v12i2.13587

Hoa, N. Q. (2023). AI and plagiarism: Opinion from teachers, administrators and policymakers. Proceedings of the Asiacall International Conference, 4, 75–85. https://doi.org/10.54855/paic.2346

Ivanjek, L., Morris, L., Schubatzky, T., Hopf, M., Burde, J. P., Haagen-Schützenhöfer, C., Dopatka, L., Spatz, V., & Wilhelm, T. (2021). Development of a two-tier instrument on simple electric circuits. Physical Review Physics Education Research, 17, 1-15. https://doi.org/10.1103/PhysRevPhysEducRes.17.020123

Kasepalu, R., Prieto, L. P., Ley, T., & Chejara, P. (2022). Teacher artificial intelligence-supported pedagogical actions in collaborative learning coregulation: A wizard-of-oz study. Frontiers in Education, 7, 1-15. https://doi.org/10.3389/feduc.2022.736194

Khan, R. A., Jawaid, M., Khan, A. R., & Sajjad, M. (2023). ChatGPT - Reshaping medical education and clinical management. Pakistan Journal of Medical Sciences, 39(2), 605-607. https://doi.org/10.12669/pjms.39.2.7653

Kim, S.-W. (2023). Change in attitude toward artificial intelligence through experiential learning in artificial intelligence education. International Journal on Advanced Science Engineering and Information Technology, 13(5), 1953–1959. https://doi.org/10.18517/ijaseit.13.5.19039

Linuwih, S. (2013). Konsepsi alternatif mahasiswa calon guru fisika tentang gaya-gaya yang bekerja pada balok. Jurnal Pengajaran MIPA, 18(1), 69-77. https://doi.org/10.18269/jpmipa.v18i1.259

Maries, A., & Singh, C. (2013). Exploring one aspect of pedagogical content knowledge of teaching assistants using the test of understanding graphs in kinematics. Physical Review Physics Education Research, 9, 1-14. https://doi.org/10.1103/physrevstper.9.020120

Phage, I. (2018). Undergraduate physics students’ conceptual understanding in the learning of kinematics using a blended approach. Ijaedu- International E-Journal of Advances in Education, 4(11), 199–204. https://doi.org/10.18768/ijaedu.455623

Phage, I. B., Lemmer, M., & Hitge, M. (2017). Probing factors influencing students’ graph comprehension regarding four operations in kinematics graphs. African Journal of Research in Mathematics, Science and Technology Education, 21(2), 200–210. https://doi.org/10.1080/18117295.2017.1333751

Polverini, G., & Gregorcic, B. (2024). Performance of ChatGPT on the test of understanding graphs in kinematics. Physical Review Physics Education Research, 20, 1-16. https://doi.org/10.1103/PhysRevPhysEducRes.20.010109

Remoto, J. P. (2023). ChatGPT and other AIs: Personal relief and limitations among mathematics-oriented learners. Environment and Social Psychology, 9(1), 1-13. https://doi.org/10.54517/esp.v9i1.1911

Suganda, T., Kusairi, S., Azizah, N., & Parno, P. (2020). The correlation of isomorphic, open-ended, and conventional score on the ability to solve kinematics graph questions. Jurnal Penelitian & Pengembangan Pendidikan Fisika, 6(2), 173–180. https://doi.org/10.21009/1.06204

Sutopo, S., Parno, P., & Angin, S. L. (2017). Pemahaman mahasiswa tentang multi representasi konsep percepatan. Jurnal Riset dan Kajian Pendidikan Fisika, 4(2), 48-53. https://doi.org/10.12928/jrkpf.v4i2.6551

Wang, T., Lund, B. D., Marengo, A., Pagano, A., Mannuru, N. R., Teel, Z. A., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 1-15. https://doi.org/10.3390/app13116716

Warsono, W., Nursuhud, P. I., Darma, R. S., Supahar, S., Oktavia, D. A., Setiyadi, A., & Kurniawan, M. A. (2020). Multimedia learning modules (MLMs) based on local wisdom in physics learning to improve student diagram representations in realizing the nature of science. International Journal of Interactive Mobile Technologies (Ijim), 14(6), 148-158. https://doi.org/10.3991/ijim.v14i06.11640

Zavala, G., Tejeda, S., Barniol, P., & Beichner, R. J. (2017). Modifying the test of understanding graphs in kinematics. Physical Review Physics Education Research, 13(2), 20111, 1-16. https://doi.org/10.1103/PhysRevPhysEducRes.13.020111




DOI: https://doi.org/10.26618/jpf.v13i1.16624

Refbacks

  • There are currently no refbacks.


Copyright (c) 2025 Jurnal Pendidikan Fisika

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

View JPF Stats

   Free counters!

Jurnal Pendidikan Fisika  is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.