Skip to main content

Advertisement

SpringerLink
Log in

A Framework for Classifying Replication Studies in Educational Technologies Research

  • Original research
  • Published:
Technology, Knowledge and Learning Aims and scope Submit manuscript

Abstract

Replicating research studies is considered one way of establishing validity and confidence of findings in a field of study. In this paper, we introduce a replication framework for classifying studies conducted in the area of educational technology as a possible guide to conducting and reporting replication studies in the field. The paper includes the benefits as well as challenges of replicating research, and proposes a categorical continuum that might be used to determine the strength of the replication of a study. Examples of replication studies and how they fit the framework are included. Implications for using this framework for conducting studies that are worthy of replication in the field of educational technology are addressed in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Availability of Data and Material

There are no data associated with this paper as the examples are from previously published papers which can be accessed for a fee or through subscription. The authors declare no conflict of interest. No ethics review was required to undertake this paper. No human subjects were included.

References

  • Adams, D. A., Nelson, R. R., & Todd, P. A. (1992). Perceived usefulness, ease of use, and usage of information technology: A replication. Management Information Stystems (MIS) Quarterly, 16(2), 227–247.

    Article  Google Scholar 

  • APA. (2020). Publication manual of the American Psychological Association (7thed.). Washington, D.C.: American Psychologial Association.

  • Barshay, J. (2014). Education researchers don’t check for errors – dearth of replication studies. The Hechinger Report. Available: http://educationbythenumbers.org/content/education-researchers-dont-check-errors-dearth-replication-studies_1762/

  • Benson, L., & Borrego, M. (2015). The role of replication in engineering education research. Journal of Engineering Education Research, 104(4), 388–392. https://doi.org/10.1002/jee.20082

    Article  Google Scholar 

  • Berliner, D. C. (2002). Educational research: The hardest science of all. Educational Researcher, 31(8), 18–20.

    Article  Google Scholar 

  • Bond, M. (2020). Facilitating student engagement through the flipped learning approach in K-12: A systematic review. Computers & Education. https://doi.org/10.1016/j.compedu.2020.103819

    Article  Google Scholar 

  • Bond, M., Zawacki-Richter, O., & Nichols, M. (2019). Revisiting five decades of educational technology research: A content and authorship analysis of the British journal of educational technology. British Journal of Educationa Technology, 50(1), 12–63.

    Article  Google Scholar 

  • Bondie, R. S., Dahnke, C., & Zusho, A. (2019). How does changing “one-size-fits-all” to differentiated instruction affect teaching? Review of Research in Education, 43, 336–362.

    Article  Google Scholar 

  • Casti, J. L. (1989). Paradigms lost: Images of man in the mirror of science. William Morrow.

    Google Scholar 

  • Chhin, C. S., Taylor, K. A., & Wei, W. S. (2018). Supporting a culture of replication: An examination of education and special education research grants funded by the institute of educational sciences. Educational Researcher, 47(9), 594–605.

    Article  Google Scholar 

  • Christensen, R., Knezek, G., & Tyler-Wood, T. (2014). Student perceptions of science, technology, engineering and mathematics (STEM) content and careers. Computers in Human Behavior, 34, 173–186. https://doi.org/10.1016/j.chb.2014.01.046.

    Article  Google Scholar 

  • Christensen, R., Spector, J. M., Thompson, A., Schmidt-Crawford, D., Bull, G. & Knezek, G. (2015). Innovation versus replication in research findings: Has the novelty of new research findings worn off? In D. Slykhuis & G. Marks (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2015 (pp. 1126–1129). Chesapeake, VA: Association for the Advancement of Computing in Education (AACE).

    Google Scholar 

  • Darley, W. K. (2000). Status of replication studies in marketing: A validation and extension. Marketing Management Journal, 10(2), 121–132.

    Google Scholar 

  • De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., Schroeder, T. V., Sox, H. C., & VanDer Weyden, M. B. (2004). Clinical trial registration: A statement from the international committee of medical journal editors. New England Journal of Medicine, 251(12), 1250–1251.

    Article  Google Scholar 

  • Dron, J. (2021). Educational technology: What it is and how it works. AI & Society. https://doi.org/10.1007/s00146-021-01195-z

    Article  Google Scholar 

  • Garzon, J., & Acevedo, J. (2019). Meta-analysis of the impact of augmented reality on students’ learning gains. Educational Research Review, 28(1), 244–260. https://doi.org/10.1016/j.edurev.2019.04.001

    Article  Google Scholar 

  • Grubisic, A., Stankov, S., Rusic, M., & Zitko, B. (2009). Controlled experiment replication in evaluation of e-learning system’s educational influence. Computers & Education, 53, 591–602.

    Article  Google Scholar 

  • Hannafin, M., Orrill, C., Kim, H., & Kim, M. (2005). Educational technology research in postsecondary settings: Promise, problems, and prospects. Journal of Computing in Higher Education, 16(2), 3–22. https://doi.org/10.1007/BF02961472

    Article  Google Scholar 

  • Hodges, C. B. (2015). Replication studies in educational technology. TechTrends, 59(4), 3–4. https://doi.org/10.1007/s11528-015-0862-2.

    Article  Google Scholar 

  • Hodges, C. B., & Cowan, S. F. (2012). Preservice teachers’ views of instructor presence in online courses. Journal of Digital Learning in Teacher Education, 28(4), 139–145. https://doi.org/10.1080/21532974.2012.10784694.

    Article  Google Scholar 

  • Hew, K. F., Lan, M., Tang, Y., Jia, C., & Lo, C. K. (2019). Where is the “theory” within the field of educational technology research? British Journal of Educational Technology, 50(3), 956–971. https://doi.org/10.1111/bjet.12770

    Article  Google Scholar 

  • Kinshuk Huang, H.-W., Sampson, D., & Chen, N.-S. (2013). Trends in educational technology through the lens of the highly cited articles published in the journal of educational technology and society. Journal of Educational Technology and Society, 16(2), 3–20.

    Google Scholar 

  • Knezek, G., & Christensen, R. (2020). Project-based learning for middle school students monitoring standby power: replication of impact on stem knowledge and dispositions. Educational Technology Research & Development (ETR&D), 68(1), 137–162. https://doi.org/10.1007/s11423-019-09674-3.

    Article  Google Scholar 

  • Knezek, G., Christensen, R., Tyler-Wood, T., & Periathiruvadi, S. (2013). Impact of environmental power monitoring activities on middle school student perceptions of STEM. Science Education International, 24(1), 98–123.

    Google Scholar 

  • Li, J., Antonenko, P. D., & Wang, J. (2019). Trends and issues in multimedia learning research in 1996–2016: A bibliometric analysis. Educational Research Review, 28(1), 100282. https://doi.org/10.1016/j.edurev.2019.100282

    Article  Google Scholar 

  • López, X., Valenzuela, J., Nussbaum, M., & Tsai, C.-C. (2015). Some recommendations for the reporting of quantitative studies [Editorial]. Computers & Education, 91, 106–110. https://doi.org/10.1016/j.compedu.2015.09.010

    Article  Google Scholar 

  • Lynch, M. (2018). Ten edtech research journals that you should read. https://www.thetechedvocate.org/10-edtech-research-journals-that-you-should-read/

  • Lynch, J. G., Bradlow, E. T., Huber, J. C., & Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333–342. https://doi.org/10.1016/j.ijresmar.2015.09.006

    Article  Google Scholar 

  • Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70, 151–159.

    Article  Google Scholar 

  • Mackey, A., & Gass, S. M. (2005). Second language research: Methodology and design. Erlbaum.

    Google Scholar 

  • Makel, M C., & Plucker, J.A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Research, 20(10), 1–13. Retrieved from http://edr.sagepub.com/content/early/2014/07/23/0013189X14545513

  • Makel, M. C., Plucker, J., & Hegarty, C. B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542. https://doi.org/10.1177/1745691612460688

    Article  Google Scholar 

  • Matthews, H. M., Hirsch, S. E., & Therrien, W. J. (2018). Becoming critical consumers of research: Understanding replication. Intervention in School and Clinic, 53(5), 267–275.

    Article  Google Scholar 

  • Morrison, S. J. (2016). Forum. Journal of Research in Music Education, 63(4), 395–396. https://doi.org/10.1177/0022429415625485

    Article  Google Scholar 

  • Morrison, R., Matuszek, T., & Self, D. (2010). Preparing a replication or update study in the business disciplines. European Journal of Scientific Research, 47(2), 278–287.

    Google Scholar 

  • National Science Foundation and Institite of Education Sciences. (2018). Common guidelines on replication & reproducibility in education research: A supplement to the commone guidelines for education research and development. National Science Foundation and the Institute of Education Sciences, U.S: Department of Education, Washington DC.

    Google Scholar 

  • Oh, Y. J., Jia, Y., Lorentson, M., & LaBanca, F. (2013). Development of the educational and career interest scale in science, technology, and mathematics for high school students. Journal of Science Educational and Technology, 22, 780–790.

    Article  Google Scholar 

  • Open Science Collaboration. (2012). Perspect. Psychological Science, 7, 657–660.

    Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 6251. https://doi.org/10.1126/science.aac4716

    Article  Google Scholar 

  • Plucker, J. A., & Makel, M. C. (2021). Replication is important for educational psychology: Recent developments and key issues. Educational Psychologist, 56(2), 90–100. https://doi.org/10.1080/00461520.2021.1895796

    Article  Google Scholar 

  • Roblyer, M. D., & Knezek, G. A. (2003). New Millennium Research for educational technology: A call for a national research agenda. Journal of Research on Technology in Education, 36(1), 60–71.

    Article  Google Scholar 

  • Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100.

    Article  Google Scholar 

  • See, B.H. & Perry, T. (2021). A call for replication studies in education special issue of educational research and evaluation. Educational Research and Evaluation. https://think.taylorandfrancis.com/special_issues/replication-studies/

  • Sherer, R., & Teo, T. (2019). Unpacking teachers’ intentions to integrate technology: A meta-analysis. Educational Research Review, 28(1), 90–109.

    Article  Google Scholar 

  • Sheridan, K., & Kelly, M. A. (2010). The indicators of instructor presence that are important to students in online courses. MERLOT Journal of Online Learning and Teaching, 6(4), 767–779.

    Google Scholar 

  • Smith, N. C., Jr. (1970). Replication studies: A neglected aspect of psychological research. American Psychologist, 25, 970–975.

    Article  Google Scholar 

  • Spector, J. M., Johnson, T. E., & Young, P. A. (2015). An editorial on replication studies and scaling up efforts. Education Technology Research & Development, 63(1), 1–4.

    Article  Google Scholar 

  • Sung, Y.-T., Lee, H. Y., Yang, J. M., & Chang, K.-E. (2019). The quality of experimental designs in mobile learning research: A systemic review and self-improvement tool. Educational Research Review, 28, 100279. https://doi.org/10.1016/j.edurev.2019.05.001

    Article  Google Scholar 

  • Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C. C. (2017). Some guidance on conducting and reporting qualitative studies. Computers & Education, 106, A1–A9. https://doi.org/10.1016/j.compedu.2016.12.002

    Article  Google Scholar 

  • van Alten, D. C. D., Phielix, C., Janssen, J., & Kester, L. (2019). Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis. Educational Research Review, 28(1), 1–18. https://doi.org/10.1016/j.edurev.2019.05.003

    Article  Google Scholar 

Download references

Acknowledgements

This paper began as a thread of conversation at the 2015 National Technology Leadership Summit organized by the National Technology Leadership Coalition and hosted by the American Association of Colleges for Teacher Education. The authors wish to thank the working group attendees who helped form this discussion: Lynn Bell, Robert Branch, Arlene Borthwick, Denise Crawford, Gerald Knezek, Meghan Manfra, Natalie Milman, David Rutledge, and Melanie Shoffner.

Funding

None.

Author information

Authors and Affiliations

  1. University of North Texas, 3940 Elm Street #G-150, Denton, TX, 76207, USA

    Rhonda Christensen & J. Michael Spector

  2. College of Education, Georgia Southern University, Campus, Box 8131, Statesboro, GA, 30458, USA

    Charles B. Hodges

Authors
  1. Rhonda Christensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles B. Hodges
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Michael Spector
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

(optional: please review the submission guidelines from the journal whether statements are mandatory).

Corresponding author

Correspondence to Rhonda Christensen.

Ethics declarations

Conflict of interest

The manuscript is not under consideration by other journals and the research meets ethical and legal guidelines.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

The categorical continuum is intended to be use as a guide for either determining what should be included in a replication study or whether a completed study might be considered a replication study of some type. Each of the research design components are listed below with questions that might be asked in determining in which category the study might belong. In each of the replication types, there are two categories that allow a little more “wiggle” room for determining the type of study.

See Appendix (Table

Table 4 Guidelines for using the categorical continuum framework for types of replication studies
Full size table

4).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Christensen, R., Hodges, C.B. & Spector, J.M. A Framework for Classifying Replication Studies in Educational Technologies Research. Tech Know Learn 27, 1021–1038 (2022). https://doi.org/10.1007/s10758-021-09532-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10758-021-09532-3

Keywords

Search

Navigation