Evaluating Assessment. Validation with PLS-SEM of ATAE Scale for the Analysis of Assessment Tasks

  1. Ibarra-Sáiz, María Soledad 1
  2. Rodríguez-Gómez, Gregorio 1
  1. 1 Universidad de Cádiz
    info

    Universidad de Cádiz

    Cádiz, España

    ROR https://ror.org/04mxxkb11

Revista:
Relieve: Revista ELectrónica de Investigación y EValuación Educativa

ISSN: 1134-4032

Ano de publicación: 2020

Título do exemplar: Monográfico: Evaluación en la Educación Superior

Volume: 26

Número: 1

Tipo: Artigo

DOI: 10.7203/RELIEVE.26.1.17403 DIALNET GOOGLE SCHOLAR lock_openAcceso aberto editor

Outras publicacións en: Relieve: Revista ELectrónica de Investigación y EValuación Educativa

Resumo

Una de las funciones esenciales del profesorado universitario se concreta en el proceso de toma de decisiones sobre los diferentes componentes que constituyen el diseño de los procesos de evaluación, siendo uno de sus elementos clave la calidad de las tareas de evaluación. En este estudio se presenta tanto la validación de un instrumento para la valoración por el estudiantado de las tareas de evaluación como el modelo que sustenta las relaciones entre los constructos que caracterizan las tareas de evaluación. A partir de una revisión de la literatura se ha elaborado un modelo teórico de las características de las tareas de evaluación y las relaciones existentes entre ellas. Para su comprobación se ha diseñado, sobre la base de un modelo de medida de carácter formativo, el cuestionario Análisis de las Tareas de Evaluación y Aprendizaje (ATAE). Mediante un diseño de cohorte se han obtenido un total de 1.166 cuestionarios cumplimentados por estudiantes de los grados de Administración y Dirección de Empresas (ADE) y Finanzas y Contabilidad (FYCO). La evaluación del modelo de medida y del modelo estructural se ha realizado mediante la técnica Partial Least Squares Structural Equation Modeling (PLS-SEM) utilizando el software SmartPLS_3. Los resultados muestran la no existencia de problemas de colinealidad y unos niveles elevados de importancia absoluta y relativa de cada uno de los ítems del cuestionario. Es de destacar, desde la percepción de los estudiantes, que el carácter retador de una tarea de evaluación se relaciona con la transferencia del aprendizaje, y cómo el uso de estrategias de comunicación y la demostración de una comprensión profunda son elementos mediadores de esta relación. Palabras clave: Tarea de evaluación; evaluación como aprendizaje; empoderamiento; PLS-SEM; mínimos cuadrados parciales; modelo de ecuaciones estructurales; PLS predictivo

€ Ver financiamento

Información de financiamento

Este trabajo ha sido posible gracias al proyecto TransEval (Ref. I+D+i 2017/01) financiado por la Universidad de Cádiz y al apoyo de la Cátedra UNESCO Evaluación, Innovación y Excelencia en Educación.

Financiadores

  • Universidad de Cádiz Spain
    • I+D+i 2017/01
  • Cátedra UNESCO Evaluación, Innovación y Excelencia en Educación
    • I+D+i 2017/01

Referencias bibliográficas

  • Ashford-Rowe, K., Herrington, J., & Brown, C. (2014). Establishing the critical elements that determine authentic assessment. Assessment & Evaluation in Higher Education, 39(2), 205–222. https://doi.org/10.1080/02602938.2013.819566
  • Ashwin, P., Boud, D., Coate, K., Hallet, F., Keane, E., Krause, K.-L., … Tooher, M. (2015). Reflective teaching in higher education. Bloomsbury. https://doi.org/10.1142/S0129183114500405
  • Bearman, M., Dawson, P., Bennett, S., Hall, M., Molloy, E., Boud, D., & Joughin, G. (2017). How university teachers design assessments: a cross-disciplinary study. Higher Education, 74(1), 49–64. https://doi.org/10.1007/s10734-016-0027-7
  • Bearman, M., Dawson, P., Boud, D., Bennett, S., Hall, M., & Molloy, E. (2016). Support for assessment practice: developing the Assessment Design Decisions Framework. Teaching in Higher Education, 21(5), 545–556. https://doi.org/10.1080/13562517.2016.1160217
  • Bearman, M., Dawson, P., Boud, D., Hall, M., Bennett, S., Molloy, E., & Joughin, G. (2014). Guide to the Assessment Design Decisions Framework. Retrieved from http://www.assessmentdecisions.org/guide/
  • Biggs, J., & Tang, C. (2011). Teaching for quality learning at university. What the students does (4th ed.). McGraw-Hill-SRHE & Open University Press.
  • Boud, D. (2014). Shifting views of assessment: from secret teachers’ business to sustaining learning. In C. Kreber, C. Anderson, N. Entwistle, & J. McArthut (Eds.), Advances and inovations in university assessment and feedback (pp. 13–31). Edinburgh University Press Ltd. https://doi.org/10.3366/edinburgh/9780748694549.003.0002
  • Boud, D. (2020, en prensa). Challenges in reforming higher education assessment: a perspective from afar. RELIEVE - Electronic Journal of Educational Research, Assessment and Evaluation.
  • Cepeda-Carrión, G., Nitzl, C., & Roldán, J. L. (2017). Mediation Analyses in Partial Least Squares Structural Equation Modeling: Guidelines and Empirical Examples. In H. Latan & R. Noonan (Eds.), Partial Least Squares Path Modeling: Basic concepts, methodological issues and applications (pp. 173–195). Springer. https://doi.org/10.1007/978-3-319-64069-3
  • Chin, W. W. (1998). The partial least squares approach to structural equation modeling. In G. A. Marcoulides (Ed.), Modern methods for business research (pp. 295–398). Lawrence Erlbaum.
  • Cubero-Ibáñez, J., & Ponce-González, N. (2020). Aprendiendo a través de tareas de evaluación auténticas: Percepción de estudiantes de Grado en Educación Infantil. Revista Iberoamericana de Evaluación Educativa, 13(1), 41–69. https://doi.org/10.15366/riee2020.13.1.002
  • Dawson, P., Bearman, M., Boud, D. J., Hall, M., Molloy, E. K., Bennett, S., & Joughin, G. (2013). Assessment Might Dictate the Curriculum, But What Dictates Assessment? Teaching & Learning Inquiry: The ISSOTL Journal, 1(1), 107–111. https://doi.org/10.2979/teachlearninqu.1.1.107
  • Dochy, F. (2009). The Edumetric Quality of New Modes of Assessment: Some Issues and Prospect. In G. Joughin (Ed.), Assessment, Learning and Judgement in Higher Education (pp. 85–114). Springer. https://doi.org/10.1007/978-1-4020-8905-3_6
  • Dochy, F., & Gijbels, D. (2006). New learning, assessment engineering and edumetrics. In L. Verschaffel, F. Dochy, M. Boekaerts, & S. Vosniadou (Eds.), Instructional Psychology: Past, present and future trends. Sixteen essays in honour of Erik De Corte. Elsevier.
  • Entwistle, N., & Karagiannopoulou, E. (2014). Perceptions of Assessment and their Influences on Learning. In C. Kreber, C. Anderson, N. Entwistle, & J. McArthut (Eds.), Advances and innovations in university assessment and feedback (pp. 75–98). Edinburgh University Press Ltd. https://doi.org/10.3366/edinburgh/9780748694549.003.0005
  • Garson, G. D. (2016). Partial Least Squares: Regression & Structural Equation Models. Statistical Publishing Associates.
  • Glofcheski, R. (2017). Making Assessment for Learning Happen Through Assessment Task Design in the Law Curriculum. In D. Carless, S. M. Bridges, C. K. Y. Chan, & R. Glofcheski (Eds.), Scaling up Assessment for Learning in Higher Education (pp. 67–80). Singapore. https://doi.org/10.1007/978-981-10-3045-1_5
  • Gore, J., Ladwig, J., Eslworth, W., & Ellis, H. (2009). Quality assessment framework: A guide for assessment practice in higher education. Callaghan, NSW Australia: The Australian Learning and Teaching Council. The University of Newcastle. Retrieved from http://www.olt.gov.au/system/files/resources/QAF FINAL doc for print.pdf
  • Gulikers, J. T. M., Bastiaens, T. J., Kischner, P. A., & Kester, L. (2006). Relations between student perception of assessment authenticity, study approaches and learning outcomes. Studies in Educational Evaluation, 32, 381–400. https://doi.org/10.1016/j.stueduc.2006.10.003
  • Gulikers, J. T. M., Bastiaens, T., & Kirschner, P. A. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52(3), 67–85. https://doi.org/10.1007/BF02504676
  • Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2017). A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM) (2nd ed.). Sage.
  • Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24. https://doi.org/10.1108/EBR-11-2018-0203
  • Hair, J. F., Sarstedt, M., Ringle, C. M., & Gudergan, S. P. (2018). Advanced Issues in Partial Least Squares Structural Equation Modeling. Sage.
  • Herrington, J., & Herrington, A. (2006). Authentic conditions for authentic assessment: Aligning task and assessment. Proceedings of the 29th HERDSA Annual Conference, 146–151.
  • Horst, S. J., & Pyburn, E. M. (2018). Likert scaling. In B. B. Frey (Ed.), The SAGE encyclopedia of educational research, measurement, and evaluation (pp. 974–976). Sage.
  • Ibarra-Sáiz, M. S. y Rodríguez-Gómez, G. (2019). Una evaluación como aprendizaje. En J. Paricio, A. Fernández e I. Fernández (Eds.), Cartografía de la buena docencia. Un marco para el desarrollo del profesorado basado en la investigación (pp. 175–196). Narcea.
  • Ibarra-Sáiz, M. S., Rodríguez-Gómez, G., & Boud, D. (2020). Developing student competence through peer assessment: the role of feedback, self-regulation and evaluative judgement. Higher Education. https://doi.org/10.1007/s10734-019-00469-2
  • Jiménez-Cortés, R. (2019). Aprendizaje de las mujeres en las redes sociales: Validación de la escala MAIA con PLS. Revista de Investigación Educativa, 37(2), 431–449. https://doi.org/http://dx.doi.org/10.6018/rie.37.2.325721
  • Johnson, R. L., & Morgan, G. B. (2016). Survey scales. A guide to development, analysis, and reporting. The Guilford Press.
  • Muñoz-Cantero, J.-M., Rebollo-Quintela, N., Mosteiro-García, J., & Ocampo-Gómez, C.-I. (2019). Validación del cuestionario de atribuciones para la detección de coincidencias en trabajos académicos. RELIEVE - Revista Electrónica de Investigación y Evaluación Educativa, 25(1), 1–16. https://doi.org/10.7203/relieve.25.1.13599
  • Nitzl, C., Roldán, J. L., & Cepeda, G. (2016). Mediation Analysis in Partial Least Squares Path Modeling: Helping researchers discuss more sophisticated models. Industrial Management & Data Systems, 116(9), 1849–1864. https://doi.org/10.1108/IMDS-07-2015-0302
  • O’Donovan, B. (2016). How student beliefs about knowledge and knowing influence their satisfaction with assessment and feedback. Higher Education, 74(4), 617–633. https://doi.org/10.1007/s10734-016-0068-y
  • OECD. (2013). Methodological considerations in the measurement of subjective well-being. In OECD Guidelines on measuring subjective well-being (pp. 61–138). Paris: OECD Publishing. https://doi.org/10.1787/9789264191655-en
  • Panadero, E., Andrade, H., & Brookhart, S. (2018). Fusing self-regulated learning and formative assessment: a roadmap of where we are, how we got here, and where we are going. The Australian Educational Researcher, 45(1), 13–31. https://doi.org/10.1007/s13384-018-0258-y
  • Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723–735. https://doi.org/10.1080/00313831.2015.1066436
  • Pereira, D., Niklasson, L., & Flores, M. A. (2017). Students’ perceptions of assessment: a comparative analysis between Portugal and Sweden. Higher Education, 73(1), 153–173. https://doi.org/10.1007/s10734-016-0005-0
  • Ringle, C. M., Wende, S., & Becker, J.-M. (2015). SmartPLS 3. Bönningstedt: SmartPLS. Retrieved from http://www.smartpls.com
  • Rodríguez-Gómez, G., & Ibarra-Sáiz, M. S. (2015). Assessment as learning and empowerment: Towards sustainable learning in higher education. In M. Peris-Ortiz & J. M. Merigó Lindahl (Eds.), Sustainable learning in higher education. Developing competencies for the global marketplace (pp. 1–20). Springer. https://doi.org/10.1007/978-3-319-10804-9_1
  • Roldán, J. L., & Sánchez-Franco, M. J. (2012). Variance-Based Structural Equation Modeling: Guidelines for Using Partial Least Squares in Information Systems Research. In Research Methodologies, Innovations and Philosophies in Software Systems Engineering and Information Systems (pp. 193–221). IGI Global. https://doi.org/10.4018/978-1-4666-0179-6.ch010
  • Sadler, D. R. (2016). Three in-course assessment reforms to improve higher education learning outcomes. Assessment & Evaluation in Higher Education, 41(7), 1081–1099. https://doi.org/10.1080/02602938.2015.1064858
  • Sambell, K., McDowell, L., & Montgomery, C. (2013). Assessment for Learning in Higher Education. Routledge.
  • Sharma, P. N., Shmueli, G., Sarstedt, M., Danks, N., & Ray, S. (2018). Prediction-Oriented Model Selection in Partial Least Squares Path Modeling. Decision Sciences. https://doi.org/10.1111/deci.12329
  • Shmueli, G., Ray, S., Velasquez Estrada, J. M., & Chatla, S. B. (2016). The elephant in the room: Predictive performance of PLS models. Journal of Business Research, 69(10), 4552–4564. https://doi.org/10.1016/j.jbusres.2016.03.049
  • Shmueli, G., Sarstedt, M., Hair, J. F., Cheah, J. H., Ting, H., Vaithilingam, S., & Ringle, C. M. (2019). Predictive model assessment in PLS-SEM: guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322–2347. https://doi.org/10.1108/EJM-02-2019-0189
  • Smith, J. K., & Smith, L. F. (2014). Developing Assessment Tasks. In C. Wyatt-Smith, V. Klenowski, & P. Colbert (Eds.), Designing Assessment for Quality Learning (pp. 123–136). Springer. https://doi.org/10.1007/978-94-007-5902-2
  • Strijbos, J., Engels, N., & Struyven, K. (2015). Criteria and standards of generic competences at bachelor degree level: A review study. Educational Research Review, 14, 18–32. https://doi.org/10.1016/j.edurev.2015.01.001
  • Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018). Developing evaluative judgement: enabling students to make decisions about the quality of work. Higher Education, 76(3), 467–481. https://doi.org/10.1007/s10734-017-0220-3
  • Thomas, T., Jacobs, D., Hurley, L., Martin, J., Maslyuk, S., Lyall, M., & Ryan, M. (2019). Students’ perspectives of early assessment tasks in their first-year at university. Assessment & Evaluation in Higher Education, 44(3), 398–414. https://doi.org/10.1080/02602938.2018.1513992
  • Wren, J., Sparrow, H., Northcote, M., & Sharp, S. (2009). Higher Education Students’ Perceptions of Effective Assessment. International Journal of Learning, 15(12), 11–23. https://doi.org/10.18848/1447-9494/CGP/v15i12/46037
  • Zhao, X., Lynch, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37(2), 197–206. https://doi.org/10.1086/651257
Fonte de datos: Dialnet