Teamwork assessment in collaborative projects through process mining techniques

  1. Juan Antonio Caballero-Hernández 1
  2. Antonio Balderas 1
  3. Manuel Palomo-Duarte 1
  4. Pablo Delatorre 1
  5. Antonio J. Reinoso 2
  6. Juan Manuel Dodero 1
  1. 1 Universidad de Cádiz
    info
    Universidad de Cádiz

    Cádiz, España

    ROR https://ror.org/04mxxkb11

    Geographic location of the organization Universidad de Cádiz
  2. 2 Universidad Alfonso X el Sabio
    info
    Universidad Alfonso X el Sabio

    Villanueva de la Cañada, España

    ROR https://ror.org/054ewwr15

    Geographic location of the organization Universidad Alfonso X el Sabio
Journal:
The International journal of engineering education

ISSN: 0949-149X

Year of publication: 2020

Issue Title: Teamwork Assessment in Engineering Education

Volume: 36

Issue: 1

Pages: 470-482

Type: Article

SCOPUS: 2-s2.0-85078471367 DIALNET GOOGLE SCHOLAR

More publications in: The International journal of engineering education

Sustainable development goals

Abstract

Teamwork is one of the key issues in engineering projects success. Unfortunately, due to the high number of interactions, theassessment of collaborative tasks remains a challenge. Wikis are web-based systems that support collaborative work in enterpriseengineering project documentation providing quantitative data from the members’ contributions and interactions. While thisobjective data is interesting for the teamwork assessment, a qualitative assessment process can provide a complementary approach.We propose an architecture that combines information from both sources to conduct a scalable assessment of the teamwork in awiki. It was implemented in a course of the degree on Computer Engineering using specific developed software tools and ProcessMining techniques. Process Mining tools automatically apply artificial intelligence algorithms to extract knowledge from realprocesses and discover models. These models provided evidence of conducted behaviour. The actual dynamics of the teams in thewiki were automatically detected and could be analysed for assessment purposes. Finally, the followed mixed approach allowed adetailed and scalable teamwork skills assessment process.

Bibliographic References

  • A. Abbott, Sequence analysis: New methods for old ideas, Annual Review of Sociology, 21(1), pp. 93-113, 1995.
  • A. Balderas, J. M. Dodero, M. Palomo-Duarte and I. Ruiz-Rube, A domain specific language for online learning competence assessments, International Journal of Engineering Education, 31(3), pp. 851-862, 2015.
  • A. Balderas, M. Palomo-Duarte, J. M. Dodero and I. Ruiz-Rube, Qualitative assessment of wiki-based learning processes, Proceedings of Ninth multidisciplinary symposium on the design and evaluation of digital content for education (SPDECE-2012), Alicante, Spain, June 2012, pp. 161-172, 2012.
  • A. Balderas, M. Palomo-Duarte, J. M. Dodero, M. S. Ibarra-Sáiz and G. Rodríguez-Gómez, Scalable authentic assessment of collaborative work assignments in wikis, International Journal of Educational Technology in Higher Education (2018), 15(1), pp. 40-61, 2018.
  • A. Bogarín, R. Cerezo and C. Romero, A survey on educational process mining, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(1), pp. 1-17, 2018.
  • A. Bolt, M. De Leoni, W. M. Van Der Aalst and P. Gorissen, Exploiting process cubes, analytic workflows and process mining for business process reporting: a case study in education, International Symposium on Data-driven Process Discovery and Analysis (SIMPDA), Vienna, Austria, December 2015, pp. 33-47, 2015.
  • A. Fidalgo-Blanco, D. Lerís, M. L. Sein-Echaluce and F. J. García-Penalvo, Monitoring indicators for CTMTC: Comprehensive training model of the teamwork competence in engineering domain, International Journal of Engineering Education (IJEE), 31(3), pp. 829-838, 2015.
  • A. Fidalgo-Blanco, M. L. Sein-Echaluce, F. J. García-Penalvo and M. A. Conde-González, Using Learning Analytics to improve teamwork assessment, Computers in Human Behavior, 47(1), pp. 149-156, 2015.
  • A. H. Cairns, B. Gueni, M. Fhima, A. Cairns, S. David and N. Khelifa, Process mining in the education domain, International Journal on Advances in Intelligent Systems, 8(1), pp. 219-232, 2015.
  • A. R. González, Percepción de los estudiantes sobre el uso de wiki. Análisis de una experiencia de aprendizaje colaborativo en la Universitat de Girona, Teoría de la Educación: Educación y Cultura en la Sociedad de la Información, 14(1), pp. 76-100, 2013.
  • AssessMediaWiki, https://assessmediawiki.forja.cica.es/indexen.html, Accessed 6 May 2019.
  • C. Argyris and D. A. Schön, Participatory action research and action science compared: A commentary, American Behavioral Scientist, 32(5), pp. 612-623, 1989.
  • C. Robson and K. McCartan, Real world research: A resource for users of social research methods in applied settings, 4th edn, Wiley, Chichester, 2016.
  • C. Romero, R. Cerezo, A. Bogarín and M. Sánchez-Santillán, Educational process mining: A tutorial and case study using moodle data sets, in S. ElAltia, D. Ipperciel and O. R. Zaïane (eds), Data Mining and Learning Analytics: Applications in Educational Research, Wiley, New Jersey, p. 3, 2016.
  • C. Schoor and M. Bannert, Exploring regulatory processes during a computer-supported collaborative learning task using process mining, Computers in Human Behavior, 28(4), pp. 1321-1331, 2012.
  • D. F. M. Rios and P. P. Robles, Desarrollo de competencias académicas utilizando la Wiki en la ensenanza de nivel superior: caso de dos universidades colombianas y sus bibliotecas, VIRTUalis, 6(12), pp. 40-59, 2016.
  • E. Kalliamvakou, D. Damian, L. Singer and D. M. German, The code-centric collaboration perspective: Evidence from github, Technical Report DCS-352-IR, University of Victoria, 2014.
  • F. Lima, A. C. Prado, A. A. Massote and F. Leonardi, Automation course for industrial engineers: An approach based on Petri Nets, software tools and laboratory experiments, International Journal of Engineering Education, 30(5), pp. 1271-1279, 2014.
  • figShare, https://figshare.com/projects/IJEE19-TeamworkAssessmentinCollaborativeProjectsthroughProcessMining Techniques/63434, Accessed 28 Oct 2019.
  • G. P. Fairbrother, Quantitative and qualitative approaches to comparative education, in M. Bray, M. Bray, B. Adamson and M. Mason (eds), Comparative Education Research. CERC Studies in Comparative Education, vol. 19, 2nd edn, Springer Cham, Hong Kong, China, pp. 71-93, 2014.
  • H.M.W.Verbeek, J. C. A.M.Buijs, B. F.VanDongenand andW.M.Vander Aalst,Prom6: Theprocess mining toolkit, International Conference on Business Process Management Demonstration Track, Hoboken, New Jersey, September 2010, pp. 34-39, 2010.
  • I. Traverso, A. Balderas, J. M. Dodero, I. Ruiz-Rube and M. Palomo-Duarte, Open Data Framework for Sustainable Assessment of Project-Based Learning Experiences, Program Electronic Library and Information Systems, 50(4), pp. 380-398, 2016.
  • J. A. Caballero-Hernández, J. M. Dodero, I. Ruiz-Rube, M. Palomo-Duarte, J. F. Argudo and J. J. Domínguez-Jiménez, Discovering bottlenecks in a computer science degree through process mining techniques, 2018 International Symposium on Computers in Education (SIIE), Cadiz, Spain, September 2018, pp. 1-6, 2018.
  • J. A. Caballero-Hernández, M. Palomo-Duarte, J. M. Dodero, G. Rodríguez and M. S. Ibarra-Sáiz, Integrating external evidences of skill assessment in Virtual Learning Environments, Proceedings of the Fifth International Conference on e-Learning (eLearning-2014), Belgrade, Serbia, September 2014, pp. 65-70, 2014.
  • J. M. Dodero, M. Palomo-Duarte, I. Ruiz-Rube, I. Traverso, J. M. Mota and A. Balderas, Learning technologies and semantic integration of learning resources, IEEE Revista Iberoamericana de Tecnologías del Aprendizaje, 10(1), pp. 11-16, 2015.
  • J. Ortega-Valiente, A. J. Reinoso and R. Munoz-Mansill, Analysis of the implementation of wiki-based platforms in university education, International Journal of Computer Information Systems and Industrial Management Applications, 5(1), pp. 41-49, 2013.
  • K. Gibbard, A. Grocutt, A. Turner, T. O'Neill, R. Brennan and S. Li, Assessment of individual and teamwork attributes in undergraduate engineering students, Proceedings of the Canadian Engineering Education Association (CEEA), Vancouver, BC, Canada, June 2018, pp. 1-4, 2018.
  • L. V. Martínez and M. E. del Moral Pérez, Webquest y wikis: búsqueda de información en red y desarrollo de competencias en colaboracion, Teoría de la Educación: Educación y Cultura en la Sociedad de la Información, 12(1), pp. 190-208, 2011.
  • M. A. Conde-González, F. J. García-Penalvo, A. Fidalgo-Blanco and M. L. Sein-Echaluce, Study of the flexibility of a Learning Analytics tool to evaluate teamwork competence acquisition in different contexts, Proceedings of the Learning Analytics Summer Institute Spain 2017: Advances in Learning Analytics (LASI-SPAIN 2017), Madrid, Spain, July 2017, no. 1925, pp. 63-77, 2017.
  • M. D. V. Urías, S. A. Ortega and M. G. Navarro, Desarrollo de competencias digitales en docencia online: la asignatura Cimientos del curso de adaptación a grado en ingeniería de edificación, Revista de Educación a Distancia, 49(8), pp. 1-29, 2016.
  • M. Mittal and A. Sureka, Process mining software repositories from student projects in an undergraduate software engineering course. Companion Proceedings of the 36th International Conference on Software Engineering (ICSE), Hyderabad, India, May 2014, pp. 344-353, 2014.
  • M. Palomo-Duarte, J. M. Dodero, A. García-Domínguez, P. Neira-Ayuso, N. Sales-Montes, I. Medina-Bulo, F. Palomo-Lozano, C. Castro-Cabrera, E. J. Rodríguez-Posada and A. Balderas, Scalability of assessments of wiki-based learning experiences in higher education, Computers in Human Behavior, 31(1), 2014.
  • M. Palomo-Duarte, J. M. Dodero, I. Medina-Bulo, E. J. Rodríguez-Posada and I. Ruiz-Rube, Assessment of collaborative learning experiences by graphical analysis of wiki contributions, Interactive Learning Environments, 22(4), pp. 444-466, 2014.
  • M. Pérez-Mateo, La construcción colaborativa de proyectos como metodología para adquirir competencias digitales, Comunicar 21(42), pp. 15-24, 2014.
  • M.A. Conde-González, R. Colomo-Palacios, F. J. García-Penalvo and X. Larrueca, Teamwork assessment in the educational web of data: A learning analytics approach towards ISO 10018, Telematics and Informatics, 35(3), pp. 551-563, 2018.
  • P. Porouhan and W. Premchaiswadi, Process mining and learners' behavior analytics in a collaborative and web-based multi-tabletop environment, International Journal of Online Pedagogy and Course Design (IJOPCD), 7(3), pp. 29-53, 2017.
  • R. Lingard and S. Barkataki, Teaching teamwork in engineering and computer science, 2011 Frontiers in Education Conference (FIE), Rapid City, SD, October 2011, pp. F1C-1-F1C-5, 2011.
  • S. D. Ravana, N. A. M. Amin and S. N. Raman, Effects of assessment on wiki activities during group work, Proceedings of The Asian Conference on Society, Education, and Technology (ACSET), Osaka, Japan, October 2013, pp. 1-14, 2013.
  • S. Hadjerrouit, Wiki as a collaborative writing tool in teacher education: Evaluation and suggestions for effective use, Computers in Human Behavior, 32(1), pp. 301-312, 2014.
  • StatMediaWiki, https://meta.wikimedia.org/wiki/StatMediaWiki, Accessed 6 May 2019.
  • W. Carr and S. Kemmis, Becoming critical: Education, knowledge and action research, Falmer Press, London, 1986.
  • W. M. Van Der Aalst, A. Adriansyah, A. K. A De Medeiros, F. Arcieri, T. Baier, T. Blickle et al., Process mining manifesto, International Conference on Business Process Management, Clermont-Ferrand, France, August 2011, pp. 169-194, 2011.
  • W.Premchaiswadi and P. Porouhan, Process modeling and decision mining in a collaborative distance learning environment, Decision Analytics, 2(1), pp. 6-40, 2015.
Data source: Dialnet