A design science approach to developing and evaluating items for the assessment of transversal professional competences

Abstract

Purpose: Relying on a design science paradigm, the purpose of this paper is to describe the development and evaluation of items for an ICT artefact that supports the assessment of transversal professional competences within the validation of prior learning (VPL). To do so, the authors build a conceptual bridge between the Occupational Information Network (O*NET) and the European Qualifications Framework (EQF). Design/methodology/approach: Design science research paradigm, in particular the participatory development of candidate items and their evaluation in a multi-stakeholder approach. Findings: The authors find that a self-assessment of professional competences should be comprised of 160 items in order to cover the breadth and depth of the O*NET in the hierarchical taxonomy. Such quantity of items sufficiently builds a conceptual bridge between the O*NET and the; EQF. Practical implications: When designing procedures for the VPL, it is imperative to bear in mind the purpose of the validation procedure, in order to determine relevant stakeholders and their needs in advance as well as the; required language proficiency of the assessment instrument. Social implications: The innovative value of this approach lies in the combination of an underlying hierarchical taxonomy with assessment items that are developed based on the qualification standards of different Austrian professions. Together with specific verbs that were adapted for each particular item, an innovative self-assessment is proposed. Thereby the authors aim to account for some of the mentioned shortcomings of the EQF. Originality/value: This paper applies a design science paradigm to develop an ICT artefact that should support the VPL. By reflecting on the design process, the authors introduce a theoretical bridge between the O*NET and the EQF. Thereby the authors aim to account for some of the mentioned shortcomings of the EQF. © 2021, Florian Fahrenbach.

Keywords

Author Keywords

Evaluation ICT Assessment Competences Artefact Validation of prior learning

Index Keywords

Author Affiliations

WU Vienna University of Economics and Business, Vienna, Vienna, Austria

Funding & Acknowledgements

Österreichische Forschungsförderungsgesellschaft, FFG

Grant: 866270

Funding text 1: The work was partially funded by the EU H2020 program under MSCA-RISE agreement 645751 (RISE BPM) and FFG Austrian Research Promotion Agency (project number: 866270). I am very grateful to anonymous reviewers that helped to improve the paper.; Funding text 2: The validation of prior learning (VPL) is defined as the process of \u201Cassessing and recognising a wide range of skills and competences which people develop through their lives and in different contexts, for example through education, work and leisure activities\u201D (, p. 216) and plays an increasingly important role in the educational policies of the European Union (EU) and beyond. It is supported by the decision to foster lifelong learning (), through the introduction of the European Qualifications Framework (EQF) () and the recommendation to validate non-formal and informal learning regardless of the route of acquisition (). Developing viable and efficient methods for the assessment of professional competences in VPL procedures may facilitate a decrease in unemployment, increase labour market mobility and facilitate social cohesion and social justice in the EU (; ; ). While legislation for the VPL are in place in most of the EU members states, developing innovative and standardised methods for the assessment of professional competences within VPL continues to challenge EU policy-makers (, ) and researchers (; ).

Horizon 2020 Framework Programme, H2020

Grant: 645751

Funding text 1: The work was partially funded by the EU H2020 program under MSCA-RISE agreement 645751 (RISE BPM) and FFG Austrian Research Promotion Agency (project number: 866270). I am very grateful to anonymous reviewers that helped to improve the paper.; Funding text 2: The validation of prior learning (VPL) is defined as the process of \u201Cassessing and recognising a wide range of skills and competences which people develop through their lives and in different contexts, for example through education, work and leisure activities\u201D (, p. 216) and plays an increasingly important role in the educational policies of the European Union (EU) and beyond. It is supported by the decision to foster lifelong learning (), through the introduction of the European Qualifications Framework (EQF) () and the recommendation to validate non-formal and informal learning regardless of the route of acquisition (). Developing viable and efficient methods for the assessment of professional competences in VPL procedures may facilitate a decrease in unemployment, increase labour market mobility and facilitate social cohesion and social justice in the EU (; ; ). While legislation for the VPL are in place in most of the EU members states, developing innovative and standardised methods for the assessment of professional competences within VPL continues to challenge EU policy-makers (, ) and researchers (; ).

References 10 References

1 Achterkamp, Marjolein C., Critically identifying stakeholders evaluating boundary critique as a vehicle for stakeholder identification, Systems Research and Behavioral Science, 24, 1, pp. 3-14, (2007)

2 Allais, Stephanie Matseleng, The impact and implementation of national qualifications frameworks: A comparison of 16 countries, Journal of Education and Work, 24, 3-4, pp. 233-258, (2011)

3 A Taxonomy for Learning Teaching and Assessing A Revision of Bloom S Taxonomy of Educational Objectives, (2001)

4 Bergsmann, Evelyn Maria, Evaluation of competence-based teaching in higher education: From theory to practice, Evaluation and Program Planning, 52, pp. 1-9, (2015)

5 Bergsmann, Evelyn Maria, The Competence Screening Questionnaire for Higher Education: Adaptable to the needs of a study programme, Assessment and Evaluation in Higher Education, 43, 4, pp. 537-554, (2018)

6 Beyer, Sylvia, Gender Differences in the Accuracy of Self-Evaluations of Performance, Journal of Personality and Social Psychology, 59, 5, pp. 960-970, (1990)

7 Biemans, Harm J.A., Towards competence-based VET: Dealing with the pitfalls, Journal of Vocational Education and Training, 61, 3, pp. 267-286, (2009)

8 Evaluating the Quality of Learning the Solo Taxonomy, (1982)

9 Vocational Training European Journal, (2000)

10 Making Learning Visible Identification Assessment and Recognition of Non Formal Learning in Europe, (2000)

A design science approach to developing and evaluating items for the assessment of transversal professional competences

How to Cite