The effectiveness of a cybersecurity awareness programme is dependent on features like its ability to comprehend evolving and emerging cyber threats, advancements in technology, and shifts in an organisation’s business missions and priorities as well as the usability of awareness material and its delivery channels in terms of the relevance of topics, the quality of the content, and the preferred delivery channels.

Factors widely used to measure the effectiveness of a cybersecurity awareness programme include assessing its reachability and touchability (i.e., the ability to reach and impact an audience) as well as monitoring improvements in an audience’s cybersecurity competencies, attitude, and behaviour through their participating in awareness programmes.

To achieve this analysis, one or multiple qualitative and quantitative methods are used, such as conducting surveys, assessment tests, and interviews of the participants, observing the participants’ behaviours, and analysing system and log data.

The existing review and evaluation approaches are mostly limited to what factors to measure and how to measure them. Unfortunately, they often do not consider when to take a measurement (i.e., before, during, or after the programme implementation) and for whom each factor is measured. For example, conducting assessment tests before and after the awareness programme can provide results that can be assumed to be due to the awareness programme; whereas conducting interviews at regular intervals helps to identify areas where people may need further support. Likewise, the meaning of effectiveness may vary according to the stakeholders involved. An audience may understand effectiveness in terms of how interesting and engaging an awareness programme is. Similarly, the cybersecurity awareness professional may perceive effectiveness in terms of reachability and touchability, as mentioned earlier. Finally, the programme’s sponsors may want to know what value the programme brought to the organisation to decide whether or not to further invest in the awareness programme. Therefore, the review and evaluation of cybersecurity awareness programmes should consider the needs of all the relevant stakeholders and obtaining results in their desired formats will assist in future decision-making and the ongoing sustainability of the programmes.

CyberSec4Europe’s Awareness effectiveness study focuses on developing cybersecurity awareness review and evaluation metrics that address the limitations mentioned above and help to make the review and evaluation processes as inclusive, complete, and unbiased as possible. We believe that such metrics are necessary to effectively carry out continuous monitoring of and enhancements to cybersecurity awareness programmes. The Awareness effectiveness study will be of value to anyone looking for designing and evaluating a cybersecurity awareness programme.