As we navigate the complexities of the modern world, it is crucial to reflect on how the advancements made in gender based analysis plus (GBA Plus), diversity, equity, inclusion, active reconciliation, accessibility, and environmental sustainability are incorporated into our evaluation practice. While we know why these lenses are important, this sub-theme challenges us to reflect, learn and share specific ways to apply these critical lenses into our work.

This session will bring together students, academics and industry professionals to share how they are incorporating lenses into their work.

Embracing a Feminist Approach to MEAL
This short presentation will provide an overview of Oxfam Canada’s Feminist Monitoring, Evaluation, Accountability and Learning (MEAL) principles. These principles are grounded in the organization’s commitments to decolonization, justice, equity, diversity, and inclusion, and how to practically apply these lenses in MEAL practices. The presentation will cover each of the seven key foundational elements to feminist MEAL:

1. An integral part of transformative social change
2. Shifts power to partners and participants
3. Highlights the evaluator as a facilitator
4. Values collective, context-driven knowledge generation
5. Provides a learning orientation to evaluations
6. Is rooted in safe programming, guided by ‘do no harm’
7. An approach rather than a strict set of tools

The presenter will provide three case studies from Oxfam Canada’s international gender justice and equity projects to demonstrate how the principles have been applied to MEAL activities. The presentation will end with 3-5 key takeaways for the audience to consider when pursuing feminist MEAL approaches in their own work.

Kimia Ghomeshi, Feminist MEAL Specialist, Oxfam Canada
Kimia Ghomeshi is a Credentialed Evaluator with 12 years of experience in research/evaluation specializing in community-based and feminist approaches. She started MKG Consulting Services in 2016 and since then completed over 25 evaluations. Since joining Oxfam Canada in 2019, she has provided technical MEAL support and oversight for international projects spanning HECA region of Africa, South and South East Asia and the Caribbean. Her professional experience spans several intersecting areas including gender equality and justice, immigrant and refugee settlement and integration, health equity, and diversity, equity, and inclusion. Kimia has a graduate diploma in Program and Policy Evaluation from Carleton University.

AI, GBA Plus, and EDI: The Alphabet Soup of the Future
As we navigate the vast landscape of artificial intelligence (AI) development, it becomes increasingly evident that biases embedded in algorithms can perpetuate and even exacerbate existing societal inequalities. Simultaneously, AI has been attributed to debunking the longstanding myth supporting researchers’ use of male mice because they were presented as being less erratic than female mice. This proposal sheds light on the transformative potential of Gender-Based Analysis Plus (GBA+) in the realm of AI, explores offering a nuanced approach to mitigate biases and foster inclusivity in machine learning models. Ultimately this can lead to improved Equity, Diversity and Inclusion (EDI) outcomes.

AI models learn from historical data, and if the data contains biases, the AI system may replicate and even amplify those biases. Mitigating the intersection of AI and perpetuating systemic biases involves adopting practices like fairness-aware algorithms, diverse and inclusive data collection, and ongoing ethical evaluations of AI systems. This will be a crucial element of the future work of evaluators. The proposed presentation discusses how algorithmic decision making can be influenced by applying GBA Plus, how training of AI systems can be taught to consider GBA Plus phases and dimensions, the practical applications such as using plug-ins to alter AI training. The presentation concludes with a discussion on the ethical implications of the responsible use of technology to ensure responsible AI development and deployment, promoting transparency, accountability, and fairness in the use of artificial intelligence across domains and individuals.

Jane Whynot PhD, Partner, Goss Gilroy Inc.
Dr. Jane Whynot is a partner at GGI and leads their social-cultural line of business. She also holds adjunct status at Carleton University teaching qualitative methods in a program evaluation graduate certificate program. She currently serves as the practice note editor for the Canadian Journal of Program Evaluation. Her research areas of interest include integrating diversity in evaluations, how various tools support this, and is a keen supporter of evaluation capacity building.

New Year – New Direction: How Public Safety is Adapting its Evaluation Processes to be More Practical and Supportive
Public Safety Canada (PS) provides Community Safety and Crime Prevention funding for communities across Canada and supports the evaluation of these funded projects. Over the last years, PS has learned, through research and community feedback that the PS evaluation process to date has been challenging for Indigenous communities because of a strong emphasis on Western-based evaluation practices, resulting in a necessary change to the PS evaluation processes.

This presentation will describe the challenges experienced and how they were discovered (research paper, anecdotal evidence, online engagement), explain the process of changing the evaluation process within the context of the Canadian Government, and share how PS is moving forward in supporting Indigenous communities with their evaluations including practical help in the form of an online community of best practice.

We will discuss the need for:

• flexibility in accepting non-academic forms of evidence and other ways of knowing
• creating evaluation guidebook and templates using simpler language and a visually appealing format, including alternative circular formats for logic models
• empowering communities by supporting data sovereignty and encouraging community-led evaluations
• defining success and indicators that work for both communities as well as government
• encouraging new forms of knowledge mobilization that are useful for the communities.

Crystal Klein, Evaluation Advisor, Public Safety Canada
Crystal has worked in the public service for the last 4 years as a program evaluator, and most recently as an evaluation advisor within the Crime Prevention Research and Evaluation Unit at Public Safety Canada. Crystal has a master’s degree in Legal Psychology from Maastricht University.

 

Examining the Connection Between Evaluation Theory, Policy and Practice Within the Canadian Federal Government
Evaluation theories are intended to serve as a guide for evaluation practice as they reflect a set of general prescriptions that specify the purpose of evaluation, how it should be done and the basis of what constitutes credible evidence. However, evaluation theories are not easily transferred into practice. It is when policymakers adapt evaluation theories into policy by situating them within a specific organizational, political, and cultural context that theory is integrated into practice. In an effort to further understand the theory-policy-practice connection, my research involves an examination of the six evaluation policies implemented by the Government of Canada since 1977 including their theoretical foundations and how they may have influenced evaluation practice. My research will also examine if and what changes occurred within evaluation practice following the implementation of the 2016 Policy on Results including the capacity to do and use evaluations, and how evaluation capacity may be strengthened to support policy interpretation and implementation.

Élyse McCall-Thomas, Ph.D. student, Faculty of Education, University of Ottawa and Evaluation Manager, Natural Sciences and Engineering Research Council
Élyse McCall-Thomas is a Ph.D. student with the University of Ottawa’s Faculty of Education, working with Dr. Isabelle Bourgeois. The focus of Élyse’s research is the theory-policy-practice connection with a specific focus the implementation and impact of the Government of Canada’s 2016 Policy on Results on the capacity to do and use evaluations. Élyse is also an Evaluation Manager with the Natural Sciences and Engineering Research Council and has over 15 years of experience in research and evaluation. Her work spans many areas including research funding, education, public health, mental health and addictions, and homelessness. Through her work, Élyse emphasizes considerations of capacity building, equity and inclusion, as well as the use of evaluation to support learning and evidence-based decision making.

Redefining Socio-Cultural Dynamics: Starting Point for Discussing the Integration of Program Evaluation in Haitian Public Management
This presentation uses the results of a study conducted to understand the obstacles to integrating program evaluation into the culture of public organizations in Haiti. The study showed that redefining stakeholder relationships, restructuring organizations, building capacity of national stakeholders and raising awareness of the benefits of evaluation are important prerequisites for the coherence of the institutionalization process. The study calls for theoretical reflection on the use of management tools that may not be adapted to a given cultural context.

Pauliana Borgella, Lead Coordinator, Research and Program Support, Centre for Research on Educational and Community Services and Centre for Interdisciplinary Research on Citizenship and Minorities, University of Ottawa 
Pauliana Borgella is an affiliated researcher at the Center for Studies in Governance and coordinator of the Center for Interdisciplinary Research on Citizenship and Minorities (CIRCEM) and the Center for Research on Community and Educational Services (CRSEC) at the University of Ottawa. She has experience in the administrative and financial management of projects and the coordination of public programs in Haiti. Her current studies focus on the challenges of implementing public administration reforms in countries of the South and on the integration of program evaluation in Haitian public administration.

Adapting, Clarifying, Learning: Lessons from a Practicum
Applying evaluation theories and frameworks is not always easy. It can be challenging to find the right combination that speaks to the program under review and the Client’s context. As first-time practitioners, we were fortunate to be matched to a supportive practicum client that was open to our ideas and welcomed our engagement. After a challenging start, we settled on the lens of behaviour change for our evaluation framework in recognition that new programs require the right conditions for public servants to become agents of change.

We believe that the lessons we learnt from this evaluation may be of value to the community as we all respond to more evaluations that centre on cultural change, requiring employees to do more with less and/ or adapt to new technologies. The setting for this evaluation was at a Canadian federal department, where the organization is seeking to modernize its Education, Training, and Awareness program.

The program theory that drives our evaluation is focused on a behaviour-based model to align it with the principles of a “good” theory of change as espoused by John Mayne. The COM-B framework for behavioural change not only informs our Theory of Change (ToC) but is used to formulate our evaluation questions and consequently shapes all the subsequent phases of our evaluation.

Helen Stitt, Federal Department of Industry, Science and Economic Development
Helen Stitt is an experienced public servant currently on leave from the Australian Department of Education while she accompanies her partner on posting to Canada. Prior to joining the Australian Public Service, Helen spent over twenty years working in the State (Provincial) Victorian Government, primarily in the Department of Premier and Cabinet and Department of Treasury and Finance, holding multiple leadership roles in economic policy. While in Ottawa, Helen is working in the Federal Department of Industry, Science and Economic Development (ISED). Helen holds a Bachelor of Arts from the Australian National University and is a graduate student in the Public Policy and Program Evaluation Graduate Diploma program at Carleton University.

Mahmoud Rahim, Health Services Administrator, Unity Health Toronto – St. Michael’s Hospital
Mahmoud Rahim, MPA, BSc, BA, is a health services administrator at Unity Health Toronto -St. Michael’s Hospital. He is also a graduate student in the Public Policy and Program Evaluation Graduate Diploma program at Carleton University and is the Student Representative on the CES-NCC Board of Directors.