Approaching the evaluation of engaged learning

Jason Slade, a Department of Town and Regional Planning postgraduate student, tells us about progress relating to the evaluation of the department’s Community Engagement Initiative which is supported by the Engaged Curriculum Funding Stream.

My first blog post introduced our use of the engaged curriculum funding, which looks to better understand Town and Regional Planning’s community engagement project in Westfield through developing more effective methods for evaluating engaged learning. The need for this, as described in the first post, comes from a perceived thinness in current evaluation models, which tend toward pro forma questionnaires, ‘designed by a central university office [to] assess the extent to which the professor kept the course to a pre-set schedule, clearly defined the assignments, and specifically defined the grading criteria in advance.’[1] Engaged learning is different, perhaps closer to ‘real life’, with correct answers within pre-set schedules elusive. The work unfolds instead through negotiation and renegotiation between the university, the community and students, that strives to enact useful projects providing mutual benefit. The need to evaluate these processes appropriately becomes important, as Dewar and Isaac note,[2] lest the disconnect between familiar modes of working and engaged learning sees students become disengaged in ways that undermine the outcomes that we hope to facilitate. In developing criteria for evaluating engaged learning, then, we consulted the work of others who had wrestled with these issues.

Winkler[3] develops an evaluative approach using the five values of participatory action research – empowerment; social justice and equity; supportive relationships and inclusion; mutual learning and reciprocal education; and respect for diversity and power sharing – as criteria for evaluating a project, through reflection on her own experience, on the reports of students, and interviews/focus groups with community members. She concluded that all do not necessarily benefit equally from service learning; highlighting the importance of developing fine-grained, micro-level understandings of and reflections on community-university partnerships; and making the important insight that less successful engagements can develop our understandings as much as successful ones.[4] Questions remain, however, for though Winkler’s schema is a helpful development it does not seem to really reveal the extent to which students have developed the reflective and critical capacities we hope they have.

The development of these reflective and critical capacities was central to Sletto’s[5] rationale for employing engaged learning, and in assessing the extent to which a course he offered developed these he looked to Freestone, Thompson and Williams;[6] and Keen and Hall.[7] Freestone et al. utilised open-ended questionnaires to solicit student feedback, this was innovative in being longitudinal – with questionnaires before, during and after a work-placement – enabling reflection on how students’ expectations and understandings developed. Keen and Hall looked at 23 universities, analysing survey responses and concluding that the most important aspect of engaged learning is, ‘the sustained dialogue across boundaries of perceived difference that happens during service and in reflection along the way’.[8] Sletto’s approach evaluated his project using students’ comments on a web-based blog during the course and two sets of survey responses, one immediately after the course and one eighteen months later. These suggested that the process had developed more reflective practitioners and that something close to a ‘genuinely reciprocal’ partnership had been initiated.[9] Again, however, Sletto finds participants not benefitting equally, consequently suggesting that we should look to create spaces that encourage students to embrace the unexpected. He subsequently takes this further, with reference to an international context, suggesting that rather than pursuing projects in the ‘invited spaces of neo-liberal governance’ educators and students should look to create community-university partnerships that challenge hegemonic power relations.[10]

We looked to build on these insights, developing ‘rules’ for evaluation that attended to specific issues of engaged learning:

  • Involving actors other than students in evaluation
  • Finding ways to encourage and gather more comprehensive and richer feedback
  • Assessing the extent to which deeper level learning has occurred around developing reflexivity and critical engagement, particularly across perceived boundaries of difference
  • Moving beyond teaching and learning to consider and improve both the relationships between universities and community partners and the activity undertaken in community contexts

These were in addition to current evaluation priorities in Higher Education:

  • Understanding how teaching and learning can be improved
  • Taking account of students perspectives and experiences
  • Ensuring that teaching meets certain criteria around quality and standards

Our approach was to stage a series of informal interviews with students and residents. Four areas – engagement, commitment, relationships and ‘the university’ – provided a loose structure but interviews were driven by the insights and understandings of interviewees, with lengths varying between 5 and 30 minutes. The approach appears to be innovative in employing interviews as a mode of evaluation and in opening the process to community participants on broadly equal terms. In this instance the use of a paid post-graduate researcher was also helpful, as my liminal status – as neither quite student nor staff member – went some way to unsettling straightforward understandings of student/lecturer and university/community roles, hopefully allowing both students and residents to speak candidly and in confidence about their experiences. Our findings will be presented in our final blog-post.

[1] Dewar, M. E. and Isaac, C. B. (1998). Learning from Difference: The Potentially Transforming Experience of Community-University Collaboration. Journal of Planning Education and Research, 17, 337.

[2] Ibid., 338.

[3] Winkler, T. (2013). At the Coalface: Community-University Engagements and Planning Education. Journal of Planning Education and Research, 33 (2), 215-227.

[4] Ibid., 223-5.

[5] Sletto, B. (2010). Educating Reflective Practitioners: Learning to Embrace the Unexpected through Service Learning. Journal of Planning Education and Research, 29 (4), 403-415.

[6] Freestone, R., Thompson, S., and Williams, P. (2006). Student Experiences of Work-Based Learning in Planning Education. Journal of Planning Education and Research, 26, 237-249.

[7] Keen, C. and Hall, K. (2009). Engaging with difference matters: Longitudinal student outcomes of co-curricular service-learning programmes. Journal of Higher Education, 80 (1), 59-79.

[8] Ibid., 77.

[9] Sletto, (2010), 410-11.

[10] Sletto, B. (2012). Insurgent Planning and Its Interlocutors: Studio Pedagogy as Unsanctioned Practice in Santo Domingo, Dominican Republic. Journal of Planning Education and Research, 33 (2), 228-240.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s