5  Semester 1 Week 9

5.1 Collecting data, developing evaluation for the introduction

In this session, I tend to cover data collection (i.e. trying to pre-empt any issues before they happen) and also developing evaluation of the Introduction section. I usually have both quant and qual projects so cover both types of data collection - I would only cover one though, if students were only doing one type. We then cover evaluation in Intros using some resources that Wil and I developed for our Research Methods 2 course. (Discussions are covered in Semester 2 Week 5)

Although I cover data collection in Week 7 (if I have time), that was an initial discussion to put it at the forefront of their minds and this is more in-depth, as they should be close to having ethics approved if they don’t already.

5.1.1 Expected progress

  • By now, ethics approvals should be imminent or through already
  • Some students may have already started data collection, or might be getting materials together in anticipation of starting
  • If doing secondary data analysis, the timeline is a bit less pressured
  • Students should be working on introductions

5.1.2 Data Collection (Quant projects)

(N.B. I only supervise survey-based quant projects, so my notes refer to this only)

  • How is data going to be collected (e.g. Qualtrics?). Student should use university email, not personal email for account.
    • Even if ethics isn’t through yet, can still familiarise themselves with it.
  • Are students clear on how to present their study on whichever system they are using?
  • Test, test and test again
    • Check the platform’s default settings for things like IP address or location tracking – some of these are switched on by default and can undermine anonymity even when ethical approval assumes an anonymous survey
    • Where possible, pilot with a few real people (e.g. friends, coursemates) before sending it to me to test, not just testing it themselves. Some problems (confusing wording, unexpected timing, issues on a different device or browser) only show up once someone unfamiliar with the study tries it.
Mistakes I’ve seen before
  • One student had to start data collection from scratch after not noticing, until midway through, that their scale order was inconsistent: Strongly Agree - Disagree - Neither agree nor disagree - Agree - Strongly Disagree
  • On another project, participants saw one of two pictures (the between-subjects condition), but it wasn’t clear from the data which picture had been shown, so data collection had to start again
  • I emphasise to students that they should not start data collection without extensive testing and I reiterate to let me know when it is ready for me to look over. Other supervisors may differ on this, but it’s caused issues in the past (see examples above).
  • Students need Emily’s extended debrief if going to recruit Level 1 students (within our School, anyway)

5.1.3 Data Collection (Qual projects)

  • How are students collecting data?
    • If online, must use GDPR-approved system (e.g. Teams)
    • If in person, will need a) appropriate recording equipment, b) a quiet space
    • Where possible, have a backup recording method running (e.g. a second device), in case of equipment failure. Unlike quant data collection, it is unlikely that you can repeat the session with the same participant if something goes wrong
  • Some supervisors differ on their opinions on this, but I ask students to send out questions in advance for all participants, as it can be more accessible for certain people (e.g. those who are neurodivergent or who don’t speak English as a first language)
  • How and when will consent be reconfirmed at the start of the session? I ask students to capture a verbal confirmation on the recording itself, rather than relying only on a form signed beforehand. We also discuss ongoing consent (i.e. checking-in throughout the interview/focus group)
  • If students are recording on a stand-alone device (e.g. audio recorder), ensure that recordings are transferred to secure, university-approved storage as soon as possible after the session, rather than left sitting on the device itself. Data needs to be deleted off the device as soon as possible.
  • I ask students how they are feeling about collecting data. Typically, they can be a bit nervous
    • Important to practise setting things up/recording etc.
    • Good to pilot things (e.g. with a friend). I’m also happy to help in office hours etc.
    • Can practise with each other (i.e. others in the group) to help get comfortable
    • Can be helpful to write a list of everything need to cover at the start and end so that they don’t forget anything
    • It’s also worth asking whether students have thought about how they’d handle a participant becoming upset or wanting to stop partway through, particularly given some topics may touch on sensitive experiences. It’s good to anticipate this, so it isn’t something they’re figuring out for the first time in the moment.

5.1.4 Evaluation in introductions

In RM2, Wil and I have made some excerpts from intros and discussions for our labs. In today’s session, I only cover intros and do discussions in a later session (Semester 2 Week 5). Materials can be downloaded from here.

We use a fictional topic (zombie apocalypse survival confidence) rather than examples from students’ own dissertations, so no one’s real topic ends up coming up, and the focus stays on structure and argument rather than content.

I show students the “not so good” paragraph first and ask them to have a go at improving it themselves (individually or in pairs) before I reveal the stronger version, so they’re practising the judgement rather than just recognising it once shown. We then have a discussion about it.

This resource has ‘not so good’ examples, then ‘stronger’ examples and then notes for things to look out for and finally some tips. The three things we look at are:

  • Synthesising evidence
  • Focused criticism
  • Why is your study worth doing? – this reinforces the same point from the bullet-point intro session: just because something hasn’t been done before doesn’t make it worth doing on its own

At the end, I ask students to look back at their own reading notes so far and identify one place where they’ve been describing one study in detail, rather than synthesising across sources, and one criticism from the literature that their own study could specifically address. This gets them applying the three lenses to material they already have, without needing a finished draft.

5.2 Internal resources

Guidance from Prof Emily Nordmann on extended debriefs for Level 1 students

For Psychology & Neuroscience students in University of Glasgow only: how to recruit on SONA

Excerpts plus notes and tipes on Developing Evaluation in Introductions by Dr Wil Toivo and I

5.3 External resources

Guidance on SONA