Welcome to the CHIIR 2021 tutorial on addressing rigour and ethics when designing Interactive Information Retrieval (IIR) experiments involving participants with disabilities. On this web page you will find information on:
We hope you enjoy the tutorial and it provides lots of issues for you to ponder on, whether you are working with users with disabilities or other vulnerable users. We provide a list of extended references at the end.
Gerd and Andy.
Research is conducted within the field of interactive information retrieval on a broad diversity of users, with more interest recently in working with minority groups, such as people with disabilities. The tutorial addresses best practice when conducting research on cohorts that might be regarded as "vulnerable participants". Previous research has showed that rigorous experiments for all types of users can be difficult, and there are many issues to be raised with regard to the tension between rigour and ethics. The tutorial examines these issues by engaging attendees with particular groups using scenarios and personas.
We have developed both scenarios and personas for the tutorial. Please download these documents in preparation for the tutorial exercises.
No particular knowledge of impairments is required, but an understanding of the process of designing Interactive Information Retrieval (IIR) experiments would be useful. The latest version of Zoom and a browser such as Chrome or Firefox is sufficient for engagement in the activities planned for the tutorial.
The tutorial activities will follow the main themes directly, and we will interleave presentation of key material with exercises for attendees to further cement the issues in the field. We will use the scenarios and personas provided in section 2 above to inform the tutorial exercises, and will put attendees into groups to allow different perspectives. We will encourage attendees to feedback their perspectives either during or after the tutorial depending on time constraints.
Attendees can feedback their exercise results via a Poll Everywhere link. This will not be mandatory, and attendees can choose another method by which they can provide feedback if they prefer e.g. email. Any responses will be collated by the tutorial presenters and made available to attendees i.e. via the tutorial website after the event.
Overview of the tutorial including timings of activities, an overview of the material to be delivered and exercises for participants is provided below. The total time for the tutorial is 3 hours including a break.
Timing: 10 minutes.
Material: Introduction to the key themes and structure of the tutorial, emphasizing the participatory nature of the event.
Timing: 15 minutes. We will spend around 10 minutes on the material and 5 minutes on exercise.
Material: Issues working with vulnerable groups will be examined, and given good practice for obtaining consent will be reviewed.
Exercise: Attendees will be placed into groups and will be asked to reflect on key themes identified given in a persona allocated to the group e.g. what is the best ethical practice for that cohort, and how would informed consent be obtained.
Timing: 30 minutes. We will spend around 15 minutes each on presenting the material and the exercise.
Material: Experimental design issues such as recruitment, topic or task selection and appropriate conduct of the experiment given the ethical issues will be outlined. For example the issue of the number and choice of topics will be examined, given appropriate ethical conduct and rigour in experimental design for IIR research.
Exercise: Attendees will be put back into their groups with the same personas they used in the previous exercise and provided all the scenarios to reflect on issues to do with experimental design for vulnerable cohorts.
Timing: 30 minutes.
Timing: 30 minutes. We will spend around 15 minutes each on presenting the material and the exercise.
Material: Issues and problems when analyzing data from experiments with vulnerable groups will be examined, together with an overview of best practice in both qualitative and quantitative methods to address the issues.
Exercise: Attendees will be put back into their groups with the same personas and scenarios in previous exercises. Attendees will reflect on results analysis for specific groups, examining appropriate methods for the specific vulnerable group allocated.
Timing: 10 minutes. We will spend around 5 minutes each on presenting the material and the exercise.
Material: The best practice for reporting results in an appropriate and respectful manner [4] will be overviewed.
Exercise: Attendees will be put back into their groups with the same personas and scenarios in previous exercises. Attendees will reflect on appropriate strategies for reporting results for vulnerable specific groups allocated.
Timing: 15 minutes.
Material: Wrap up of tutorial, reminder of key themes and examination of potential ways forward.
Dr. Gerd Berget is an associate professor at Oslo Metropolitan University, Department of Archivistics, Library and Information Science (gerd.berget@oslomet.no). Her research fields are interactive information retrieval, universal design and human computer interaction. Her PhD investigated the information searching behaviour of people with dyslexia.In recent years, she has been working with the information needs of people with intellectual impairments. Gerd has a special interest in research ethics and vulnerable user groups, with a particular focus on people with disabilities.
Dr. Andrew MacFarlane is a Reader at City, University of London, Centre for HCI Design (andym@city.ac.uk). His PhD in Information Science was supervised by Prof Stephen Robertson. He has a wide range of research interests that range implementing IR systems to evaluating them in user studies. He has a particular interest in disabilities and search, particularly dyslexia that he helped to pioneer and maintains an active interest in the field. He is also active in the field of the impact of aphasia and search.
Berget, G. and A. MacFarlane, 2019. Experimental methods in IIR: The tension between rigour and ethics in studies involving users with dyslexia. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval (CHIIR). 2019. ACM: New York, 93-101.
Berget, G. and A. MacFarlane, 2020. What is known about the impact of impairments on information seeking and searching? Journal of the Association for Information Science and Technology, 71, 5, 596-611.
Berget, G., A. MacFarlane, and N. Pharo, 2020. Modelling the information seeking and searching behaviour of users with impairments: Are existing models applicable?. Journal of Documentation.
Bracken-Roche, D., Bell, E., Macdonald, M. E., & Racine, E. 2017. The concept of 'vulnerability' in research ethics: An in-depth analysis of policies and guidelines. Health research policy and systems, 15(1), 8-8. https://doi.org/10.1186/s12961-016-0164-6
Burden, R., 2008. Is dyslexia necessarily associated with negative feelings of self-worth? A review and implications for future research. Dyslexia, 14, 3, 188-196.
Cameron, L. and J. Murphy, 2007. Obtaining consent to participate in research: The issues involved in including people with a range of learning and communication disabilities. British Journal of Learning Disabilities, 35, 2, 113-120.
Carawan, L.W., B.A. Nalavany, and C. Jenkins, 2016. Emotional experience with dyslexia and self-esteem: The protective role of perceived family support in late adulthood. Aging & Mental Health, 2016, 20, 3, 284-294.
Frankfort-Nachmias, C., Nachmias, D., & DeWaard, J., 2015. Research methods in the social sciences. Worth.
Hatcher, J., Snowling, M. J., & Griffiths, Y. M. 2002. Cognitive assessment of dyslexic students in higher education. British Journal of Educational Psychology, 72(1), 119-133. https://doi.org/10.1348/000709902158801
Hill, H., 2013. Disability and accessibility in the library and information science literature: A content analysis. Library & Information Science Research 35, 2, 137-142.
Høien, T., & Tønnesen, G. 2008. Ordkjedetesten [The Word Chain Test]. Logometrica.
Honey, A., E. Emerson, and G. Llewellyn, 2011. The mental health of young people with disabilities: Impact of social conditions. Social Psychiatry and Psychiatric Epidemiology, 46, 1, 1-10.
Ingwersen, P. 1992. Information retrieval interaction, London, Taylor Graham.
International Test Commission. 2001. International Guidelines for Test Use [Article]. International Journal of Testing, 1(2), 93.
MacFarlane, A., Al-Wabil, A., Marshall, C.R., Albrair, A., Jones, S.A., & Zaphiris, P. 2010. The effect of dyslexia on information retrieval: A pilot study. Journal of Documentation, 66(3), 307–326.
MacFarlane, A., Albrair, A., Marshall, C.R., & Buchanan, G. 2012. Phonological working memory impacts on information searching: An investigation of dyslexia. In J. Kamps, W. Kjraaj, & N. Fuhr (Eds.), Proceedings of IIiX 2012 the Fourth Information Interaction in Context Symposium, Nijmegen, the Netherlands (pp. 27–34). New York: ACM.
MacFarlane, A., Buchanan, G., Al-Wabil, A., Andrienko, G., & Andrienko, N. 2017. Visual analysis ofdyslexia on search. In R. Nordlie & N. Pharo (Eds.), Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (pp. 285–288). New York: ACM
Mortimore, T., & Crozier, W. R. 2006. Dyslexia and difficulties with study skills in higher education. Studies in Higher Education, 31(2), 235-251. https://doi.org/10.1080/03075070600572173
Mueller, S. T. 2010. The Psychology Experiment Building Language, Version 0.11. In http://pebl.sourceforge.net
Nario-Redmond, M.R., J.G. Noel, and E. Fern, 2013. Redefining disability, re-imagining the self: Disability identification predicts self-esteem and strategic responses to stigma. Self and Identity, 12, 5, 468-488.
Nind, M., 2014. What is inclusive research? London: Bloomsbury.
Novita, S., 2016. Secondary symptoms of dyslexia: A comparison of self-esteem and anxiety profiles of children with and without dyslexia. European Journal of Special Needs Education, 31, 2, 279-288.
Quinn, J. M., & Wagner, R. K. 2015. Gender differences in reading impairment and in the identification of impaired readers: Results from a large-scale study of at-risk readers. Journal of Learning Disabilities, 48(4), 433-445. https://doi.org/10.1177/0022219413508323
Sheppard, K. and T. Badger, 2010. The lived experience of depression among culturally Deaf adults. Journal of Psychiatric and Mental Health Nursing, 17, 9, 783-789.
Smythe, I., & Everatt, J. 2001. Adult checklist. https://www.bdadyslexia.org.uk/common/ckeditor/filemanager/userfiles/Adult-Checklist.pdf
Strauss, E., Sherman, E. M. S., & Spreen, O. 2006. A compendium of neuropsychological tests: Administration, norms, and commentary. Oxford Press.
Warmington, M., Stothard, S. E., & Snowling, M. J. 2013. Assessing dyslexia in higher education: The York adult assessment battery-revised. Journal of Research in Special Educational Needs, 13(1), 48-56. https://doi.org/doi:10.1111/j.1471-3802.2012.01264.x
Wilkinson, G. S., & Wilkinson, G. J. 2006. Wide Range Achievement Test. Psychological Assessment Resources.
Williams, P., 2020. "It all sounds very interesting, but we're just too busy!": Exploring why 'gatekeepers' decline access to potential research participants with learning disabilities. European Journal of Special Needs Education, 35, 1, 1-14.