GRACE GASPARDO UX RESEARCHER






PORTFOLIO
OTHER PROJECTS
ABOUT






RESUME
PROBING FOR FEEDBACK
Probes that were designed and deployed as a method of eliciting feedback on museum exhibit technologies from children with communication delays and their caretakers for the Children’s Museum of Atlanta.


Methods
  • ethnographic research
  • field research
  • interviews
  • surveys
  • probes
  • graphic design
Goals
  • Work on a research-driven project
  • Find opportunities within the research and design process to be more inclusive
  • Fulfill the project requirements to complete GeorgiaTech’s Human-Computer Interaction masters degree
Duration
8 months,  Sept 2019-April 2020
Advisor
 Carrie Bruce

BACKGROUND

 When I approached my masters project, I wanted to focus on investigating ways children with communication delays and their caretakers could be included and empowered in the design process. I decided on this research intent based on three  things that came up in early-stage research.
  • Children and their caretakers have very intertwined interactions when engaging with products designed for the child. 
  • There was a gap in the literature that involves both children with special needs and their caretakers in the research and design process.
  • Most research and design methods rely heavily on a participant’s ability to communicate verbally. 

I partnered with the Children’s Museum of Atlanta to provide a framing and focus to my research activities. With the context of the museum, I wanted to explore how children and their caretakers could be involved in museum exhibit design to make them more inclusive.

RESEARCH
Observations
 I had done general observational work to familiarize myself with museum exhibits and the ways the children and adults interact with one another. While I knew I would be conducting my research at the Children’s Museum of Atlanta, it was important to visit other museums to have a general sense of what kinds of exhibits are displayed, what and how technologies are integrated in exhibits, and how exhibits and the museum cater to children and their caretakers.
The museums I visited were Fernbank Science Center, Fernbank Museum of Natural History, Tellus Science Museum, and the Southern California Children’s Museum. At the museums, I focused on the interactions between children and adults and how they engaged with museum exhibits, focusing on communication styles.
I also worked on developing a set of taxonomies for museum exhibits based on the types of interactions they supported and the ways they communicated information.
In addition to the museums, I also visited a program center that supports families with Down Syndrome, Gigi’s Playhouse, and the local chapter of the Maker Faire, Atlanta Maker Faire, for additional observations of child-caretaker interactions. From the observational work I did at the museums, I found a couple of trends between types of exhibit interactions and the communication the facilitated:
Exhibits were considered to facilitate active engagement if they required the visitor to interact with the exhibit in some way in order to complete the experience. An example of an exhibit with active engagement would be the Ball Machine exhibits in the Children’s Museum of Atlanta. These exhibits require the visitor to manipulate artifacts in order to achieve an enriching experience. 



 Exhibits with passive engagement were exhibits that required very little interaction to complete the experience. An example of an exhibit with passive engagement would be the exhibit describing the Atlas moth in the Children’s Museum of Atlanta. All information is presented with no interaction required. There are some other exhibits that might require moving a card or pressing a button, and these would also be considered having more passive engagement since the interaction required is minimal and information is presented through basic text.

 The Children’s Museum of Atlanta hosted a Sensory Friendly Session the first Saturday every month for one hour. I was able to attend three mornings, but went during other times to supplement my observational data (i.e. looking at communication styles, methods of interactions, and exhibit features associated with both). Due to limited access to the target user group, most of my early observational work was done with other groups of children with differing needs and abilities. While it was informative, having more time with my user group would have been yielded stronger data to compare results with.
Interviews
 In addition to my observational work, I also interviewed an occupational therapist at Kidz Therapy Networks to gain some insights into working with children with special needs and how they design and structure activities for them. Although I had only one formal expert interview, I had conversations with other individuals who work with children with special needs in various capacities throughout my research for this project. The insights they’d shared contributed to my work and colored how I approached this project.
From the activity-oriented interactions I observed and from my conversations with experts, I gathered other findings and insights that would drive how I approached preparing activities and materials:

  • Adults default to helping and supportive roles, i.e. preparing materials, fine motor skill tasks
    • More likely to do activities alongside their child if there aren’t things to prepare for the child
  • Give children agency to direct their activities and time spent on the activities
    • Offer a variety of choice
    • Activities should support breaks
  • Adults often need to be invited to participate with their child
    • Easier for adults to develop interest in what their child is doing versus children developing interest in what their adult is doing
  • Content needs to appeal to children first, adults second
  • Some children work better in familiar environments with familiar people
  • Information should be communicated in multiple modalities
  • Children respond best to photographs
    • Easier to connect to real life experiences and instantiations of concepts
 I also interviewed representatives from the Children’s Museum of Atlanta (Karen Kelly, Director of Exhibits and Education, and Liz Nevil-Smith and Connor Lynch, the Managers of Exhibits over the course of this project) to understand the methods they’ve been using to gather feedback from visitors and the successes and limitations of some of those methods. Understanding how they’ve been working with their visitors and their needs informed my approach to make sure I was considerate of their needs. From my discussions with the Children’s Museum of Atlanta, I learned of some of the obstacles they’ve faced in their work to elicit feedback:

  • They work primarily with adults to gather feedback
    • Input from children is generally limited to observational research around exhibits and is infrequent after efficacy has been established
    • Adults are invited to fill out surveys at stations heading towards the exist and through email
    • Adults are invited to focus groups for more direct interactions and in depth conversations
  • It’s difficult to schedule multiple participants for focus groups, especially for families with children with special needs
  • Reported behavior is different from observed behaviors
    • Adults seem to project their preferences onto their child when they speak for them
Pivot
 In addition to doing general observational work and interviewing experts, I conducted a few in situ interviews and conducted some observational research during the Sensory Friendly mornings at the museum. In situ interviews would allow me access to both adult and child at the same time. As I conducted interviews, it quickly became apparent that there were drawbacks that I hadn’t fully considered:
  • Adult caretakers dedicated most of their attention to their child (as they should), and it was at times difficult to retain their attention to answer questions.
  • Adults often spoke for their child, even when I asked for their personal opinions and experiences. 
They often assumed I didn’t want to talk to their child, interpreting my requests in a way that didn’t focus on their child, and as a result children weren’t as included as I wanted (which was largely the reason for the in situ interviews). Those insights coupled with the ones I gained from the observational work and interviews with my contacts at the Children’s Museum of Atlanta, it became evident that I should look at finding another way to conduct research with my target user group.


RESEARCH QUESTION


How might we elicit feedback from children with communication delays and their caretakers?
Probes
Probes are a package of artifacts that prompts participants to do tasks that reveal cultural values and inspire new approaches to design. Within my literature review, I came across this method and it addressed a lot of the limitations I felt there were with the previous methods I had employed and those that came up when speaking with my contacts at the Children’s Museum of Atlanta.
 Probes were chosen as the feedback elicitation method due to its strengths and how it can meet many of the needs from my target user group and address the shortcomings of other methods.
The focus of the probes overlapped with my early discovery-oriented research: exploring participants’ relationship and impressions of museum exhibits and technologies.

DESIGN
Participants
I focused my recruiting efforts by reaching out to local Atlanta-based organizations that support families and children with special needs and therapy clinics that cater primarily to children with special needs. I sent flyers with a brief description of my research goals and the requirements associated with participation for this study:
  • Ability to complete a kit of activities within a specified time frame
  • Ability to visit the Children’s Museum of Atlanta within a specified time frame (admission would be provided)
  • Adult caretaker and child with communication delays (dyad)
  • Child between the ages of 5 and 10 years old
 The age range of the child was chosen based on work by Allison Druin working with children and co-design as well as the target age range of the Children’s Museum of Atlanta and the ages of the children who came during the Sensory Friendly Sessions.
Probe Prototype
Materials
 The supplies that were in the probe were chosen to offer the participants choice in what medium they could use to complete an activity and as well as be familiar in type and brand. The supplies included in each probe included:

  • Booklet of activity instructions
  • Booklet on exhibits and exhibit technology
  • Activity sheets
  • Blank lined notebook
  • Construction paper
  • Pre-cut construction paper shapes
  • Cardstock
  • Stickers
  • Crayola Markers
  • Crayola Colored Pencils
  • Crayola Crayons
  • Ticonderoga Pencils
  • Scissors
  • Play-Doh
  • Elmer’s Glue Stick
Activities
 Activities were designed to elicit preferences and context around experiences of exhibit technology, preference of exhibit technology, experience with exhibit technology, and impact of exhibit technology on exhibit experiences. Each activity fell within one of these areas and participants were offered multiple activities within a topic area. The content was derived from previous observational work and reflected the questions asked in the pre-study survey and interview. The activities varied in assumed level of ability, involvement, and modality. The probe contained a book that was included to help prime the participants, giving them some context about what the probe was about and what I was interested in learning from them. It was about museum exhibits and exhibit technologies. Participants were asked to read it for the first activity. The second activity was to get a sense of the types of exhibits and exhibits features they've seen, liked, and are familiar with.
 The third activity prompted participants to create a new exhibit. The goal was to get an idea of what combinations of things the participants liked or wanted to see.
 The fourth activity asked participants to share an exhibit that they had seen before and liked. 
 The last activity prompted participants to reflect on how the exhibit technologies affect their visits to museums and express that in some way.
Research Plan
The first part of the research plan was to do shepherded visits with the dyads, having them look at predetermined exhibits and exhibits of their choice, noting how engaged they are with the exhibits, the technologies they interacted with, their reactions to the exhibits, and tracking their time spent at exhibits. This was to be a reference point to compare the accuracy of results of the probes and other methods.
 To parallel the methods commonly used by the museum, one group of the adults would be given a survey to complete after the museum visit, another an interview, and the last group would have neither. They would then all get a probe, have a week to do the activities within it with their child, and then I would collect the probes, analyze the outputs, and get outside evaluation to review the content of the probes, the types of activities (their appropriateness and relevance), and get feedback on the value the probes might have for the Children's Museum of Atlanta. The weekend of the scheduled museum visits, the Atlanta Children’s Museum closed out of an abundance of caution. I was unable to follow my initial research plan and needed to modify it to fit the situation. The most drastic change was not having the museum visit and missing the resulting observational data that would have been used for validating the probe outputs and providing shared reference points and knowledge for the participants. Additionally, I shifted the focus to exhibits and exhibit technology more generally instead of focusing specifically on the exhibits at the Children’s Museum. This allowed me to broaden the scope of museum exhibit types and technologies and reference exhibits I had seen through my observational visits at other museums and cover as many potential experiences as I could.  I modified my plan to have the pre-study interview and survey, distribute the probes, compare the outputs of the probes with the interview and survey, and evaluate the probes. Luckily, I was able to distribute the probes prior to Atlanta’s shelter-in-place order, but I would not be able to pick them up once they were complete. I created a set of instructions for participants to have them document and send me their probe outputs so that I could analyze them.
The Initial Research Plan consisted of a museum visit with dyads, a comparison of methods (survey and interview), probe distribution, probe collection, analysis, and evaluation. The research would have focused on exhibits and exhibit technologies currently in exhibits at the Children’s Museum of Atlanta. The Modified Research Plan consisted of a comparison of methods (survey and interview), probe distribution, participants documenting probe outputs and sending digital files back for analysis, analysis, and evaluation. The research would instead focus on exhibits and exhibit technologies I had seen across multiple museums.
 The questions for the interviews and surveys were written to address the main question the probe was trying to address: what uses of technology would be appropriate for these users. The activity themes for the probe were derived from these intents.

RESULTS
Pre-Study Interview and Survey
 Both the pre-study interview and survey were designed to parallel each other and the themes of the probe. The questions addressed the main question the probe was trying to answer: what uses of technology would be appropriate for this user group. 
As expected, interview responses had more context and detail, but in both the interview and survey the technology stated as most engaged with was touchscreens. This is not that surprising based on the ubiquity of touchscreens and screen-based interactions. Touchscreens were also the technology claimed to be what their child preferred the most. 
 Additional insights that came from the interview were how the usage of technology in an exhibit could help support engagement and how they appreciated when exhibits had elements that could create an opportunity for conversation and discussion. More generally, exhibits that show the familiar in a new way are memorable. Children are also more likely to be engaged when they know something will happen if they interact with it.
Probe Outputs
 In choosing and combining features of exhibits, there were some differences between child and adult preferences. The asymmetry wasn’t unexpected based on responses from the interview and survey as well as observational data. The adult tended to choose modalities and features that lend themselves to more embodied interactions while the child tended to pick modalities and features lending themselves more towards smaller touch-based interactions and usage of finer motor skills. One interesting result was one of the dyads including recordings of their activity sessions. In the recordings, they referenced specific exhibits they had seen at Fernbank Museum of Natural History.
Comparison of Methods and Results
    The probe provides opportunities for the child to give their own responses rather than relying solely on the adult to speak on their behalf. It also allows for the participants to be more reflective and be primed with the context of the study. It is more time intensive, but can yield more engaged responses.     In comparision with the interview and survey, participants (both child and adult) chose a broader selection of exhibit types and features while doing the probe activities. The survey listed the same types and features that were in the probe activities, but fewer were selected via survey. There was also more of a reflection on how technology contributed to their personal experiences with museum exhibits.
    Post-Study Survey
     A feedback survey was distrubed after probe outputs were returned.  They reported generally having a positive experience for both adult and child, and felt that the activities took an acceptable amount of time. They found the package complete and expressed interest in doing it again.
    They also shared that the book was too high of a level for the child, that they weren’t sure how impactful their work was, and that they would have preferred having a wider variety of activities for the child’s skills and abilities.

    EVALUATION

    Children’s Museum of Atlanta
    I presented the probe and its results to the museum as a possible feedback method that could be incorporated with their current practices, which relies largely on online feedback and surveys.
     Their reaction to the probes was positive; they could see it being a valuable method for getting feedback from children. Currently, their primary methods for feedback were surveys and focus groups. Probes could be an alternative between the two that would allow them to generate richer insights than surveys and without the scheduling challenges and other factors they faced with focus groups.

    Their current feedback methods targeted adults primarily, but the probes would allow them to elicit more feedback directly from the children. After efficacy of an exhibit is established, generally through observational work, children become less of a participant in their research practices.
    They did have some reservations, but many of them were due to the constraints at the time of implementation:
    • Wanted to see a comparison of behaviors within the museum against reported responses and activity outputs
    • Wanted more use of photographs rather than illustrations
    Study Evaluation
    There are a number of modifications I would make given the chance to continue this project in the future. Generally, I would like to follow the initial research plan, and I would also want to have more involvement from others.

    Participants I would like more participants involved to get richer data outputs, but due to various constraints for this project, I was limited. I would recruit more aggressively in the future. While the intention of the method I explored was to make the research process more inclusive, I think there would have been value in being more broad with the user-group, especially for a pilot study. I could see running studies with children across the spectrum of support needs and language abilities to test the validy of the probes.

    Experts Additionly, more involvement from other experts would be beneficial for a project like this. For a large part, I was the only one involved in this project with any regularity. A more extensive evaluation process would be really beneficial to making this more robust. I’d want someone more familiar with creating education materials for children to review the content and rate how appropriate it is for the participants, and I’d like to see how different experts would analyze and evaluate the outputs and how consistent their analyses would be. 

    MuseumsHaving more museums involved would also be interesting. Getting their feedback on what kind of value they see in this method and how probes could be adapted to fit their research needs could help refine the design of the probes.

    Moderated Museum Visits Having participants visit the museum with me would also help the robustness of this project. It would help create a shared reference for all the participants, create a better sense of context for evaluation, and be a strong point of comparison for their activities and reference for validity of probe data.
      Probe Evaluation
      Going forward, there are a number of modifications I’d make to the probes themselves. These changes primarily target the materials within the probe, activity options, and focus of the probes.
       Beginning with the informational materials (i.e. reference booklet and instructions), they need to be more accessible to the participants. While I tried my best to write content that was easy to understand and sought out feedback, I needed to reach out to people who were less familiar with the content and more familiar with working with children. Additional visual aids might also help (while the Book of Exhibits had illustrations, the instruction booklet did not). I’d also be curious to try a digital version as that would also allow for videos and animations of certain concepts or examples.

      • Make the content more accessible to young children.
        • Simplify language and concepts
      • Provide visual instructions for certain tasks
      • Provide online version of the booklet that has animations/videos as examples and to better illustrate concepts


       There are many things I want to modify with the activity sheets. Foremost, I would want more variety in the types of activities a child could do. While I tried to include some variety based on the abilities of the child, there wasn’t enough across different tasks. It would also be good to try out different types of activities and validate which ones work best in which contexts.

      • A wider a variety of activities for each skill and ability level
      • Explore different types of activities
        • Which types of activities work best for which ages and abilities?
        • Which activities are easier to have consistent analysis with among evaluators?
        • Which activities contribute richer data for a particular focus?
       Coupled with the change in method (i.e. having a museum visit), I’d want the activities to feature exhibits in the museums more explicitly. While I used these exhibits as the references for the activity sheets, they were abstracted to a certain degree and I did not use photographs as I originally intended based on the conversation I had with the occupational therapist. I felt compelled to create illustrations due to my not having good reference photographs of these exhibits, and I prioritized clean graphics and visual cohesion. I didn’t plan on using the photos I took for personal reference to be seen by anyone else and therefore didn’t prioritize taking quality photos. I would take better reference photographs in the future. I’m also curious about how having a concrete example of an exhibit but speaking about it more generally would affect the outputs of the probe.

      • Reference actual exhibits to ground it in a real experience.
      • Using photographs instead of illustrations
       The activity sheets could also use some updates. Including the activity name and instructions on the activity sheet would help make doing the activities easier. Participants wouldn’t have to reference instructions in a separate location. Originally I wanted to offer variation in the types of activity sheets they could use, but in doing so may have put a higher workload on the participants which isn’t something I want to do going forward.

      Additionally, I noticed that participants filled in every box on a lot of the sheets and that made me wonder if the layout made the participants feel like they needed to fill in all the spots. To remove some of that influence, I’d want to create sheets that allowed for a little more freedom, such as having a large box versus the multiple small ones in a line.

        I would also modify some of the activity sheets to accommodate more variability in ability and give the participants an opportunity to provide more information.

        • Create better opportunities for participants to elaborate
          • Provide more options for differing levels of ability to provide more information
         I distributed the survey after the probes came in, but to make it easier on the participants, it might help to have one included in the probe. 
        • Include a post-activities survey to be completed and submitted with all other materials
          • Feedback on overall experience doing the activities
          • Feedback on materials
            • Clarity
            • Accessibility
            • Ease
         I would also want to vary the focus of the probes, being more specific about what the probe would cover. It would also be interesting to do more direct comparisons of exhibits and their features. This would also pull in some of my taxonomy work more explicitly.
        • Target more specific aspects of exhibit interactions related to technology
          • What impact do people want technology to have on their experience?
            • Assistive
            • Facilitative
        • Compare different exhibits with similar features
          • Why are some uses more successful than others?
          • How are they used?
          • What are they used in combination with?
        Reflections
         Probes provide an alternative method for museums to collect feedback and insights from their visitors. Its asynchronous nature allows for a lot of flexibility while producing rich outputs, especially for continuous feedback. The priming aspects of the probe also provide the opportunity for the museum experience to continue within the participants’ homes and work with their schedules. I think there is a lot of potential for its applicaion in these settings and it would be interesting to see how they can be even more tailored for museums. While this project didn’t pan out as I had originally planned, I learned a lot through the course of this project and it still remains something I’d enjoy revisiting again.  I outlined things I would change if I were to go forward, but more personally there are changes I would have made or make in the future. In hindsight, I think I would have adjusted my scope and/or early research schedule. I spent a lot of time giving room for the research to guide my direction and as a result the project didn’t feel like it came together until the back end of my timeline. This created a crunch towards the latter half of the semester. Coupled with the quick changes that the pandemic necessitated, it led to a lot of excess stress. Scoping more broadly, particualy the user group, while not addressing what initially drew me and motived the direction this project took, would have made parts of it easier and thus would have sped up some aspects of the research processes. This project challenged me in ways I didn’t expect. I quickly realized that while I am able to do a lot independently, I thrive when in collaboration with others. I enjoy processing through discussion and I believe it facilitates stronger outputs. So much of this project stayed in my head for a long time that it made organizing and articulating my thoughts around it difficult.