Tips & Resources

Should I pilot my survey before launching it?

Yes! Given Survconf relies on multiple underlying services (Amazon Web Services, Heroku, Agora, etc.), it is a good idea to pilot your survey and make sure interactions are properly being captured immediately before launching to ensure that none of the services are down. You can pilot your survey yourself by accessing your survey link from two different browsers (for text chat) or two different web-cam enabled devices (for video/audio) to be paired with yourself to pilot the interaction. To ensure everything is working correctly with your survey set up, you should also pilot being able to download your test videos from AWS to make sure they were uploaded correctly.

Do you have any recommendations on whether it is best to use text, audio, or video in studies?

Although video interactions will provide the richest data, there are tradeoffs (with regard to added complexity) that you should consider. Using text/chat interactions in SurvConf is the easiest to set up, the easiest to recruit participants for, and minimizes the likelihood of technology issues. Alternatively, the way we recommend thinking about using video interactions in a study (especially via platforms like MTurk/Prolific), is to imagine if you were to put 4 people who don’t use video conferencing tools regularly in a video conference for the first time; the odds of at least one out of those four people having some technology issue (e.g., webcam, microphone, or connectivity issues) is non-trivial. Consequently If you opt to use video or audio interactions (especially with >2 interaction partners), you will want to make sure to over-sample to account for dyads/groups due to technology issues that you may need to drop from analyses and/or potentially use check/filter questions following the video interaction to gauge whether participants (or anyone else in their group) and alter their survey questions following the interaction accordingly.

Why can’t I get enough participants through Amazon Mechanical Turk or Prolific Academic to enter into my survey simultaneously as dyads/groups?

Given many potential participants are hesitant (or are unable) to go on video, we found the following options will help to result in a greater number of pairs/groups. 

• Make sure there is nothing enabled that is limiting the flow of participants into your study (examples below):
  • If you are using Prolific, you’ll want to reach out to Prolific support to turn off the ‘rate limiter‘ for your account.
  • If you are using CloudResearch/TurkPrime to run your Amazon Mechanical Turk study do not choose the ‘MicroBatch‘ setting (although ‘HyperBatch’ is okay).
• You can double the amount of time that participants wait for a partner by offering them a bonus to wait longer. Specifically, if a participant is not matched within that wait time period that you set, you could provide participants with the option to either receive a small base compensation and conclude the study immediately, or a bonus to go back into the waiting room for the same additional time as the original wait. 
• Increasing participant pay will attract a larger flow of participants quickly.
• Make sure you that you don’t use too many filters thereby limiting the potential participant pool too much.
• Make sure you describe in your study listing that being on webcam is required.
• Keep an eye on your study to ensure that an ample flow of participants are entering at a time to be paired. If it starts slowing down too much, stop and relaunch your study (potentially at a different time) so that it goes back to the top of the list of studies that participants can join (for larger sample-size studies, this is often required).
• Consider using smaller groups (e.g., it will be harder to successfully match groups of 8 participants joining the survey within a few minutes of each other as compared to groups of 2).
• You can conduct a pre-survey, where you get a large number of participants to select one of a small number of time slots when they would be available for entering in the second (main interaction) survey, which would increase the flow of participants entering into the interactions within short periods of time.

* NOTE: Before planning your studies on Prolific or MTurk, please also take a look at this helpful document, put together by Dean Baltiansky, a doctoral student at Columbia GSB!

Can I create pages within a SurvConf Qualtrics survey that force participants to stay on the interaction page for a certain amount of time or auto-advance them after a set time period?

Yes, all of Qualtrics normal functionality works. To accomplish this, you can use a combination of altering the conf_length variable (as described in the instructions) and/or add a normal Qualtrics timing question to the video/audio/text chat interaction page and set the relevant criteria through the timing question (e.g., auto-advance or delay showing submit button).

I want to compare the effect of differing modes of communication, do you have any recommendations for study design?

SurvConf works well for comparing differing modes of communication (text vs audio vs video). Our one recommendation is that to avoid confounding participants being aware of each other’s physical appearances versus the effect of the mode within a particular interaction, at the beginning of the survey, you have an introductory video session where participants get to see each other irrespective of condition. Then, later on in the survey, for the interaction of interest, you manipulate the mode of interaction used. Given SurvConf’s ability to have multiple interactions with the same partner within a study, this design is easily accomplished. 

Why are some participants having technology issues that result in a pair/group not working properly? 

Like any video-conferencing software, participants may have technology issues for a variety of reasons, such as malfunctioning devices or privacy/webcam settings that do not allow the survey access. To limit the potential of these issues, SurvConf includes a technology check for webcams and microphones at the beginning of the survey (note: this is not needed for text-only surveys) that can be used to filter out participants who may have issues before they are even paired with a partner. Additionally, in the instructions document, we provide step-by-step information on how to ensure that SurvConf is not blocked by firewalls that may exist in certain universities or enterprises (e.g., by running SurvConf via a cloud proxy). However, some participant technology issues are still unavoidable. As a result, we suggest you over-recruit in the expectation that some participants will not be able to interact with their partners due to their technology issues. 

Does SurvConf work for participants who are accessing the survey through their smartphone? 

SurvConf is not currently optimized for smartphone use. Although SurvConf does work on some (but not all smartphones) we advise against allowing participants to use smartphones as: a) some participants will not be able to properly access their smartphones video camera via SurvConf within a survey, and b) the participant experience of engaging in a multiple person video call will be fundamentally different on a smartphone (e.g., where not all other matched participant videos are visible and/or are much smaller). By default, SurvConf does not restrict smartphone users from participating in case you wish to include them, but if you would like to restrict smartphone users from entering the survey, this page provides details on how to do so https://www.qualtrics.com/support/survey-platform/survey-module/mobile-survey-optimization/.

Why is Amazon Web Services still charging me even though my study was completed a while ago?

If your study collects video/audio that exceeds Amazon’s free storage teir level (5 GB at the time this post was written), you will be charged their excess storage rate (~$0.02-$0.03 per GB) for storage each month. To avoid being continually charged this amount, log into your Amazon Web Services/S3 account and delete the videos/audio that you collected once you have downloaded your own copies of the videos.

Are there any other sources like SurvConf?

Yes, SurvConf is built using part of the code from SMARTRIQS (https://smartriqs.com/), which also enables text-based interactions in surveys. SMARTRIQS is also set up to allow for economic games (e.g., the dictator and trust game) between participants. 

Additionally, ChatPlat (https://chatplat.com/#) creates the code for you to include participants’ text-chats within the survey and stores the text-chat data in a user-friendly form. Their program offers functionalities for text-chat that SurvConf currently does not (e.g., providing emoticons).  

Do you have any resources that you recommend for analyzing data from SurvConf interactions? 

ZoomGroupStats (http://zoomgroupstats.org), an R-package created by Andrew Knight, provides a variety of analyses for video, audio, and text interactions, including sentiment and conversation analyses.