By Bella Ratmelia, Senior Librarian, Research & Data Services
When you share Qualtrics surveys on social media or other public platforms, you're opening the door for more responses from your target participants, but also to unwanted spam bot. These automated programs can flood your surveys with fake or nonsensical responses, skewing your data, wasting resources (such as your time, as you will have to spend extra time and energy to sift out the bot responses!), and compromising the integrity of your research.
The good news for Qualtrics users is that the platform offers security features to combat these digital invaders. Even with Qualtrics' powerful built-in security features, the evolving sophistication of bots means a multi-layered 'defense system' is crucial to be able to reduce the impact of bots and collect the high-quality data you need.
SMU subscribes to Qualtrics, which you can access here: https://smusg.au1.qualtrics.com/homepage/ui
Strategy #1: Utilize the Bot Detection feature
Qualtrics comes with built-in Bot Detection feature that utilizes CAPTCHA features. CAPTCHA stands for Completely Automated Public Turing test to tell Computers and Humans Apart. Qualtrics specifically uses Google's reCAPTCHA that's less intrusive and works in the background. To enable this feature, follow these steps:
- Go to Survey options.
- Go to Security.
- Enable Bot Detection.
- (Optional) Publish the survey when you're ready.
Remember to turn on this feature before you start collecting your data!
Once this feature is enabled, a new field called "Q_RecaptchaScore" will be added to your survey's embedded data. This field will have a value between 0 to 1, with 0 closer to bot, and 1 closer to humans. This field can be used in reporting or during data cleaning to indicate whether a response is more likely a bot or a human.
Additional features you can turn on under "Security" tab includes:
- Prevent Indexing - Block search engines from including your survey in their search results, which should help to prevent unwanted respondents from accessing the survey.
- Prevent Multiple Submissions - This setting uses browser cookies (and sometimes IP address) to determine whether that user has already submitted a response, helping to prevent multiple submissions from the same user. You may also choose to redirect them to the end of the survey straightaway, or simply have their responses flagged as potential duplicate.
Strategy #2: Add a CAPTCHA verification question (or two)
Bot Detection as with Strategy #1 provides passive protection and data for later analysis. In a way, the bots can still submit a response to your survey. If you want a more active defense that stops the suspected bots from taking the survey and get their response recorded, you can place a Captcha question at the beginning of your survey.
While effective at stopping bots, keep in mind that explicit CAPTCHA challenges can add a minor step for legitimate users, so consider your audience and the potential impact on completion rates.
This is how it will look like in your survey:
Strategy #3: "Attention Check" Questions
Beyond automated detection, strategically crafted questions can serve as useful bot detectors. Qualtrics' flexible question types and logic make these easy to implement!
Some examples:
- Attention Checks (Instructional Manipulation Checks): These verify if respondents are truly paying attention and a human. Use a question that gives specific instruction for the answer.
- Complicated Question Types: Qualtrics offers advanced question types like "Drag and Drop," "Rank Order," or "Matrix Table" questions that can be harder for simple bots to navigate and answer correctly.
With these questions, you can set a condition (e.g. through branching / skip) that if the incorrect answer is chosen, the respondent is either terminated, or have their responses flagged instead. Either way, you will have additional flags that you can use to filter out low quality responses.
Filtering for bot or spam responses post-collection
Qualtrics provides "Response Quality" features, sometimes part of "ExpertReview," which automatically check for potential issues like "speeders" and "possible bots." You can see a snapshot of this under "Data & Analysis" tab, at the right side of the page.
When clicked, a popup will appear with a report and flags if Qualtrics spot any suspicious responses.
Some potential issues that may get flagged include:
- Speeders: Qualtrics flags responses that are completed in an unusually short amount of time, e.g. in 10 seconds for a 50-questions survey.
- Bot Detection: This leverages the Q_RecaptchaScore to identify likely bot responses. Note that this may not apply for spam responses from click farms manned by humans (or simply inattentive respondents)
- Duplicate Detection (RelevantID): Qualtrics also offers Q_RelevantIDDuplicate and Q_RelevantIDFraudScore fields, which assess respondent metadata to determine the likelihood of duplicate or fraudulent submissions. These can be enabled in Survey Options > Security
Conclusion
While these tips primarily target bots, they can also help identify low-quality responses from human "click farms" or inattentive participants by checking for consistency, attention, and thoughtful input.
By being proactive and leveraging the many features available within Qualtrics, you can significantly reduce spam bot responses, ensuring your survey data is clean, reliable, and truly representative of your target audience.
As always, if you have questions or need help with Qualtrics, please do not hesitate to contact me at bellar@smu.edu.sg
Happy surveying!
