Qualtrics Tips and Tricks: Randomisation for Question Order Effects

By Bella Ratmelia, Senior Librarian, Research & Data Services

Question order effects occur when the wording or ideas provoked by a survey question linger in respondents' minds and influence their responses to subsequent questions. Multiple studies, including one study by Siegelman (1981), have shown that the sequence of survey questions influences how people respond. When question order skews data, it creates response bias—a systematic error rooted in how surveys are structured.

Common examples of this bias include:

• Priming effect is a cognitive bias where exposure to one stimulus (the "prime") influences the response to a subsequent stimulus without conscious guidance or intention. I.e. the initial questions "activate" certain concepts, memories, or emotions in the respondent's mind, making them more accessible and likely to influence later answers.
  • Example: if you ask questions about personal finance difficulties, it might make respondents more critical when evaluating incumbent politicians
• Anchoring is a bias where individuals rely too heavily on an initial piece of information (the "anchor") when making decisions. In surveys, this anchor is often a number, or a specific reference point presented in an earlier question, which then influences the estimation or judgment in a later question.
  • Example: Consider a rating scale from 1-10. If the first question is "Rate the President's performance on the economy (1=Terrible, 10=Excellent)" and the President is very popular, a respondent might give a rating of 8. If the next question is "Rate Congress' performance," the high rating of 8 might anchor their subsequent ratings, making them less likely to use the lower end of the scale for Congress.
• Assimilation-Contrast effects: This effect describes how a respondent's judgment on a question can be pulled closer to (assimilation) or pushed further away from (contrast) the context of a preceding question.
  • Example: Rating your satisfaction with local services after questions about national problems might inflate local satisfaction scores. The poor state of "national problems" provides a negative standard of comparison, making "local services" look much better in contrast.

SMU provides university-wide access to Qualtrics, making it easy for faculty, staff, and students to implement sophisticated survey features like question randomisation (among others that will be covered in the next articles) without additional software or complex coding. You can access Qualtrics through your SMU credentials at https://smusg.au1.qualtrics.com/

Basic Question Randomisation Setup in Qualtrics

Here's the step-by-step process:

1. Create or select a block containing all questions you want to randomize
2. Click on the block to select it (you'll see a blue outline)
3. Navigate to "Block Options" in the left panel
4. Select "Question Randomization" from the dropdown

5. Choose "Randomise the order of all questions"

You can see a preview example here for your reference: Preview Example

Tracking Randomisation in Your Data

You can view the exact order in which questions were displayed to each respondent by enabling Export Randomised Viewing Order before exporting your data. You'll find this option by going to "Data & Analysis" main tab at the top, and then click on the "Export & Import" button on the right side. Once the export window pops up, click on "More Options" to show more available options.

The display order appears as a new column in your data file, with multiple questions separated by a vertical bar "|" in order of their appearance. This is crucial for your analysis stage.

Best Practices and common pitfalls to avoid:

• Always preview your survey multiple times to see different randomisation patterns before launch.
• Test thoroughly: recruit a few test respondents to ensure randomisation works as expected. Very important to do before you launch your survey.
• Consider partial randomisation: Sometimes you want certain questions (like demographics) to always appear in the same position.
• Use block-level randomisation for related questions. Instead of randomising your entire survey, group related questions into logical blocks and randomise only within each block. This preserves the natural flow respondents expect while still reducing order effects. You can also set these blocks to be randomised as well.
• Use the Advanced Randomisation option for more sophisticated randomisation such as having a fixed number of questions to be displayed first, excluding certain questions from randomisation, or making sure that the questions are evenly shown during the randomisation.

Good luck with your survey research! If you need assistance or have questions about Qualtrics, contact us at library@smu.edu.sg

References:

Oldendick, R. (2008). Question order effects. In Encyclopedia of survey research methods (Vol. 0, pp. 664-665). Sage Publications, Inc., https://doi.org/10.4135/9781412963947.n428

Sigelman, L. (1981). Question-Order Effects on Presidential Popularity. The Public Opinion Quarterly, 45(2), 199--207. http://www.jstor.org/stable/2748791