Helping Give Away Psychological Science/Standard Operating Procedures/ Qualtrics Assessment Guide

= Qualtrics guide to making assessments = All questionnaires in HGAPS’ Assessment Center are presented using Qualtrics, an online program for making and administering surveys/questionnaires/assessments. All UNC students and employees have free access to Qualtrics. This SOP reviews research and evidence-based assessment related to Qualtrics' information and policies. For a general introduction to Qualtrics, see the quick guide put out by the company; for a thorough discussion of the Qualtrics software, or as a reference, see the official training handbook.

Question Types
Qualtrics offers a fair amount of flexibility in question format. The type of question format used for a certain question can be chosen by selecting the question and then selecting the green dropdown under “Change Question Type,” found in the right-hand menu. Text in quotes here is taken from the Qualtrics question type overview.


 * Static Content: Because all elements in a Qualtrics survey block are designated as “Questions,” if you want to put descriptive text or graphics into the survey, you create a new question and select the type as static content.
 * Descriptive Text: “used to add an introduction or instructions to your survey, or wherever you need to display content to the respondent, without asking for feedback”
 * example: survey instructions
 * Graphic: “used to display an image to your respondents without asking for feedback”
 * example: displaying stimuli
 * File: used to display a link to a certain file type; accessed through setting a “Static Content” question to type “File”
 * not commonly used
 * Standard Questions: In almost all cases, your survey can be built using primarily these basic question types.
 * Multiple Choice: “can be displayed as a traditional list of choices or as a dropdown menu, select box, etc.”
 * example: race or ethnicity demographics
 * Matrix Table: “used to collect multiple pieces of information in one question”
 * example: a survey that uses same Likert-type scale to rate all statements
 * Text Entry: “used to gather open-ended feedback from respondents”
 * example: asking for participant feedback on the user-experience of the questionnaire
 * Slider: used to “let respondents indicate their level of preference with a draggable bar rather than a traditional button or checkbox”
 * example: participant rates agreement with the statement on a continuous scale; like the level of perceived pain
 * Rank Order: “used to determine each respondent’s order of preference for a list of items”
 * not commonly used
 * Side by Side: used to “ask multiple questions in one condensed table”
 * typically, not used; can appear too compact/dense
 * Specialty Questions: These questions are used in special circumstances when advanced functionality is needed. Your best resource for this is the Qualtrics documentation. Here is the link to the “Question Types Overview”; from there you can navigate to information on individual “Specialty” questions.
 * Advanced Questions: These questions collect metadata; most of them are invisible to the participant. See official Qualtrics documentation for more information. You can use the “Question Types Overview” link provided above to navigate to the documentation on the “Advanced” question types.

Question Options
In addition to the “Question Type,” each question has several other features that can be customized. These are available underneath the “Question Type” dropdown in the menu on the right side of the page. Below are general options available for nearly all types; you can find information about the special options available only for specific types in the appendix entry on Qualtrics Options.


 * Choices
 * Number: the total number of choices available
 * Edit Multiple: brings up a window with each choice’s text on one line; useful for easier editing of many options at once
 * Automatic Choices: check this box to gain access to a library of premade responses, like those describing satisfaction, agreement, etc.; typically better to simply write in your responses to ensure they match the survey you are using
 * Validation Options: validation features to require or encourage participants to answer survey questions
 * Force Response: requires the participant to respond before continuing to the next page; useful for mandatory questions
 * example: obtaining informed consent
 * Request Response: prompts the participant to respond before proceeding to the next page; presents as “There are unanswered questions on this page. Would you like to continue?” pop up
 * example: general questionnaire items (by ethical standards participants can choose to decline to answer)
 * Unchecked: a participant may choose to leave this question unanswered
 * rarely preferable to “Forced Response” or “Request Response”
 * Validation Type:
 * None: any combination of question responses is considered valid
 * Custom Validation: opens the editor for determining what counts as a valid response, by determining what combination of question responses constitutes a valid response

Response Coding
Many questionnaires we use are provided to us along with information on how to score participant responses. For questionnaires that use scale ratings for participants, the values of each choice/option along the scale may vary. (For example, a scale including “Strongly Disagree” to “Strongly Agree” might contain values indexed from 1 to 5, or from 0 to 4.) Our choice values in Qualtrics must match those provided by the questionnaire documentation. Otherwise, printed out results/scoring points will be off and could be the difference between the participant’s score falling in a clinical range or not leading the clinician to making clinical decisions based on faulty scores.

To avoid this problem, make sure values are coded correctly before you publish the survey. To change choice-value coding:


 * Right-click on the question. It will become highlighted in light blue.
 * Select “Recode Values…”
 * Check the “Recode Values” box. Yellow fields, numbered by default from 1 upwards, will appear beside the question choices.
 * Input the correct values. Click in each yellow box, delete the default value, and insert the correct value (according to questionnaire scoring standards).
 * Close the pane. Click “Close” in the bottom right of the editing pane. Your changes will save automatically.

Page Breaks
Questions are all displayed on one page unless page breaks are added. To add a page break:


 * Select the question you want to place the page break after.
 * Click “Add Page Break.” You can find this option in the “Actions” submenu at the bottom of the right-hand menu.

You will want to have each questionnaire appear on a separate page. Add more page breaks as necessary to enhance clarity.

Conditional Questions
It only makes sense to ask certain questions based on the answer provided to a previous question. For example, if a participant answers “Yes” to “Have you experienced racial discrimination?” we would show them a series of related questions. If a participant answers “No” to that question, it is illogical to show them the related questions. You can use Display Logic to determine under what circumstances a question is shown:


 * 1) Make sure the question you wish to apply Display Logic is on a separate page from the question you wish to evaluate.
 * 2) If they are currently on the same page, insert a page break between them or move the Display Logic question to a different page.
 * 3) If they are placed on the same page, Display Logic will not work correctly. The question the Display Logic is applied to will simply never appear.
 * 4) Right-click on the question. It will become highlighted in light blue.
 * 5) Select “Add Display Logic…”
 * 6) Fill out the condition that must be met to display the question. The first field refers to the type of information that will be checked (“Question” answer, “GeoIP Location,” etc.), and the second field varies according to the option selected in the first field.
 * 7) When you select “Question,” the second field will be the question you wish to evaluate. The third field will be the answer choice you wish to evaluate, and the fourth field will be chosen from “Is Selected,” “Is Not Selected,” “Is Displayed,” or “Is Not Displayed.” (The last two options are used much less often than the first two.)
 * 8) Add additional conditions if necessary. Press the green “+” to add a new logical statement. This one will start with the choice of “And” or “Or.” Selecting “And” means this requirement and the previous requirement must both be met; selecting “Or” means at least one of the multiple requirements must be met.
 * 9) Press “Save.”

Skip Logic
The Skip Logic feature can allow you to alter survey flow according to a certain question’s answer. You can cause a certain question’s response to end the survey or skip to a later part of the survey. To use Skip Logic:


 * 1) Right-click on the question. It will become highlighted in light blue.
 * 2) Select “Add Skip Logic…”
 * 3) Fill out the condition you want this question to meet for Skip Logic to be activated. You will be able to select one of the answer choices available in this question, and then decide whether Skip Logic is applied when this answer “Is Selected,” “Is Not Selected,” “Is Displayed,” or “Is Not Displayed.”
 * 4) Decide where the participant should be redirected. When the Skip Logic is activated, one or more questions will be skipped (not shown to the participant). You can choose the following options for the related “Skip To:” field:
 * 5) Another question in this block: skip all questions after this one up until the question selected in this field
 * 6) End of Block: skip all questions after this one in the same block; participant starts answering again at beginning of next block
 * 7) * can be used to serve similar purposes to Display Logic (described above)
 * 8) End of Survey: skip all other questions; the survey is complete
 * 9) * can be used to disqualify a participant early on who answers a question that precludes them from participating in the rest of the survey
 * 10) Note: Only questions shown on separate pages from the question inducing Skip Logic can be skipped. That is if Question A and Question B are shown on the same page, and A is set to skip B if the answer to A is given as “100,” nothing will happen. Question B will not disappear whenever A is answered as “100”. For this to work properly, B must be shown on a different page from A.

Survey Question Design
There are many ways the same questionnaire can be built on Qualtrics. One major choice is whether to group similar questionnaire items, via a question of type “Matrix Table” or “Side by Side,” or to enter each questionnaire item individually, via a question of type “Multiple Choice,” etc. The differences concern:


 * Aesthetic: grouped questionnaire items often appear closer together than individualized questionnaire items and are more likely to be perceived as a cohesive unit
 * Functionality: you cannot use Skip or Display Logic to navigate amongst questionnaire items displayed in the same group as one question (e.g. you can’t use Item 1A’s answer to determine if Item 1B shows) since they are all one question rather than individual questions within the same block
 * Data output: questions with multiple “parts” (i.e. grouped questionnaire items) output their data as coming from the same question with subparts (i.e. Q1_1, Q1_2, Q1_3) while questionnaire items entered as individual questions appear as coming from different questions (i.e. Q1, Q2, Q3)

Considering these differences, it is preferable to group items coming from the same questionnaire into a single “Matrix Table” or “Side by Side” question. If the questionnaire is extremely lengthy, you may want to group subscales rather than the entire survey. Under ordinary circumstances, do not group items from different questionnaires into single questions.

Also keep in mind that “Slider” questions, which can be used to create visual analog scales, cannot be combined with other sub-question types in a “Matrix Table” or “Side by Side” question. The only way to use a continuous rating scale like this is to place it in its own question. However, the “Slider” question permits multiple sub-questions (“Choices”) of the same type, so you can present multiple continuous rating questionnaire items within the same Qualtrics question.

Repeat Headers
Sometimes a questionnaire will be so long that the anchor labels (i.e. the meaning of “1,” “2,” “3,” etc.) will be hidden as the participant scrolls past them down the page. If this is the case for your questionnaire, use a “Side by Side” question type to present the questionnaire items. As you will see in the appendix entry on Qualtrics Options, this question type has a parameter called “Repeat Headers.” This allows you to determine how many time the anchor labels (headers) are shown: not at all; once, at the top; twice, at the top and middle; twice, at the top and bottom; three times, at the top, middle and bottom; or before every questionnaire item.

Survey Flow
The interface for determining the progression of a Qualtrics questionnaire is the Survey Flow view. By default, different questionnaire blocks appear in the order they are arranged in the general view. With the Survey Flow view, you have more power over how and when your various questionnaire blocks appear. (Note that the only way to organize individual questions within the blocks is from the general view. Survey Flow view is only for managing the order of whole blocks.) Access Survey Flow view by clicking “Survey Flow” along the top menu when you are designing a survey.

Survey Flow supplies an overview of how your survey will be presented to the participant, moving from beginning to end, top to bottom. Each item along this progression is called an Element. Survey Flow is arranged such that each Element coming off the main flow (shown as a grey “backbone” along the left side, with arrows coming from it) will always occur. You can include “Branch” and “Randomizer” Elements to make some blocks show only conditionally. In summary, the different Element types and their primary options are:


 * Block: used to present a question block that you created in the general view
 * Branch: used similarly to Display Logic to create large-scale conditional display; when the conditions of this Element are met, the survey flows down to its nested Elements
 * conditions are set similarly to conditions in Display Logic
 * the first field in all conditional statements beyond the first will have the familiar “And” and “Or” options, but also the “Move to a new Logic Set” option; this allows you to create multiple overarching Logic Sets of conditions, which may have an “And” or “Or” relationship amongst themselves
 * Randomizer: used to randomly present Elements nested underneath
 * Randomly present X of the following elements: set value anywhere from 0 to the total number of nested Elements; allows you to randomly display only a few of the nested Elements or all of them
 * Evenly Present Elements: when checked, Elements will be presented evenly (i.e. randomization will result in an even split of all presentations)
 * Conjoint: used to join back together various diverging flows, created by various “Branch” or “Randomizer” Elements; after the termination of previous Elements, all flows will continue to the element immediately after the Conjoint
 * End of Survey: used to end the survey; can be customized via “Customize → Override Survey Options”
 * Custom end of survey message allows you to insert a different survey message than the one Qualtrics displays by default
 * draft a custom message by using “Select a message → My Library → New Message…”
 * Embedded Data, Web Service, and Authenticator are more advanced Element types; see the full Qualtrics documentation for more information

Randomizing Questionnaire Content
For some studies, there may be a good reason to administer questionnaires in a fixed order across participants (i.e., when we expect some questionnaires to influence responses to other questionnaires and want to put the more influential questionnaires last). However, for other studies, it may make sense to randomize the order of questionnaires, while keeping the set of questions that make up each questionnaire together (e.g. if we are trying to control for general order effects). You would use “Question Randomization” to randomize the order of questions within a questionnaire; typically, you won’t want to do this when using a previously validated questionnaire, because question order will affect the reliability and validity of that measure. You would use Randomizer Elements under Survey Flow to present the questionnaires themselves in random order.


 * To randomize the questions within a questionnaire or questionnaire subsection:
 * Be sure to group all the questions you wish to randomize into the same block.
 * At the top of this block, select “Block Options → Question Randomization.”
 * Choose “Randomize the order of all questions.”
 * Also, keep in mind that question randomization overrides and ignores page breaks, skip logic, and rules about the number of questions per page. Strategically separate the questions into blocks to avoid this becoming an issue.
 * To randomize the questionnaires within an entire survey:
 * Go to Survey Flow view and add a Randomizer Element.
 * Add elements to display each block that contains a whole questionnaire.
 * If one single questionnaire is broken up into multiple subunits, you still want to display those subunit blocks together, rather than interspersed with the other blocks holding whole questionnaires.
 * To present questionnaires divided into blocks like this, add a new Randomizer Element under the initially added overarching Randomizer Element.
 * Within this second “Randomizer” element, add blocks to display the subunits of the broken-up questionnaire.

Measuring Timing
A useful feature for recording data on the way participants responded to the survey is the hidden “Timing” question. To gain access to this feature:


 * 1) Add a question of type “Advanced → Timing” to a page.
 * 2) This question is invisible to the participant and does not require any action on their part, but records four measures regarding their activity on that page:
 * 3) First Click: seconds between page load and first-time participant clicks something
 * 4) Last Click: seconds between page load and last time participant clicks something (other than “Next” button)
 * 5) Page Submit: seconds between page load and clicking the “Next” button
 * 6) Click Count: total number of clicks on the page

* Note that these measures are by page, not by block or question.


 * 1) The “Timing” question also has a few other useful features.
 * 2) You can set “Seconds to disable submit,” which hides the “Next” button for a certain amount of time.
 * 3) This may be useful for small numbers of questions in which you wish to delay submission,
 * 4) In general, it may be easier to use JavaScript to do this (see Presenting stimuli → Code 1 below).
 * 5) You can also set “Seconds to auto-advance” the page, which can be useful in controlling study pacing.
 * 6) The last feature is “Show Timer” which shows a timer that counts up or down (your choice) to show participants how long they are spending on the page.
 * 7) The total survey duration is measured automatically by Qualtrics and is included in the output data.
 * 8) All “Timing” question data is exported along with participant data from the rest of the survey questions.

Proofreading and Piloting the Survey
All surveys must be free of typing and grammatical errors, which detract from our image of professionalism and can negatively affect assessment results. You should also pilot the assessment package on various devices: tablets, smartphones, common computer browsers (e.g. Chrome, Internet Explorer, Firefox, Safari). Ask other lab members to take the survey, as well, to make sure content shows up correctly for other people and to estimate survey duration. They may also notice typos and grammatical errors that you had not noticed.


 * 1) If you’d like to display duration time to pilot testers so that they can report the time taken on the survey to you, you can include a block towards the end that reveals this information:
 * 2) Place a page break before the question
 * 3) Set the question type as “Descriptive Text”
 * 4) Put the following as the text: “The survey took ${e://Field/Q_TotalDuration} seconds.” Be sure to delete this question/block before releasing the survey to actual participants.

Qualtrics Error-Checking

 * 1) Before continuing with the task, check the Qualtrics survey data output and feasibility.
 * 2) Complete the entire Qualtrics questionnaire yourself. Answer the questions seriously, as if you were a real participant. Time yourself to estimate the questionnaire duration.
 * 3) Have several others complete the entire Qualtrics questionnaire. Ask them to answer the questions seriously, as if they were real participants. Ask them to time themselves to estimate questionnaire duration, or have Qualtrics output their time.
 * 4) Check through the Qualtrics questionnaires to make sure all text matches original questionnaires. If possible, have other members do this as well.
 * 5) Double-check that question response values in Qualtrics match original questionnaire answer choice values. (For example, the default value for the first answer choice in Qualtrics is 1, but for some questionnaires, this value should be 0.) Adjust response values in Qualtrics as needed to match questionnaire values and avoid recoding during data analysis.
 * 6) Make sure that you and at least one other person in the lab have taken the entirety of the last version of your Qualtrics questionnaires before collecting any real data.
 * 7) Have someone else that is qualified to evaluate your work and check your raw questionnaires for errors and examine the Qualtrics survey flow.

= Permissions = Only the owner of a Qualtrics survey can assign permissions to other users, via the “Share Project” option in the right dropdown list beside the project name. Because of this, it is possible that once you graduate or otherwise leave HGAPS, new personnel will be unable to share the project (i.e. give editing rights) to new personnel. There are a few options to avoid or correct this issue:


 * 1) Have Eric create the survey and share it with you. This way, Eric will be the owner and can always assign permissions to new personnel.
 * 2) Create a copy of the survey. Create a clone of the survey, so that you are now the owner of it.
 * 3) Find the survey. View your list of shared surveys in the main “Shared with Me” window.
 * 4) Make a copy. In the far-right dropdown menu, select the “Copy Project.”
 * 5) Transfer copies of responses. Enter the original project. Navigate to “Data & Analysis → Export & Import.” Choose “Export Data… → Export Data with Legacy Format.” Now, enter your copied project. Navigate to “Data & Analysis → Export & Import.” Choose “Import Data…” and upload a copy of the responses you just created.

= Appendix: Qualtrics Options =