Automated test runs (ATRs) in modern surveys

An automated test run (ATR) is a system job that generates test responses in bulk, allowing you to test your survey's logic, functionality, and reliability without having to manually click through the survey.

Using ATRs, you can:

• Test various survey paths more efficiently.

  The test responses simulate real participants' profile variable values and responses, so you don't have to retake the survey in preview mode dozens of times.

• Anticipate potential problems with your survey logic and flow.

  When you view your test data in reporting, look for places where the number of responses decreases unexpectedly, or questions where the data seems skewed. They may indicate issues with the survey structure that will impede real participants later.

• See what your reporting data structure will look like.

  Does your report look how you expect it to look? Can you segment data and analyze subgroups the way you want to? The ATR may reveal that you need to adjust your survey structure to collect the data you want.

Note:

• This feature is available in modern surveys only.
• While ATRs are useful for validating survey logic and reporting data structure, you should still manually preview and test your survey to spot any errors and possible changes you want to make.

Create an ATR

1. Go to the Surveys app and open a modern survey.
2. In the Survey Builder, on the Build toolbar, click Automated Test Run.
   

3. Click New Test Run.
4. Type a name for the ATR (100 characters maximum).
5. Type the number of test runs to complete, or use the slider to select a number.
6. Select a locale.
7. Optional: Turn on the Custom Parameters toggle.

   This setting allows you to generate test responses based on the profile variables referenced in your survey. For example, if you select Gender as the profile variable and select Male and Female, the ATR will generate test responses that simulate both types of responses.

   If you are testing the path for nonmembers, custom parameters are not needed.

   1. Select the profile variable.
   2. Select the profile variable values.
8. Click Create Test.
   The ATR is listed in the Automated Test Run table with a status of Created.
9. To return to the Survey Builder, click the close button in the top right corner to close the Automated Test Run dialog.
   

Run an ATR

Note: Only one ATR can be run at a time.

1. In the Survey Builder, on the Build toolbar, click Automated Test Run.
   
2. Beside an ATR, click Run Test.
   The ATR's status changes from Created to Pending to Running. Depending on the number of responses requested and the size and complexity of the survey, the ATR may take a while. When the test finishes running, the status updates to Completed.

View ATR data

ATR data appears in standard reports and modern reports. It does not appear on the Monitor tab.

1. Click the Report tab.
2. View ATR data in standard reports:
   1. Click the Standard Reporting tile.
   2. In the Report table, click Standard Report.
   3. On the report toolbar, click Data and select Test Data.
   4. On the report toolbar, click Data and select the statuses you want to display:
      • Complete
      • Incomplete
      • Disqualified
      • Over Quota
3. View ATR data in modern reports:
   1. Click the Modern Reporting tile.
   2. Open a modern report.
   3. In the toolbar, click Filters.
   4. Set the first filter condition to This Survey Dataset Type Is Test Data.
   5. In the second filter condition (This Survey Response Status Is), select the statuses you want to display:
      • Over Quota
      • Disqualified
      • In Progress
      • Completed
   6. Click Apply & Exit.

What ATR data looks like

The total number of completed test responses in the report may not equal the sum total of your ATRs. This is because:

• Manual test responses (for example, previewing and testing the survey manually) are included.
• If you deleted test data from past ATRs at any point, the total number of test responses may be lower than expected.
• The ATR simulates incomplete, disqualified, and over quota responses as well, so the number of completions may be lower than expected.

In normal question types with defined answer options, the ATR randomly chooses answers. Therefore, the test data approximates what real data may look like.

In Highlighter questions, random parts of the image are highlighted so that the ATR can proceed to the end of the survey.

In Open Ends, random text is generated so that the question can be marked as complete.

In Image Upload, Appointment Booking, and Video Feedback questions, nothing is submitted or uploaded.

In Usability Task questions, the ATR marks the question as a successful response in order to proceed past it.

Delete an ATR

1. In the Survey Builder, on the Build toolbar, click Automated Test Run.
   
2. Beside an ATR, click the options menu and select Delete.
   
   The ATR is no longer listed in the Automated Test Run dialog. However, any test data generated through the ATR still remains.

Delete all test data

Delete All Test Data deletes all test data generated through ATRs and manual testing.

1. In the Survey Builder, on the Build toolbar, click Automated Test Run.
   
2. Click Delete All Test Data.
3. Click Delete to confirm.

Note: Deleting test data does not reset quotas. You must reset the test quota counts manually.