Important: To avoid additional charges from oversampling, we recommend adding quotas for all surveys with purchased sample.
This document describes the process for adding quotas to a survey using the XML Editor. Using the Survey Editor? Learn more about the Quota Element.
Quotas are often used by researchers to obtain a sample of participants that is statistically significant to the population that they are analyzing. They are also used to track and monitor the number of qualified completes in a survey.
The quotas for any given project can be viewed in the "Quotas" tab of the Response Summary.
If quotas are used to target a sample of a specific size, then quota limits can be set to prevent an excess number of qualified completes for any given participant group. On the other hand, if quotas are only used to monitor the number of completes for a given category, then quota limits can be set to infinite.
When a quota is called for a participant, that participant must fall into a quota bucket to continue the survey. If a participant does not qualify for a quota bucket, they are terminated and a special marker named "NQ" is set.
1: The Quota System
Quotas are created using Excel spreadsheets. For example, a simple gender quota may look like the following:
| A | B | |
|---|---|---|
| 1 | # = Gender | |
| 2 | Male | 500 |
| 3 | Female | 500 |
Before you can create a quota table, however, you must define all of the quota markers that are available.
1.1: Defining the Quota Markers
All quota markers should be specified in a sheet named "defines". The syntax for declaring quota markers is below:
| A | B | C | |
|---|---|---|---|
| 1 | Name | CONDITION | ALT. NAME |
For example:
| A | B | C | |
|---|---|---|---|
| 1 | qual | plus | |
| 2 | Male | Q1.r1 | |
| 3 | Female | Q1.r2 | |
| 4 | Age_1 | Q2.check('18-24') | 18-24 |
| 5 | Age_2 | Q2.check('25-34') | 25-34 |
| 6 | Age_3 | Q2.check('35-44') | 35-44 |
The marker names are declared in column A. The condition in which a participant should qualify for a given marker is declared in column B. Optionally, you can specify an alternative name for any marker to appear in the reports in column C.
In addition to the condition logic that you are used to, column B may also be set to "plus" for any marker. A marker with a "plus" condition is a created marker that any participant can qualify for. Plus markers should be used when you need to synchronize randomly assigned markers across multiple tables; otherwise, a condition of 1 or True is standard practice. See the section below on Cross-Table Markers for more information.
1.2: Creating the Quota Tables
Once you have defined all of the possible markers to be used in the survey in a sheet named "defines", you can create additional sheets with any naming convention to add your quota tables to. Quotas may consist of many sheets containing many quota tables.
Note: Multiple quota tables in a single sheet should be separated by a blank row.
The syntax for a simple quota table is below:
| A | B | |
|---|---|---|
| 1 | # = TABLE NAME | |
| 2 | MARKER NAME | LIMIT |
For example:
| A | B | |
|---|---|---|
| 1 | # = Qualified | |
| 2 | qual | inf |
| 3 | ||
| 4 | # = Gender | |
| 5 | Male | 500 |
| 6 | Female | 500 |
| 7 | ||
| 8 | # = Age | |
| 9 | Age_1 | 350 |
| 10 | Age_2 | 350 |
| 11 | Age_3 | 300 |
Tip: A marker's LIMIT can be set to an integer, range, percentage, or "inf" for no upper limit.
Quotas can be nested to further refine the limit in place for sample groups. There are several different ways to nest quotas and the syntax for each is shown below:
| A | B | C | |
|---|---|---|---|
| 1 | # MARKER NAME = TABLE NAME | ||
| 2 | MARKER NAME | LIMIT | |
| 3 | MARKER NAME | LIMIT | |
| 4 | |||
| 5 | # = TABLE NAME | MARKER NAME | MARKER NAME |
| 6 | MARKER NAME | LIMIT | LIMIT |
| 7 | MARKER NAME | LIMIT | LIMIT |
| 8 | |||
| 9 | # = TABLE NAME | # | |
| 10 | MARKER NAME | MARKER NAME | LIMIT |
| 11 | MARKER NAME | LIMIT | |
| 12 | MARKER NAME | MARKER NAME | LIMIT |
| 13 | MARKER NAME | LIMIT |
The emboldened MARKER NAME cells above represent markers that a participant must also qualify for in addition to the regular markers listed in column A.
For example:
| A | B | C | |
|---|---|---|---|
| 1 | # Male,Female = Age | ||
| 2 | Age_1 | 350 | |
| 3 | Age_2 | 350 | |
| 4 | Age_3 | 300 | |
| 5 | |||
| 6 | # = Age x Gender | Male | Female |
| 7 | Age_1 | 175 | 175 |
| 8 | Age_2 | 175 | 175 |
| 9 | Age_3 | 150 | 150 |
| 10 | |||
| 11 | # = Gender x Age | # | |
| 12 | Male | Age_1 | 175 |
| 13 | Age_2 | 175 | |
| 14 | Age_3 | 150 | |
| 15 | Female | Age_1 | 175 |
| 16 | Age_2 | 175 | |
| 17 | Age_3 | 150 |
The "Age" table above will only be called for participants who qualify as "Male" and "Female". Since this is not possible, this "Age" table will never be called.
Note: This "Age" table above is considered a conditional quota table because it is only called if a participant is eligible for all of the markers listed between "#" and "=". A single marker may be specified or multiple markers separated by a comma.
The "Age x Gender" table above is set up to evenly distribute the age groups across each gender. For instance, there can only be 175 qualified males within the age range 18 - 24.
The "Gender x Age" table above is set up slightly different than the "Age x Gender" table but the limits are exactly the same.
You may also specify limits as percentages. The syntax for quota tables that use percentages is below:
| A | B | |
|---|---|---|
| 1 | # TOTAL = TABLE NAME | |
| 2 | MARKER NAME | LIMIT % |
For example:
| A | B | |
|---|---|---|
| 1 | # 1000 = Age | |
| 2 | Age_1 | 35% |
| 3 | Age_2 | 35% |
| 4 | Age_3 | 30% |
In the quota table above, a hard limit of 350 (35% of 1000) is specified for "Age_1" and "Age_2". A hard limit of 300 (30% of 1000) is set for "Age_3".
1.3: Calling the Quotas from the Survey
After you have defined all of the quota markers for the survey and created the necessary quota tables, you can save the Excel workbook to a file named quota.xls and upload it to the project's directory.
Use the following syntax to add a quota call to your survey:
<quota label="LABEL" sheet="SHEETNAME" />
Note: Labels are required for quota elements in surveys with compat of 143 or higher.
For example:
<quota label="quota_age" sheet="age_gender" />
If the quota file is named something other than quota.xls, then you can use the filename attribute to read a different file.
<quota label="quota_age" filename="FILENAME" sheet="SHEETNAME" />
For example:
<quota label="quota_age" filename="my-quota.xls" sheet="age_gender" />
Important: This attribute will make the project incompatible with the user interface.
If you want the participant to go somewhere else in the survey if they are overquota, you can specify overquota="LABEL" in the <quota> element to direct them to an element with label="LABEL". For example:
<quota label="quota_age" overquota="Outtro_Message" sheet="age_gender" />
Just like survey questions, survey quotas will be called as soon as a participant reaches the <quota> element within the survey. For example:
<radio label="Q1">
<title>Are you...</title>
<row label="r1">Male</row>
<row label="r2">Female</row>
</radio>
<suspend/>
<number label="Q2" size="3" verify="range(1, 125)">
<title>Please enter your age below:</title>
</number>
<suspend/>
<term label="term_age" cond="not Q2.check('18-44')">Q2: Age lt 18 or gt 44</term>
<quota label="quota_age" sheet="gender_age" />
<html label="Introduction" where="survey">
Congratulations! You've qualified for this survey.
</html>
If a <quota> element is placed after a question or comment element on the same page, the quota will be called before displaying the element.
2: Examples
2.1: A Simple Quota
In this example, there is a simple quota to limit participants based on their gender and age. The quota specifications are below:
- N = 2000 qualified participants.
- 50/50 split between males & females.
- Split between the following age groups:
- 600 for ages 18-24
- 600 for ages 25-34
- 400 for ages 35-44
- 400 for ages 45-54
The gender and age questions that you will use to create the quota logic are provided below:
<radio label="Q1" optional="0">
<title>Are you...</title>
<row label="r1">Male</row>
<row label="r2">Female</row>
</radio>
<number label="Q2" size="3" verify="range(1,125)">
<title>Please enter your age below:</title>
</number>
<suspend/>
<term label="term_age" cond="not Q2.check('18-54')">Q2: Age lt 18 or gt 54</term>
Given the questions above, the next step is to define the quota markers. In a new Excel workbook, create a sheet named "defines" and enter the following:
| A | B | C | |
| 1 | qual | plus | |
| 2 | Male | Q1.r1 | |
| 3 | Female | Q1.r2 | |
| 4 | Age_1 | Q2.check('18-24') | 18-24 |
| 5 | Age_2 | Q2.check('25-34') | 25-34 |
| 6 | Age_3 | Q2.check('35-44') | 35-44 |
| 7 | Age_4 | Q2.check('45-54') | 45-54 |
With your markers defined, you can move on to creating the quota table named "general" based on the specifications provided:
| A | B | |
| 1 | # = Qualified | |
| 2 | qual | 2000 |
| 3 | ||
| 4 | # 2000 = Gender | |
| 5 | Male | 50% |
| 6 | Female | 50% |
| 7 | ||
| 8 | # = Age | |
| 9 | Age_1 | 600 |
| 10 | Age_2 | 600 |
| 11 | Age_3 | 400 |
| 12 | Age_4 | 400 |
We can save the two sheets above, "defines" and "general", into a workbook named "quota.xls" and upload it to our project. Once it's in place, we can add a <quota> element to our survey and call the quota sheet:
<radio label="Q1" optional="0">
<title>Are you...</title>
<row label="r1">Male</row>
<row label="r2">Female</row>
</radio>
<number label="Q2" size="3" verify="range(1,125)">
<title>Please enter your age below:</title>
</number>
<suspend/>
<term label="term_age" cond="not Q2.check('18-54')">Q2: Age lt 18 or gt 54</term>
<quota label="quota_age" sheet="general" />
Good to go.
2.2: A Nested Quota
Similar to the previous example, we'll create a quota with the following specifications:
- N = 2000 qualified participants
- Even distribution between genders of all age groups
- Age groups: 18-24, 24-34, 35-44, 45-54
Using the following questions, we'll create a nested quota table to achieve an even distribution of genders and ages.
<radio label="Q1" optional="0">
<title>Are you...</title>
<row label="r1">Male</row>
<row label="r2">Female</row>
</radio>
<number label="Q2" size="3" verify="range(1,125)">
<title>Please enter your age below:</title>
</number>
<suspend/>
<term label="term_age" cond="not Q2.check('18-54')">Q2: Age lt 18 or gt 54</term>
The "defines" and "general" sheets with the nested quotas are specified below:
defines
| A | B | C | |
| 1 | qual | plus | |
| 2 | Male | Q1.r1 | |
| 3 | Female | Q1.r2 | |
| 4 | Age_1 | Q2.check('18-24') | 18-24 |
| 5 | Age_2 | Q2.check('25-34') | 25-34 |
| 6 | Age_3 | Q2.check('35-44') | 35-44 |
| 7 | Age_4 | Q2.check('45-54') | 45-54 |
general
| A | B | C | D | E | |
| 1 | # = Qualified | ||||
| 2 | qual | 2000 | |||
| 3 | |||||
| 4 | # 2000 = Gender x Age | Age_1 | Age_2 | Age_3 | Age_4 |
| 5 | Male | 12.5% | 12.5% | 12.5% | 12.5% |
| 6 | Female | 12.5% | 12.5% | 12.5% | 12.5% |
After we save the quota sheets above into a file named "quota.xls" and upload it to our project's directory, we can call the "general" quota sheet from the survey using a <quota> element:
<radio label="Q1" optional="0">
<title>Are you...</title>
<row label="r1">Male</row>
<row label="r2">Female</row>
</radio>
<number label="Q2" size="3" verify="range(1,125)">
<title>Please enter your age below:</title>
</number>
<suspend/>
<term label="term_age" cond="not Q2.check('18-54')">Q2: Age lt 18 or gt 54</term>
<quota label="quota_age" sheet="general" />
The code above produces the following result:
Learn more: Assigning Balanced Concepts in Quotas with Multiple Variables
2.3: Multi-Dimensional Quotas
When working with quotas, the Survey Editor is somewhat limited to the amount of nesting we can create for our quota cells. While the UI quota element can only support 1 level of nesting (adding the rows and columns for our Quota element), creating our quota definition in Excel allows us to add further levels of nesting. In the following example, we will build a nested quota based on three criteria - Age, Gender, and Income.
2.3.1: Setting Up the Quota Definitions
The first step we need to take when creating any Excel quota, is to add our Defines sheet, where we will define our marker names, corresponding logic and Response Summary display text:
| Male | s1.r1 | Male |
| Female | s1.r2 | Female |
| 18-24 | s2.r1 | 18-24 |
| 25-34 | s2.r2 | 25-34 |
| 35-44 | s2.r3 | 35-44 |
| 45-54 | s2.r4 | 45-54 |
| 55+ | s2.r5 | 55 and over |
| lt_15k | s3.r1 | Less than $15 000 |
| 15-24k | s3.r2 | $15 000 to $24 999 |
| 25-34k | s3.r3 | $25 000 to $34 999 |
| 35-49k | s3.r4 | $35 000 to $49 999 |
| 50-74k | s3.r5 | $50 000 to $74 999 |
| 75k+ | s3.r6 | $75 000 and over |
2.3.2: Setting Up the Quota Table
On a seperate sheet you will need to create the quota table will define the nesting and limits. For this example we will use AgeXGenderXIncome:
| #1500 = Age Gender Income |
Once we define our quota table, we will start adding its dimensions one by one. While quota markers can be listed as rows or columns for readability purposes we will want to add each dimension in our quota as a separate column in our table. Let’s start with defining the gender dimension first:
We can define our Male and Female cells underneath the table definition.
| #1500 = Age Gender Income |
| Male |
| Female |
Now, let’s add the second dimension of our quota, based on participant age groups. Usually, we would add the age quota markers as the columns for our table, but since we are building a multi-dimensional quota, we will add them as rows within our second table column. Note that defining an extra dimension for our quota requires us to add a # symbol as a header for the column.
| #1500 = Age Gender Income | # |
| Male | 18-24 |
| 25-34 | |
| 35-44 | |
| 45-54 | |
| 55+ | |
| Female | 18-24 |
| 25-34 | |
| 35-44 | |
| 45-54 | |
| 55+ |
Notice that we now spread out the gender markers, so we can nest the possible age groups inside each gender marker. To add another dimension to represent our income levels, we can add another column with a # header. Because we will be adding several income categories, our age groups will also need to be spread out, so that we fit each income category in each age group. The below example illustrates this set-up for ages 18-24, and 25-34.
| #1500 = Age Gender Income | # | # |
| Male | 18-24 | lt_15k |
| 15-24k | ||
| 25-34k | ||
| 35-49k | ||
| 50-74k | ||
| 75k+ | ||
| 25-34 | lt_15k | |
| 15-24k | ||
| 25-34k | ||
| 35-49k | ||
| 50-74k | ||
| 75k+ |
Finally, what we need to define is the limit for each possible Gender/Age/Income combination is. We can add the limits as a 4th and final column. Note that specifying cell limits does not require # symbols:
| #1500 = Age Gender Income | # | # | |
| Male | 18-24 | lt_15k | 125 |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 75k+ | 125 | ||
| 25-34 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 75k+ | 125 |
2.3.3: Final Result
After аpplying the above method of nesting for our full quota setup, we should have the following as our quota sheet:
| #1500 = Age Gender Income | # | # | |
| Male | 18-24 | lt_15k | 125 |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 25-34 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 35-44 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 45-54 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 55+ | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| Female | 18-24 | lt_15k | 125 |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 25-34 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 35-44 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 45-54 | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 | ||
| 55+ | lt_15k | 125 | |
| 15-24k | 125 | ||
| 25-34k | 125 | ||
| 35-49k | 125 | ||
| 50-74k | 125 |
Which will produce the following quota table within our Response Summary:
2.3.4: Adding Multiple Dimensions
Following the principle of nesting quota dimensions we established, each new dimension that we want to add can be added as a new column defined using the # symbol. When defining the quota markers within our new dimension. In addition to that, the set of markers for our new dimension should be nested for each marker in the prior dimension.
2.4: Lowest-Bucket Fill (Concept Picker)
The quota system is used to present concepts evenly to all participants. In this example, we'll create the necessary quotas to properly assign a participant one out of the three possible concepts. Additionally, we'll create a question named "vConcept" to track which concept each participant was assigned.
Specified below are two sheets: the mandatory "defines" sheet and a "concept" sheet to pick one of the concepts for each participant.
defines
| A | B | |
| 1 | Concept_1 | plus |
| 2 | Concept_2 | plus |
| 3 | Concept_3 | plus |
concept
| A | B | |
| 1 | # = Concept Pick | |
| 2 | Concept_1 | inf |
| 3 | Concept_2 | inf |
| 4 | Concept_3 | inf |
All three of the concept quota markers we declared are "plus" markers, which means that all participants are eligible for each one.
By default, a quota table will assign only one of the markers that a participant qualifies for. If a participant qualifies for more than one marker, the quota system will choose the marker with the lowest percentage of completes.
This means that if the limits are the same (as they are in the example above), then the marker with the lowest number of completes will be chosen.
If the limits are different, however, then the marker with the lowest percentage of completes will be chosen. For example, a marker with 10 out of 100 completes has a lot more work to do to fulfill its requirement than a marker with 4 out of 5 completes, so the marker with 10 completes will be chosen.
If the limits are equal and the quota system is choosing markers based on the lowest count, then the quotas will maintain an even distribution throughout the study.
With the quotas specified above uploaded to our project's directory, we can call the "concept" sheet in our survey and create a question to track which concept marker was assigned:
<quota label="quota_concept" sheet="concept" />
<radio label="vConcept" where="execute">
<title>HIDDEN: Assigned Concept</title>
<exec>
for x in xrange(3):
if hasMarker('Concept_{}'.format(x+1)):
vConcept.val = x
break
</exec>
<row label="r1">Concept 1</row>
<row label="r2">Concept 2</row>
<row label="r3">Concept 3</row>
</radio>
<suspend/>
<pipe label="Concept">
<case label="c1" cond="vConcept.r1">CONCEPT 1</case>
<case label="c2" cond="vConcept.r2">CONCEPT 2</case>
<case label="c3" cond="vConcept.r3">CONCEPT 3</case>
<case label="c99" cond="1">undefined</case>
</pipe>
<html label="Show_Concept">
Now you will be asked about [pipe: Concept].
</html>
The code above produces the following result:
Notice that the concepts are being evenly distributed. If the quota system is assigning the next lowest bucket, then "Concept 1" is up next! If we were to run through this survey with QA codes on, we would see the following:
The hidden question above is populated to reflect the marker/concept we were assigned.
That's it! The proper concept was assigned and displayed to the participant.
The only missing component of this example is a quota to cap the total number of participants entering the survey. See the first example where we added a "qual" marker to set an upper-limit.
2.5: Assigning Multiple Markers
Continuing from the previous example, what if we needed to assign two out of the three possible concepts for each participant to see?
We can assign multiple quota markers by using a special syntax when we define the quota table. The revised "concept" sheet is below:
defines
| A | B | |
| 1 | Concept_1 | plus |
| 2 | Concept_2 | plus |
| 3 | Concept_3 | plus |
concept
| A | B | |
| 1 | # cells:2 = Concept Pick | |
| 2 | Concept_1 | inf |
| 3 | Concept_2 | inf |
| 4 | Concept_3 | inf |
Notice that we added "cells:2" just after the "#" in the "Concept Pick" quota table.
By default, a quota table will choose only one of the qualifying quota markers. Specifically, the one with the lowest count.
If we specify "cells:#" in the quota table, then the quota system will choose # of the qualifying markers. If a participant qualifies for more than # markers, then the first # markers with the lowest count will be chosen. If a participant qualifies for less than # markers, then all of the qualifying markers will be chosen.
Since we are choosing multiple concepts to display, we need to update our hidden question and pipes to account for the multiple marker assignment:
<quota label="quota_concept" sheet="concept" />
<checkbox label="vConcept" exactly="2" shuffle="rows" where="execute">
<title>HIDDEN: Assigned Concept</title>
<exec>
for x in xrange(3):
if hasMarker('Concept_{}'.format(x+1)):
vConcept.rows[x].val = 1
</exec>
<row label="r1">Concept 1</row>
<row label="r2">Concept 2</row>
<row label="r3">Concept 3</row>
</checkbox>
<suspend/>
<exec>ConceptLoop_expanded.order = vConcept.rows.order</exec>
<loop label="ConceptLoop" vars="concept" randomizeChildren="1">
<block label="ConceptStage" >
<html label="Show_Concept_[loopvar: label]">
Now you will be asked about [loopvar: concept].
</html>
</block>
<looprow label="1" cond="hasMarker('Concept_1')"><loopvar name="concept">CONCEPT 1</loopvar></looprow>
<looprow label="2" cond="hasMarker('Concept_2')"><loopvar name="concept">CONCEPT 2</loopvar></looprow>
<looprow label="3" cond="hasMarker('Concept_3')"><loopvar name="concept">CONCEPT 3</loopvar></looprow>
</loop>
The code above produces the following result:
The distribution of concepts is still evenly distributed. With 112 qualified completes, we can see that the quotas are working correctly since the total number of markers distributed shows 224 (e.g., 112 participants * 2 markers each = 224).
If we were to run through the survey with QA codes turned on, we would see the following:
The hidden question is a <checkbox> question to account for the number of concepts to be assigned.
The <loop> element will cycle through and show each concept appropriately.
2.6: Random Selection / Monitoring a Variable
The quota system cannot randomly select quota markers. We can, however, monitor the selections made at a specific question whose answers are randomly populated.
Below is a hidden question whose responses are randomly generated:
<radio label="vRandomPick" where="execute"> <title>HIDDEN: Random Choice</title> <exec> vRandomPick.val = random.choice(vRandomPick.rows).index </exec> <row label="r1">1</row> <row label="r2">2</row> <row label="r3">3</row> <row label="r4">4</row> </radio>
The quota specifications below are set up to monitor the selections made for the question above:
defines
| A | B | |
| 1 | Random_1 | vRandomPick.r1 |
| 2 | Random_2 | vRandomPick.r2 |
| 3 | Random_3 | vRandomPick.r3 |
| 4 | Random_4 | vRandomPick.r4 |
random_monitor
| A | B | |
| 1 | # = Random Monitor | |
| 2 | Random_1 | inf |
| 3 | Random_2 | inf |
| 4 | Random_3 | inf |
| 5 | Random_4 | inf |
We can add a <quota> element to our survey that calls the "random_monitor" sheet above just after the "vRandomPick" question:
<radio label="vRandomPick" where="execute"> <title>HIDDEN: Random Choice</title> <exec> vRandomPick.val = random.choice(vRandomPick.rows).index </exec> <row label="r1">1</row> <row label="r2">2</row> <row label="r3">3</row> <row label="r4">4</row> </radio> <suspend/> <quota label="quota_random" sheet="random_monitor" />
This quota tracks the random selections made at the "vRandomPick" question, so the data table for this question should be exactly the same as the "Random Monitor" table in the "Quotas" tab of the Response Summary.
For example, the data table for this question looks like this:
And the "Random Monitor" quota table looks like this:
The data is populated in exactly the same way. Even though the quota system didn't randomly assign a marker, we are achieving random-ness by using a hidden question. We are monitoring the selections made at this question by calling the <quota> element immediately after the random question.
If we want to, for instance, assign a random concept based on this selection, we can reference the question itself or the marker that is assigned (e.g.,cond="vRandomPick.r2" or cond="hasMarker('/random_monitor/Random_2')").
2.7: Cross-Table Markers
By default, the quota system does not allow you to specify the same "plus" marker across multiple tables. For example, the following quota specifications generate an error:
defines
| A | B | |
| 1 | MarkerA | plus |
| 2 | MarkerB | plus |
| 3 | MarkerC | plus |
plus_table
| A | B | |
| 1 | # = Plus Marker Table #1 | |
| 2 | MarkerA | inf |
| 3 | MarkerB | inf |
| 4 | MarkerC | inf |
| 5 | ||
| 6 | # = Plus Marker Table #2 | |
| 7 | MarkerA | inf |
| 8 | MarkerB | inf |
| 9 | MarkerC | inf |
The "plus_table" sheet above generates the following error:
In order to force the quota system to allow these "cross-table markers", we need to add a very special attribute to ALL <quota> elements in our survey. Instead of this:
<quota label="quota_plus" sheet="plus_table" />
We need to set the doit="1" attribute like this:
<quota label="quota_plus" sheet="plus_table" doit="1" />
With the special override doit="1" in place, our survey and quotas work as expected. The resulting quotas table is shown below:
Cross-table markers are often used to balance concept assignment across multiple variables. For example, if you needed to balance numerous concepts across participants by age, gender, ethnicity, and any other variable, you must use cross-table markers and the special doit="1" attribute.
Learn more: Assigning Balanced Concepts in Quotas with Multiple Variables
When using doit="1", do not use multiple sets of plus markers in multiple tables (e.g. markers A1, A2, A3 in one dimension and B1, B2, B3 in another), and cells:X will not choose multiple plus markers when utilizing cross-table (plus) markers to synchronize across tables; only one plus marker in each table will be selected. It is still possible that multiple non-plus markers can be chosen per table. Please test your project to verify it is working per your expectation.
2.8: Month-to-Month Tracking
Special surveys, such as tracker studies, sometimes need to field for long periods of time. Quotas often need to be updated after each period in order to establish new or reset existing limits.
In this example, we'll go over a method for creating monthly quotas that can easily switch between limits for each month.
There many ways to accomplish this task. The method described here should be very easy to implement using the Survey Editor.
At the very top of the survey, we will create an <exec> element that will set a marker relative to the current month:
<exec>
setMarker('May')
</exec>
Optional: We can automate the code above to automatically set the current month marker:
<exec>
current_month = datetime.datetime.now().strftime("%b")
setMarker(current_month)
</exec>
Using the code above, we can declare our quotas for the entire year!
defines
| A | B | |
| 1 | Jan | hasMarker('Jan') |
| 2 | Feb | hasMarker('Feb') |
| 3 | Mar | hasMarker('Mar') |
| 4 | Apr | hasMarker('Apr') |
| 5 | May | hasMarker('May') |
| 6 | Jun | hasMarker('Jun') |
| 7 | Jul | hasMarker('Jul') |
| 8 | Aug | hasMarker('Aug') |
| 9 | Sep | hasMarker('Sep') |
| 10 | Oct | hasMarker('Oct') |
| 11 | Nov | hasMarker('Nov') |
| 12 | Dec | hasMarker('Dec') |
| 13 | Male | Q1.r1 |
| 14 | Female | Q1.r2 |
| 15 | qual | plus |
general
| A | B | C | D | E | F | G | H | I | J | K | L | M | |
| 1 | # = Limits | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec |
| 2 | qual | 800 | 800 | 800 | 800 | 800 | 800 | 800 | 800 | 800 | 800 | 800 | 800 |
| 3 | |||||||||||||
| 4 | # = Limits | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec |
| 5 | Male | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 |
| 6 | Female | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 | 400 |
We can add the quotas above to our survey:
<exec>
setMarker('May')
</exec>
<radio label="Q1">
<title>Are you...</title>
<row label="r1">Male</row>
<row label="r2">Female</row>
</radio>
<suspend/>
<quota label="quota_general" sheet="general" />
<html label="Introduction" where="survey">
Congratulations! You've qualified to take this survey.
</html>
The code above produces the following two tables in the Quotas tab of the Response Summary:
When we need to begin fielding for a new month, we just need to update the <exec> element to set the new month marker (e.g., change "May" to "Jun").
If you've automated the month marker setting, then you're all set! You can field this survey all year long and never go over 800 completes per month!
2.9: Soft Quotas
So far we've seen quotas with hard limits set. That is, all of the quotas have a maximum, upper-limit that cannot be exceeded.
It is possible to declare a minimum, lower-limit that a quota must achieve. These are called "soft quotas".
To use soft quotas, you must specify a total for the quota table as well as the minimum and maximum limits for each quota marker.
For example, given the following quota specifications:
- N = 100
- At least 40 male participants
- At least 50 female participants
We can create a soft quota table to account for the specifications above:
defines
| A | B | |
| 1 | qual | plus |
| 2 | Male | Q1.r1 |
| 3 | Female | Q1.r2 |
general
| A | B | |
| 1 | # = Qualified | |
| 2 | qual | 100 |
| 3 | ||
| 4 | # 100 = Gender | |
| 5 | Male | 40-50 |
| 6 | Female | 50-60 |
Since we need at least 40 males, we specified a soft quota of 40-50. The 40 represents the lower-limit and the 50 is the upper-limit. Why did we choose 50 as the upper limit? Because the upper-limit should be set to a number that allows for all other soft quotas to be reachable.
Use the following formula to figure out the upper limit for a soft quota:
UPPER-LIMIT = TOTAL - THE SUM OF ALL OTHER LOWER-LIMITS
For example, the sum of all other lower limits is 50 for the male marker (because there's one other marker with a lower limit of 50). So 100 - 50 = 50.
For the female marker, the sum of all other lower limits is 40. So 100 - 40 = 60.
This means that the maximum number of female participants allowed into this survey is 60 in order to leave room for the minimum amount of male participants needed, 40.
When using soft quotas, the “Quotas” tab in the Response Summary will automatically update the upper-limits based on how many participants have entered the survey. As soon as a single bucket meets its lower-limit and begins to go over, the other marker's upper-limits will decrease to account for the decreasing number of buckets available.
2.10: Priority Quotas
By default, quota markers have a priority level of 0 and they're all considered even. The level of quota marker priority/importance can be set per quota marker using a special syntax. For example:
defines
| A | B | |
| 1 | Item_1 | Q3.r1 |
| 2 | Item_2 | Q3.r2 |
| 3 | Item_3 | Q3.r3 |
| 4 | Item_4 | Q3.r4 |
| 5 | Item_5 | Q3.r5 |
| 6 | Item_6 | Q3.r6 |
items
| A | B | |
| 1 | # cells:3 = Item Picker | |
| 2 | Item_1 | inf:5 |
| 3 | Item_2 | inf:5 |
| 4 | Item_3 | inf |
| 5 | Item_4 | 250:4 |
| 6 | Item_5 | 250 |
| 7 | Item_6 | 250 |
Using the special "LIMIT:#" syntax, we are able to prioritize the assignment of certain markers. Higher priority # markers will be assigned first. Markers without a priority number set default to 0, the lowest.
In the quota above, "Item_1" and "Item_2" will always be chosen if the participant is eligible for those markers. "Item_4" will always be chosen over "Item_(3, 5, 6)" until it meets its upper-limit of 250.
Assuming none of the markers have met their limit, here are a few examples of which markers would be assigned given the priority quota above:
- Eligible for Item 1, 2, 3, 4
- Item 1, 2 & 4 are assigned.
- Eligible for Item 1, 2, 3
- Item 1, 2 & 3 are assigned.
- Eligible for Item 3, 4, 5, 6
- Item 4 is definitely assigned (unless limit has been met).
The remaining two assignments will be based on the lowest counts for each Item 3, 5 or 6. - Eligible for Item 3, 5, 6
- Item 3, 5 & 6 are assigned.
- Eligible for Item 1
- Item 1 is assigned.
2.11: Quota Hooks
You can use a quota hook to modify the possible cell assignments for a participant.
Click here to learn about using quota hooks in a survey.
3: Data & Reporting
3.1: Quota Data Tables
If "quota" is added to the setup attribute of the main <survey> element (e.g., setup="time,term,quota"), then two data tables are automatically generated for every quota table in the project.
Each data table will contain all of the markers for that particular quota table and the number of participants who qualified for each marker. One data table will reflect the number of overquotas (e.g., voqtable1) and the other will reflect the number of qualified completes (e.g., vqtable1).
For example:
If you need to manually create these data tables, use the createQuotaTables() mutator function.
3.2: Editing Quotas Online
The limits for all quota markers can be adjusted from the "Quotas" tab in the Response Summary. Click on the "Edit Quotas" button and you'll have the ability to:
- raise/lower a marker's limit
- raise/lower the total limit
- set a lock on a specific quota maker, stopping it from accepting future participants
Learn more: How to Pause Quotas
3.3: Accessing Quota Markers from the Survey
Quota markers are named using the following convention: /SHEETNAME/MARKER.
For example:
- /general/Male
- /concepts/Female/Age_2/Concept_1
If a marker is a "plus" marker, then the name of the marker by itself will also exist in the set of markers (e.g., "Concept_1").
Markers are stored in the persistent list, p.markers. For instance, you can print p.markers to see all of the markers assigned or check if a marker exists in this list with "/general/Female" in p.markers.
In the "Markers" report located in the "Other Reports.." found in Report (2010), you can see all of the markers currently assigned for a project (e.g., https://tes.decipherinc.com/report/se...1234?markers=1).
You can also access information about quota marker limits from within the survey.
For example:
<exec> cells = gv.survey.root.quota.getQuotaCells() current, limit, overquota = cells["/general/Male"] isStopped = "/general/Male" in gv.survey.root.quota.stoppedCells </exec>
The stoppedCells code above cannot be used in projects located in the /selfserve directory.
The getQuotaCells() function call above returns a dictionary with marker names as the keys and a tuple consisting of (current, limit, overquota) as the values (e.g., {'/general/Male' : (38, 40, 0)}).
If soft limits are in place, then the values will look slightly different (e.g., {'/general/Male' : (38, (40, 40, 50), 0)}).
You can use this information to manually assign quota markers using the setMarker() function without going over the quota limit.
The setMarker() function assigns markers to a participant while the hasMarker() function checks if a marker already exists for a particular participant. Click here to learn more about these and other Python marker functions.