Target Population, Access Frame, Sample
2.2. Target Population, Access Frame, Sample¶
An important initial step in the data life cycle is to express the question of interest in the context of the subject area and consider the connection between the question and the data collected to answer that question. It’s good practice to do this before even thinking about the analysis or modelling steps because it may uncover a disconnect where the question of interest cannot be directly addressed with these data. As part of making the connection between the data collection process and the topic of investigation, we identify the population, the means of accessing the population, instruments of measurement, and additional protocols used in the collection process. These concepts help us understand the scope of the data, whether we aim to gain knowledge about a population, scientific quantity, physical model, social behavior, etc.
The target population consists of the collection of elements that you ultimately intend to describe and draw conclusions about. By element we mean those individuals that make up our population. The element may be a person in a group of people, a voter in an election, a tweet from a collection of tweets, or a county in a state. We sometimes call an element a unit or an atom.
The access frame is the collection of elements that are accessible to you for measurement and observation. These are the units by which you can access the target population. Ideally, the frame and population are perfectly aligned; meaning they consist of the exact same elements. However, the units in an access frame may be only a subset of the target population; additionally, the frame may include units that don’t belong to the population. For example, to find out how a voter intends to vote in an election, you might call people by phone. Someone you call, may not be a voter so they are in your frame but not in the poulation. On the other hand, a voter who never answers a call from an unkown number can’t be reached so they are in the population but not in your frame.
The sample is the subset of units taken from the access frame to measure, observe, and analyze. The sample gives you the data to analyze to make predictions or generalizations about the population of interest.
The contents of the access frame, in comparison to the target population, and the method used to select units from the frame to be in the sample are important factors in determining whether or not the data can be considered representative of the target population. If the access frame is not representative of the target population, then the data from the sample is most likely not representative either. And, if the units are sampled in a biased manner, problems with representativeness also arise.
For each of the following examples, we provide a diagram for the data scope. The target population, access frame, and sample are represented by circles with shaded interior, and for each example, the configuration of their overlap represents the scope.
EXAMPLE: Informal Rewards and Peer Production. Content on Wikipedia is written and edited by volunteers who belong to the Wikipedia community. This online community is crucial to the success and vitality of Wikipedia. In trying to understand how to incentivize members of online communities, researchers carried out an experiment with Wikipedia contributors as subjects [Restivo and van de Rijt, 2012]. The target population is the collection of active contributors—those who made at least one contribution to Wikipedia in the month before the start of the study. Additionally, the target population was further restricted to the top 1% of contributors. The access frame eliminated anyone in the population who had received an informal incentive that month. The access frame purposely excluded some of the contributors in the population because the researchers want to measure the impact of an incentive and those who had already received one incentive might behave differently. (See Fig. 2.2).
The sample is a randomly selected set of 200 contributors from the frame. The sample of contributors were observed for 90 days and digital traces of their activities on Wikipedia were collected. Notice that the contributor population is not static; there is regular turnover. In the month prior to the start of the study more than 144,000 volunteers produced content for Wikipedia. Selecting top contributors from among this group limits the generalizability of the findings, but given the size of the group of top contributors, if they can be influenced by an informal reward to maintain or increase their contributions that is a valuable finding.
In many experiments and studies, we don’t have the ability to include all population units in the frame. It is often the case that the access frame consists of volunteers who are willing to join the study/experiment. \(\blacksquare\)
EXAMPLE: 2016 Presidential Election Upset. The outcome of the US presidential election in 2016 took many people and many pollsters by surprise. Most pre-election polls predicted Clinton would beat Trump by a wide margin. Political polling is a type of public opinion survey held prior to an election that attempts to gauge who people will vote for. Since opinions change over time, the survey starts with a ‘‘horse-race’’ question, where respondents are asked for whom would they vote in a head-to-head race if the election were tomorrow: Candidate A or Candidate B.
In these pre-election surveys, the target population consists of those who will vote in the election, which in this example was the 2016 US presidential election. However, pollsters can only guess at whether someone will vote in the election so the access frame consists of adults who have a landline or mobile phone (so they can be contacted by the pollster) and are determined to be likely voters (this is usually based on their past voting record, but other factors may be used to determine this). The sample is those people in the frame who are chosen according to a random dialing scheme. (See Fig. 2.3).
Later in Chapter 3, we discuss the impact on the election predictions of people’s unwillingness to answer their phone or participate in the poll. \(\blacksquare\)
EXAMPLE: Pollution and Health. The CalEnviroScreen project studies connections between population health and environmental pollution in California communities. The California Environmental Protection Agency (CalEPA) and the California Office of Health Hazard Assessment (OEHHA) and the public developed the CalEnviroScreen project. The project uses data collected from several sources, including demographic summaries from the U.S. census, health statistics from the California Office of Statewide Health Planning and Development, and pollution measurements from air monitoring stations around the state maintained by the California Air.
Ideally, we want to study the people of California, and assess the impact of these environmental hazards on an indivudal’s health. However, in this situation, the data can only be obtained at the level of a census tract. The access frame consists of groups of residents living in the same census tract. So, the units in the frame are census tracts and the sample is a census–all of the tracts–since data are provided for all of the tracts in the state. (See Fig. 2.4).
Unfortunately, we cannot disaggregate the information in a tract to examine what happens to an individual person. This aggregation impacts how we analyze the data and the conclusions that we can draw. \(\blacksquare\)
These examples have demonstrated some of the configurations a target, access frame, and sample might have, and the exercises provide a few more examples. When a frame doesn’t reach everyone, we should consider how this missing information might impact our findings. Similarly we ask what might happen when a frame includes those not in the population. Additionally, the techniques for drawing the sample can impact how representative of the population the sample is. We begin to address this topic in Section 2.5.2. When you think about the generalizability of your data findings, you will also want to consider the quality of the instruments and procedures used to collect the data. If your sample is a census that matches your target, but the information is poorly collected, then your findings will be of little value. This is the topic of the next section.