what is a good survey?

A well designed survey with good questions leads to trustworthy results that inform good decision-making.

While responding to the survey, we noticed a number of problematic questions that would result in biased responses and misleading data that could ultimately misinform US Fencing’s decision making about NACs.

Question 5 is a leading question that elicits a response bias against Salt Lake City as a NAC location

Response 1 is a positive response to Salt Lake City as a future NAC location conditioned on the respondent being from the east coast.

Response 2 conditions SLC not as a 1st choice.

Response 3 indicates SLC as too far away to travel to for the respondent

Response 4 rejects SLC as an option.

The choice of responses are biased against Salt Lake City as a NAC location.

The response options do not allow a non-east coast resident to give a positive response to SLC.

To get an unbiased survey result for SLC that can inform future good decisions, US Fencing must re-phrase the response options to remove the bias.

See the definition of leading question from Stat Treks Statistics Dictionary below:

“Leading questions. The wording of the question may be loaded in some way to unduly favor one response over another. For example, a satisfaction survey may ask the respondent to indicate where she is satisfied, dissatisfied, or very dissatified. By giving the respondent one response option to express satisfaction and two response options to express dissatisfaction, this survey question is biased toward getting a dissatisfied response.”

GET 25% off all fanatics gear with code bunny

April 21 and 22

Information received for Question 12 will not yield collatable data useful for decision-making

The question is imprecise, leaves too much to individual interpretation and and invites subjective personal preferences, without actually addressing the heart of the issue for NAC locations.

Example 1:

Person A picks location as the most important thing without elaborating. Person B picks location and says the location must be within 2 hour driving distance from where they live, Person C picks location and specifies that the location must be close to interesting tourist sites, and Person D says the location of the venue must be close to shops and restaurants. All these possible answers refer to location, but they are too varied to form the basis of data that can inform good decisions for US Fencing.

Example 2:

Person E says hotels are the most important thing and they must have Hilton and Marriott in the choice of hotels. Person F says that hotels with 24 hour room service are a must. Person G says hotels with full gyms are the most important to them. And so the responses continue.

It will be very difficult to aggregate responses that are so varied in any usable way. It would be very unscientific to pick what seems like the most common response and take action on it.

Almost 80% of all tournament travel to NACs is undertaken by busy students and their working parents during the school year. Whether we live on one of the coasts or in the middle of the country, every parent cares about missed school days, extra days out of work, and the cost of travel to NAC locations. Getting in and out as fast as possible at the lowest possible cost is a priority. US Fencing has to select locations that are fair to all, and not swing back and forth between penalizing one coast or the other.

The survey does not address these issues of flight options, travel costs and travel time which are the real priorities. The survey does not ask if NAC attendees care about these priorities to inform US Fencing’s decisions in selecting NAC locations.

Question 13 will not yield usable data

How would a reasonable respondent answer this question? You guessed it! By picking their own geographic location. The results will pretty much mirror where the respondents live. How useful is that data when deciding where to locate a NAC?

April NAC demographics are not a representative sample of all NAC attendees

Approximately 38% of April NAC attendees in SLC were Veteran fencers. As a percentage of overall NAC attendees, Veteran fencers represent less than 9% of all fencers who have competed in NACs between 2017 and 2019.

The selection bias in the April NAC survey will render the results less reliable. This is what the experts call “undercoverage” of the overall demographics that goes to NACs.

See: Who Goes to NACs and Pays the Fees 2017 - 2019?

Presumably, US Fencing’s goal is to inform its future decisions about NACs.

A better way to survey the NAC going population would be to send the survey to a random sampling of NAC attendees, and not to attendees of a single NAC.

See the definition of Undercoverage from the Stats Trek Statistics Dictionary below:

“In survey sampling, undercoverage is a type of selection bias . It occurs when some members of the population are inadequately represented in the sample.

A classic example of undercoverage is the Literary Digest voter survey, which predicted that Alfred Landon would beat Franklin Roosevelt in the 1936 presidential election. The survey sample suffered from undercoverage of low-income voters, who tended to be Democrats. Undercoverage is often a problem with convenience samples .”

To get good results to inform its future NAC related decisions, we strongly suggest that US Fencing re-do the survey and send it out to a representative sample of all NAC attendees.

Please share this article with everyone you know in the fencing community