Designing Your Data Narrative

Clarity and Transparency: Be clear with respondents about why their opinion matters and how their data will be used. This includes informing them about who will have access to the data and what purposes it will be used for.

Confidentiality: If you’re collecting personally identifiable information, inform respondents about the confidentiality of their responses. This could include stating that their responses won’t be used for fundraising or shared with any other organization.

Contact Information: Provide a contact point (like a general info email) where respondents can go if they have questions or concerns about the survey.

Sharing Results: Consider providing a way for respondents to see what you’ve learned from the survey. This could involve providing a link where they can see key highlights of the survey results after a certain period.

Informed Consent: If you plan to share the dataset itself or release full data tables, it’s important to inform respondents about this potential data use or transfer.

There are a lot of datasets available from government sources, such as census data. Sharing data across institutions can be helpful for answering similar questions. For example, if multiple organizations within the same city are all collecting some kind of data that could be in conversation with each other, they might not need to all be collecting that data. They could trade insights to avoid duplication of effort.

Before thinking about the big survey you want to write, it is useful to think about what data you already have. Consider your registration form as a potential source of data. 

There are many alternatives to surveys for gathering insights, such as qualitative research with new attenders or subscribers. Smaller sample sizes can still provide valuable insights for comparing different groups within a population.

There are many alternatives to surveys for gathering insights. These include qualitative research methods such as conducting interviews or focus groups with new attendees or subscribers. She also suggested that organizations can use these methods to check their assumptions and improve the quality of data they’re getting.

For example, organizations can conduct a debrief after an event or program to gather feedback and insights. This can be done informally, such as through a conversation, or more formally through a structured interview or focus group.

Organizations can also use existing data sources to inform their research. This could include analyzing operational data, financial data, or data from registration forms.

Maintaining privacy and securely storing information during data collection is key. The first step is to identify whether you’re collecting personally identifiable information. If you are, it’s important to create walls of access between that information and the data you’re using or analyzing. This can be done by separating out the identifiable information, saving a version of the dataset that doesn’t have those fields but has a key identifier so it can be reconnected if needed, and then doing all the analysis on the de-identified dataset.

It is recommended that you get rid of the data after some period of time, for example, six months after a study is complete, to prevent potential future breaches. For more sensitive studies, you can use a “blind” approach where nobody has access to the full dataset and multi-party authentication is required to access the data.

Being clear and transparent with survey respondents about how their data will be used and shared is vital, and it is good practice to provide them with a link to see the key highlights of what the organization learned from their survey responses.

Maintaining transparency by informing respondents about how their data will be used and shared is very important if one wants to make the data openly available. This includes providing the actual underlying raw data set, releasing tables of responses, and informing users of potential data use or transfer. By providing respondents with a link to see the key highlights of what the organization learned from their survey responses, participants are given a sense of payoff and understanding of their contribution,

Leading Questions: These are questions that prompt or encourage the desired answer. For example, “Isn’t it great that x, y, or z?” or “The program you attended was fantastic, right?” These types of questions can influence how respondents answer.

Double-Barreled Questions: These are questions that ask two things at once. For example, “Was your experience meaningful and enjoyable?” When you include two concepts in one question, it’s hard to interpret which one the respondent is answering.

Question Effect Order: The order in which questions or response options are presented can influence how people respond. The first item on a list is a little more likely to be selected than others. To control for this bias, some survey platforms allow you to randomize the order of response options.

Overloading Questions: Sometimes people try to include too many ideas into one question, which makes it hard to interpret the responses. For example, a question might include 15 or 20 different response options for people to read through. In such cases, respondents are likely to find the first option that sort of applies and pick it.

To mitigate these biases, use clear and concise language, avoide double-barreled questions, and constrain response options to 5-7 options. 

Check out the survey toolkit from Of/By/For All that helps organizations design surveys for different constituencies.

• Understand the target population and sample size needed to be representative. Consider using census data for comparison.
• Separate personally identifiable information from the dataset for analysis to protect privacy. Remove this information after the study.
• Be transparent about how data will be used and shared. Inform respondents about confidentiality, data access, and opportunities to see results.
• Incentivize survey participation through timely responses, gift cards, or experience-based incentives like behind-the-scenes tours.
• Test surveys with colleagues and cognitive interviews to improve question design and catch biases.
• Use clear, concise, and unbiased language. Avoid double-barreled questions and constrain long lists of responses.
• Consider qualitative research methods like interviews alongside surveys to understand reasons behind responses.

Qualitative and quantitative data are powerful when used together. Alternate between the two methods, using quantitative data to identify patterns and then qualitative research to understand the “whys” behind those patterns.

Qualitative research is important for understanding program implementation and addressing potential “blind” spots in quantitative data. Open-ended survey questions and testing surveys qualitatively are some ways to incorporate qualitative aspects into quantitative research.

• Using incentives like gift cards, as this is a recognized way to boost responses.
• Sending reminders, as response rates often increase with each reminder email sent. Katie noted their surveys rarely get responses on the first email.
• Making the survey invitation timely by asking about a recent experience or relationship, while it’s still fresh in people’s minds.
• Providing an interesting incentive experience, as the speaker shared research found a behind-the-scenes tour incentive was more motivating than a gift card.
• Communicating the value and impact of participation upfront, so people understand how their input can help the organization.
• Testing different incentive options to see what motivates your particular participants or audience the most. Incentives that resonate can vary.