Phase One: Mass Culture’s Evaluation Approach & Building a Creative Space for Experiential Learning
Bringing together a group of funders for an honest discussion to reflect on the first year of a project is both a privilege and a daunting experience. Not long ago, I was fervently convincing them of its brilliance and worthiness as an investment. Don’t get me wrong—I stand firmly by the purpose and goals of Mass Culture’s Data Narratives for the Arts (DNA) initiative. I knew from the start that this would be as much a learning experience for myself and the organization as it would be for those we engaged. What I didn’t anticipate was just how profound the learning would be, or the many intentional steps of observing, documenting, learning, and adapting we would need to take to embrace it fully.
This practice of developmental evaluation, which I’ve come to know through the coaching I’ve done with Jamie Gamble on several projects now, is instrumental in how Mass Culture does its work. In the project’s initial phases, we collaboratively designed an evaluation framework with the funders to steer our efforts. The funder discussion mentioned at the beginning of this article served as a sensemaking session, a benchmark in our evaluative process that runs alongside our programming. It allows us to jointly reflect on the feedback received so far and discuss what has been effective and where improvements are needed. (For more about what a sensemaking session is check out episode four of this podcast on our Research in Residence initiative.
This practice of developmental evaluation, which I’ve come to know through the coaching I’ve done with Jamie Gamble on several projects now, is instrumental in how Mass Culture does its work. In the project’s initial phases, we collaboratively designed an evaluation framework with the funders to steer our efforts. The funder discussion mentioned at the beginning of this article served as a sensemaking session, a benchmark in our evaluative process that runs alongside our programming. It allows us to jointly reflect on the feedback received so far and discuss what has been effective and where improvements are needed. (For more about what a sensemaking session is check out episode four of this podcast on our Research in Residence initiative.
To tackle the challenge of gathering continuous feedback from project participants, we transformed it into a creative assignment. Each of the arts organizations participating in the learning intensive received a DNA Learning Log on Jamboard. This allowed them to not only share which learnings resonated most, but also express their feelings about the learning experience. It also served as an invitation to establish that we were open to and encouraged ongoing feedback, which we received from them through various channels.
To establish the DNA Learning Intensive as a creative and experimental learning journey, Mass Culture created a Miro Board called the DNA Resource Centre to share all project information. Additionally, we mailed out data kits (see image 1) to each arts organization in the DNA Data Cohort. These kits included an activity for visualizing their organization’s data. Although they were all given the same coloured glass shapes, they all used them to express data differently. This creative kickoff not only set the tone, adorning the first page of each of Learning Log, but also demonstrated that each organization already possesses and analyzes valuable data.
Some highlights of what we learned during the Project Launch Phase (09/23 to 12/23):
- We were cautioned that arts organizations, especially smaller ones, might lack the desire or capacity to engage in data utilization learning opportunities. That assumption proved to be entirely incorrect, at least in terms of their appetite for engagement. Over 80 arts organizations of all sizes, representing eight different artistic disciplines from across the country, applied.
- Our creative approach to this expansive subject matter served as an effective starting point. Despite participants in the cohort varying widely in their readiness and specific needs, beginning with a creative exercise helped establish a common starting ground for all involved.
- While the Arts Data Platform has sparked interest within the arts community, further efforts are needed to develop detailed use cases and showcase strategies for leveraging the data within it.
Phase Two: Refining The Art of Inquiry - Implementing the Learning Intensive
Mass Culture’s learning intensive began in September with three open online sessions, followed by seven learning modules for a cohort of fifteen arts organizations. We intentionally kept some modules flexible and adaptable in order to tailor them to the specific learning needs of the group later. It was during the second cohort-specific learning module that Mass Culture truly embarked on its own learning journey in how best to support arts organizations to utilize data effectively. Image 2 illustrates the original design of the learning modules (left) compared to the actual implementation (right), highlighting the significant insights gained through this shift. This experience has led Mass Culture to reconsider whether a cohort-designed consecutive set of learning modules is the most effective way to support arts organizations in enhancing their data practices. We have come to the conclusion that this model is not optimal.
So, what happened to cause a shift? We anticipated the challenge arts organizations would face in formulating specific questions to ask of their data. It’s overwhelming when you have numerous potential questions and limited clarity on what data is available to answer them. While we initially asked arts organizations to outline their data questions in the application form, it became clear during the second module that more structured support was needed to help them refine and workshop these questions effectively.
Initially, I believed that the data in the Arts Data Platform would inspire data narratives among participants. However, what they truly sought—and rightfully so—was guidance on how to collect, analyze, and creatively present their own data. While financial data on the arts sector in platforms like the Arts Data Platform and CADAC, in their current forms, can provide valuable context, such as provincial revenue trends or financial comparisons between regions, it often serves as a backdrop rather than the centrepiece of the stories arts organizations wish to tell through their data.
Until now, most people in the arts sector have engaged with this type of data primarily for reporting purposes rather than for deep self-understanding. Let me be clear, though: collecting and analyzing this data is very important, and we must continue to do so while acknowledging its limitations. This process helps us learn to ask better questions of ourselves and our practices.
Here are some insights we will carry forward from the Learning Intensive Implementation Phase (11/23 to 04/24):
- Building a data narrative must begin from a point of inquiry. Defining the right research question can be exceptionally challenging, yet it forms the bedrock when constructing a Data Narrative. (The Charity Insights Canada Project housed at Carleton University produced this short video (below), which offers a concise explanation of why this starting point is such an important first step. )
- Real-life case studies of similar organizations implementing data practices are genuinely inspiring. We invited Arts Etobicoke to share their data work during module six, but looking back, I wish we had involved them right from the start. Stories, especially those demonstrating the creative approaches that are possible, are inherently more inspiring than simply seeking inspiration from a dataset. Mindblowing, I know.
- To better support data narratives, future iterations of the Arts Data Platform should incorporate additional data sources and present information in formats that facilitate clearer and more accessible data storytelling.
- Setting a timeline for the learning intensive should be considered as a preliminary guideline to provide initial parameters, but flexibility and aligning with others’ workflows are essential to overcoming capacity challenges. This work requires time, not just due to its complexity, but also because it must fit into the daily schedules of participants. Mass Culture is dedicated to supporting an ethical and responsible data culture within the arts, which necessitates flexibility within our programming approaches. Even though the learning intensive has concluded, Mass Culture is continuing to provide ongoing support for many of the cohort participants, ensuring they receive the assistance they require and that we committed to. While I recognize that this kind of flexibility is rare in a funding environment that, due to financial and time-based limitations, makes it challenging for organizations to continue projects beyond their initial (proposed) timelines, the benefits yielded reinforce my belief that this is something we have to push for.
- Empowering arts organizations to operationalize data within their structures—whether through mapping out their data, crafting effective surveys, grasping the principles of data visualization, or establishing long-term tracking mechanisms—is a more significant achievement than simply producing a series of data narratives.
- Supporting arts organizations in harnessing their data practices requires high-level engagement and customization.
This final insight brings me to DNA’s most notable lesson: “We need data coaches”
Working closely with funders on this project has provided invaluable constructive feedback. They hold a comprehensive overview of various processes across numerous projects. Through ongoing dialogues with funders, a pivotal idea emerged to guide DNA’s Learning intensive: engaging data coaches to offer personalized support.
We all benefit from expertise and support beyond our own capabilities. The concept of data coaches surfaced from open collaboration and seeking diverse perspectives. It was clear from the outset that data coaches were not intended to act as consultants who simply take data, work independently, and return with a polished solution. Instead, this additional layer of support was designed to help build self-sufficiency, broaden perspectives, enhance data cultures within organizations, and encourage ongoing relationships that will exist beyond the conclusion of this learning series.
Among the 14 arts organizations that engaged with a data coach (with some still continuing the working relationship), not all experiences were equally successful. However, those who did succeed achieved truly remarkable outcomes – beyond my own expectations.
We are often our own toughest critics (except when we’re submitting final reports to our funders). There are always aspects that could have been handled differently or improved upon. The key takeaway here is the importance of implementing a developmental evaluation process as part of our work. This is what encourages us to continually learn from our actions and share those insights with our community. Thank you for taking the time to read this article and reflect with us on these insights.
