I admit: I didn't think it was that hard to get a survey translated.
Over the past few weeks, I learned just how wrong I was -- and ate a big piece of humble pie in the process.
With colleagues, I'm working on a landscape analysis of how families and educators in California feel about family engagement and the state's requirements for incorporating stakeholder feedback into district plans for improvement. We're designing a training program around these topics, but to make sure our program will be relevant, we wanted to hear from the people who would be participating in it. We designed a survey and planned for focus groups, and I naively thought we were good to go.
Although Baltimore has a growing -- but fairly localized -- population of English Language Learners, the families at the schools where I worked were predominately Black and English-speaking. When I worked at the district, we had a cadre of interpreters we regularly contracted with for events, and we used large-scale survey software that easily facilitated (mostly adequate) translations.
So when we decided to translate the California survey into nine additional languages, I didn't anticipate just how difficult that would be.
Our survey was fairly basic and brief, so I built it out in Google Forms ... only to learn that despite the widespread availability of their free translation technology, there was no mechanism for translating surveys in their tool. (I'm honestly still scratching my head about this.) The most straightforward (ha!) way I found to create a multilingual survey in Google was to independently translate the survey into each language, build a separate page in the survey for each language, copy and paste each line of the translated surveys, and then use skip logic to direct people to the page with the language they selected.
We gave up on Google. We found out that our client had a Survey Monkey account that included the ability to create multilingual surveys. I was excited. Finally - a logical way to complete this seemingly simple task!
Nope. I was still wrong.
While this platform at least offers a dropdown menu of languages on the survey page (thereby making it easier for respondents and avoiding the skip logic silliness on the back end), it turns out that this paid feature was just as cumbersome to use as the Google option. What I ended up having to do was download a coded text file for each language, pay to independently translate each of the languages (Thank you, Stepes Translation, for coming to our rescue!), copy and paste each line of the translations into specific sections of the text file, and then upload the translated file to the system. NINE TIMES.
With my hand cramping from all of that Ctrl-C and Ctrl-V action, I was stunned by how technically difficult and frankly, inaccessible the survey translation process was. Who is actually going to go through all this? More importantly, what does this mean for the voices of those who are not native English speakers? Without access to a large, institutional subscription to a powerhouse survey software, my gut tells me that very little translation is likely to happen. As a result, many important voices are being silenced.
I don't have a solution to offer here, but I'm glad that this is a lesson I learned. This has opened my eyes to the institutional roadblocks that prevent equitable language access in our country... and I know I've just scratched the surface. Translation services, albeit not 100% reliable, are widely accessible and free online, yet they are not integrated into lower-cost survey platforms. This not only causes a huge headache for survey designers, but it inhibits the ability to hear from non-English speakers about important issues. As I seem to say in a lot of my blog posts, we have to do better.
If anyone has a better solution than the relay race I just ran, please share in the comments! I do hope that a more accessible and user-friendly option exists.
It's easy to feel discouraged and upset when you turn on the news these days. So much is happening to progress the fight for racial equity and justice in this country that even a global pandemic seems to have taken the backseat. The truth is that without a precedent for our current events, we are all making sense of them as they come. We face tremendous uncertainty in the days and months ahead, especially in the education sector. No one knows what school should or will look like when the 2020-2021 year begins in the fall. And that is scary.
But here's why I'm feeling encouraged. Without a doubt, the Black Lives Matter movement is bringing critical and often unheard voices to the forefront. I'm also starting to see this happening in schools, with many districts really lifting up the voices of parents and families as they make decisions for what reopening schools will look like. I've seen multiple districts just this past week sharing surveys with families about reopening. How can we truly serve children and families if we don't know what they fear, what they want, or what they need?
So if your school, organization, or district is trying to imagine what school will look like in August, and you haven't talked to families and students, now's the time to use some simple evaluation strategies to give power to their perspectives. Here are a few tips to get started.
Think about what you really need and want to know from your stakeholders. Make a list of what your team is wondering about or what the impact of proposed plans might be before you draft your survey questions. For example, many districts are considering alternate schedules to accommodate all students in socially distant ways. Here are a few things to think about:
Encourage your survey respondents to commit to an answer. Whenever I take a survey, and I don't really know or care about the answer, I always select the non-committal, middle option. Most people do - it's human nature. However, during this especially important time, we can't risk having a whole bunch of middle of the road responses. Consider using a four-point (instead of a five-point) scale that encourages respondents to indicate if they're feeling negatively or positively about what you're asking. Instead of a neutral/not sure answer choice in the middle, have them choose from a scale like this: Strongly Disagree, Disagree, Agree, or Strongly Agree. Include a "not applicable" option if you feel that's relevant - we don't want to force answers that don't make sense - but this type of scale will give your team a better sense of which way your stakeholders are leaning and help you make more informed decisions.
Make it equitable and accessible. Hopefully it goes without saying that not all students and families can access a survey that is online and only in English. To embrace and reflect the diversity of our nation's school districts, we must try to reach our stakeholders in multiple ways. Of course, an online survey is the easiest way to collect information, and many families can at least access the internet on their phones. However, some families cannot, and to truly understand what your families and students are feeling about reopening, we need to try to reach them as well. Think about mailing surveys or distributing them at food giveaways or other local gathering places. Or, if you're unable to translate the survey into every language spoken in your district, hire a few bilingual staff members or outside interpreters to do brief phone surveys with families whose native language isn't English.
For the most successful reopening possible in the fall, districts need to know what families and students are thinking now. Brief surveys are an easy, cost-effective way to reach a large percentage of your stakeholders. Schools and districts need to think creatively to hear from as many families as they can and make their understanding of student and family needs as inclusive and diverse as possible. You'll be amazed at how much more informative your results can be!
I still remember how scared I was during the 2015 riots/uprising in Baltimore after the death of Freddie Gray.
I understood the pain and outrage in the black community over the death of a young, black man in police custody, but looking back, I didn't really get the depth of the collective trauma that was at the heart of it.
At the time, I was a Community School Coordinator at a predominately black elementary school about two miles down the road from Mondawmin Mall, the epicenter of the protests. As word spread of a student-led protest at Mondawmin that afternoon, staff and families began to fear for the safety of our students as they left school for the day. I had to take a different route on my way home, since I always passed the mall on my drive. On my detour, I passed armored National Guard tanks and heavily armed soldiers. It was clear that this was much more than a student-led protest. I was rattled.
That night, I holed up in my apartment and spent the entire evening glued to the news, unable to do anything but watch my city burn. Living close to downtown, I heard a lot of noise outside that night, but I was shocked to wake up and find buildings across the street boarded up, having been broken into the night before.
I felt that same initial shock this weekend when I saw images of my new city - Columbus, OH - on fire. While still frightening, I had a different understanding of the situation than I did five years ago.
In the five years since that night, I have continued to study, listen, and learn about the deeply institutionalized racism in this country and the violence that still accompanies it in modern-day America. I now have a better understanding of how intensely traumatized our black communities are from centuries of oppression, discrimination, and brutality, and I also know that I can never truly know that pain for myself.
So the recent surge in senseless and hate-fueled killings of black men and women in this country has rested heavy on my heart. I've felt sick over the horrific and unnecessary deaths of Ahmaud Arbery, Breonna Taylor, and George Floyd in recent weeks ... and so many others before them. I've heard the friends and colleagues I love question their own self-worth and the safety of their children just because of the color of their skin. Dyjuan Tatro, featured in the moving series College Behind Bars, about the Bard Prison Initiative, summed this up so eloquently:
And yet, police brutality has only proliferated this week. I've watched videos of officers driving their cars into crowds of protesters, pulling down a young man's mask and pepper spraying him, and shooting protesters and journalists in the face with "non-lethal" bullets.
There is no reason for it, and there is no excuse. The blatant racism and hate crimes in our country must be prosecuted and condemned.
But let's think for a moment about our country's black youth. On top of this historical and racial trauma that so many of them carry before they are old enough to understand it, they also attend under-resourced, hyper-segregated schools ... and yet, we expect them to learn and function like those who are lucky enough to live without these burdens.
It doesn't make sense. We have to do better.
I'm still grappling with my role in all of this and how I can try to make the world a bit better. I will continue to serve and support communities and school districts serving black and brown youth, but somehow that doesn't feel like enough. I hope that as a nation, we can collectively remember that there is a common good, and that it is far easier to stand for that than to stand for hate.
It's funny how things work out sometimes.
Tamara Hamai and I have been sowing the seeds for our new program, Evidence for Engagement, for months. Our partnership happened so organically - a meeting of the minds for two evaluators who have experience with and a passion for organizations that serve youth and families. We'd been toying with the best way to support the organizations that we serve and help them use evaluation to improve their access to funding and the children and families they serve.
Then COVID hit.
The pandemic has caused all of us to pause and re-evaluate how our work fits into a very new, very different reality. Tamara and I know that small organizations, especially those who work in schools, are struggling right now. Their access to the people they serve has been essentially cut off. We realized that organizations may need our help even more than before.
Our solution: We're running a totally free, three-week email series that will help small youth- and family-serving organizations build their evidence base (which is required under the Every Student Succeeds Act for any organization receiving federal education funds). Through videos, worksheets, frameworks, and success stories, Tamara and I will walk participants through the process of becoming evidence-based organizations and help them see this as an opportunity, not a burden.
The goal: We want to help vital, community-based organizations plan for the future, open themselves up to new opportunities, and become more sustainably funded. We're hoping that this opportunity will help them better serve youth and families, not only during this difficult period of time, but also for a long time afterward.
For us, this is also about equity. We know that for many community-based, minority-owned organizations, budgeting for evaluation is out of the question. We also know that these grass-roots organizations are having a profound impact on their communities -- and that their communities need all the support we can give. We're hoping that we can get more small, local organizations approved as evidence-based programs in their districts and begin to level the playing field.
If you think this program will benefit you and your organization, sign up below! If you know of someone else who could use this support, encourage them to join.
Ready to start your evidence journey?
Sign up below to get the Evidence for Engagement mini-course sent to your inbox.
You have successfully joined our subscriber list.
Last week, I watched a powerful webinar from Ohio State's Wexner Medical Center about health inequity and COVID-19. One of the first speakers, Dr. Nwando Olayiwola, started to talk about vulnerable populations but quickly corrected herself. She called them, "populations that have been made vulnerable."
What a difference such a small change made. Instead of assuming that people in those populations are inherently vulnerable, her corrected phrase shows that inequity is the result of intentional decisions that negatively affect specific groups of people. Her quick self-correction stuck with me.
This morning, I was reading the education news that comes to my inbox each day and saw a headline from the Wall Street Journal last week that read, "Some School Districts Plan to End the Year Early, Call Remote Learning Too Tough." Another entry in the same newsletter suggested that based on a national poll of teachers and administrators, 65% want to start the school year as normal in the fall, without adjustments to the curriculum or schedule. My heart sank, and I instantly thought of Dr. Olayiwola's revised phrase.
Now, I am no longer a K-12 teacher, so it is unfair for me to pass judgment on educators, whose jobs were already difficult, and who had to do a 180-degree shift in their daily practices within days or weeks. That is an incredible challenge, and it will take time to adjust. I empathize with teachers and cannot imagine what I would have done if I had to shift my middle school instruction online in a heartbeat. However, we're also faced with a growing educational crisis.
Evaluators and researchers have been doing some great work to illuminate the educational ramifications of COVID-19 on disadvantaged communities. Researchers have studied the concept of "summer slide" for many years and have shown students regress in math and reading skills without educational opportunities during the summer. There is even a National Center on Time and Learning, whose work revolves around reducing inequities related to the inflexible and insufficient school year schedules that predominate in our country. Recently, the Collaborative for Student Growth found that an even more significant "COVID slide" is likely to occur when students return to school in the fall, having retained only about 70% of the reading growth and 50% of the math growth they would have typically made in a school year. Recommendations to mitigate the COVID slide include summer school and additional learning time for students.
Yet, the Wall Street Journal article discussed how districts across the country are choosing to end the school year up to three weeks early in order to have more time to prepare for the fall. One superintendent even stated, "It made sense to us to get rid of the stress and get ready for the following school year." We certainly need more supports for teachers and greater access to technology for students in order to make online learning more functional and effective. But does that justify giving up entirely?
Decades of research have shown that more time in school leads to better outcomes for students, especially those from low-income communities. In a time of increased risk for widening the achievement gap, is it ethical to throw in the towel? By justifying inaction with the feeling that it is too difficult to educate students from home, we are making our student populations -- in many cases -- more vulnerable than they already were. I hope that evaluators and researchers can come together with educators to study the data, fully understand the problem and its drivers, and develop policy recommendations that will not only support teachers but will also do right by students who need time with teachers to thrive.
It's not always easy to measure the impact of family and community engagement efforts. Some aspects of education -- like test scores or report card grades -- (notwithstanding the wide variety of controversies around their use) are pretty straightforward to measure. They're already quantified. They're known entities. We can tell that story. But when we talk about measuring the impact of, say, a super successful family science night, our minds go blank.
Thinking of our present situation, in which family and community engagement is more critical than ever, how can we tell the story of how a school or organization engaged and supported its students and families and even its neighbors during the pandemic? How do we collect and measure both the individual or personal impacts and the large-scale results?
Erica Green and Lola Fadulu did such an effective job of this in a recent New York Times story about how school cafeterias across the country have become "relief kitchens" for not only students and their families, but also community members in need. The cafeteria manager featured in the story effortlessly rattles off the incredible numbers of people she and her team have served and also shares stories of individuals she's encountered. Not only is she a hero in her community, but she's a perfect example of one of my core beliefs -- that you don't need to be a statistician to effectively use data in your work.
If your team is not sure how to start measuring your COVID-19 impact and telling its story, here are three steps to get started:
This is just a short list of ways to use data to measure your impact and involve your community in doing so. Of course, if your school or organization needs some guidance, I'm here for you. I offer three different packages to support schools, districts, and organizations with this work. I'm also happy to do phone consultations to help you brainstorm.
Regardless of how you do it, this is the time to make data a priority. What will be your story?
For the past few weeks, one image has been recurring in my mind. I keep picturing the living conditions of a student whose home I visited a few times when I was working as a community school coordinator in Baltimore. This student had struggled at other schools but was thriving at ours. He had repeated second grade, so his maturity compared to his peers was notable, but overall, he was just a really sweet kid. We did a number of home visits for him that year because he missed a lot of school, and as a black child from a low-income home with documented learning disabilities, school was even more important for him than most.
Every time I get frustrated with having to stay home - in my very comfortable apartment, with my husband and dog, and with fairly steady work - I've been trying to check myself. I keep thinking of my former student and imagining how difficult it must be to be confined to a space that may not be healthy, safe, or developmentally appropriate. I keep thinking about all the students I've known who love coming to school because there are people there who love them, two to three meals a day, and a sense of community. I keep thinking that I wish I knew how to help them all right now.
There is no question that low-income students struggle to get to school. (I wrote about this in my last blog post as well.) According to Attendance Works:
"Children living in poverty are two to three times more likely to be chronically absent—and face the most harm because their community lacks the resources to make up for the lost learning in school. Students from communities of color as well as those with disabilities are disproportionately affected."
Unfortunately, they're struggling to access school online as well. The New York Times reported this week that large percentages of low-income students in districts across the country are absent from the virtual education being provided as a result of COVID-19. So not only are students from disadvantaged backgrounds missing out on the resources that many of them so desperately need and want to access, but they are also disconnected from their school communities. Since it is unclear when we will be able to return to work and school, there is the possibility that our highest-need children could be without school for many months, only exacerbating already-existing gaps in achievement and opportunity.
Making matters worse, black communities are disproportionately becoming victim to COVID-19. As one of the many social determinants of health, education joins other critical factors such as adequate housing, socioeconomic status, access to and coverage of healthcare, and more to comprise health outcomes for people and communities. As you can see in the chart below from the Kaiser Family Foundation, these factors have a profound impact on a person's ability to live a healthy life.
For low-income black communities in particular, the collective impact of these factors has not only disastrous outcomes but also clear roots. Dr. Camara Jones, a physician and epidemiologist, is cited in the article linked above about the disproportionate impact of COVID-19 on black communities:
“COVID is just unmasking the deep disinvestment in our communities, the historical injustices and the impact of residential segregation... This is the time to name racism as the cause of all of those things. The overrepresentation of people of color in poverty and white people in wealth is not just a happenstance."
Our unequal and unfair society is how it is by design and not by chance. COVID-19 is showing us how this is even more urgently a matter of life or death. Other than overhauling our government systems and laws entirely, I struggle with not knowing how these issues can be fixed or what I as an individual can do to make things better for others. I just hope that opening up the dialogue about these issues will start to lead to changes for students like mine and the families and communities in which they live.
I naively said to my family a few weeks ago that I was lucky that my business wouldn't be affected in the way that so many restaurants and other small businesses are by the pandemic. I couldn't see then, but it has become incredibly clear since that every business and every industry have been and will be impacted on a massive scale by this pause in how we normally function. I am so fortunate that I can continue much of my work from home; yet every day, my heart breaks for other small business owners who are doing everything they can to keep their businesses afloat during this unprecedented time.
I spoke to one of my former clients recently, and I expressed how badly I felt that the cancellation of schools would drastically affect his work. His response to me was this: "This is a time to pivot!"
All week, I've been thinking about what he said and what a resilient attitude he had. I've also been pondering how this affects not only my work but the organizations and communities I serve. Take the issue of chronic absenteeism in schools. Chronic absenteeism is typically defined as missing 10% of more of a given school year, and it has been empirically tied to a host of negative outcomes for students, including reading levels, special education identification, suspensions, dropout rates, and more. Students from vulnerable and underserved populations are at the highest risk. The good news about this issue is that it is both preventable and reversible, and I've found in my work that regular and collaborative data tracking on student absences and related interventions can make a huge difference for kids and schools. A recent article, Chronic Absenteeism in the Time of Coronavirus, discussed the implications of shuttered schools on how schools and systems typically address attendance issues, but also on what closed schools mean for the students themselves.
In thinking of how to pivot from the typical accountability measures associated with attendance, as Jordan's article suggests, how can we in the education field work together during this time to address the root causes -- the underlying reasons why so many students miss school? I am loving all of the positive news articles out there about school districts employing bus drivers to deliver meals, offering wifi hotspots to those without internet access at home, and teachers driving around students' neighborhoods to help them feel connected.
From my lens, I think this break from traditional schooling is an opportune time to go deep with our data and determine all that we can about which students are missing school the most and why. Once we've done that, we can get creative about our interventions for kids while they are in their homes and communities. I've got a few ideas in development for how my work can pivot to best serve our schools, districts, and non-profit organizations who have the most direct lines to children and families (more information on that to come!). In the meantime, if your organization is trying to better support its chronically absent students or more effectively engage with families during this time, let's chat. Shoot me an email here so that we can set up a conversation.
How are you planning to pivot? Share your ideas in the comments below - I'd love to start a conversation and channel our collective creativity!
I've been training in Muay Thai (kickboxing) for a few years. I am always learning something new and being pushed outside of my comfort zone ... and I love it. However, I am and have always been a perfectionist. It's something I've struggled with my whole life: sometimes, I'm really proud that I've been lenient with myself, and other times, the perfectionism rears its ugly head. Lately, I've noticed it manifesting at the gym. As I'm trying to apply a new skill in work with a partner or coach, I've been getting frustrated and self-critical. My self-protective instincts (ironically not working to appropriately block a punch or kick) have made me think, "I don't like this aspect or that skill," instead of allowing me to see that this is a process of growth and that there is no place for perfection in that process.
I think organizations (and the people within them) can be the same way when it comes to evaluation. We get used to our routines, we think we've perfected them, and then one of a few scenarios happen that push us out of our comfort zones. Maybe we are required to learn a new system or skill, or -- even worse! -- we get feedback that doesn't match our own perceptions. Now at the gym, my feedback can be a simulated sparring round that doesn't end so well for me. But in our workplaces, while we are focused on serving the people we care about, feedback that we're not doing so well is upsetting to hear and painful to accept. That upset and pain is followed by questions -- "What could we do differently?" Why is so-and-so doing well at this when we're struggling?" or even "Is this feedback accurate or reliable?" Our self-protective instincts kick in.
The anticipation of negative feedback -- in whatever form -- is a huge barrier for people (including myself!) to try new things, reflect on their own performance, or seek help and other perspectives. Certainly, the accountability culture in education has only made these innate fears and insecurities worse.
Today at the gym was different though. The past few days, I've been more reflective about why I'm getting so frustrated and how that is keeping me from truly learning and growing. So today, I tried to pay attention to the moments when I got frustrated (ie. I collected some data on myself!). I worked with my coach to talk through those negative feelings and develop some strategies I could try in those situations. Then, I practiced and stayed open to more feedback... and by the end of the session, I felt more resilient and confident in my skills than I had in awhile.
Terms and methods like "continuous improvement" and "improvement science" get used a lot in both education and evaluation, and they are proven methods for making institutional (or personal) changes on all levels. I'm sure that what I did at the gym today was just a tiny Plan-Do-Study-Act cycle. Yet for me, these formal frameworks for self-assessment and reflection can sometimes be hard to grasp - and they can feel like another thing we're accountable for doing. However, we can look at them more simply: sometimes, all we need to do is recognize that we're passionate but not perfect, allow ourselves to be open to feedback, and develop an authentic plan for how we can improve. This is true for individuals and organizations.
As an evaluator, I love the moments when conversations about data lead to a-ha moments instead of feelings of defeat. (Data visualization is especially helpful here.) Sometimes, when we take a step back and think about why we're assessing or evaluating, we can see that it's not all about accountability and funding requirements (and not about our individual or collective insecurities either). Sometimes, it is just about putting our guard down (or up, if you're at the gym), remembering that we can always do better, and learning to see our imperfections as a sign of growth in the making.
I had some great conversations this week with colleagues about establishing a culture of data in organizations and training organizations who are new to evaluation and data. These conversations reminded me about one of my favorite old blog posts, that I originally wrote for the National Association for Family, School, and Community Engagement (NAFSCE) in 2017. Given this week's discussions, it felt like a good time to bring it back into the rotation (with a few updates!).
Don't Be Scared of Data - How it Can Guide Family Engagement and Attendance Interventions
When I was a teacher, conversations around instructional data were baffling to me. Fresh out of policy school, I was eager to use what I had learned about data analysis to monitor how my students were performing, but as a social studies teacher, this task was more difficult than I had anticipated. I was required to keep a data binder, and administrators would periodically check to confirm that, well, it existed. However, I struggled to figure out what to put inside of it. My administrators did not help me understand how – absent standardized test data – I could track progress on specific standards outside of my grade book. It often felt like the conversation ended after the word “data” was uttered.
As I have focused my career on family engagement efforts, I have seen how conversations about using data to improve engagement are often greeted by the same blank stares I encountered as a teacher of a non-tested subject. On other days, talking about data elicits looks of panic or skepticism. At one particularly memorable training, community school coordinators were led in a debate about the utility of data. Sitting from my seat on the pro-data side of the room, I was amazed by arguments from the anti-data group. What resonated most is that these capable and talented colleagues understood data to simply be numbers on which their performance review was based, not as a tool to discover context and unlock insights about the families being served.
I think this belief system exists for a number of reasons. First, many educators are tired of increasing demands for data without sufficient training. Professionals need to understand how data can be collected, ways in which it should be analyzed, and how it can actually make their work easier. I have found that on-the-ground staff are often the last to receive the proper supports and professional development around understanding and using data. It becomes a symbol for all of the things we don’t like about accountability instead of the asset that it truly can be.
Perhaps more importantly, the work of engaging families – understanding needs, forming trusting relationships, and helping people when they are vulnerable – is incredibly difficult to quantify. Often, we know we have made progress or achieved results – not because of a spreadsheet or heat map – but because a family had enough food for the weekend or because a child stopped acting out as much in class. How do we tell those stories? How do we show our value as professionals when these important markers seem impossible to put into a spreadsheet? These are the critical questions we need to answer.
For these reasons, it is my mission to help educators and professionals realize that data does not have to be scary or intimidating. It does not require complex coding skills or mathematical know-how to track how clients are being served. If you would have been sitting across from me in the data debate, here are some tips to get you started:
Using your organization's qualitative and quantitative data can give you amazing insight into both the ongoing needs and continuing growth of the students and families you serve. With a little less reticence towards this approach, we can make a lot more progress in engaging families to help their children succeed.
Of course, if your organization is unsure of how to get started in this area, I'd love to be of assistance. Learn about the new Build Your Evaluation Capacity training package!
The goal of this blog is to highlight relevant issues that impact students, families, and communities and spark engaging discussions about how to address those issues through evaluation.