Predicting Ambitious Instruction at CPS
(Github Project Repo)
What predicts where you will see 'good' teaching?
It's easy to imagine that this question has a simple answer - wherever you have good teachers, you see good teaching. All we have to do is identify good teachers and get rid of bad ones, and education will be fixed!
Teachers will cringe at this type of thinking, because we see our own effectiveness fluctuate constantly, from year to year, from day to day, based on a huge number of contextual factors - some within our control, some without, some predictable, some not.
I personally experienced a huge change in my teaching effectiveness a little over a year ago, when I changed schools. At my previous school, I had found it relatively easy to use 'best teaching practices' - pushing my students to take on more of the cognitive load, to explain themselves in detail throughout class, and to persist through struggle and failure as we engaged in difficult projects. At the new school, I encountered strong resistance from my students to these teaching practices, and I found myself falling back slightly to more 'traditional' teaching just to make my days more bearable.
In the following project, my guiding hypothesis is that 'ambitious instruction' is not a quality that is embodied by a particular teacher. I propose that 'ambitious instruction' is a complex aspect of school culture that can be nurtured or undermined, depending on the school environment.
Every year in Chicago Public Schools, teachers, parents and students fill out the "My School, My Voice" surveys collected by University of Chicago's Urban Education Institute. These surveys fall under what are considered "5 Essential Ingredients" a school needs to have in place to be considered well-organized for improvement.
I wanted to know what predictors would most influence the metric called 'Ambitious Instruction.' This metric is based on student survey results, and essentially corresponds to how often students say they are asked to do rigorous, minds-on activities in class, and how much of the thinking load is placed on students.
The 5Essentials data is not readily available for public download, so I scraped it from the UEI website for each school in CPS. From the CPS data portal, I was able to get information about the type of school, whether the school is a charter school or board school, what grades are served. The dataset also included demographic data, such as % of students in special education, % of students with bilingual designation, and % of students on free and reduced lunch.
As I initially began developing a regression model, I experimented with different kinds of regressions. Since I had a lot of data features I was working with, I used a Lasso regularization model to identify features that could be eliminated, as shown in the figure.
Perhaps surprisingly, neither SQRP points (used to determine a CPS school's level rating, which is relevant for decisions about closure) nor being a charter school, ended up being particularly predictive for the Ambitious Instruction rating. The percent of students with IEPs also had low predictive power, as did how well teachers rated 'parental involvement' at the school. I dropped these features and optimized my model based only on the most critical parameters, eventually settling on a Ridge regression.
My final model had a typical error of ± 7.3 points. The 5Essentials scale change their qualitative level (the color from the graphic at the top) every 20 points, so in general my model will predict the correct level for a given school most of the time.
Critical Feature 1: Supportive Environment
+8-fold effect on 'Ambitious Instruction'
Supportive Environment is one of the only 5Essentials data features that persisted in my model. This feature is based on a combination of data from students and teachers. In general, the survey questions that build this feature are broken down into a few primary categories.
Safety - example: how safe do you feel in the hallways at school?
Student-Teacher Trust - example: do you agree - my teachers treat me with respect?
There are also questions that differ slightly for high school vs. elementary school - in high school, students are asked how much the school is oriented towards preparing for life and college after graduation. In elementary school, students are asked how invested their peers are in their school work.A supportive environment made a lot of sense to me as a critical feature of a school that can enable ambitious teaching.
Safety -> A student who does not feel safe in school does not have the real estate in their brain to engage in cognitively challenging classwork.
Trust -> The fact that respect and relationships are prerequisites for teaching students effectively is a truism in teacher education programs, but relationships seem to be particularly critical to getting students to take on the kind of challenging work that will be most beneficial for them in the long run. This correlation probably goes both ways, as students may have increased trust for teachers who challenge them and hold them accountable.
Critical Feature 2: Percent of Students on Free and Reduced Lunch
+3.5-fold effect on 'Ambitious Instruction'
This feature was encouraging, since the correlation between %FR-lunch and 'ambitious instruction' was POSITIVE - meaning as the frequency of low income students in the population increases, 'ambitious instruction' becomes more likely. My personal interpretation of this is that teachers in low-needs schools may approach students with more traditional teaching methods, without suffering professional consequences or classroom consequences. Teachers in high-needs schools do not have this luxury.
Case in point - The Lab School (sorry not sorry)
Critical Feature 3: Percent of Bilingual Students
-3.3-fold effect on 'Ambitious Instruction'
This was the only critical feature in my model that was negatively correlated with ambitious instruction. The importance of this feature matches my own experience - I had honed my teaching style and curriculum in a school where fewer than 20% of my students spoke a language other than English at home. I then tried to make it work wholesale in a school where well over half my students spoke something other than English at home. Needless to say, it did not work, and my early attempts to force it to work probably reduced my student's trust in me (see critical feature 1).
What predicts where you will see 'good' teaching?
It's easy to imagine that this question has a simple answer - wherever you have good teachers, you see good teaching. All we have to do is identify good teachers and get rid of bad ones, and education will be fixed!
Teachers will cringe at this type of thinking, because we see our own effectiveness fluctuate constantly, from year to year, from day to day, based on a huge number of contextual factors - some within our control, some without, some predictable, some not.
I personally experienced a huge change in my teaching effectiveness a little over a year ago, when I changed schools. At my previous school, I had found it relatively easy to use 'best teaching practices' - pushing my students to take on more of the cognitive load, to explain themselves in detail throughout class, and to persist through struggle and failure as we engaged in difficult projects. At the new school, I encountered strong resistance from my students to these teaching practices, and I found myself falling back slightly to more 'traditional' teaching just to make my days more bearable.
In the following project, my guiding hypothesis is that 'ambitious instruction' is not a quality that is embodied by a particular teacher. I propose that 'ambitious instruction' is a complex aspect of school culture that can be nurtured or undermined, depending on the school environment.
Every year in Chicago Public Schools, teachers, parents and students fill out the "My School, My Voice" surveys collected by University of Chicago's Urban Education Institute. These surveys fall under what are considered "5 Essential Ingredients" a school needs to have in place to be considered well-organized for improvement.
 |
| Each school gets a rating in each of the 5 Essential metrics of a school. This image shows rating a high school in CPS that has high ambitious instruction despite relatively low ratings in other categories that correspond to a high functioning school. |
I wanted to know what predictors would most influence the metric called 'Ambitious Instruction.' This metric is based on student survey results, and essentially corresponds to how often students say they are asked to do rigorous, minds-on activities in class, and how much of the thinking load is placed on students.
The 5Essentials data is not readily available for public download, so I scraped it from the UEI website for each school in CPS. From the CPS data portal, I was able to get information about the type of school, whether the school is a charter school or board school, what grades are served. The dataset also included demographic data, such as % of students in special education, % of students with bilingual designation, and % of students on free and reduced lunch.
As I initially began developing a regression model, I experimented with different kinds of regressions. Since I had a lot of data features I was working with, I used a Lasso regularization model to identify features that could be eliminated, as shown in the figure.
 |
| This graph shows which features drop out of a regression model as the rigor of the model increases. The graph should be read from right to left. The features that drop out of the model are, in order of least importance, SQRPpts, CTU vs charter school, % special education students, and 'Involved Families'. |
Perhaps surprisingly, neither SQRP points (used to determine a CPS school's level rating, which is relevant for decisions about closure) nor being a charter school, ended up being particularly predictive for the Ambitious Instruction rating. The percent of students with IEPs also had low predictive power, as did how well teachers rated 'parental involvement' at the school. I dropped these features and optimized my model based only on the most critical parameters, eventually settling on a Ridge regression.
My final model had a typical error of ± 7.3 points. The 5Essentials scale change their qualitative level (the color from the graphic at the top) every 20 points, so in general my model will predict the correct level for a given school most of the time.
Critical Feature 1: Supportive Environment
+8-fold effect on 'Ambitious Instruction'
Supportive Environment is one of the only 5Essentials data features that persisted in my model. This feature is based on a combination of data from students and teachers. In general, the survey questions that build this feature are broken down into a few primary categories.
Safety - example: how safe do you feel in the hallways at school?
Student-Teacher Trust - example: do you agree - my teachers treat me with respect?
There are also questions that differ slightly for high school vs. elementary school - in high school, students are asked how much the school is oriented towards preparing for life and college after graduation. In elementary school, students are asked how invested their peers are in their school work.A supportive environment made a lot of sense to me as a critical feature of a school that can enable ambitious teaching.
Safety -> A student who does not feel safe in school does not have the real estate in their brain to engage in cognitively challenging classwork.
Trust -> The fact that respect and relationships are prerequisites for teaching students effectively is a truism in teacher education programs, but relationships seem to be particularly critical to getting students to take on the kind of challenging work that will be most beneficial for them in the long run. This correlation probably goes both ways, as students may have increased trust for teachers who challenge them and hold them accountable.
Critical Feature 2: Percent of Students on Free and Reduced Lunch
+3.5-fold effect on 'Ambitious Instruction'
This feature was encouraging, since the correlation between %FR-lunch and 'ambitious instruction' was POSITIVE - meaning as the frequency of low income students in the population increases, 'ambitious instruction' becomes more likely. My personal interpretation of this is that teachers in low-needs schools may approach students with more traditional teaching methods, without suffering professional consequences or classroom consequences. Teachers in high-needs schools do not have this luxury.
Case in point - The Lab School (sorry not sorry)
Critical Feature 3: Percent of Bilingual Students
-3.3-fold effect on 'Ambitious Instruction'
This was the only critical feature in my model that was negatively correlated with ambitious instruction. The importance of this feature matches my own experience - I had honed my teaching style and curriculum in a school where fewer than 20% of my students spoke a language other than English at home. I then tried to make it work wholesale in a school where well over half my students spoke something other than English at home. Needless to say, it did not work, and my early attempts to force it to work probably reduced my student's trust in me (see critical feature 1).
This is not to say that a large bilingual population makes ambitious teaching impossible. Rather, I suspect that there simply are not a lot of teachers in CPS who are well-trained in how to execute ambitious teaching practices in a classroom with lower English fluency. Frankly, having spent a lot of time pursuing such training, it's hard to find.
In conclusion, it seems there are a number of levers that schools can use to try to move the teaching culture of their school. By far, the most important thing a school should focus on is building a school with a strong community, where relationships are prioritized and time is dedicated to building relationships amongst staff and students. This may require taking steps like hiring teachers that are more representative of the student body, having advisory periods, and devoting time to assemblies and town halls that can build whole school culture and a future orientation.
A high percentage of bilingual students is somewhat anti-correlated with ambitious teaching - schools with high percentages of bilingual students may be well served by creating dual language programs, where students have the opportunity to learn subjects in their home language during the day. These are only feasible in schools with low diversity amongst their bilingual students, so most schools will probably need to pursue training for teachers that specifically address how to implement discussion in classrooms with mixed language fluency.
In conclusion, it seems there are a number of levers that schools can use to try to move the teaching culture of their school. By far, the most important thing a school should focus on is building a school with a strong community, where relationships are prioritized and time is dedicated to building relationships amongst staff and students. This may require taking steps like hiring teachers that are more representative of the student body, having advisory periods, and devoting time to assemblies and town halls that can build whole school culture and a future orientation.
A high percentage of bilingual students is somewhat anti-correlated with ambitious teaching - schools with high percentages of bilingual students may be well served by creating dual language programs, where students have the opportunity to learn subjects in their home language during the day. These are only feasible in schools with low diversity amongst their bilingual students, so most schools will probably need to pursue training for teachers that specifically address how to implement discussion in classrooms with mixed language fluency.
☆☆☆☆
ReplyDelete