Burn the computer: personalized assessment on paper

How can I adaptively target content to students’ needs without restricting assessment items to the lame formats — like multiple choice — that computers are able to read?

Given the richness of the Math Twitterblogosphere, it’s pretty hard to share something new that makes a substantial contribution to our online community.  I think I have something worth sharing here.  It answers the question above.   In other words, it’s a way to get (most of) the advantages of adaptive learning systems without all the drawbacks.

Dan Meyer has chronicled those drawbacks in many blog posts, for example, this one.  I like the way commenter Dan Anderson summed up the limitations of letting a computer assess student work:

A big advantage with meatsacks [read: human teachers] over computers is the ability of a human to look at the work. Computers can only indirectly evaluate where the student went wrong; they can only look at the shadow on the ground to tell where the flyball is going. Meatsacks can evaluate directly where the student is going awry.

And yet computers do have an advantage: it’s very easy for them to keep track of what each student needs to work on and to deliver practice or assessment that’s targeted to those needs.  Can we have the best of both worlds?

I’ve created a system that can make a unique printable mini-quiz for each student, depending on what skill they need to be assessed on.  It draws on an item bank, categorized by skill, that can be as large as you want so questions won’t be repeated on successive retakes.  Quizzes also print in order by students’ position in the seating chart, so you can simply walk down each row and breezily hand each student a personalized quiz.  (Not every quiz should be personalized, though.  At least half the time, I pick the topic and everyone gets the same quiz.  Personalized quizzes are for efficient retakes.)


The system is free, of course, and fully editable by anyone who knows how to work a spreadsheet.  Here’s how it works.  Each video is only a few seconds.


Step 1: Students select the skill they want to be quizzed on.  


Step 2: You display students’ current choices on-screen.  The screen updates live, so students who change their minds can see their most recent selection.


Step 3: You just copy and paste the Google Form responses into the quiz generator.


Step 4: With a simple CTRL + P, you print the entire class set.  It automatically prints in order by seating chart.


Step 5: Updating your seating chart is easy.  Changes to the seating chart automatically update the printing order of the quizzes.


Step 6: On the next quiz, increase the “quiz generator key” by 2.  This will change the questions given for each skill.

Step 7: Grading tool.  This speeds up your grading process by more than a factor of 10. Duuuuude.  A factor of 10.  (Turn the volume on to listen to this screencast).

the files you need

If this post gets decent page views, I’ll come back in and write some tech support pieces to explain how to use all the features: how to add assessment items with images (it’s not trivial to add images into a cell of a spreadsheet); how to link up the spreadsheets correctly; and how to toggle all the various options in the program.

is this not overkill?

I’m pretty sure it’s not.  Let me just nail my 95 theses to this door here and see what you think.  Here goes:

  1. Students should not grade their own formative assessments. An expert needs to grade them.
  2. That expert should be a human, not a computer, for reasons given above.
  3. With my current grading and prep load, I’m already maxed out on how much grading I can do. I can’t check huge numbers of ungraded formative assessments in addition to grading the tests/quizzes I already give.
  4. Therefore, formative assessments must replace some of my existing grading load, not add to it.  They have to count in the gradebook.
  5. But if they’re graded, they won’t really be formative unless students can do retakes and earn credit for improving.
    • [My conclusion]  Formative assessments must be graded tests or quizzes that students can retake.

  6. Should they be tests, or should they be quizzes? Many teachers use a formative assessment system with tests.  Here’s Dan Meyer’s version.  Let’s think about that. (If you think Dan’s is not the best example, let me know in the comments.  I don’t want a straw-ish man here.)
    • Advantage: Tests can be comprehensive. Each test can assess the full range of skills covered so far.
    • Big disadvantage: Tests aren’t very frequent. Ideally, students would be able to relearn something and then earn credit for demonstrating proficiency within a couple of days, instead of waiting for the next test.
      • Of course, you could have a policy that students may always come in informally outside of class to demonstrate mastery, but many teachers find that students don’t really bother to come after school to do that.  In fact, I think if all the students who should come really did, it would overwhelm my ability to informally generate assessments after school.
    • Here’s a bureaucratic reason that tests might be the wrong vehicle for formative assessments: in many districts, teachers don’t have control over the tests they give.  There tends to be more flexibility and independence around teacher-generated quizzes.
  7. Okay, let’s consider using quizzes as formative assessments.  Advantages: they’re more frequent, and you can still use your district’s tests.   But lots of disadvantages, too.
    • Advantage: Shorter, more frequent assessments are better for learning.  Or so says Marzano.
    • Big disadvantage: How will retakes work?   If you have 10 skills this quarter, and you quiz a different skill each day, a student might need to wait up to 10 class days for the chance to retake the skill they’re ready to re-do. That’s unacceptably long.
    • Logistical disadvantage:  Even though frequent assessment is good for learning, how can I squeeze quizzes into the last 10 minutes of class consistently without losing too much instructional time?  These quizzes need to be very quick to hand out (and to pass back, once they’re graded).
    • Solution: you need a way to let students pick the quiz topic they want to retake, so that on some days, different students can take different quizzes. If this happens frequently, students can relearn and reassess in a tight loop lasting no more than a few days.
    • Logistical problems:
      • Imagine laying out 10 stacks of quizzes on the counter, or on your teacher desk, and inviting each row of students come up and pick a quiz. If these quizzes are short (4 questions or so), the first students may be done by the time the last students have picked their quiz.
      • Entering grades in the gradebook is a challenge. Try typing 120 grades into the gradebook, in up to 10 different columns, while overwriting old grades (with a grading program that has no “undo” button), without making a single mistake.  Not easy!
      • In addition, managing answer keys is a huge problem here. Try grading 5 class sets of quizzes in 10 minutes per set, when you need to make 10 different answer keys and then flip between those 10 keys to check students’ quizzes.
  8. So maybe my assessment tool is not overkill after all.
    • Even if I want to assess a single skill, I can toggle an option to print out 2, 3, 4, or more different versions of the quiz, to reduce opportunities for cheating.
    • There’s a tool to help organize grade entry.
    • The tool to manage answer keys when grading a class set has not has been created.  See screencast 7 above.
    • Here’s how I handle passing back daily quizzes quickly: students turn their papers into a tray specific to their seating area (left, middle, or right).  When I grade the papers, I keep them grouped like that.  Then when I want to pass them back, they’re already grouped by seating area, and I’m not traversing the room 10 times to pass them all back.  I can pass back a class set in 1 minute.

does this fix the real problem?

The root problem is that it’s hard to get kids to take the initiative and fill in their own skill gaps, even when you identify them.  Here’s Michael Pershan, over at his blog

Second, I don’t think the feedback itself given in SBG [Standards-Based Grading] is helpful to kids. What’s the path from “You’re a beginner at solving linear equations” to actually learning to solve a linear equation? Some say that kids will go home and study linear equations more if you tell them they’re bad at them, which doesn’t fit with what I know about high school students. But maybe your kids are different than mine.

Not only do I agree with Michael here…I also designed this entire project as a response to his critique and Dan Meyer’s larger criticism of adaptive learning systems.

Here’s why, in my classroom, the system I’m presenting seems to avert the pitfall Michael’s pointing to.  After letting students choose their retake skill on the Google Form, I let students go to different stations with study guides for their chosen skills.  There’s something about signing up for a skill’s retake, and then immediately diving into that skill’s study guide (starting with circling the ones you got wrong last time) that seems to lead students to feel there’s a point in trying to relearn the skill.  That what’s being asked of them is a manageable bite.

And I don’t mind making everyone do a retake, even those who had 100’s on everything.  Short, frequent quizzes are good, thanks to the testing effect.

Cri de coeur

I’ve never taken a coding class.  Millions of people out there could have done this better than I did.  But even if I felt like waiting a few more years for a good formative assessment solution, I don’t even see one on the horizon.  So I made my own.  In the last 3 years, I’ve created this quiz generator, written all the quiz items (most of which I’m not publishing here for test security), made the Khan Academy grading tool in the previous blog post, and tried to rewrite as many lessons as possible to make them better.  That’s a lot of time spent on tools and resources.  As a teacher I’d prefer my extra time be spent on the kids rather than the tools.

Relatedly, it’s not really my dream that lots of other teachers start to use this program.  My dream would be for assessment companies like MasteryConnect to include these features in their own programs so doofuses like me didn’t have to build their own quiz generator (and so teachers had a convenient platform for sharing quiz questions instead of writing them all from scratch).  But almost every edtech company out there is pushing for everything to be done online.  A paper-based assessment system with human graders just isn’t that interesting to them.

*Note about the title of this post: if you know me, you know I’ve worked hard to find a way to make Khan Academy a useful tool for my Algebra 1 students.  So I only want to burn the computer when it comes to real assessments.  As a practice tool, computerized exercises are fine with me.

Yes, this post needs an image so it looks better on twitter


Genuinely differentiated grading with Khan Academy

I hope this is my last post on Khan Academy for a while.  It’s not that central to my teaching (I’d rather be writing about Desmos or something).  But I do think the tool I’ve designed to differentiate grading with Khan Academy may be useful to some folks out there.

I’ll link you to the screencasts for how it works, and then to a Google link for the actual spreadsheet file in Google Drive, but here’s the gist:

  • I don’t let Khan Academy automatically recommend exercises for my students to practice.  I want to be in charge of selecting what kids work on.
  • Khan’s main value is its memory quizzes, called “Mastery Challenges”, that check if a student has forgotten something we’ve learned (if they’ve forgotten a skill, it gets added back onto the student’s agenda).
  • But different students need to be held to different standards of retention and accuracy, per IEP’s and observations.
    • My new spreadsheet allows me to exempt some students entirely from these memory quizzes, and allows other students to earn full credit with reduced expectations of retention & accuracy.  Meanwhile, most students are still held to the full standard.
    • In addition, students can be exempted from the hardest exercises on an assignment.
    • On the opposite end, students who are really advanced can go ahead and earn extra credit by working on Khan Academy’s automatically recommended skills…but only after they have completed the assigned skills.
  • Here are the screencasts for my new grading tool. One thing to know: this year’s improvements have made it a very easy to system to maintain.
  • This spreadsheet explains how the teacher-facing grading system works.  The student-facing end is different.  It’s a technique for hacking around Khan Academy’s automatic recommendations and instead forcing kids to do the exercises you want them to do.  You can find a description here at this blog post.  It’s different than using Khan’s “Teacher recommendation” tool.  That tool does not re-add a skill you’ve recommended when the student fails the retention quiz on the skill.  So if your goal in using Khan Academy is to focus on retention, their “teacher recommendation” tool is useless.
  • Khan Academy’s content started weak, and some of you may not feel it’s ready for your use yet.  Depends on the course.  Apparently, AB and BC Calc were rewritten this summer, though I haven’t checked them out.  Algebra 1 is currently being rewritten, but those changes have not gone live yet.  In my spare time, I work as a volunteer to help them identify Algebra 1 improvements that need to be made.  There are many.  In August, I created a 30-page document suggested changes for about 25% of the course.  We’ll see how many of my suggestions they take.

Here is the link to the actual spreadsheet.

Future work: How can we add a feature that automatically pairs students up so each member of the pair has an assigned skill they’re able to teach the other?  All the required data is there in the spreadsheet, but I can’t figure out an algorithm that makes it work.

Creating Intellectual Need for Multiplying Binomials

I’ve always needed a way to motivate the study of quadratics.  In the past, I’ve used materials from some of Dan Meyer’s 3-Acts: Super Mario to get students to realize that linear predictions are sometimes wrong, and Will It Hit the Hoop? to specifically focus students on quadratic graphs.  But even to my teacher ears, the jump to actual quadratics skills sounded cheap: “Now that we all agree quadratic functions are important, let me teach you to multiply things like (x+1)(x+2), because it’s really important for understanding parabolas, and I’ll explain why later.”  Groan.

I’d like to share a new lesson that I really liked because it:

  • Naturally focuses students on area models of quadratic expressions;
  • Shows that quadratics are the way to model something that’s speeding up or slowing down;
  • Has a really low barrier to entry.

A low barrier to entry means students can dabble their toes in this concept pretty easily at the start, without encountering hard math until they’ve played around a bit.  Before we go on, let’s check that this blog post is worth your time.  Here is the whole lesson I’m about to describe, fast-forwarded to be just 2 min long:

Still interested?  Cool.

The most direct way to (a) introduce area models of quadratic expressions, and (b) make it seem like quadratic expressions are useful is to pose a question that’s directly related to area.  Something like: Farmer Joe has 100 feet of fence and wants to make the largest sheep pen he can.  What length and width should he use for the pen?   [The answer is to model area as A = (L)(W) = (L)(100-2L) = 100L – 2L2 , graph the quadratic function, and find its vertex].

In my experience, the Farmer Joe question doesn’t arouse much natural curiosity from students, and I think I know why: even students who naturally enjoy math puzzles have no inkling at the outset of their inquiry that their solution method will also help them understand the many faces of quadratics: projectiles, cars speeding up or slowing down, the famous handshake problem, etc.  It’s not until you’re well into the problem, and you see that the graph of area vs length looks like the flight path of a projectile, that you have a chance of recognizing how significant quadratics might be.  And by that point, you’ve already done enough hard math that you might be a bit tired or grumpy.  Learning quadratics should be like hiking to a beautiful vista: look at all the things I can see from up here!  The ahhhh experience of arriving at that vista needs to come sooner in the introduction or students end up feeling the way I felt on my last hiking trip in Montana: that’s a great view, but OMG I hate mosquitos–let’s get the #@%! out of here.

If you’re learning quadratics after learning linear functions, then the best way to notice you’re at a pretty awesome vista is to see that you’re looking at a pattern that’s accelerating.  A pattern that’s accelerating is noticeably very different than all the patterns we’ve done so far.  My class starts linear function  by looking at dot patterns like Fawn’s–specifically, we focus on ones that visually distinguish the y-intercept from the slope.  For example, looking at the pattern below, how many dots would be in Stage 10?  Stage x?

Dot pattern example

Students get really used to asking, “How fast is the pattern growing?” or “How many dots does it add each stage?”  We also do modified versions of Stacking Cups and Barbie Bungee to keep emphasizing that finding the rate is crucial for making a prediction.

In addition, the narrative in my room is that algebra is a way to predict the future by finding and expressing patterns.  For example, when we study direct varation early in the year, students actually make short videos of a prediction experiment in their own lives.

Okay, against that backdrop, I present students with the following lesson to try a prediction that finally breaks the constraint of using constant-rate patterns and motivates area models for polynomial multiplication.  Here’s the full, narrated video overview of the lesson.

Update 6/21/16: Here’s a Desmos activity to go with the visual “dot pattern” section at the end.

Room for improvement:  As I was transitioning students to (x+2)(x+3) and drill problems, I felt that even though I’d gotten students to the vista, I need to do a better job of showing them everything they can see.  What if they think these area patterns only work when the first difference in the pattern goes like +1, +3, +5, etc?  I should show that if the first differences go +2, +6, +10, etc, then you can use 2x2…visually, just draw two of the x2 patterns.  If you wanted +1, + 2, +3, you could use (1/2)x2  by drawing the x2 dot pattern and then cutting it in half.  I should also make the connection to accelerating cars, psychology’s inverted U-shaped graph of stress vs performance graph, Farmer Joe, and everything else that’s quadratic.   However, I think that’s best saved for the next lesson.  We teach roughly 90-minute blocks, and I like each block to have some conceptual development and some practice.  When you see kids every other day as it is, you need to squeeze in some practice to each lesson.  So in the future, we’ll transition to (x+2)(x+3) and do drill just as shown above, but the following lesson I’ll take time point out all the landmarks you can see from this vista.

CCSS thought: I’m not sure how this lesson would play in a Common Core state.  Do you do arithmetic series in Algebra 1, and if so, do you do them before quadratics?  That would probably make this whole shtick might seem kind of lame.  We don’t do CCSS here.  Our state test doesn’t really assess comprehension much, so I’m not sure how much this lesson will even improve my students’ standardized test scores. My students have always been able to multiply binomials without experiencing an intellectual need for doing so.  But this lesson just felt so satisfying.  I hope it’s been worth your time to read about it.

Sharing the file: I’m happy to share a copy of the powerpoint file to anyone who’d like it.  Just ask in the comments.

How I made Khan Academy less adaptive (and way better)

I want to share 2 tricks I have come up with for making Khan Academy a really great homework system.  The first trick is very simple, and I’ll describe it here.  The second involves a really complicated spreadsheet, but now that I’ve made it I think you should be able to start using it almost immediately.

The adaptive aspect of Khan Academy makes it almost unusable for me in the classroom.  Because the adaptive software picks students’ next exercise, what the system picks may have nothing to do with what I’m teaching this week in class.  Now, KA does have a way for teachers to add an exercise to students’ dashboards: you “recommend” an exercise to a student, and it shows up on top of their agenda like this:

KA recommendation

But here’s the thing: the way this feature is implemented actually defeats the main advantage KA offers over traditional pencil-and-papeer homework.  What is that advantage?  While it’s terrible for teaching new concepts to students, Khan Academy is pretty great at detecting when they’ve forgotten something.  The system includes a built-in generator of adaptive quizzes (called “mastery challenges” in Khan parlance) that check whether a student still remembers something she may have learned a few months ago.

Mastery challenges intro

So here’s the problem with the teacher recommendation feature of Khan Academy: yes, it lets you add an exercise to the top of a student’s agenda — but once the student achieves that initial success, she no longer sees that exercise on her dashboard, even if she later shows that she has forgotten the skill and needs to re-do it.

Here’s a really simple trick for getting around this: first, have your students add their own usernames in their list of “coaches”.  Once they do this, you can post a link to a coach report that is filtered for just the exercises you want them to do.  For example, here is a link: http://bit.ly/1SzQw8F.  You will not be able to access the link unless you have a Khan Academy account and have at least 1 “student”; if you don’t have any students on KA, just add your own username as your coach, and you’ll be able to view the link.  I’ve found that Bitly is a good way to post the link because the length of the links overwhelms my school’s website hosting platform.  Students will click on that link and pull up a report that shows their progress on only those exercises.

Coach report intro

All non-assigned exercises are filtered out, and the report updates (with a browser refresh) as soon as a Mastery Challenge changes the skill level in any exercise.

In my class, I post 3 links per week: 20-point exercises, 4-point ones, and 2-point ones.  There are usually about 4 exercises in the 20-point category per week.  These are new exercises, and they are the core that I need everyone to learn for the week.  The 4-point exercise link is cooler, from a teacher perspective, because it contains every 20-point exercise I’ve ever assigned the class.   If a Mastery Challenge shows that you have forgotten a skill, then that skill’s bar may turn gray on the coach report for 4-point exercises.  In that case, you’d need to go back and re-do the skill from scratch before trying to level up on it again.  That’s really where Khan Academy pays off: it has this great built-in detector of student retention and forgetting.  And, increasingly, it has high-quality practice on skills your students should already have learned through your lessons.

The 2-point exercises are challenging ones I’ve selected for ambitious students to try if they’re done with everything.  They’re related to what we’re learning in class but go beyond our expectations.  Students who complete the 2-point exercises can earn extra credit by working on exercises automatically recommended by Khan Academy on the student dashboard.

So that’s the simple trick.  In a later post, I’ll describe how to use the spreadsheet I’ve designed to assign points for different exercises based on the downloadable report in the top right corner of the “Student Progress” report on Khan Academy.  [That post is here.]  Perhaps I shouldn’t say this, but I do hope at some point some KA people actually read these ideas. There’s no reason why it should take so much hacking to expose (what I think is) their site’s main benefit to students.

11/9/15 Update: For those interested, one of the KA employees in charge of the Mastery Challenges system describes the way they work here.

10/25/16 Update: i significantly changed description of the point allocations (20 points, 4 points, 2 points) to match what I now do.  It’s been an improvement.  I also deleted the description of the 2-week cycle (Week A and Week B) of each assignment, because I now require students to go from grey to dark blue in a single week.

Is Cognitive Load Theory helpful?

This is a follow-up to Dan Meyer’s twitter conversation a few days ago about cognitive load theory:
My thoughts: I suspect that the difference between germane and non-germane cognitive load can be detected on an fMRI machine.  You’d first need to see what parts of the brain light up when a student is thinking about something germane.  Then just check whether the activity in question makes those (germane) areas light up more, or whether it makes those light up only a little and instead mostly consumes the region of your brain that helps you interpret a cumbersome computer interface.
This kind of stuff is not that far-fetched.  For example, here is an artificial intelligence program using nothing but fMRI input to predict what algebraic steps a student is taking.  So not only is it determining whether the student is thinking about something germane, it’s actually identifying exactly what the student is thinking…and (here’s the kicker), often BEFORE the student has actually recorded those steps on the computer screen.  Basically: mindreading.
And for some context, here is the researcher’s descriptions of what the split screens represent in the video, and here is a link to the research project.

Teaching for Understanding vs. Teaching for Reasoning Skills

In my last post, Dan Meyer and I discussed whether having students make and test their own conjectures can lead to poor long-term content learning.  I think it easily can, not because of poor teaching but because of humans’ limited working memory.  Dan’s reply captured the inquiry-learning perspective perfectly, and I made a first pass at replying in the comments but promised to reply with more later.  Here goes:

Conceptual Understanding is Fragile

Dan writes:

If calculating LCMs were my highest goal here, I would turn to other strategies, including lecture and definition. But calculating LCMs is secondary to conjecturing and testing your conjectures. That’s the higher goal here.

Can you tell me what help you see direct instruction offering me there?

In short, Dan seems to be saying that lecture is great for teaching computational skills, but that he’s willing to sacrifice some efficiency in computational learning in order to develop students’ reasoning skills (e.g., testing conjectures).  That’s great, but it skirts the question of improving conceptual understanding, which is a totally separate dimension.  Even students with good reasoning skills (the habits of mind that lead to productive inquiry) and strong computational fluency can have poor conceptual understanding.  This happens when they regularly go through the instructional sequence Dan lays out: inquiry until students make the desired discovery, followed by notes and drill practice on the skill.  The assumption underlying this approach is that once students have made a discovery for themselves, they understand it deeply enough to move on to application and drill practice–that having discovered a concept naturally leads to having a strong, durable conceptual understanding of it.

In reality, conceptual understanding is fragile.  Students need practice retrieving the reasons for their conclusions in different contexts to establish them in long-term memory and get them connected to related conceptual schema that are already there.  The mere fact of having made a discovery doesn’t guarantee I’ll remember the reasons for it tomorrow, nor that I’ll think to transfer that understanding to related situations.  I still need lots of practice explaining “why?” and “how would it be different if…”, as well as “would the same pattern apply in this situation?” and “how could you represent that another way”.

Too often in inquiry lessons (including my own), this practicing of the reasons is relegated to the whole-class debrief in which small groups describe their thinking while sharing out from the investigation.  I know this is an attempt to provide conceptual practice, but it’s nowhere close to what’s needed.  For any given “how would it be different if…” question, at least 50% of students probably don’t understand, but because they successfully made the desired discovery, they (and the teacher, often including me) accept it.   How much formative assessment do teachers do in this debrief phase?  If incorrect reasons pop up in the discussion, do they just let another student speak up to correct the record, or do they stop and reteach that reason to mastery?

In short, the discovery is not the lesson.  It’s just the set-up for the real lesson, which is when we rehearse the reasons for what we’ve concluded and how it’s connected to everything else we know.

 Dan’s Challenge: How Can Direct Instruction Teach Conjecture-Making?

 To go back to Dan’s comment,

But calculating LCMs is secondary to conjecturing and testing your conjectures. That’s the higher goal here…Can you tell me what help you see direct instruction offering me there?

My short answer is that since cognitive load is the issue, you’d want to instruct students directly on any techniques they could use to reduce their own cognitive load: making organized lists, searching through the problem space in an organized way, etc…the sort of things we encourage students to do anyway.

Secondly, I’d say that direct instruction of conceptual understanding (the sort I referred to as the “real lesson” above) probably helps students make and test conjectures.  Better understanding leads them to ask better questions during the inquiry phase.  I haven’t done a lot of reading on this, but one paper I remember from grad school shows that people with poor content knowledge tend to ask shallower or less relevant questions.  Here’s the abstract:

Questions should emerge when a person studies a device (e.g., a lock) and encounters a breakdown scenario (“the key turns but the bolt doesn’t move”). Participants read illustrated texts and breakdown scenarios, with instructions to ask questions or think aloud. Participants subsequently completed a device-comprehension test, and tests of cognitive ability and personality. Deep comprehenders did not ask more questions, but did generate a higher proportion of good questions about plausible faults that explained the breakdowns. An excellent litmus test of deep comprehension is the quality of questions asked when confronted with breakdown scenarios. 

I’m sure there’s better research out there on this topic, but the main idea would basically be that students at a low Van Hiele level would be inhibited from making good conjectures by being literally unable to perceive the deep features of the scenario.  Wouldn’t those students develop their conjecture-making and testing abilities more if they had better conceptual understanding?

Please note that I’m not an advocate for direct instruction, just for attending to what students are really thinking, even when they appear to have made the right discovery.  If I’m an advocate for anything, it’s for paying attention to cognitive factors in learning, not because they’re more important than motivational ones, but because I think the MTBoS community sometimes gives them short shrift.  So cognitive load is my thing, much more than direct instruction.

Have They Encoded the Wrong Rule?

See Dan’s comment in my previous post for the context, but no, I don’t think they have.  When you learn the rule that for numbers like 2 and 10 (in which one is a multiple of the other), the LCM is just the larger number, you’re actually learning two separate facts: the rule, and when the rule works.  Learning the first and not yet knowing the second doesn’t mean you have the rule wrong, it means you’re ready to make your next discovery.

Dan’s “Shipping Routes” and Cognitive Load

During Dan’s series of lesson makeovers last summer, I intended to write up a critique of the Least Common Multiple makeover he and Dave Major created. Dan’s write-up seemed to illustrate his view of the respective roles of inquiry and direct instruction, and I’ve always thought it showed a misinterpretation of the cognitive research that cautions against unguided inquiry. I didn’t bother chiming in until Grant Wiggins wrote this blog post, bringing up the issue of cognition, working memory, and inquiry vs. direct instruction again. So here are my thoughts.

For context, Shipping Routes shows students the clip below and asks whether the two boats will ever get back to port at the same time:

Students are then sent to a simulator programmed by Dave Major, allowing them to choose different round-trip times for the two boats and showing their cycles back and forth. Students can discover for themselves when the boats get back in sync, and what that has to do with least common multiples. For example, boats with round-trip times of 2 min and 5 min will be back in sync after 10 min. Sadly, Dave’s simulator seems to be offline right now, so you can’t try it for yourself.

Dan’s approach

In his write-up, Dan shies away from direct instruction right from the get-go:

I could tell students what to look for here and how to approach the problem. I could show a few worked examples…

Two problems there:

  1. Some students will need more than just three examples to determine a pattern.
  2. My selection of those particular examples – that is, my decomposition of the entire solution space into just three categories – did a lot of the intellectual heavy lifting for my students. They need to decide on those three categories and come up with a rule that takes them all into account.

Regular readers of Dan’s blog (cuz, yeah, there really aren’t regular readers of my blog) may know that worked examples are an interest of mine. Here’s how the stuff I’ve learned applies to Shipping Routes.

Science of Memory and Learning

Humans have 2 kinds of memory, working memory and long-term memory, and they function completely differently. Retrieving information from long-term memory is almost effortless, and we can process huge amounts of information from long-term memory simultaneously. In contrast, working memory can only hold on to 4-7 chunks of information, and only for about 30 seconds or so. The word “chunks” is important. For a demonstration, watch the first 1:30 minutes of this video:

There are various theories about how information gets encoded in long-term memory, but the main idea is that the information has to stay in working memory for long enough to get practiced/recalled several times before the working memory dumps it–each practice opportunity strengthens its foothold in long-term memory. Working memory will dump it when new information comes in, so if you keep throwing new information at someone who’s still encoding the previous information, you make it very hard for them to form long-term memories, even if they seem to be processing what you’re saying as you’re saying it. It could have made sense to them when you said it and then disappeared from their mind when you said the next thing. (Sound familiar to any teachers?)

For that reason, inquiry instruction can make it very hard to encode information into long-term memory. As Sweller et al (2006) say in paper that spawned much controversy, “Inquiry-based instruction requires the learner to search a problem space for problem-relevant information. All problem-based searching makes heavy demands on working memory. Furthermore, that working memory load does not contribute to the accumulation of knowledge in long-term memory because while working memory is being used to search for problem solutions, it is not available and cannot be used to learn.”

Grant Wiggins really does not like this paper–much of his most recent post (which inspired me to dust off my keyboard here) is spent picking it apart, particularly the “ludicrous” emphasis Sweller et al place on novice vs. expert learners. I think Wiggins misunderstands the authors on that point. Granted, I’m a layman, but I think the difference between novice and expert learners is pretty simple: isn’t it the difference between someone who’s encoded the information being studied into long-term memory and someone who’s still processing it from working memory?

The strong opinions (not limited to Wiggins by any means) reflect the fact that this theory of memory and cognition points out shortcomings of inquiry learning. As I’ll describe at the bottom of this post, it does not actually mean that you should never use inquiry, but some people on both sides of the debate say it does, particularly when the theory itself is misunderstood.

Critiquing Shipping Routes

How would this play out in Shipping Routes? In order to be successful with Dan’s lesson, students need to try different pairs of round-trip times on the simulator to discover what governs when the boats are back in sync. Students can set times to the tenth of a second, e.g, 3.2 min for the first boat and 4 min for the second boat. These boats would by in sync for the first time after 16 minutes. Would they notice how 16 arises from 3.2 and 4 as the least common multiple?

Let’s give the lesson the benefit of the doubt and say that a teacher would suggest (or have a student suggest) that everyone try whole-number times to start. So one group of students might, for example, try 2 min and 6 min. and discover that the answer, 6 min, is the larger number. At this point, Dan would have the teacher challenge this group to see if that rule always works, with an eye toward finding counterexamples. Dan says,

If a student just tries the first example and says, “It’s easy. It’s always the longer of the two times.” I can then say, “Great. But try that on several more examples and make sure it works.” (It won’t.) Or I can suggest one of the other two categories. But I’d rather not offer those categories before the student has even considered why she might need them, or even the fact that there are different categories.

The other categories are coprime numbers (like 2 and 5) or numbers with a common factor (like 6 and 10). Students have to identify the three situations and find the rule for each, with minimal guidance.

Do you see how this might overtax working memory to the point of inhibiting long-term memory formation…how students might successfully find one or two of the three rules on Monday and then walk in on Tuesday having forgotten what they’d discovered? Or find all three rules, but be unable to remember the first one by the time they find the last one? If you don’t think this would tax their memories, keep in mind that many students have to stop and think or count on their hands just to remember that (3)(9) = 27.

Modifying Shipping Routes

How could Shipping Routes be adapted so it accounted for working memory limitations? Here’s what I would do. Whatever rule students discover first, before encouraging them to find cases where their new rule DOESN’T work and where they need to create a new rule, I’d have them fully digest the math behind the rule that they have found. They’ve just found this rule. It’s fragile, sitting in their memory buffer temporarily–this is our chance to have them encode it into their long-term memory and harden it by connecting it to other conceptual schema already in their brains. (Yes, I do believe strongly in developing conceptual understanding in addition to mere skill development). If we have them switch to a new search for a new rule, we throw away most of the benefit of the discovery they just made.

How would I help them practice it and connect it to other concepts? After students played around with the simulator for a while, I’d present them with text input boxes like this and ask students to fill in the blanks:

Boat 1: 2 min

Boat 2: ______

Time until they’re in sync: _____

A computer could easily analyze student responses and figure out which rule they’ve discovered. Let’s take the example of a student who discovered that for coprime numbers like 2 and 5, you multiply the numbers. Now I’d want to have the students really dig into that rule.

For example, I could ask them to re-represent the combination they came up with on a double number line:Double Number Line

Then I’d ask them to self-explain the connection via drop-down menus, as in Martina Rau’s paper finding that multiple representations didn’t help students learn fractions unless students were prompted to self-explain the connections between the representations. (The picture shows drop-down menus for self-explaining adding fractions, but you could do something similar for explaining why 2 and 5 have a common multiple at 10):

Martina Rau

Then I’d have students practice this rule a minimum of 5-10 times, perhaps following the approach of this other paper by Martina Rau.

And then, finally, I’d have the online lesson challenge these students to find a case that breaks their rule, and the learning cycle would begin again.

So When & How Should We Use Inquiry?

As I said above, understanding working memory limitations doesn’t mean you should never use inquiry. Rather than spelling out my views on this here, I’ll just point you all again to Dan’s posting of my thoughts on this topic over at his blog. Incidentally, I did try to convince some researchers in the learning sciences to use Dan’s makeover lessons as a test-bed for studying instructional principles, and I still think that would be a great idea (see my first blog post ever). I mean, researchers, if you want teachers to actually pay attention to your work, why not conduct your studies in the context of the lessons we’re all talking about online?

Update: Dan’s reply from the comments:

“Whatever rule students discover first, before encouraging them to find cases where their new rule DOESN’T work and where they need to create a new rule, I’d have them fully digest the math behind the rule that they have found. They’ve just found this rule. It’s fragile, sitting in their memory buffer temporarily–this is our chance to have them encode it into their long-term memory and harden it by connecting it to other conceptual schema already in their brains.”

But you have them encoding an incorrect rule!

If I have any recurring complaint about the literature around direct instruction it’s that it doesn’t account for the effect of direct instruction on student motivation. If I have a /second/ complaint it’s that it doesn’t adequately account for the difference between what teachers /say/ and what students /learn/. It’s like, “If the students didn’t learn what the teacher said, the teacher either needs to say it again or say it better.”)

“Do you see how this might overtax working memory to the point of inhibiting long-term memory formation…how students might successfully find one or two of the three rules on Monday and then walk in on Tuesday having forgotten what they’d discovered?”

I wouldn’t count on students being able to calculate LCMs based on this activity alone. They’d need plenty of fluency practice also, which would give me more opportunities for direct instruction. If calculating LCMs were my highest goal here, I would turn to other strategies, including lecture and definition. But calculating LCMs is secondary to conjecturing and testing your conjectures. That’s the higher goal here.

Can you tell me what help you see direct instruction offering me there?

My initial reply (more to come later):

@Dan, there’s lots to chew on in your reply. I’ll have to reply in pieces, because I need to pick up a certain little guy from daycare.

I agree that direct instruction literature often fails to account for student motivation–that’s important, and it’s what led me to slowly adopt lots of your techniques. I’m more skeptical of your second complaint. You say that the direct instruction literature “doesn’t adequately account for the difference between what teachers say and what students learn.” But this is a cognitive concern, and cognition is where direct instruction literature has is strongest results. That’s the area where we find numerous studies showing stronger or more efficient learning than you get from inquiry learning.

Perhaps you’re saying that direct instruction is efficient at teaching skills, but that it bombs when it comes to teaching what those skills mean and how they’re connected conceptually. If so, that’s pretty much how I interpret Grant Wiggins’ post, too.

Here’s why I disagree with that. The way I see the original Shipping Routes lesson playing out is that we’d do that activity, there’d be lots of good conjecturing and testing/discussing of conjectures, and then we’d start taking notes and doing fluency practice. For the reasons I described above, I think lots of students would be starting essentially from zero at that point. Lots of good conceptual wrestling would have been done during the investigation, but little of the understanding would have stuck in students’ long-term memory, so what would actually end up sticking would be the explanations delivered in note-taking and fluency practice. Too little guidance during the inquiry phase forces me as teacher to do too much “knowledge dumping” at the end, when I just try to pour the concepts into kids’ heads.

If you really want to attend to the differences between what teachers say and what students learn, you have to give them lots of feedback at the moment they’re constructing the concepts. This calls for inquiry that is more guided. We still want to achieve the Generation Effect and the euphoria of figuring things out for yourself, but we want it to stick, too.

I’ll have to respond later to your first objection, that they’d be encoding an incorrect rule, and to your closing question about how direct instruction could help with the metacognition you prioritize. Happy Friday, and thanks for taking the time to read the post and respond.

Anybody else feel like responding, or do I have to get my mom to comment on here 🙂

Update^2: For the rest of the reply, see my next blog post here: Teaching for Understanding vs. Teaching for Reasoning Skills