This material is in early beta: over 300 suggestions and corrections are waiting to be folded in, some quite significant. Changes should be in place by July 2018, at which times printed copies and downloadable electronic copies will be made available.

Individual Learning

After reading this chapter, you will be able to

  • Explain what metacognition is and why it is important to learning.
  • Explain what near and far transfer are, and correctly identify which one occurs most often.
  • Name and explain six strategies learners can use to accelerate their learning.
  • Explain why working long hours reduces productivity.
  • Define calibrated peer review and explain its benefits for learning.
  • List common myths about computing education.

The previous three chapters have looked at what instructors can do to help their learners. This chapter looks at what learners can do for themselves by changing their study strategies and getting enough rest.

The key to getting more out of learning is metacognition, or thinking about one’s own thinking processes. Just as good musicians listen to their own playing, and good teachers reflect on their teaching (Chapter 8), learners will learn better and faster if they make plans, set goals, and monitor their progress. It’s difficult for learners to master these skills in the abstract—for example, just telling them to make plans doesn’t have any effect—but lessons can be designed to encourage certain study practices, and drawing attention to these practices in class helps them realize that learning is a skill that can be improved like any other [McGu2015].

The big prize is transfer of learning, which occurs when something we have learned in one field helps us learn more quickly in another. Researchers distinguish between , which occurs between similar or related areas like fractions and decimals, or loops in different programming languages, and far transfer, which occurs between dissimilar domains—the idea that learning to play chess will help mathematical reasoning or vice versa.

Near transfer undoubtedly occurs—no kind of learning beyond simple memorization could occur if it didn’t—but as [Barn2002] showed, discussion of far transfer is muddied by conflicting definitions. [Sala2017] recently analyzed many studies and concluded:

the results show small to moderate effects. However, the effect sizes are inversely related to the quality of the experimental design We conclude that far transfer of learning rarely occurs.

When far transfer does occur, it seems to happen only once a subject has been mastered [Gick1987]. In practice, this means that learning to program won’t help you play chess and vice versa.

Six Strategies

Psychologists use a variety of approaches to study learning, but have wound up making the same recommendations about what actually works. [Mark2018] The Learning Scientists have catalogued six core strategies and summarized them in a set of downloadable posters. Teaching these strategies to students, and mentioning them by name when you use them in class, can help them learn how to learn faster and better [Wein2018].

Spaced Practice

Ten hours of study spread out over five days is more effective than two five-hour days, and far better than one ten-hour day. You should therefore create a study schedule that spreads study activities over time: block off at least half an hour to study each topic each day rather than trying to cram everything in the night before an exam [Kang2016].

You should also review material after each class (but not immediately after—take at least a half-hour break). When reviewing, be sure to include at least a little bit of older material: for example, spend 20 minutes looking over notes from that day’s class, and then 5 minutes each looking over material from the previous day and from a week before. (Doing this also helps you catch any gaps or mistakes in previous sets of notes while there’s still time to correct them or ask questions: it’s painful to realize the night before the exam that you have no idea why you underlined “Demodulate!!” three times.)

When reviewing, make notes about things that you had forgotten: for example, make a flash card for each fact that you couldn’t remember, or that you remembered incorrectly. This will help you focus the next round of study on things that most need attention.

The Value of Lectures

According to [Mill2016a], “The lectures that predominate in face-to-face courses are relatively ineffective ways to teach, but they probably contribute to spacing material over time, because they unfold in a set schedule over time. In contrast, depending on how the courses are set up, online students can sometimes avoid exposure to material altogether until an assignment is nigh.”

Retrieval Practice

Researchers now believe that the limiting factor for long-term memory is not retention (what is stored), but recall (what can be accessed). Like any other skill, recall of specific information improves with practice, so taking practice tests, or summarizing the details of a topic from memory and then checking what was remembered and what wasn’t, improved outcomes in real situations. For example, [Karp2008] found that repeated testing improved recall of word lists from 35% to 80%.

Research also shows that recall is better when practice uses activities similar to those used in testing; for example, writing personal journal entries helps with multiple-choice quizzes, but less than doing multiple-choice quizzes [Mill2016a]. This is called .

One way to exercise retrieval skills is to solve problems twice. The first time, do it entirely from memory without notes or discussion with peers. After grading your own work against a rubric supplied by the instructor, solve the problem again using whatever resources you want. Your score on the second version shows you how well you are able to retrieve and apply knowledge.

Another method (mentioned above) is to create flash cards. In physical form, a question or other prompt is written on one side, and the answer is written on the other; in digital form, these are ideal for deployment on mobile devices like phones. If you are studying as part of a group, you can exchange flash cards with a partner; this also helps you discover important ideas that you may have missed or misunderstood.

A quicker version of this is read-cover-retrieve: as you read something, cover up key terms or sections with small sticky notes. When you are done, go through it a second time and see how well you can guess what’s under each of those stickies.

Whatever method you use, don’t just practice recalling facts and definitions: make sure you also check your understanding of big ideas and the connections between them. Sketching a concept map and then comparing it to your notes or to a previously-drawn concept map is a quick way to do this.

Hypercorrection

One powerful finding in learning research is the hypercorrection effect [Metc2016]. Most people don’t like to be told they’re wrong, so it’s reasonable to assume that the more confident someone is that the answer they’ve given in a test is correct, the harder it is to change their mind if they were actually wrong. However, it turns out that the opposite is true: the more confident someone is that they were right, the more likely they are not to repeat the error if they are corrected.

Interleaving

One way you can space your practice is to interleave study of different topics: instead of mastering one subject, then the next, then a third, shuffle study sessions. Even better, switch up the order: A-B-C-B-A-C is better than A-B-C-A-B-C, which in turn is better than A-A-B-B-C-C [Rohrer2015]. This is effective because interleaving fosters creation of more links between different topics, which in turn increases retention and recall.

How long you should spend on each item depends on the subject and how well you know it, but somewhere between 10 and 30 minutes is long enough for you to get into a state of flow (Section 5.2) but not for your mind to wander. Interleaving study will initially feel harder than focusing on one topic at a time, but that’s a sign that it’s working. If you are making flash cards for yourself, or doing practice tests, you should see improvement after only a couple of days.

Elaboration

Explaining things to yourself as you go through them helps you understand and remember them. One way to do this is to follow up each answer on a practice quiz with an explanation of why that answer is correct, or conversely with an explanation of why some other plausible answer isn’t. Another is to tell yourself how a new idea is similar to or different from one that you have seen previously.

If you are studying with a partner, you can each pick an idea, then try to find a series of connections that lead from one to the other. For example, Saskatchewan is a province of Canada; Canada is a country; countries have governments; governments are elected, and people try to predict election results using statistics, so there’s a five-step chain from Saskatchewan to statistics.

Talking to yourself may seem like an odd way to study, but [Biel1995] explicitly trained people in self-explanation, and yes, they outperformed those who hadn’t been trained. An exercise that builds on this is to go through code line by line with a group, having a different person to explain each line in turn and say why it is there and what it accomplishes.

Explaining things to others even works on exams, though the extent of the benefits are still being studied. [Cao2017a,Cao2017b] looked at two-stage exams, i.e., a normal (individual) exam which is then immediately followed by a second exam in which students work in small groups to solve a set of problems. They found significant short-term gains for students doing exams collaboratively, but not long-term gains, i.e., the benefits visible a couple of weeks after the mid-term had faded by the final. They also found that students in the middle of the class benefited strongly, and that homogeneous-ability groups benefited, while heterogeneous groups did not.

Concrete Examples

One specific form of elaboration is so useful that it deserves its own heading, and that is the use of concrete examples. Whenever you have a statement of a general principle, try to provide one or more examples of its use, or conversely take each particular problem and list the general principles it embodies. [Raws2014] found that interleaving examples and definitions made it more likely that learners would remember the latter correctly.

One structured approach way to do this is the ADEPT method: give an Analogy, draw a Diagram, present an Example, describe the idea in Plain language, and then give the Technical details. Again, if you are studying with a partner or in a group, you can swap and check work: see if you agree that other people’s examples actually embody the principle being discussed, or which principles are used in an example that they haven’t listed.

Another useful technique is to teach by contrast, i.e., to show learners what a solution is not, or what kind of problem a technique won’t solve. For example, when showing children how to simplify fractions, it’s important to give them a few like 5/7 that can’t be simplified so that they don’t become frustrated looking for answers that don’t exist.

Dual Coding

The last of the Learning Scientists’ six core strategies is to present words and images together. As discussed in Section 4.2, different subsystems in our brains handle and store linguistic and visual information, and if complementary information is presented through both channels, then they can reinforce one another. (However, learning is more effective when redundant information is not presented simultaneously in two different channels [Maye2003], because then the brain has to expend effort to check each channel against the other.)

One way to take advantage of dual coding is to draw or label timelines, maps, family trees, or whatever else seems appropriate to the material. (I am personally fond of pictures showing which functions call which other functions in a program.) Drawing a diagram without labels, then coming back later to label it, is excellent retrieval practice.

Time Management

I used to brag about the hours I was working. Not in so many words, of course—I had some social skills. Instead, I’d show up for class around noon, unshaven and yawning, and casually mention how I’d been up ’til 6:00 a.m. hacking away at some monster bug or other.

Looking back, I can’t remember who I was trying to impress. Instead, what I remember is how much of the code I wrote in those all-nighters I threw away once I’d had some sleep, and how much damage the bugs I created in those bleary-eyed stupors did to my grades.

My mistake was to confuse “working” with “being productive”. You can’t produce software (or anything else) without doing some work, but you can easily do lots of work without producing anything of value. Scientific study of the issue goes back to at least the 1890s (see [Robi2005] for a short, readable summary). The most important results for learners are:

  1. Working more than eight hours a day for an extended period of time lowers your total productivity, not just your hourly productivity—i.e., you get less done in total (not just per hour) when you’re in crunch mode than you do when you work regular hours.
  2. Working over 21 hours in a stretch increases the odds of you making a catastrophic error just as much as being legally drunk.

These facts have been reproduced and verified through hundreds of experiments over the course of more than a century. The data behind them is as solid as the data linking smoking to lung cancer. However, while most smokers will admit that their habit is killing them, people in the software industry still talk and act as if they were somehow exempt from these findings.

It’s very easy to go backward when programming: it only takes me a couple of minutes to create a bug that will take hours to track down later—or days, if someone else is unlucky enough to have to track it down. This is summarized in Robinson’s first rule:

Productivity varies over the course of the workday, with the greatest productivity occurring in the first four to six hours. After enough hours, productivity approaches zero; eventually it becomes negative.

It’s hard to quantify the productivity of programmers, testers, and UI designers, but five eight-hour days per week has been proven to maximize long-term total output in every industry that has ever been studied. There’s no reason to believe that software development is any different, or that student programming is different from full-time programming in industry.

Ah, you say, that’s “long-term total output”. What about short bursts now and then, like pulling an all-nighter to meet a deadline? Well, that’s been studied too, and the results aren’t pleasant. Your ability to think drops by 25% for each 24 hours you’re awake. Put it another way, the average person’s IQ is only 75 after one all-nighter, which puts them in the bottom 5% of the population. Two all nighters in a row, and their effective IQ is 50, the level at which people are usually judged incapable of independent living.

The catch in all of this is that people usually don’t notice their abilities declining. Just like drunks who think they’re still able to drive, people who are deprived of sleep don’t realize that they’re not finishing their sentences (or thoughts). They certainly don’t realize that they’re passing parameters into function calls the wrong way around, or that what they’re typing in will all have to be deleted and re-done tomorrow, when it will take longer than it would have if they’d just gone home and gotten a good night’s sleep.

And despite what many people want to believe about themselves, none of us are good at multi-tasking. What we can become good at, though is automaticity, which is the ability to do something routine in the background while doing something else [Mill2016a]. Most of us can talk while chopping onions, or chew while reading; with practice, we can also take notes while listening, but we can’t study while doing other things.

When You Just Can’t Say No

Research has shown that our ability to exert willpower runs out, just like our ability to use muscles: if we have to resist eating the last donut on the tray when we’re hungry, we are less likely to fold laundry and vice versa. This is called ego depletion [Mill2016a], and an effective counter is to build up habits so that doing the right thing is automatic.

“But—but—I have so many assignments to do!”, you say. “And they’re all due at once! I have to work extra hours to get them all done!” No: in order to be productive, you have to do two things: prioritize, and focus. The first is important because people are naturally very good at spending hours on things that don’t need to be done, and then finding themselves with too little time for the things that actually count. It can actually be expressed as an algorithm:

  1. Make a list of the things you have to do.
  2. Weed out everything that you don’t need to do right away. Notice that I said “need”, not “want”: if you want to mess around with a new blogging tool, that’s fine, but that’s play time, not work time, and we’re talking about getting work done.
  3. Sorting what’s left so that the most important tasks are at the top. (I don’t worry about getting the stuff below the first three or four lines into exact order, since I’m going to re-check my list before I get to them anyway.)
  4. Make sure you have everything you need to see the first task through: the assignment description, a comfortable chair, etc. Don’t give yourself an excuse to interrupt your own work: the world will provide enough of those.
  5. Shut down your email and turn off your cell phone. Don’t panic, it’s only for an hour—most people can’t stay focused longer than that, and anyway, you’ll need to stretch your muscles and get rid of that coffee you drank.
  6. Set an alarm to go off in sixty minutes, and focus. Don’t switch tasks in that hour unless you absolutely have to. Instead, if you remember an email message that needs to be sent, or discover a couple of tests that really should be written, add a note to your to-do list. (This is one reason I keep mine in a lab notebook: the few seconds it takes to pick up a pen and jot something down gives my hands a rest from the keyboard.)
  7. When your hour is up, take a break: check mail (but don’t reply to anything that isn’t urgent), go to the washroom, stretch a little, and then re-order your to-do list and start the next round.

If any task on your list is more than an hour long, break it down into into smaller pieces and prioritize those separately. Keep in mind that the future is approaching at a fixed rate of one day every 24 hours: if something’s going to take sixty hours to do, you’d better allow at least ten working days for it (assuming you’re only interrupted 25% of the time), which means you’d better tackle the first piece two working weeks before the deadline. And since breaking large tasks down into small ones takes time, don’t be embarrassed about putting “plan XYZ” in your to-do list.

The point of all this organization and preparation is to get yourself into the most productive mental state possible. Psychologists call it flow [Csik2008]; athletes call it “being in the zone”, while musicians talk about losing themselves in what they’re playing. Whatever name you use, you will produce much more per unit of time in this state than normal.

That’s the good news. The bad news is that it takes roughly ten minutes to get back into a state of flow after an interruption, no matter how short the interruption was. This means that if you are interrupted half a dozen times per hour, you are never at your productive peak.

Designed to Fail

The effect of interruptions on productivity begs the question: if timeslicing is bad, why are schools set up to require you to do it all the time?) Doing nothing but the project course eight hours a day for three weeks would be more efficient. However, it would be harder on instructors, and would be difficult to integrate with courses in subjects like math and languages that take time to soak in.

Making lists and setting one-hour alarms will seem a little earnest at first, but trust me: your friends will stop mocking you once they see that you’re able to finish your assignments and still have time to play some badminton and catch a movie. They may even start to imitate you.

Peer Assessment

Asking people on a team to rate their peers is a common practice in industry. [Sond2012] surveyed the literature on student peer assessment, distinguishing between grading and reviewing. The benefits they found included increasing the amount, diversity, and timeliness of feedback, helping students exercise higher-level thinking, encouraging reflective practice, and supporting development of social skills. The concerns were predictable: validity and reliability, motivation and procrastination, trolls, collusion, and plagiarism. However, while these concerns are real (and as we have recently seen, can be damaging to society as a whole), the evidence shows that they aren’t significant in class. For example, [Kauf2000] compared confidential peer ratings and grades on several axes for two undergraduate engineering courses and found that self-rating and peer ratings statistically agreed, that collusion (i.e., everyone giving their peers the same grades) wasn’t significant, that students didn’t inflate their self-ratings, and crucially, that ratings were not biased by gender or race.

One important variation on peer assessment and review is contributing student pedagogy, in which students produce artifacts to contribute to other students’ learning. This can be developing a short lesson and sharing it with the class, adding to a question bank, or writing up notes from a particular lecture for in-class publication. For example, [Fran2018] found that students who made short videos to teach concepts to their peers had a significant increase in their own learning compared to those who only studied the material or viewed the videos.

Another is , in which a student reviews one or more examples using a rubric and compares their evaluation against the instructor’s review of the same work. Only once student’s evaluations are close enough to the instructor’s are they allowed to start evaluating peers’ actual work.

As long as evaluation is based on observables, rather than personality traits, peer assessment can actually be as accurate as assessment by TAs and other outsiders. “Observables” means that instead of asking, “Is the person outgoing,” or “Does the person have a positive attitude,” assessments should ask, “Does the person listen attentively during meetings,” or, “Does the person attempt to solve problems before asking for help.” The evaluation form in Appendix J shows a sample to get you started. To use it, rank yourself and each of your teammates, then calculate and compare scores.

Final Thoughts

A lot of what people believe about programming isn’t true, or isn’t proven [Oram2010,Stef2017], and the same is true for computing education. [Guzd2015a] list of the top 10 myths is:

  1. The lack of women in Computer Science is just like all the other fields of science, technology, engineering, and medicine.
  2. To get more women in CS, we need more female CS faculty.
  3. A good CS teacher is a good lecturer.
  4. Clickers and the like are an add-on for a good teacher.
  5. Student evaluations are the best way to evaluate teaching.
  6. Good teachers personalize education for students’ learning styles.
  7. High schools just can’t teach CS well, so they shouldn’t do it at all.
  8. The real problem is to get more CS curriculum into the hands of teachers.
  9. All I need to do to be a good CS teacher is model good software development practice, because my job is to produce excellent software engineers.
  10. Some people are just born to program.

The last of these is the most pervasive and most damaging. As discussed in Section 10.3, Elizabeth Patitsas and others have shown that grades in computing classes are not bimodal [Pati2016], i.e., there isn’t one group that gets it and another that doesn’t. Many people who would be excellent programmers have convinced themselves that they just don’t have what it takes to do so; if all you do is dispel that belief, you will have made the world a slightly better place.

Challenges

Learning Strategies (individual/20 minutes)

  1. Which of the six learning strategies do you regularly use? Which ones do you not?
  2. Write down three general concepts that you want your learners to master, and then give two specific examples of each. (This uses the “concrete examples” practice).
  3. For each of those concepts, work backward from one of your examples to explain how the concept explains it. (This uses the “elaboration” practice).

Convergent Evolution (pairs/15 minutes)

One practice that wasn’t covered above is guided notes, which are instructor-prepared notes that cue students to respond to key information in a lecture or discussion. The cues can be blank spaces where students add information, asterisks next to terms students should define, etc.

Create 2–4 guided note cards for a lesson you have recently taught or are going to teach. Swap cards with your partner: how easy is it to understand what is being asked for? How long would it take to fill in the prompts?

Changing Minds (pairs/10 minutes)

[Kirs2013] argues that myths about digital natives, learning styles, and self-educators are all reflections of the mistaken belief that learners know what is best for them, and cautions that we may be in a downward spiral in which every attempt by education researchers to rebut these myths confirms their opponents’ belief that learning science is pseudo-science. Pick one thing you have learned about learning so far in this book that surprised you or contradicted something you previously believed, and practice explaining it to a partner in 1–2 minutes. How convincing are you?

Flash Cards (individual/15 minutes)

Use sticky notes or anything else you have at hand to make up a dozen flash cards for a topic you have recently taught or learned, trade with a partner, and see how long it takes each of you to achieve 100% perfect recall. When you are done, set the cards aside, then come back after an hour and see what your recall rate is.

Using ADEPT (whole class/15 minutes)

Pick something you have recently taught or been taught and outline a short lesson that uses the five-step ADEPT method to introduce it.

The Cost of Multi-Tasking (pairs/10 minutes)

The Learning Scientists blog describes a simple experiment you can do without preparation or equipment other than a stopwatch to demonstrate the mental cost of multi-tasking. Working in pairs, measure how long it takes each person to do each of these three tasks:

  • Count from 1 to 26.
  • Recite the alphabet from A to Z.
  • Interleave the numbers and letters, i.e., say, “1, A, 2, B, ” and so on.

Have each pair report their numbers: you will probably find that the third (in which you are multi-tasking) takes significantly longer than either of the component tasks.

Changing Your Mind (small groups/20 minutes)

Working in groups of 3–6, go through Guzdial’s list of myths in computing education [Guzd2015a]. Which ones do you believe? Why? What would convince you to change your mind? Does [Guzd2015a] present or link to such evidence?

Calibrated Peer Review (pairs/20 minutes)

  1. Create a 5–10 point rubric for grading programs of the kind you would like your learners to write that has entries like “good variable names”, “no redundant code”, and “properly-nested control flow”.
  2. Choose or create a small program that contains 3–4 violations of these entries.
  3. Grade the program according to your rubric.
  4. Have your partner grade the same program with the same rubric. What do they accept that you did not? What do they critique that you did not?