WEDNESDAY, SEPTEMBER 25, 2013
Part 3—With a weak intellectual culture: This past Sunday, the Washington Post gave major play to Amanda Ripley’s new book, The Smartest Kids in the World.
A cynic would say it’s easy to see why the Post did that. Jay Mathews’ review of the book topped the front page of the Post’s Outlook section. Mathews, a nationally known education writer, drew a conclusion from Ripley’s new book which extended a treasured Post theme.
(“The most consistent U.S. failing Ripley discovers is our way of selecting and training teachers.”)
That’s what a cynic would say about the high profile this book received from the Post. On the other hand, a fair-minded person might compliment Mathews on the somewhat murky way he ended his review of the book.
The PISA is the international testing program around whose findings Ripley builds her book. Mathews ends like this:
MATHEWS (9/22/13): Ripley seems to realize toward the end that she put too much faith in the PISA as a measure of creativity and critical thinking, a controversial issue among experts. Upbeat statements about the test tend to disappear later in the book. What the PISA is measuring is probably not creativity. If you believe the exam does capture that elusive quality, then you have to accept the notion that it can be pounded into students as is done in South Korea.
One thing the PISA almost certainly assesses is how much students know. In countries that want them to learn about the world, students get higher scores. They need that command of content, as educators put it, or they won’t have anything to be creative about.
Ripley rightly concludes that we need “a serious intellectual culture in schools.” But watch cable news, eavesdrop in a student cafeteria or attend a local PTA meeting, and you will see that such a culture is something that so far doesn’t interest us much.
We’ll agree with Ripley, a hundred times over. This country would gain from “a serious intellectual culture in schools.” (Our schools would also gain from cultures of enjoyment and exploration.)
As he rolls his eyes at cable news and the rest, Mathews seems to suggest, at the end of his piece, that our whole country lacks “a serious intellectual culture.” Before he aims this barb at American students and PTA members, we would suggest that the gent try healing himself—and the talented Ripley too.
Early on in his review, Mathews describes Ripley as “a talented writer.” We’ll agree with that assessment, though only up to a point. Ripley’s book is very readable, loaded as it is with human interest about a very small number of exchange students and their teachers and principals.
There are favorable things to be said about Ripley’s book. But from what “intellectual culture” has this book, and Mathews’ review of it, sprung?
Consider the second page of The Smartest Kids in the World, where Ripley lays out her basic framework. Why did she decide to write a book about education, a topic she rather snootily says she once found, “well, kind of soft?”
(Did you see the talented writing there?)
Why did Ripley change her mind? What made her decide to write a book about education? The talented writer posts a chart, which she describes in this manner:
RIPLEY (page 2): Then one day I saw this chart, and it blew my mind.
The United States might have stayed flat over time, but that was the exception, it turned out. Look at Finland! It had rocketed from the bottom to the top of the world, without pausing for breath. And what was going on in Norway, right next door, which seemed to be slip sliding into the abyss, despite having virtually no child poverty? And there was Canada, careening up from mediocrity to the heights of Japan. If education was a function of culture, could culture change that dramatically—that fast?
This is our introduction to miraculous Finland, one of the three countries Ripley explores in her book. According to Ripley, the chart which blew her mind shows the small but miraculous Nordic nation “rocket[ing] from the bottom to the top of the world, without pausing for breath.”
The chart to which Ripley refers appears next door, on page 3. But uh-oh! Quite plainly, it doesn’t show miraculous Finland doing any such thing.
Anyone who looks at this chart can see that Ripley’s description of Finland’s rocket trip is grossly, dramatically wrong. Within what sort of “intellectual culture” do such errors reside?
The chart to which Ripley refers is, within the context of her book, largely useless. Ripley’s caption for the chart reads as follows:
RIPLEY (page 3): Dance of the Nations: Over a half century, different countries gave eighteen different tests to their children. Economists Ludger Woessmann and Eric Hanushek projected kids’ performance onto a common measuring stick. The results suggest that education levels can—and do—change dramatically over time, for better and worse.
We don’t mean this as a criticism of Woessmann and Hanushek’s work. Their chart may well make perfect sense, once it has been explained.
But within the context of Ripley’s book, the chart is never explained. For that reason, it is basically impossible to interpret. On its vertical axis, it shows a scale of scores from those “eighteen different tests” which extends from 460 up to 560. Recorded scores for the fifteen nations extend from a low score of roughly 470 to a high of perhaps 555.
Along its horizontal axis, the time frame of the chart extends from 1965 to 2009.
At no point does Ripley explain what those test scores mean. More precisely, she makes no attempt to explain how large the observable score gains actually are in practice. If a nation moves from a score of 480 to a score of 520, does that correspond to a large change in learning and academic skill? Or is such a score change just a minor blip?
Without some attempt to answer those questions, we can see nations moving up and down on the “test score” scale, but we have no idea if the changes involved are significant. Similarly, we have no way to estimate the size of the gaps in learning and achievement between the fifteen nations.
More significant is what we plainly can see concerning miraculous Finland.
According to Ripley, miraculous Finland “rocketed from the bottom to the top of the world, without pausing for breath.” It was this type of miraculous change which created Ripley’s interest in talentedly writing about education!
But uh-oh! On this largely unexplained chart, Finland is never anywhere near the bottom of the world. On a murkily undifferentiated curve, Finland is shown in the 1960s and 1970s scoring at levels which few other countries on the chart have achieved even today.
Uh-oh! If this chart means what it seems to mean, Finland seems to have been at the top of the world, among these countries, from the earliest days of national and international testing. According to Ripley’s chart, only Japan and Belgium outscored Finland back in the 1960s. Only Japan outscored Finland in the 1970s.
Finland’s scores have improved over time, from roughly 510 in those earlier decades to roughly 545 today. But there is no way of knowing how much extra learning is represented by that type of score gain. And Ripley’s basic account of Finland’s history seems to be flagrantly wrong:
Sorry, Chaarli! If we go by this largely unexplained chart, Finland didn’t rocket from the bottom of the world to the top. On page two of her talented book, Ripley’s basic account of her interest in education seems to be flagrantly wrong.
What kind of “intellectual culture” guides the writing of such a book? A book where an obvious, foundational error appears right on page two? We’re not sure, but we were similarly puzzled when Ripley explained why Poland was chosen as one of the countries with the world’s “smartest kids.”
In fairness, Ripley never quite says that Poland is a high-scoring country. She frequently makes it sound that way, and every reviewer in the land seems to think that’s what she said. Are book reviewers ever required to take a reading test?
Ripley sometimes makes it sound like Poland's a high-scoring nation. But when she gets precise with her language, she describes Poland as “a country on the ascent,” an example of “the metamorphosis model” (page 24).
To what extent is or was Poland “a country on the ascent?” When Ripley describes Poland’s score gains, she focuses on the jumps that occurred the first two times Poland took part in the PISA.
With apologies, we have to give you a slightly truncated version of her account. (Never a lender be!) We expect to be able to fill in this account tomorrow:
RIPLEY (page 135): [I]n 2000, Polish fifteen-year-olds took the PISA...Polish fifteen-year-olds ranked twenty-first in reading and twentieth in math, below the United States and below the average for the developed world. Two-thirds scored in the rock-bottom lowest literacy level. Three years later, in 2003, a new group of Polish fifteen-year olds took the PISA again...Poland, the punch line for so many jokes around the world, ranked thirteenth in reading and eighteenth in math, just above the United States in both subjects.
In full context, those gains are treated as an example of what it means to be “a country on the ascent.” That said, even the least observant reader may notice something slightly odd about this thrilling account.
Moving from twenty-first to thirteenth in reading seems like a very good gain in the space of three years. But according to Ripley, Poland only moved from twentieth to eighteenth in math during this same period.
As a reader, we were puzzled by that passage. Were we supposed to be blown away by that degree of success—by the fact that Poland jumped all the way from 20th place to 18th? Was Ripley hoping her readers just wouldn’t notice the meagerness of that ascent?
Using the PISA Data Explorer provided by the NCES (and cited by Ripley), we decided to double-check Poland’s ascent during this period. We don’t know what measure Ripley is using when she describes Poland’s international rankings in math in 2000 and 2003. But if we’re going by average scores, Poland went from 21st out of 28 OECD countries in math in 2000 to a tie for 21st out of 29 OECD countries in 2003, as you can see for yourself by using the Data Explorer.
Poland did gain ground on the OECD average in math during that period. But, again with apologies, that passage on page 135 represents Ripley’s attempt to explain why Poland, “a country on the ascent,” is one of only three countries on which she chose to focus in this high-profile book.
We’ll be candid—Poland’s gains on the PISA don’t seem ginormous to us, especially if you read closely enough to see that Ripley is only claiming a very slight ascent in ranking in math in her basic presentation. And not only that! In this confusing passage from his review, Mathews refers to a situation which may make matters worse:
MATHEWS: The deeper Ripley goes, however, the less certain she is of the answer to our school problem. Teachers in the high-scoring countries give their students more rigorous assignments and get more support from parents, principals and students for demanding work than teachers do in the United States. Ripley embraces that key concept. But some of those nations share the American habit of thinking that not all students need rigor.
The PISA is given to 15-year-olds. Ripley cites a testing expert’s discovery that Poland gave that age group a boost by holding back pupils on their way to vocational school for an extra year of academic studies. Then, as an experiment, the PISA was given again to a sampling of those students when they were 16 or 17 and attending vocational schools. Their scores had dropped significantly. The extra year of academics had no lasting effect. Poland had an American-like gap between kids who were heading for college and those who weren’t.
In that rather murky passage, Mathews alludes to a basic change Poland made between 2000 and 2003. (In her book, Ripley explains this change with a bit more clarity, though even there, questions remain.)
Traditionally, lower-achieving Polish kids got “tracked” into vocational schools before their fifteenth year. After the disappointing PISA results of 2000, the country decided to keep those kids in more challenging academic programs for an additional year.
When Poland’s average scores rose in 2003 (if not its international rankings), did that represent a one-time reaction to this change in basic procedures? There’s no way of knowing, but Poland’s scores didn’t change a great deal between 2003 and 2009, the last year for which PISA scores are available.
Quite plainly, Poland isn’t a high-scoring nation, not even on the PISA. Beyond that, we’re not real sure why Poland is cited as “a country on the ascent,” and Ripley doesn’t really bother explaining in her 230-page book.
Given our nation’s “intellectual culture,” she seems to assume that readers will gasp when she tells us that Poland jumped from 20th place in math all the way up to 18th. (By 2009, Poland ranked 19th out of 34 OECD nations in math on the PISA.)
Let’s review:
Judging from Ripley’s chart, miraculous Finland didn’t start at the bottom of the world. A simple glance at Ripley’s chart seems to show that her statement is wrong.
Poland, a country on the ascent, only went from 20th to 18th in math in the period Ripley chose to highlight—and it achieved that gain in ranking on some undisclosed measure.
When a book is built around such claims, we can’t say we’re blown away by its “intellectual culture.” And we still must account for Ripley’s most significant decision.
Like many other nations, the United States and Poland take part in three major international testing programs—the PISA, the TIMSS and the PIRLS. No one makes these countries take part in these programs. Presumably, countries take part in the TIMSS and the PIRLS because they think the TIMSS and the PIRLS are valuable testing programs.
This causes a bit of a puzzle. The talented Ripley’s well-written book covers more than 200 pages. But a reader is never even told about the TIMSS and the PIRLS, testing programs on which American students have scored better, in recent years, than they have done on the PISA.
In an endnote, Ripley explains her decision to restrict herself to the PISA—to refer to data from one testing program when three sets of data are available. In our view, the intellectual culture displayed in this note isn’t especially high:
RIPLEY (page 258): There are other tests besides the PISA, each of which provides valuable data in its own right; for the purposes of this book, I was most interested in which countries prepared students to think, learn and thrive in the modern economy. PISA was designed with this purpose in mind. The OECD’s 1999 report, Measuring Student Knowledge and Skills, describes the difference between PISA and other international test this way...
Ripley goes on to provide a quote from that OECD report. She doesn’t explain that the PISA is a program which is developed and run by the OECD—that in this report, the OECD is describing and praising one of its own programs.
Is the PISA a better measure in some major way than the TIMSS and the PIRLS? We can’t answer that question, in part because the parameters of our “intellectual culture” are set by people like Ripley. But as a general matter, our nation displays a weak “intellectual culture,” just as Mathews suggested.
Part of that weak intellectual culture is put on display when reviewers like Mathews describe Poland as a high-scoring nation, thereby making the Standard Story told by this book seem just that much better. Meanwhile, the weak “intellectual culture” of Ripley’s book goes on display right on page 2, when she offers a weirdly bogus account of a chart which sits on page 3.
As with Mathews, so with Ripley. Her bogus account makes the Official Standard Story sound a great deal better. Miraculous Finland rose from the bottom all the way to the top!
If Ripley’s chart means what it seems to mean, Finland didn’t “start at the bottom of the world,” then shoot to the top, “without pausing for breath.” But wait a minute! If miraculous Finland didn’t do that, what got Ripley involved in education, the subject of this book?
Tomorrow, we’ll discuss past errors by Ripley and Mathews, errors which seem to keep advancing and improving the Official Establishment Line. Have you noticed that our intellectual culture is built around tightly-scripted, memorized tales? That our public discourse tends to be novels, novels all the way down?
Tomorrow: Extremely easy to con
With apologies: By tomorrow, we expect to provide a fuller chunk from page 135 about Poland’s rapid ascent.