By Carl Straumsheim
The world of learning analytics is full of metaphors. Educational-technology companies are “islands,” disconnected from one another. Data are locked away in “silos.” As an initiative to standardize the collection and reporting of learning analytics nears a public launch, can colleges and vendors learn to speak the same language?
“Analytics” is one of the hottest buzzwords in education. For ed-tech companies, it is also a selling point. By using vendors’ suites and solutions, colleges will gain access to data about why some students succeed and where and when others stumble — or so the pitch goes.
But there’s a catch, said Linda Feng, senior product manager for analytics and SIS integration at Instructure. “The whole premise is that all that data has to be in their world,” she said.
Step outside that world and try to take the data to another platform, and it quickly becomes a conversation where both sides are speaking in different languages.
Colleges are rarely tied to a single vendor. They may be tracking course enrollment patterns with one system, retention with another and tools to measure engagement with yet another — and those systems all produce data. Extracting data from those tools and platforms is one issue; making the data say something meaningful about what students are doing is another.
“All these systems have their own languages that are written for their own needs,” said John Whitmer, director for platform analytics and research at Blackboard. “In order to talk a common language, we have to agree to a common vocabulary.… If you don’t have a common vocabulary, you can’t write the world’s most beautiful poem.”
Blackboard, Instructure and the more than 320 other vendors and universities that make up the IMS Global Learning Consortium have for years been working to agree on which words go into that vocabulary, and their work is finally nearing its version 1.0 release. Known as Caliper, the vocabulary — called metric profiles — and the mechanisms to detect the words in it — sensors — will serve as a framework for tracking and reporting learning analytics.
Should Caliper be supported by a large number of colleges and vendors, it could become the standard for how student learning data are collected. But Caliper is still only available as a release candidate to members of the consortium and is still “a couple of months” away from public availability, said Rob Abel, CEO of IMS. The larger market of colleges and vendors, which will help determine if Caliper becomes a widely adopted standard, has yet to take a look at the framework.
Until the public release, IMS and its members are therefore managing expectations.
“Whenever standards are sold as the magic that solves all problems, that’s a concern to me,” Abel said, before resorting to another metaphor. “They’re not. They’re just plumbing.”
‘What Data Can I Get?’
It is difficult to speak concretely about how Caliper will be used when that decision ultimately rests with colleges and vendors — which is perhaps why people resort to metaphors to explain what the framework is. On its own, Caliper is not a dashboard or an app that automatically explains how students learn. Those tools must be built on top of the framework.
“[Caliper is] certainly not going to figure out what the best analytics approach is, but it’s going to make it easier for institutions to get data from applications so they can actually understand what it is and process it,” Abel said. “We think that’s about right for the market where it is now.”
That approach is similar to how the consortium developed the Learning Tools Interoperability framework. LTI didn’t create Khan Academy, ProctorU and Wikipedia, but it created a standardized way for those organizations to make their tools embeddable in learning management systems and other platforms.
Caliper builds on the interoperability framework, Abel said. Now that apps and platforms can easily connect to one another, he said, “Then the next natural question you ask is, ‘Well, what data can I get?’”
While only a handful of colleges and vendors are ready to ask themselves that question, Abel said, some Caliper work group members have begun to identify potential use cases. One common idea is the potential they see in capturing and reacting to real-time data.
The University of Kentucky is perhaps farthest along with its plans. Vince Kellen, a senior vice provost and chief information officer there, said the university plans to extend its early warning system using data collected from when students engage with course content. Kellen co-chairs a Caliper work group tasked to look specifically at how learning analytics could power real-time messaging systems.
“If a student disengages in the middle of a term, we sometimes don’t know if it’s because of finances, if classes are too difficult or if they’re not sure if they belong here from a social standpoint,” Kellen said. “By getting that interaction data, we can do a better job of detecting that sooner. If we can interact with the student sooner, we can help them better.”
Another co-chair, the University of Texas at Austin’s Phillip D. Long, expressed interest in a similar system. “The thing we have done the least well at most large universities is in providing feedback,” said Long, an associate vice provost at UT-Austin.
Such a system would first have to learn the pathways students take through content on their way to a passing grade before it would know when to flag when students are headed in the wrong direction, Long said. Once it has learned those patterns, he said, the system could, for example, notify an adviser when a student fails to review before an important quiz.
“When students are actually engaged in activities and assignments is when the opportunity is most present for being able to influence their thinking,” Long said. “We’ve never been in a position where the technology has had this degree of temporal responsiveness.”
The universities are also motivated to participate in IMS on Caliper out of a concern that sitting out would mean vendors get to decide who controls the data collected by tools and platforms, Kellen said.
“There’s a danger that the vendor community can say, ‘This is our data, not yours, and you have to pay us to learn how you teach students,’” Kellen said. “I disagree with that. The academy needs to have the line of sight to the student in digital form.”
Becoming a Movement
Dan Rinzel, senior product manager for analytics at Blackboard, said that wariness of vendors is one of the reasons why the company is a member of the consortium.
“It is important for us to participate and be early adopters in the specification for the purpose of participating well in the ecosystem and being good data stewards,” Rinzel said. “There’s some undercurrent of unease that some institutions have about where the data resides, who has access to it, and we definitely see ourselves … making sure that there’s no sense that data is being locked away.”
Blackboard last month became the first company to have one of its products complete the Caliper certification process. The company doesn’t have any immediate plans to build anything on top of the framework, but decided to support Caliper “in the interest of supporting standards and standards-based work,” Whitmer, the Blackboard director, said.
D2L has yet to run its learning management system, Brightspace, through the process, but it helped lead the Caliper work group. The company has already announced plans to support Caliper.
In some cases, vendors’ plans for Caliper are not that different from those of colleges and universities. Instructure, which develops the learning management system Canvas, is also exploring the uses of real-time data. In a proof of concept demonstrated earlier this year, the company showed a dashboard updating in real time as a fake student navigated to a quiz, submitted it and received a grade. The company is working on a “live event stream” that builds on that proof of concept, Feng, the Instructure senior product manager, said.
Instructure plans an official uptake of Caliper later this year, Feng said, and the company intends to build an “ecosystem” of partners that will develop tools that build using the data reported out from Canvas.
“Similar to how LTI tools became a movement, Caliper also needs to become a movement,” Feng said. “No one vendor by themselves is going to be motivated to do Caliper on their own.”
Early interest from Blackboard, D2L and Instructure is “almost a sure sign that [Caliper is] going to succeed,” Abel predicted. Normally, vendors are “reticent” to support new standards, he said. Before the public release, more vendors need to test their products, and more how-to documentation needs to be written, he said.
“We’re the first to say we think analytics in higher education is a 20-year project, and we’re in year one or two,” Abel said.
Technology
Editorial Tags:
Technology
Image Source:
iStock
Source: Inside Higher Ed
