The efficacy study on MatheZeit

When people talk about the success of mathematics teaching, marks and test results are usually at the centre. They matter, because they show whether children have understood mathematical content. But learning covers more than that: the confidence to take on difficult tasks, an interest in the subject, and how children deal with mistakes. Educational research groups these characteristics under the term affective outcome criteria, and they are not a side issue. According to the well-evidenced Reciprocal Effects Model, subject-specific self-concept and achievement reinforce each other over time: higher confidence predicts later gains in achievement, even once prior achievement is accounted for.

There is now a study of MatheZeit that measured both. In this article we summarise what it shows, and equally how it shows it, because the two parts differ in what they can tell us.

The study in brief

The Institute of Mathematics at Europa-Universität Flensburg examined the effect of MatheZeit in an independent study that we did not commission: a randomised controlled trial with 530 children in years 3 and 4 at 33 primary schools in Schleswig-Holstein. The randomly assigned MatheZeit group worked with the platform for six weeks, roughly 45 minutes per week within regular mathematics lessons; the control group had regular lessons with the same time budget. In addition, 250 children were surveyed on self-concept and interest; this accompanying survey ran without a control group and should be read as descriptive.

The competence effect

Children who learned with MatheZeit achieved a greater gain in competence than the control group. Translated into learning time, the advantage corresponds to around three to four months of additional learning progress, achieved after only six weeks and without a single extra minute of learning time. The finding remained stable across several statistical checks, and it is substantial: most classroom interventions tested in large field studies achieve considerably smaller effects.

The gain appeared in both year groups. In terms of content, it was concentrated in the structure-related areas, most strongly in “Patterns and Structures”, followed by “Measurement and Quantities”. This profile is notable because the revised German national education standards (KMK) highlight precisely the area “patterns, structures and functional relationships” as an overarching guiding idea of the subject: MatheZeit has its clearest effect where the standards locate the structural foundation of the subject.

What changed alongside achievement

The accompanying survey, which had no control group and is therefore descriptive, shows a positive picture of affective development. The children's mathematical self-concept rose measurably over the period of use, and interest rose slightly. This is notable because longitudinal studies consistently show that self-concept and enjoyment of learning normally tend to decline over the course of primary school; a rise within six weeks runs counter to that developmental trend.

The clearest change was in agreement with “I can solve difficult tasks too” (+8 percentage points), that is, exactly the task-level confidence that adaptive practice aims at. On interest, the largest rise was in looking forward to mathematics lessons (+5.5 percentage points), the situational experience that research on interest identifies as the first thing to change; deeper attitudes towards the subject only grow out of that through repeated positive experience.

Enjoyment, even when things get hard

One finding from the process data deserves particular attention. In short surveys taken directly during learning, 72 per cent of children reported enjoyment even after tasks they had solved only after several failed attempts. Even learning units that children perceived as too hard were still rated positively on average. Demanding tasks are otherwise a typical trigger for frustration and drop-out; the fact that enjoyment is maintained here is the most interesting part of the picture from a teaching perspective.

A plausible mechanism for this lies in the completion rates: of more than 23,000 Knobelbox tasks worked on, 99.9 per cent were completed successfully, and where children requested help from Mazio, that help almost always led to a solution. Children therefore hardly ever end a task in failure. Research on learning from mistakes shows that mistakes only become productive when the child receives corrective feedback immediately and does not experience the situation as an assessment; a tutoring system can create exactly these conditions structurally, because it responds to every failed attempt immediately, individually and without social judgement from classmates.

How children and teachers rate the platform

Children's approval of MatheZeit was consistently high. A good four in five children want to continue working with MatheZeit after the period of use, say that it makes maths more fun, and report liking maths more since using it. In a direct comparison with an established learning platform, surveyed in the same classes with identical questions, MatheZeit scored considerably better on enjoyment and concentration: on enjoyment, 87 per cent of the MatheZeit children gave the top rating, compared with 58 per cent for the comparison platform. Revealingly, both platforms were rated almost identically on usability; the difference therefore lies in the learning experience itself rather than in the interface.

The 33 participating teachers also judged the use of the platform positively: on average they would recommend it with 9.7 out of 10 points, and they particularly highlighted the children's independent work, the adaptive tasks and the diagnostic value for their own teaching. This perspective is central for practice, because a platform's effect is not determined by the technology alone but by how it is embedded in lessons, and that rests with the teachers.

Context and limitations

The study is one of the few independent, controlled demonstrations of efficacy for an adaptive mathematics learning platform in German-speaking primary education. It also has clear limitations, which the report names itself: the period was six weeks, the affective measures were collected without a control group, and whether the effect holds over a full school year is a question for further research.

Two things follow from this for us. First, the gain in competence does not come from more practice time but from the design of the learning; that supports the approach of building adaptivity and feedback into the tasks themselves. Second, achievement, confidence and interest develop together, and a learning environment should be measured against all three, with the same honesty about what is demonstrated and what is observed.