Some Observations about TIMSS 2011

TIMSS (Trends in International Mathematics and Science Study) is an assessment of the maths and science skills of fourth grade and eighth grade students around the world. The Mathematics data in the table below are taken from TIMSS 2011Exhibit 2.2: Performance at the International Benchmarks of Mathematics Achievement 4^th Grade. http://timssandpirls.bc.edu/timss2011/downloads/T11_IR_M_Chapter2.pdf

GNI is gross national income. Thus GNI per capita is the national income per person. The data in the the table below are from Wikipedia.  http://en.wikipedia.org/wiki/List_of_countries_by_GNI_(PPP)_per_capita

Country	Maths score	World Bank GNP per capita (PPP)	Notes
Singapore	43	59790
Korea, Rep. of	39	30290
Hong Kong SAR	37	51490
Chinese Taipei	34	41385
Japan	30	35510
Northern Ireland	24	36970	GNI per capita for UK, not Northern Ireland
England	18	36970	GNI per capita for UK, not England
North Carolina, US	16	48890	GNI per capita for US, not North Carolina
Florida, US	14	48890	GNI per capita for US, not Florida
Russian Federation	13	20050
United States	13	48890
Finland	12	37990
Australia	10	36910
Denmark	10	42350
Flanders, Belgium	10	39300	GNI per capita for Belgium, not Flanders
Hungary	10	20380
Lithuania	10	19690
Ireland	9	33310
Serbia	9	11640
Portugal	8	24530
Kazakhstan	7	11310
Ontario, Canada	7	39,830	GNI per capita for Canada, not Ontario
Romania	7	15140
Quebec, Canada	6	39,830	GNI per capita for Canada, not Quebec
Azerbaijan	5	9020
Dubai, UAE	5	48220	GNI per capita for UAE, not Dubai
Germany	5	40170
Italy	5	32350
Netherlands	5	43770
Slovak Republic	5	22610
Czech Republic	4	24190
Malta	4	24170
New Zealand	4	29140
Slovenia	4	27110
Turkey	4	16730
Alberta, Canada	3	39,830	GNI per capita for Canada, not Alberta
Sweden	3	42350
Armenia	2	6140
Austria	2	41970
Chile	2	16160
Croatia	2	19330
Georgia	2	5390
Norway	2	58090
Poland	2	20450
Qatar	2	87030
Saudi Arabia	2	24870
United Arab Emirates	2	48220
Abu Dhabi, UAE	1	48220	GNI per capita for UAE, not Abu Dhabi
Bahrain	1	21240
Iran, Islamic Rep. of	1	11400
Oman	1	25770
Spain	1	31930
Thailand	1	8390
Botswana (Grade 6)	0	14560
Honduras (Grade 6)	0	3840
Kuwait	0	53820
Morocco	0	4910
Tunisia	0	9090
Yemen (Grade 4)	0	2180
Yemen (Grade 6)	0	2180

 Three countries entered Grade 6 students for the study. Somewhat confusingly Yemen provided Grade 4 and Grade 6 students. At first glance, it seems unusual to lump Grade 6s in with Grade 4s, but on reappraisal, what is Grade 4 anyway? Is it an internationally recognized standard unit like the kilogram or the ampere? Actually, no. Finns and Croatians start Primary School (Grade School) at 7 years old, while English and Australian children start Primary School at age 5. (Data from the World Bank: http://data.worldbank.org/indicator/SE.PRM.AGES)

Some of the data are for entire countries while others are for, using TIMSS own terminology, ‘benchmarking participants.’ All of the benchmarking participants (e.g. , North Carolina) are sub-units of sovereign states (eg United States of America) , but (confusingly) not every sub-unit (eg North Ireland)  is classified as a ‘benchmarking participant.’  (Well, it provides a possible lesson digression on Venn diagrams.)

The original table (TIMSS 2011Exhibit 2.2) actually provides four different benchmarks (Advanced, High, Intermediate and Low) for each country or ‘benchmarking participant’. The original table also gives the standard error for each data point. Completely arbitrarily, I chose just the ‘Advanced’ benchmark and omitted the standard error.

The choice of economic data was, again, arbitrary. What choices did I have? For one, I might have used either ‘GDP per capita’ or ‘GNP per capita’ instead of GNI per capita.  Secondly, the data I used are not ‘raw’ data but have been ‘put into a blender- chopped, diced and reassembled’ as purchasing power parity (PPP) data using ‘international dollars’. PPP is an expression of the belief that countries manipulate their currency’s exchange rate relative to the US dollar. Therefore, another option would have been to use actual GNI per capita (without PPP).  For another thing, the data are not equally up-to-date. Australia’s GNI data are given as “2009 or 2010” whereas most of the other countries GNI data is for 2011.  For another, England and North Ireland have separate entries for the TIMSS data, but are a single entity (both are part of the United Kingdom) in the Wikipedia economic data. I could go on and on, but my last point will be that some countries are given more weight than others in my data.  For example, both the USA and Canada get three times the weight of Lithuania. The USA is counted as itself and again as North Carolina and Florida. Canada, as such, does not appear in the table but gets entries for Alberta, Ontario and Quebec.

Enough with the caveats, what does the data actually look like?

Remarkably, there does appear to be a correlation between the Mathematics benchmark at Grade 4 and the GNI per capita.  Using my Excel spreadsheet, I calculate the regression coefficient to be r = 0.34. Statisticians would generally consider this to be an example of ‘weak’ correlation.  They almost certainly would go on to calculate r², which would allow them to say that the “mathematics benchmark” explains 12% of the “GNI per capita.”

Remember, ‘explains’ does not mean ‘causes.’  Today’s Grade 4 Maths students cannot possibly determine the size of today’s economy!  Of course the present Grade 4s could be a proxy for the Grade 4s of one or two generations ago. The mathematics ability of Grade 4s of the 1970s could be a determiner of today's GNI. Another possibility is that the causation works in the other direction. Rich economies might spend more on Grade 4 education and better paid teachers might teach in such a way that the children do better on a standardized test of Mathematics.   Or there may be a third variable that causes both of the graphed variables.  

Unlike the data comparing height and shoe size (yesterday’s blog), there should be plenty to think about.  How big is the leap from the impersonal “How much maths education should country X provide to its citizens to ensure its economic success?” to the much more personal “How much maths education do I  need to do ensure my economic success?”  Students might even wonder if substituting “musical ability” or “prowess in sports” in Grade 4 in place of  “maths” would  have produced  a higher correlation coefficient.  I do think it is unlikely that most students would they take the same line as the Guardian newspaper. The Guardian appears unwilling to takes sides but instead errs on the side of caution by writing “The IEA's studies, much like similar intermittent league tables compiled by the OECD, generally contain so much data as to support a series of different, even contradictory, arguments.”

http://www.guardian.co.uk/education/2012/dec/11/english-schools-slip-science-rankings

In the light of the weak correlation seen in the data, does it make sense for politicians to push for better maths and science scores and for more STEM (science, technology, engineering, and mathematics) grads?  Brian Vastag who is a science reporter at The Washington Post doesn’t think so. In a story entitled “U.S. pushes for more scientists, but the jobs aren’t there” he writes:  “There are too many laboratory scientists for too few jobs.” Taking the pharmaceutical industry as an example he goes on to say: “Since 2000, U.S. drug firms have slashed 300,000 jobs, according to an analysis by consulting firm Challenger, Gray & Christmas. In the latest closure, Roche last month announced it is shuttering its storied Nutley, N.J., campus — where Valium was invented — and shedding another 1,000 research jobs.” “(http://www.washingtonpost.com/national/health-science/us-pushes-for-more-scientists-but-the-jobs-arent-there/2012/07/07/gJQAZJpQUW_story.html) 

Data in the Maths Text Book

 In a typical maths text book, data are ‘clean’, easy to use and rather boring. The example below comes from Chapter 10 of STP National Curriculum Mathematics 9B which just happened, along with a calculus text, to be within easy reach of my computer as I was typing.  

 

http://www.nelsonthornes.com/shop/nt/stp-national-curriculum-mathematics-pupil-book-9b

Since there wasn't any data in the calculus text, the STP text became the scapegoat.  For the most part, I quite like the STP National Curriculum Maths series and I am sure that I could have made my point equally well with any of 10 or 15 other texts on my shelves.

Scatter graphs are introduced by considering the hypothesis: “Tall people have larger feet than short people.”  The authors give the following data:

Person	1	2	3	4	5	6	7	8	9	10	11	12
Height (cm)	160	166	174	158	167	161	170	171	166	163	164	168
Shoe size (continental)	36	38	40	37	37	38	42	41	40	39	37	39

I used an Excel spreadsheet to obtain the following graph.  (The authors’ graph is similar but does not include a regression line.)

The authors go on to say that “The points do not all fit on a straight line… Taller people tend to have larger feet but the relationship between height and shoe size is not strong enough to justify the original statement.”  Who could disagree? My Excel spreadsheet calculates the correlation coefficient, r = 0.73, which is indicative of strong correlation. As a correction, we simply need to insert the phrase “tend to” into the original hypothesis to get, “Tall people tend to have larger feet than short people.”  A good teacher will advise the students that correlation is not causation. When x and y show strong correlation on a scatter graph x could cause y, y could cause x, or both could be the effect of an underlying cause. Most students will understand that height is not the cause of shoe size but that genetics and nutrition underlie both.

Having dispensed above – in reverse order–with ‘boring’ and ‘easy to use’ I move on to ‘clean.’ Cleanliness is a complex topic, e.g see http://en.wikipedia.org/wiki/Data_cleansing .

My point is that the text (and presumably the teacher and the class) just assume that ‘height’ and ‘shoe size’ are well defined.  But is that the case? How were the heights obtained? Are all the heights up-to-date? Was an effort made to prevent slouching or standing on tippy-toes? Could it be that some people were wearing their shoes while others were in bare feet? Were some heights measured in feet and inches and afterwards converted to centimeters? What about hair styles?  If someone has a ‘bee-hive’ hairstyle how do we measure her height? 

Amy Winehouse: http://en.wikipedia.org/wiki/File:WinehouseLA.jpg

Next consider the shoes.  Currently, I have three pairs of shoes from different manufacturers. Their sizes are 45, 45, and 47 but they all fit!  It is unlikely that everybody was wearing the same shoe. Unless it is part of a school uniform or they were all interviewed while shopping at Bata. 

Bata shoes: http://bata.com.my/webbata/faces/tiles/product.jsp?productID=1423534&catalogueID=68&categoryID=5137&parentCategoryID=5003

Of course it is possible interviewees were asked for their average shoe size.  Is that a statistic that you know about yourself? An interested student could find a project here: what is the percentage of people who know the range and central tendency (mean, median and/or mode) of their shoe size?

On the other hand, real data tend to be untidy, more difficult to interpret and potentially more exciting.  I hope to follow up tomorrow by looking at some data from TIMSS (Trends in International Mathematics and Science Study). TIMSS is an assessment of the maths and science skills of fourth grade and eighth grade students around the world.