Research Summaries: Role of test motivation in intelligence testing
3 months ago
IQ is used synonymous and interchangeably with intelligence; however in this paper [pdf] Angela Duckworth et al argue that non-cognitive factors like test motivation also affect the IQ scores and have differential predictive validity.
Raven’s Progressive Matrices Example (Photo credit: Wikipedia)
- Intelligence, which is the ability to flexibly adapt to complex situations, is usually measured using IQ scores on intelligence tests. IQ scores however do not measure juts the raw intelligence; they also measure how motivated someone is to take the test and achieve a high score.
- Intelligence tests, that lead to IQ Scores, are supposed to measure the maximal intelligence ability that a person has and not the typical intelligence that he/she uses. In all intelligence testing it is assumed that the person will devote his entire attention and exert the maximum effort possible so as to achieve the highest score possible.
- While the assumption that IQ measures maximal intelligence may be true in high-stake testing situations, where the IQ results would be used for academic admissions, job placement or promotions; in normal measurement of IQ, say in a typical school setting, the stakes are quite low (there are no real/tangible repercussions of doing bad or well on the test) and hence IQ does not typically measure the maximal intelligence, but is confounded by test motivation.
- Test motivation refers to the fact that some people will be less motivated to take the test or continue with it and may display behaviors that indicate low motivation. While others may be highly motivated to take the IQ test. Thus, there would be individual differences at trait level on test motivation.
- Test motivation is also a state variable that can be manipulated by incentivizing getting high scores on the tests. When such incentives are in place, the IQ score should increase from the baseline level or when the test was given under non-incentivized conditions.
- Intelligence, as measured by IQ, has been associated with a number of good outcomes. Non cognitive factors as measured by test motivation are also theoretically linked to important life outcomes. For the purposes of this paper, two academic outcomes (years of education and academic achievement) and two non-academic outcomes (employment and criminal conviction) were measured and analyzed.
- The study 1 performed a meta-analysis of various independent samples where a comparison was made between the IQ scores received in standardized conditions vis-a-vis under incentivized conditions. For analysis the sample was divided in high IQ (those with IQ greater than 100) and low IQ (those with IQ less than 100). The main results were that incentives did result in higher IQ scores, the effect was stringer for low IQ group and there was dose-response effect with larger incentives leading to greater IQ points gains.
- Thus, for low IQ group, the lower IQ scores in standardized conditions could be due to lower intelligence or lower test motivation. If you increase the test motivation, you could bump up the IQ score of some of them. High IQ group, on the other hand had higher scores because they had both higher intelligence and higher test motivation.
- In study 2, a thin-slice video of children giving the intelligence test was behaviorally rated for signs of low test motivation. This was a longitudinal study and the IQ scores, test motivation and four types of outcomes were analyzed to find the differential impact of IQ/intelligence and test-motivation/ non-cognitive factors on life outcomes.
- The main finding was that test motivation had a significant impact, independent of IQ, on important life outcomes. This was specially pronounced for nonacademic outcomes like employment and criminal convictions. Intelligence as measured by IQ still had significant effect on all adult outcomes. They also found that test motivation predicted IQ scores, thus IQ score measures both intelligence and test motivation.
This is an important paper [pdf] that shows that IQ scores need to be interpreted with caution, and that both cognitive and non-cognitive factors are important for life outcomes.
Research Summaries: Mental contrasting facilitates academic performance in school children
3 months ago
Today’s research summary builds on the work of Gabrielle Oettingen on WOOP/mental contrasting with implementation intentions. The paper [pdf] is co-authored by Angela Duckworth et al and successfully demonstrates the utility and incremental benefit of mental contrasting over mere positive thinking in achieving desired outcomes.
The Power of Positive Thinking (EP) (Photo credit: Wikipedia)
- When one wants to achieve goals, then the first step is to clearly articulate the desired goal. It has been shown that merely having a goal vs not having a clear goal is instrumental in goal achievement. Another process that is usually implicated in successful goal achievement is positive thinking, where you clearly visualize the positive outcomes from having achieved the goal.
- The exercise ‘Best possible future selves’ is predicated on the same premise that visualizing a better future self leads to increase in hope and optimism and positive striving to achieve the goal.
- In popular parlance though, positive thinking is equated with not thinking about any negatives at all, including the possible obstacles that may lie in the path. This obsession with just the positive aspects of future, to the ignoring of the current reality, may have detrimental effects as one’s commitment to the goal may not change with mere positive future visualizing.
- Mental contrasting is a technique whereby a positive future outcome visualization is contrasted with current reality and the client encouraged to think about internal obstacles within them that may hamper the goal achievement.
- Goal commitment is hypothesized to be made of two components: Goal desirability( which apparently does not change with either mental contrasting or positive future visualizing) and Goal feasibility (goal commitment increases in mental contrasting if the goal is considered of high feasibility as the current reality/obstacles become surmountable in one’s mind’s eyes; on the other hand if goal feasibility is low then goal commitment becomes less as the obstacles seem insurmountable and the goal is disengaged from while doing mental contrasting)
- While the exact mechanism of how mental contrasting works in not known, it is believed to work by increasing efforts (towards overcoming surmountable obstacles) , by using better strategies ( for example to remain focused and not get distracted) or by seeking help from others.
- The current studies consisted of making the class 2, 3 or 5 grade students learn foreign language words, and this learning was incentivized by promises of candy bag or small monetary reward (5 $). The gap between learning and recall varied from 2 weeks to 4 days. There were two conditions: in the positive future condition, the students filled out a section in which they listed the best possible outcomes from having mastered the foreign vocabulary words. In the mental contrasting condition, the students besides writing the best possible outcome, also reflected and wrote, what within them may prevent their achieving the goal of mastering the foreign language vocabulary.
- The foreign language vocabulary task was something that was within the capability of the students and was thus considered a task with high goal feasibility and thus should have led to greater goal commitment in the mental contrasting condition.
- Across two studies they found that indeed there was significant difference in recall of foreign language words between the two conditions, with mental contrasting leading to better learning/ recall.
- One big limitation of the study , which is acknowledged by authors in the limitations section, is that they did not include a neutral control condition in which neither positive future visualization nor mental contrasting was used. It would have been interesting to know how big an impact positive visualization has and how big an impact mental contrasting has over and above that.
- This paper is of immense practical utility as it showed that mental contrasting can also be used in group settings and is effective with minimum instructions and for a common goal. This enables tools like WOOP which build on this research to be extended to group settings. I myself use WOOP in my work with school children and have found it very useful.
Overall it is a pretty decent paper [pdf] that shows the benefits of mental contrasting over mere positive future visualization.
Research Summaries: The Effect of Self-Distancing on Adaptive Versus Maladaptive Self-Reflection in Children
4 months ago
Today’s research summary is based on a shortish paper [pdf] by Angela Duckworth et al (Walter Mischel of Marshmallow effect fame is a co-author!) which focuses on how viewing oneself from a distance, or from a third person perspective, a previous emotional experience, can lead to better and more adaptive outcomes.
Out of body experience (Photo credit: Wikipedia)
- Bad stuff happens. And we make it worse by brooding about it. There is some research that shows that thinking or ruminating about negative experiences can lead to bad outcomes in the present like compromised health or impeded cardiovascular recovery following exercise etc. Ruminative thinking style is known as a precursor and risk factor for depression.
- On the other hand there is a rich tradition of expressive writing (for e.g. Pennebaker’s work) in which people write about their negative experiences and traumas and seem to benefit (boosts in long term mood and well-being) from such an expressive act.
- Different sort of mechanisms are hypothesized in both the above cases. In the first case, one may be reliving the negative experience or recounting it and thus get overwhelmed once more in the present by such a recollection. In the second case, one may be reinterpreting the situation and making fresh sense of the events or reconstruing the events. So reflecting in a negative experience per se may not be bad or good but may lead to a good outcome only when reconstruing happens more than recounting.
- Putting a distance between oneself or seeing events from a detached third person perspective have been shown to increase one’s self control and control one’s impulses and also helpful in alleviating depression by enabling better cognitions. It has been hypothesized that self-distancing or viewing things form a detached third person perspective will lead to better and more adaptive outcomes while self-reflecting, as one will not recount or relive the experiences but will be better able to reconstrue or make new sense of the experiences.
- The current study looked at ~ 100 fifth grade students and asked them to recollect a negative angry outburst/ interaction which was interpersonal in nature. They were then instructed either to feel the event as of it was happening in the present and they were at the center of the action, or that they were watching the event unfold from a distance and observing the distant self. After they had recalled the experience in both conditions, they filled a brief survey measuring their emotional reactivity (how much power the vent still holds over them) and avoidance behavior (do they avoid talking/ thinking about that issue) . They were also asked to write an essay about their reflection and the essay was content analyzed for recounting thoughts, reconstruing thoughts and blame attributions.
- The results showed that when you put a distance between self while recollecting a negative experience, then the emotional reactivity is lesser than when you feel as if you are reliving the experience. Thus, if you want to make a negative experiences hold smaller on you recollect it while putting a distance from self. Thus it was clear that self-distancing was a more adaptive outcome.
- They also found that those students who had put a distance between their earlier self while reflecting on their angry interaction, had fewer recounting statements in their essays and more reconstruing statements. They also made fewer blame attributions.
- They also did a path analysis and found that self-distancing had its impact on more adaptive outcomes (less negative affect and emotional reactivity) via the mediating variables of more reconstruing statements than recounting statements, which in turn led to lesser blame attributions and thus a closure that led to lesser emotional reactivity.
- The take home message, children can benefit form self reflective exercises that make them reflect on negative experiences as long as they are supported in putting a distance between themselves and their past self, so that they don’t merely recount the experience but are able to reconstrue the experience.
Overall, a pretty decent paper [pdf] that stresses the importance of self-distancing while reflecting about past negative experiences.
Research Summaries: Personality Measurement and Assessment in Large Panel Surveys
4 months ago
Today’s research summary looks at another paper [pdf] by Angela Duckworth et al this time focusing on whether it makes sense to include personality variables in long national longitudinal surveys/studies like the MIDUS/ Dunedin/ HRS.
Nonconcordant traits (Photo credit: Wikipedia)
- Personality differences can be conceptualized to be either differences in ability (like cognitive ability), traits (stable patterns of thinking, feeling, acting) , motives or narratives and this paper focuses on traits to the exclusion of other measures of personality. Even in traits, the traits of concern are the Big Five traits of Neuroticism, Extraversion, Agreeableness, Conscientiousness and Openness.
- Personality, in general, and these traits, in particular, are known to predict a range of outcomes like health, achievement, and relationships. The authors believe that large panel surveys should measure these traits to find the correlations with other outcomes being measured. They review research on how traits predict wealth and health and are predicted by underlying genetic polymorphisms or variations.
- For elaborating the association between traits and genes they look at candidate gene studies as well as GWAS. Extraversion is associated with polymorphisms in Dopamine subsystem related genes. Nueroticism is primarily associated with serotenergic genes. Agreeableness and Conscientiousness are both affected by polymorphsism in genes related to dopamine as well as serotonin. Openness to COMT variation. Read the paper to get additional nuances.
- When it comes to economic outcomes, more introverted and more emotionally stable (less in neuroticism) individuals were more likely to save over the lifetime and borrow less; reverse was found for those high in agreeableness. Emotional stability was the best predictor of earnings; extraversion had a complex relation but overall positively predicted earnings; while agreeableness had a very slight negative impact on earning.
- In terms of health, traits like Conscientiousness had a direct effect on health as well as indirect effects mediated by healthy behaviors and educational attainments. In general it is safe to conclude that personality traits do not affect health outcomes directly but by their impact on problematic or protective behaviors. Personality traits have also been linked to mortality.
- The authors recommend that personality traits should be measured in large panel studies, and measured as far as consistently, using say BFI, so that they can be used to predict important life outcomes. Moreover they recommend that as personality traits can change , they should be treated as dependent variables too and measured in each subsequent measurement time.
- One recommendation they have is to keep such trait measures short and relevant; also they recommend multiple measures using informant reports or cognitive tests like go-no go task. However I ‘m not sure if that may be practical in large surveys.
- They also highlight the concerns about ‘flush-right’ responding where some unmotivated participants who are juts going through the motions of filling the survey may keep choosing the extreme right option making the survey results suspect. The instruments should have something inbuilt to detect such responding just like one detects social desirability.
Overall its a pretty decent paper to understand some of the antecedents (genetics) as well as consequents (health and wealth) of Big Five traits and makes a strong case for incorporating big five measures in such large scale studies and surveys. Check the paper here [pdf] .
Research Summaries: Establishing Causality Using Longitudinal Hierarchical Linear Modeling: An Illustration Predicting Achievement From Self- Control
4 months ago
Today’s research summary is slightly technical. It is based on this paper [pdf] by Angela Duckworth et al that shows a causal relation between self-control and academic achievement.
The Illustrated Sutra of Cause and Effect. 8th century, Japan (Photo credit: Wikipedia)
- Some personality variables like self-control predict important life outcomes. It is well know that self-control as measured at age 4 (using the marshmallow test) can predict important life outcomes years later. However, prediction may not imply causality as a third factor may be responsible for causing both the phenomena under consideration.
- The test for causality is a) causal variable must precede the effect in time; b) the causal variable and outcome variable should be correlated; and c) any third party confound or variable should be ruled out. This is easy to achieve in double blind randomized placebo controlled experiments, but personality traits like self-control are hard to manipulate as trait variables in experimental settings.
- Typically personality traits and their outcomes are studied using a longitudinal study design where changes in say self control at time T1 are correlated with outcomes like academic achievements at a later time T2, of course measure other confounding variables and factoring their effects; thus self-control, along with IQ, may be measured at the beginning of a school session and at the end of session the CGPA obtained will be used to find whether and how much self-control led to academic achievement. This however cannot establish causality in a strict sense as not all variables of interest can be identified and measured. Often the dependent variable (CGPA in our case) is itself controlled for to ensure that a higher CGPA at point T1 does not lead to higher CGPA at time T2 independent of self-control at T1.
- To take care of third party confounding variables, Angela et al used growth curve analysis with Hierarchical Linear Modelling (HLM). This involves taking multiple measures of say self -control at different times and also multiple measures of the outcome say CGPA. The independent variable is considered a time varying co-variate and used to figure the within-person relationship between the two variables of interest. Consider a between subjects confound like socio economic status (SES) that could potentially lead to different outcomes (CGPA) – if not controlled for the self control- CGPA relation arrived at by analysis of between subjects data might lead to erroneous conclusions. However, a stable thing like SES (which doesn’t change with time and is constant for an individual) will have no impact on the correlation or causal relation between how changes in self-control affect CGPA over time in the same individual.
- The direction of causality can also be ascertained by using HLM with reversed time lagged, time varying co-variates. What this means os that we can try to see of the causal arrow runs in other direction by taking measures of CGPA as predictor and self control as outcome variable.
- In this study, self control was measured using self-report, parents and teachers ratings of students for four consecutive academic years (as they moved from fifth grade to eighth grade) using the Brief Self-Control Scale ; CGPA was measured each year as the outcome variable. Self-esteem and IQ was also measured and so was gender, ethnicity etc.
- They found that self control measured 6 months earlier predicted CGPA six months later; average self-control predicted the baseline CGPA as well as the slope of CGPA changes (how fast the CGPA increased or decreased over time). Howsoever, the reverse analysis whereby short term CGPA was used to predict self-control gave negative results thus establishing the causal direction.
- It was thus established that self-control does indeed cause or lead to higher academic outcomes like higher CGPA. A limitation of the study was that a time varying third variable that increased and decreased in tandem with self-control can still account for the relationship between self control and academic achievement.
I liked the paper, though its more methodological. You can find the full paper here [pdf].
