Good Stats Make Us Uncomfortable (IO Psychology)

Topic: Organizational Performance, Statistics, Strategic HR
Publication: Harvard Business Review (OCT 2012)
Article: The True Measures of Success
Authors: M. J. Mauboussin
Reviewed By: Megan Leasher









human resource management, organizational industrial psychology, organizational management





source for picture:


Topic: Assessment, Discrimination, HR Policy, Statistics

Publication: Journal of Business Psychology (JUN 2012)



Reviewed By: Megan Leasher


Adverse impact occurs when neutral-appearing employment practices have an unintentional, discriminatory effect on a protected group. The Equal Employment Opportunity Commission is charged with enforcing all federal legislation related to employment discrimination and adheres to the 1978 Uniform Guidelines on Employee Selection Procedures for “rules of thumb” on inferring whether adverse impact is present.

However, it’s tricky for a plaintiff to present conclusive evidence that adverse impact is present in an organization’s practices.  Statistical evidence is needed to demonstrate whether employment practices are truly discriminating against a protected group.  Jacobs and colleagues investigate a common statistical method, known as “significance testing,” which is often used in courts to demonstrate evidence of adverse impact.  Significance testing compares the difference between the proportion of majority candidates selected and the proportion of protected class candidates selected in an employment decision.  If the test finds the difference between these proportions to be “statistically significant,” courts generally interpret this to mean that adverse impact is present.

This method seems to make sense from a high level, but problems arise when you look under the surface.  The outcome of significance testing is greatly influenced by the number of people who are included in the analysis.  Specifically, the more people you include in a significance test, the greater the likelihood of finding a statistically significant difference between groups. So, if you have an organization with many people included in the analysis, you are much more likely to yield a significant difference between majority and protected groups than you would with a smaller organization with fewer people to include in the exact same analysis.  This is the primary argument of the authors; why do courts use significance testing to demonstrate adverse impact when, by nature of the test, the results would almost always find that big organizations are discriminating and smaller ones are not?

The authors conducted a series of studies to determine sources of differences in adverse impact significance testing.  They found that the number of people included in the analysis was the strongest predictor in whether or not a statistically significant difference was found between groups.  Size  accounted for 49% of the final outcome of the analysis, which was almost five times greater than what any other factor (e.g., score differences on assessment in question, proportion in each group selected, etc.) accounted for.  They also discovered an interesting threshold:  When an adverse impact significance test is conducted with 500 or more people in the analysis, very small differences between the groups’ selection proportions will be statistically significant (yet below 500 these same comparisons would not be significantly different).

These findings support the powerful impact of sample size on determinations of adverse impact via significance testing, but they do not tell us if members of majority and protected class groups are really experiencing systemic, differential outcomes in employment practices.  Unless a statistical method can accurately assess the latter issue, it is meaningless.  This oversimplification leads us to believe that virtually all larger organizations are guilty of discrimination and virtually all smaller organizations are not.  This common practice in courts only serves to make small organizations feel impervious and invincible and leave large organizations running in fear.

The authors close by asserting that regulatory standards should always reflect current scientific knowledge, yet the Uniform Guidelines on Employee Selection Procedures still reflect the science of decades past.  They advocate for not only alternative methods to more appropriately measure adverse impact, but also for a more dynamic definition of adverse impact; one that considers multiple, interactive factors before a determination can be made.  Current practice is supporting the message that to be big is to be bad and to be small is to be nice, which goes directly against the spirit of anti-discrimination legislation.

Jacobs, R., Murphy, K., & Silva, J. (2012, June).  Unintended consequences of EEO enforcement policies: Being big is worse than being badJournal of Business and Psychology

human resource management, organizational industrial psychology, organizational management




source for picture:


Publication: Harvard Business Review (APR 2012)
Article: Good Data Won’t Guarantee Good Decisions
Reviewed By: Megan Leasher







human resource management, organizational industrial psychology, organizational management



source for picture:

Internet-based Data Collection: Just Do It Already!

Topic: Measurement, Statistics
Publication: Computers in Human Behavior
Author: M.J. Naus, L.M. Phillipp, M.Samsi
Featured by: Benjamin Granger

Although many organizations have jumped onto the internet-data collection bandwagon, several issues still need to be addressed. For example, are paper-pencil and internet-based tests of the same trait (e.g., personality questionnaire) or ability (e.g., cognitive ability test) really equivalent? Similarly, are there any reasons to believe that employees respond to internet-based tests differently than they would a paper-pencil test of the same trait or ability?

Naus, Philipp, and Samsi (2008) set out to investigate these questions using three commonly used  psychological scales (Beck Depression Inventory, Short Form Health Survey, and the Neo-Five Factor Inventory).

Although Naus et. al found that the paper-pencil and internet-based survey formats performed equivalently for the Beck Depression Inventory and the Short Form Health Survey, there were differences for Neo-Five Factor Inventory (a commonly used personality assessment tool). What’s going on here?

One possibility is that responses were more socially desirable for the paper-pencil format, since a researcher was present at the time. That is, in the presence of an authority figure (i.e.,  researcher) participants may have responded in order to appear more self-controlled and self-focused. This is likely much less of a concern when completing the same survey on a computer at home (in PJs!).

Overall, respondents perceived the internet-based format to be convenient, user-friendly, comfortable and secure (All great things!). So what can we conclude about these findings? Although internet-based data collection methods have some advantages over paper-pencil methods, there are some caveats to their use. In some cases, the tests may operate differently due to the particular format. Unfortunately, not much is known about how they might differ. However, Naus et al.’s findings suggest internet-based  methods receive good reactions from employees and can save an organization time and money!

Naus, M.J., Philipp, L.M., & Samsi, M. (2008). From paper to pixels: A comparison of paper and computer formats in psychological assessment. Computers in Human Behavior, 25, 1-7.

Is interrater correlation really a proper measurement of reliability?

Topic: Measurement, Research Methodology, Statistics
Publication: Human Performance
Blogger: Rob Stilson

Interrater reliability (still with me?, Ok good) is often used as the main reliability estimate for the correction of validity coefficients when the criterion is job performance. Issues arise with this practice when one considers that the errors present between raters may not be random, but due to bias, while agreement between raters may also stem from bias instead of actual consistency. In this study, the authors’ main goal was to explore the relationship between interrater correlations and validity and also to explore the relationship between the number of raters and validity.

In order to do this, the authors gathered information from 3072 Israeli policemen from 281 work teams who took part in peer rating. The average size of each of these work teams averaged about 12 people and ranged from 5 all the way to 33. The measure used was overall performance (on a 7-point Likert scale). The predictor employed in this study was the ICC (C,k) model, which is equivalent to Cronbach’s alpha. Measurement indices were computed on the team level as rating only took place within work teams.

The predicted variable for the study was the validity coefficient for each work team. This is the part of the study where you could really feel the sweat involved. Here the authors gathered information on
supervisor evaluations, absenteeism data, and discipline data collected over several years (for over 3000 policemen)! The authors then converted this information into z scores with higher scores indicating better performance.

Results showed a weak positive linear relationship between interrater correlations and the various validity indexes. This is not what you want to hear if you are doing peer rated performance evaluations. The authors’ stipulate that the correlation between raters is a conglomeration of factors
having different theoretical relationships with validity (i.e. bias and other idiosyncrasies).

Practical implications from the information gleaned here include the adjustment of validity due to attenuation. If the measurements used in the calculation included non random error estimates, the ensuing calculations will be off. A positive finding for the work world was validity in small units (less than 10 people) was about the same as those for larger units. The authors’ believe this finding may be due to observation opportunity level, which is seemingly greater in smaller work units.

Kasten, R., and Nevo, B. (2008) Exploring the relationship between interrater correlations and validity of peer ratings. Human Performance, 21(2), 180-197.