July 10, 2011
Dunning-Kruger Effect Heightens Dangers Of AI?

Even if more human geniuses get produced by genetic engineering before artificial intelligence is realized does AI doom us anyhow because geniuses will underestimate the threat? This becomes a plausible idea if even geniuses suffer from the Dunning-Kruger Effect. The Dunning-Kruger Effect? Yes, the Dunning-Kruger Effect. It could doom us to being wiped out by hostile A.I.'s.

As you can read at that link, Cornell professor of social psychology David Dunning and his then grad student Justin Kruger did some cool experiments showing that incompetent people are not competent enough in self evaluation to know they are incompetent. People don't know their limits. People assume they can model what's important about their place in the world and make decisions wisely.

What I want to know about the Dunning-Kruger Effect: At higher levels of human intelligence does the mind do a much better job of understanding its own limitations? Will geniuses develop deep enough understandings of limits of their software development skills that they won't overestimate their ability to control artificial intelligences? Or will their excessive regard for their own abilities doom us as they try to make systems too complex for them to understand and control?

Share |      Randall Parker, 2011 July 10 06:19 PM  Dangers Artificial Intelligence

datarimlens said at July 13, 2011 4:51 AM:

1. "I know that I know nothing" by Socrates certainly points to smart people being aware of serious limitations in what they know, even from historical time.
2. Smart people may underestimate threats where the masses are involved, since the average may not be as smart as they are themselves. Safe Management of Nuclear Reactors comes to mind. Humans, as biological beings, are good at the average of what they do, and they live with the flexibility necessary to cope with lots of deviations. However, we are prone to miss, misinterpret, or otherwise fail at exceptions.
3. Ethics: Humans are constructed selfish and the impact of their personal behavior on the remainder of the system is too often not ignored intentionally or unintentionally.

I Dont Exist said at July 13, 2011 11:46 AM:

Let's ask Ben Bernanke.

Assistant Village Idiot said at July 13, 2011 1:24 PM:

I was a member and president of the Prometheus Society, IQ 164+, in the 1980's.

We are often friggin' knuckleheads. When we aren't just absolute lunatics, that is. When you are that smart, you grow up being able to keep believing stupid things, because you are smart enough to out-argue any opposition. Rather like forcing in a screw that is cross-threaded.

I will guess that we are worse than non-geniuses. I had said for years that we are better ruled by the people who got 1200-1400 on their SAT's (old, old scoring. That would be 1250-1450 on recentered scoring, and about 1900-2200 (?) new scoring) than the 1400-1600 crew. The people above 1400 tend not to listen. Modification: Over the last decade, I have begun to think that applies more to the High V, low M people. SATV over 700 may be the worrisome sign. The math people may listen better. Steve Hsu, a physics prof at UOregon has a fascinating blog, Information Processing, that touches on these subjects often.

Randall Parker said at July 13, 2011 7:27 PM:


I suspect there are different kinds of genius personalities that are more or less driven by empirical evidence. Some geniuses live on mental clouds in intellectual castles. Other geniuses obsess with the real and are more outward looking. Sure, M versus V has a lot to do with it. Can a genius model reality or just build up huge rationalizations? Depends on where their strengths lie. Someone with strong spatial and mathematical reasoning abilities and curiosity is more likely to build up realistic models.

There are other personality characteristics that are going to matter. What makes for religious zealots? There's an overlap between that and whatever makes zealots for secular belief systems. People who do not feel such a need to believe in a body of doctrine are going to model reality more accurately.

Lono said at July 14, 2011 9:01 AM:

Assistant Village,

It is funny that you say that - as someone who falls into the former range I find I am often much better at seeing the bigger picture than those in Mensa who are also members of some of the more exclusive High IQ societies.

I have wondered if it is due to the fact that their memories are often so accurate that the sheer number of details they can recall on demand tends to interfere with their ability to think more abstractly - and they often tend to either be, or have similar symptoms to, those who have Asperger's syndrome.

I often find that not only do they often lack a healthy humility but they also seem to get so bogged down in semantics or other superfluous technical details as to come off as less wise than those with slightly lower IQ's.

If you have a second I would love if you would contact me at tempcontactmeemail ATT yahoo DOT com - as I have a question regarding an interesting High IQ think tank whose existence I have discovered just recently - I would be curious to hear your take on it.


I think the Dunning-Kruger Effect surely is present in all of our species - but I think those with higher IQ's probably tend to have a somewhat more realistic understanding of their limitations. Not that they always accept them mind you - challenging one's limitations is often the key to intellectual growth.

Paul R said at August 1, 2011 1:06 PM:

Dunning-Kruger Effect will be found to generalize. It will apply to AI.

xd said at February 14, 2012 8:36 AM:


Good observation. It's just anecdotal, but I recently worked with a boss who could not see the big picture but had a *perfect* memory of *every little detail*.
I could generally speaking guess the right soltion 80% of the time when I only had 80% of the facts. When I was wrong I was *way* wrong but what was interesting was
even though this boss could recite little details I had forgotten that would change my analysis from wrong to right he himself didn't seem to be able to join the dots as quickly as I could. Luckily, however, he wasn't argumentative like many high IQ people are. My own IQ is 141 so I'm hardly a slouch but I find my short term memory limits my ability to argue my position defensively in many cases and humbles me in *specific* situations.

What I find really interesting is the crowd vs experts scenario vis a vis James Suriowicki's "Wisdom of Crowds" whereby a bunch of isolated averagely intelligent non-experts if presented with a problem and a sufficient description of the situation can come up with a solution that is *better* than the expert's solution once averaged out.

It seems that the expert gets closer to the optimal solution than any individual from the crowd but the crowd as a whole nails it even though the expert could out-argue to the point of mockery each individual's thought processes...

Randall Parker said at February 15, 2012 8:12 PM:


In some cases I can imagine a group average answer being better. But for a lot of engineering problems the correct answer is too complex and there's no practical way to average together lots of wrong answers to come up with a useful answer. Don't get a crowd of average people to do diagnoses of tricky disease symptoms either. You are likely to end up dead.

Post a comment
Name (not anon or anonymous):
Email Address:
Remember info?

Go Read More Posts On FuturePundit
Site Traffic Info
The contents of this site are copyright