“There is no more dangerous mistake than the mistake of supposing that we cannot have too much of a good thing.” Thus spake George Bernard Shaw. Excess is as damaging as shortage in most things that are considered good. More is better but only up to a point of satiation. Beyond the satiation point, the marginal utility of a good is negative, as an economist may put it. Particular instances of that generalization are not hard to find.

Food, for instance, is a good that in excessive quantities is a bad as the success of the dieting industry so starkly demonstrates. Yet tens of millions poor people around the world dying of malnutrition and starvation every year is the horrible demonstration of the problem at the other extreme.

The same holds for information.

What brings all this to mind is the so-called information revolution occurring globally. Information is a good which is also subject to the law of negative marginal utility beyond the satiation point. Information overload can be as debilitating as too little. For much of the world now the problem is no longer a shortage of information but rather a surfeit.

All living beings acquire information from their surroundings for survival. Information needs vary depending on the nature of the being and the sense-organs have co-evolved to perform the required function. The more complex the entity, the more sophisticated the sense organs are. The sophistication has two dimensions: the bandwidth and the filter.

Higher bandwidth means that more information per unit of time is sensed by the entity. But the information gathered is filtered severely for relevance and only a small percentage is passed on to the processing unit (the brain) for internalization and for response. Therefore the filtering mechanism had to evolve in step with the evolution of high bandwidth sense organs.

Here is my conjecture. Higher intelligence is marked by possessing very high bandwidth sense organs (channel capacity), and a very sophisticated filtering system which rejects most of the inputs and processes only a very small portion of the total information received

We humans sense the world around us mainly through our eyes and ears. Most of the megabits per second of information we receive is rejected and only a minute fraction is processed by our brains. The filtering mechanisms developed over our evolutionary history and did so gradually. Even the brain has an inbuilt capacity to forget. There are a few who suffer from a pathological condition which does not allow them to forget anything that they have ever seen or heard.

Now we are faced with a world where information is being generated and accumulated at an exponentially increasing rate and we face the possibility of information overload that could overwhelm our capacity to filter and meaningfully process it.
. . .
I conjecture that the age of information dieting industry is upon us. The day is not far off when you would have to go on an information diet.

In a follow up post, I addressed the issue of sorting and searching the large stock of public information and claimed that “in post-industrial world and increasingly so in the future, sorting and searching through information will occupy the role that manufacturing did in the growth of the old economy.”

Sorting and searching through information are uniquely human activities because only humans have an external store of information which needs to be accessed and acted upon. The notion of acting on information stored externally is not associated with non-human animals.

The larger the stock of information, the more expensive it is to search through it to locate the precise bit that is relevant at any particular instance. To make the task of searching more tractable, ordering the information in some fashion—called sorting—becomes paramount. Computer scientists have worked on the problem of sorting and searching for decades with phenomenally successful advancement in our understanding in this regard.
. . .
When the quantity supplied of any good is in excess of anything reasonably required or demanded, the variable of importance is quality. Basically a person’s information needs are really very simple. One can only read so much, listen to so much talk and music, watch so much video, and wish to know only so much about what is going on around in one’s neighborhood and in the world at large.
. . .
The bottom line is this: there is already so much information out there that even if no additional information were generated, each one of us could be occupied a little longer than forever to finish it. Information, as we well know, is a non-rival good. That is, my “consuming” a particular piece of information will not diminish the amount available to you. Compare this to a rival good such as food. Stock of food is enough to last the six billion humans for about 3 months. In other words, if we produce no additional food, all together humans would finish the stock in three months. Or, a single human can therefore finish this in 1.5 billion years. But it is not so in the case of information. Each of us would take the estimated 18 billion years to finish the information we already have before we ask for more.

Clearly, for an average human, about 0.00000000001 percent of the total information stock is more than enough. About 99.9999999999 percent of the available information is worthless. So how does one go about searching out the teaspoonful of useful information from the oceans of available information. That is the challenge and therein lie the opportunities. That is why firms like Google will make the big bucks. The opportunity is not so much in making information available but making the right information available.

Which brings me to the point which I started off with. Searching is only part of the story when it comes to information. The other part is sorting. If one can sort the information along some relevant dimension, then you have meaningful information. What is meaningful can only be defined in the context of the entity processing the information. From the same stock and flow of information, different entities define different subsets that are relevant and meaningful. This subset can be labeled private information as opposed to the vast store of public information. Private information is the top of the sorted list of public information. Internalizing the private information leads to what we can call a stock of knowledge associated with the individual.

It is useful at this point to remind ourselves of the distinction between information and knowledge. Information is a public good the stock of which is growing exponentially. Knowledge is a private good and its primary raw material is the private information which is a very vanishingly small subset of the available public information. Even though public information has no known bounds, there are limits to how much private information can processed by a human brain and thus there are limits to the acquisition of the private good we call knowledge.

Conflating knowledge and information is distressingly too common these days and so I would like to dwell on this distinction for a bit. Some say that today we have a knowledge economy. It is trivially true because it has always been a knowledge economy ever since humans evolved brains capable of processing information into knowledge and began using knowledge to organize and coordinate economic activities. What is novel is the unfathomably huge stock of information we have available today. What distinguishes one individual from another today is the capacity to figure out what is relevant information and to internalize it efficiently into knowledge. That capacity is one of the basic skills imparted by what we call education.

To summarize the story so far: from the vantage point of an individual, this is an age of superfluous information; only a tiny fraction is relevant and meaningful; searching through the information can be automated but efficiently sorting for relevance is a private skill; imparting that skill is a primary function of education.

And now a bit from the Economist article linked above.

[Researchers of the dense data fog] raise three big worries. First, information overload can make people feel anxious and powerless: scientists have discovered that multitaskers produce more stress hormones. Second, overload can reduce creativity. Teresa Amabile of Harvard Business School has spent more than a decade studying the work habits of 238 people, collecting a total of 12,000 diary entries between them. She finds that focus and creativity are connected. People are more likely to be creative if they are allowed to focus on something for some time without interruptions. If constantly interrupted or forced to attend meetings, they are less likely to be creative. Third, overload can also make workers less productive. David Meyer, of the University of Michigan, has shown that people who complete certain tasks in parallel take much longer and make many more errors than people who complete the same tasks in sequence.

Curbing the cacophony

What can be done about information overload? One answer is technological: rely on the people who created the fog to invent filters that will clean it up. Xerox promises to restore “information sanity” by developing better filtering and managing devices. Google is trying to improve its online searches by taking into account more personal information. (Some people fret that this will breach their privacy, but it will probably deliver quicker, more accurate searches.) A popular computer program called “Freedom” disconnects you from the web at preset times.

OK, I have added to your information overload. Sorry.

[I even have a category called "Information Overload." How cool is that!]

No surprise to me as I have observed the same sort of insanity in the case of the children of friends and family.

Kids are pretty much forced to do school related activities nearly 14 hours of the day. They go to school for classes, and then go to “tuitions,” and then come back to get homework done, and . . . By the time they are done for the day, they have had no free time. They don’t have time ever to just sit and stare. There is no time for self-reflection. They are becoming narrowly focussed and self-absorbed through all the constant doing of required things.

The textbooks are horrors.

I saw the economics textbook of a kid in the 10th grade. He goes to an IB school in Mumbai. It is a fat tome of about 600 pages long and has within its covers every conceivable topic on economics. It is dense with information. If I had to read that book for comprehension, it would be work for me. I only have a PhD in economics and I would find it hard going.

The kid was struggling with “effect of expansionist monetary policy on aggregate demand” or some such nonsense. Dear god in heaven. What the hell was wrong with these idiots who try to teach macro to 14-year olds? I read the relevant pages and I could not make too much sense of it. The kid of course was so out of his depths that he didn’t know even the most basic of concepts, forget “aggregate demand.” I asked him to explain to me what a demand curve was and got a look of helplessness and worry. He tried in vain to recall from memory some definition of a demand curve.

It was a pitiable situation. I explained that I was not looking for a definition. What I wanted was an explanation of what it was. Then I took a few minutes to explain what it was. By starting at the beginning, I said that it’s an observed relationship between this and that. What do we mean when we say that two things are related? It was a series of questions and answers, slowly building up to the concept. It took so little time but at the end he got the concept so thoroughly that you could shake him up from deep sleep and he will be able to explain to you what a demand curve is and not have to struggle for a definition which he could parrot without comprehension.

I felt that all that his economics course and the book had achieved so far was demonstrate to him that the subject was incomprehensible and that he was inadequate. It had turned him off the subject.

Schools are turning the kids into uninteresting and uninterested people. Their learning so shallow that it all evaporates at the slightest disturbance. In all the furious teaching, what is lost is learning.

What’s the way out? I think there’s a need for a “back to the basics movement.” One of the things that needs to be done is to reduce the amount that is force-fed to the kids. It may appear paradoxical but I believe that school curricula have become obscenely obese.

They say that perfection in a work is achieved not when you have nothing more to add to it but rather when you have nothing more to subtract. The irreducible core is what matters. And one needs all the time in the world to understand that core.

There was a time when it was easy to keep the focus of teaching and learning to that irreducible core. We did not have fat tomes with excruciating details on every conceivable topic. Neither we nor the system could afford fat books. We did not have access to a gazillion web pages with dancing doo-daahs on every topic in the known universe. We were not under the constant pressure of learning new stuff every single day. We did not feel overwhelmed by it all. It was really a very relaxed time. Sit in a few classes every day and when school was over, just run around the neighborhood. Maybe do a bit of homework every now and then but not every day.

We could just sit around and not feel that we were not going anywhere. These days they have designed the Alice-in-Wonderland school system where kids have to run as fast as they can just to stay in the same place.

I think I know what the problem is: people with pre-www mindsets have wandered into a www-world and are totally lost. Their pre-www mindset says that you must commit to memory whatever information you have access to; in the www-world, memorizing available information is not only impossible but it is also totally unnecessary.

Technological change is rapid and is accelerating at an accelerating pace. That’s a function which is differentiable at least twice. Humans adapt to change much more gradually. Human institutions are even slower than humans in adapting to change. The education system is perhaps the slowest human institution when it comes to adapting to change.

Information technology has created an ocean where previously there used to be at most a small deep well. The educational system, a heavy concrete structure anchored to the ground, was designed for a world where water was limited and precious and could only be drawn from a small well. That all changed without much warning. Now the ocean of information has drowned that building since it cannot float. What we need are boats, not buildings. Where previously one had to work at drawing water out of the well, now the task is to keep afloat in the ocean and keep the water out of the boat. Take in too much water and you are sunk.

So here’s an opportunity. Forget the submerged huge buildings. You cannot uproot them from the ground, and even if you did, they will be impossible to float. Those who are stuck in the buildings — the people from the ministry of education mainly — are beyond rescue anyway. Instead, build boats that can float on the ocean of information.

Rebuilding education is perhaps the greatest challenge and the greatest opportunity the world has had in a long time. Technology matters of course and it is easy to note that the most important and innovative technology companies are located in the developed world. That’s so because they co-evolved. Since technology will be at the core of the rebuilding of the education system, it is quite likely that the best new education system will also be created in the developed world. But the real need for a new education system is in the developing countries because they have the populations and the demand is huge.

India could be the home of the new education revolution. India passed up an opportunity for becoming a manufacturing giant; China got that one. I just hope and pray that it does not pass up on this one. The large corporations in India could do it but the government of India will not allow that to happen. Why? The Congress government has not allowed Indians to become fully literate. It cannot afford an educated citizenry because it depends on the poor and illiterate to vote for it.

It’s all karma, neh?

Related posts:

1. The World is (Information) Fat. June 2005.

2. The Age of Superfluous Information. Oct 2005.

3. The High-tech Puzzle. July 2007.

4. Infinite Information, Infinite Ignorance. March 2008.

On the 24-hour news channels, the presenters have to keep talking non-stop about whatever is the breaking news. Naturally, it is humanly not possible to say something meaningful about any event without some time to think about it. So the need to keep talking incessantly about an event which can only be described in a few words results in verbal diarrhea. There is so little content in the story being told that the TV screen has to be filled with all sorts of other items: there are two or three lines of scrolling texts relating to different issues, some totally meaningless video occupying part of the screen, another part of the screen given to some advertisement, etc.

The nature of news is that it has little information content. News is froth on the surface of a deep ocean of interrelated events that unfold over time spans that range from weeks and months to years and decades. Merely being told about the news — even if the same little bit is repeated ad nauseam — when one lacks some familiarity with the bigger more persistent issues, is useless. It gives one information without any increase in knowledge and understanding.

I don’t like news programs that look at the world from the minute-to-minute perspective. Tennis matches and soccer games lend themselves to that but not real news.

People have different preferences, thank god for that. It’s a matter of taste and while I would recommend what I like, I don’t expect others to agree. I like programs that help me better understand what happened, how it is related to the past, what impact that will have in the future, and provoke me to think about the unanswered questions related to the event.

I would certainly like a program which is titled “Not News”. In that program, I would like to hear a couple of people who have a deeper understanding of the world than I do talk about what is going on. I note that “breaking news” often dig up talking heads but it is an ugly sight. It is always a lack of time: too little time to have pondered the issue at length before being thrust in front of a mic or a camera, and too little time to actually say what they have to say. The best they can do is to make some inane observation and it is back to the incessant chatter of the tv presenter.

I am not as familiar with Indian tv programming as I am with American tv. I think American commercial tv news programming is basically crap, and I suspect that Indian tv programming is heavily influenced by American tv (like most of the other garbage that the US produces and the rest of the world readily adopts.) But there is great stuff on American tv as well. I really like the PBS news programs like “The News Hour with Jim Lehrer”. For thoughtful views, I like programs like “Charlie Rose” and Terry Gross’s “Fresh Air” on public radio is an all-time favorite.

I think that there is a market for such programs in India. One of these days, some people in India will make a pile of money and make a significant difference by starting such channels. I suggest “Not the News” as the title of one such program.

Related posts:

Information Overload (May 2007).

The Age of Superfluous Information — Part 1. Part 2. (Oct 2005)

The World is Information Fat. (June 2005)

The world has come a long way since the 1960s when the future was defined by one word – “plastics” – as Mr McGuire advised the young graduate Ben. Now the future is defined by another word and the word is “information.” Plastics was a wonder product of the world of industrial technology which fundamentally transformed the world of objects. Information is the new thing, the product of information technology, which is going to transform the world of ideas. Actually, information is not a “thing” in the usual sense of the term. So it is the new non-thing which defines the new and exciting future.

Let me enumerate some fun facts about information. First, people produce information. So now that more people are producing information, a lot of information gets produced. Second, information accumulates. Once produced, unless every copy disappears, it persists. Third, it is a “public good.” One person’s use of a particular bit of information does not preclude another person from using the same information. Fourth, when information is “internalized” it becomes knowledge in a human brain. So the monotonically increasing stock of information raises the potential of acquiring knowledge by other humans. Processing information is one of the necessary steps in the acquisition of knowledge. Knowledge in turn is a necessary ingredient in the process of generating ideas. Ideas eventually fuel the engine that drives human civilization.

So this note is about information, knowledge, ideas, human civilization, and the rest of it. A pretty large subject which I will necessarily deal with fairly superficially given my own limitations. First I will explore the subject from a micro perspective and then move to the macro. The objective is to draw some plausible conclusions about where we as a collective of humans are headed.

Rejecting Information

The object of analysis at the micro level is the individual human. At the bare minimum, a human has to have a brain and a set of sense organs for acquiring information. Mostly it is through hearing and seeing that one receives input information – touch, smell and taste are not as important in the modern world as it would have been in our pre-literate past. Only if one is blind, or cannot read and is unable to comprehend language, do touch, smell and taste predominate – with the possible exception of tasters, noses, lovers and toddlers. Observe a toddler and note how he or she acquires information.

Physiologically the sense organs take in a huge amount of information that gets filtered and most of it is rejected. For example, from the total visual input from the eyes only a tiny fraction of the information gets processed by and stored in the brain. What we perceive is much smaller than what we see. Our brains would be overloaded if it were to process every bit of information that is presented to it. The different kinds of living organisms filter out different bits of information from the environment. Who you are determines what you perceive.

Biological versus the Artificial

A person acquires information from the environment and also the ever-increasing stock of created information. At this point it is useful to distinguish between what we can call the biological (or natural) environment and the cultural (or artificial) environment. The natural environment is that world which our species evolved in over evolutionary time scales. Our sense organs and our brains are in a strict sense biologically fit to deal with the natural world. The ability to deal with the information from the natural environment is hard-coded within us. We don’t have to go to school to learn how to process the information.

The artificial environment is created by human action. The information from it comes in terms of language and words. We have to go to school to learn, so to speak, how to process that information. An artist and a neurologist could see the same brain scan images but perceive it entirely differently because their training is different. The neurologist has over the years taken in a lot of information about brains and internalized it into knowledge. That knowledge allows the neurologist to process the information of the brain scan differently and thus acquire additional knowledge. The artist also acquires additional knowledge from the brain scans but that knowledge is different from that of the neurologist.

Sequencing

The point is that what you know already determines what you are additionally capable of knowing. There is a path dependency in the knowledge sphere that is tied to the sequence in which information was presented. Though the information available may be comprehensive (in the sense that it is complete), if the sequence of presentation of that information is out of order, it will not be comprehended. Graduate level physics information has to be presented after the undergraduate level physics has been internalized for it to make sense.

Knowledge accumulates in a human brain to the extent it is presented information in the correct sequence. It is not even theoretically possible for an external agency to determine what the correct sequence for a particular individual is. It is so because an external agent cannot fully know what the knowledge base of an individual is at a specific time. The solution is therefore to let the individual himself or herself pick out the next bit of information to internalize from a reasonably broad set of information.

Teaching versus Learning

This is where we need to distinguish between teaching and learning. Traditionally “teaching” is when an external agent presents information and expects the individual to internalize it into knowledge. “Learning” is when the individual picks up the next bit of information from the available collection. Learning can never be out of sequence. Teaching often fails in its attempt to impart knowledge because it is not even theoretically possible for an external agent to fully comprehend the internal knowledge state of the student and therefore competently present the information in the right sequence.

Summing up the points so far: information is the basis for knowledge in the brain; knowledge accumulates by internalizing information in the correct sequence.

Infinite Ignorance

The totality of information available to humans is enormous. Let’s call that “public information.” From that collection, each human being internalizes whatever little bit it is able to. That is “private information” leading to “private knowledge.” Since there are around 6 billion brains in the world, each brain has unique private knowledge but derived from the same public information. The larger the population, the greater is the stock of public information. But given the limitations of the human brain, progressively any human’s private information shrinks relative to the public information. In other words, a person becomes more ignorant relative to what is potentially knowable. All of us are privately ignorant in a world awash in information. Some time ago – perhaps as recently as a few hundred years ago – a person could potentially know a reasonable fraction of the available public information. Today that percentage would be approximately zero.

A world of infinite information is also necessarily a world of infinite individual ignorance.

This poses enormous challenges for the individual as well as humanity as a whole. As individuals, we have to accept that we cannot know everything that we potentially know. A trivial example. A few decades ago, you could have enjoyed watching within the year every movie made anywhere in the world that year. The trouble would have been that you would have had to be fabulously rich to go see them. You had the time but accessing the movies would have been costly. Today, it is fairly trivial to have access to all movies produced. But you just don’t have the time to watch even the good ones produced in just one year. World enough but time.

The Challenges and Opportunities

The challenge for the individual is how to choose which bits of the public information to consume and in which sequence. We are biologically equipped to filter out the massive amount of information coming at us from the natural world. We are not equipped to naturally filter out the currently massive amount of information coming at us from the artificial world. An individual’s success in doing so determines how successful one is in this artificial world. One of the primary jobs of the education system we need is to give us that skill. We did not need that ability and therefore our current educational system which was created for a different environment is totally ill-equipped to handle this task.

That brings us to the macro level. Any organization which does the filtering of the public information for individual use is going to be phenomenally successful. The largest corporations will be those that deal with information in the future. One can be accused of Monday morning quarterbacking for saying that. You could point to information technology giants of today and say that the lessons are plainly evident. But I don’t think that we have fully understood what the real lesson is. The point isn’t making a lot of information available to the individual. The point rather is that any institution that most efficiently and effectively reduces the information available to an individual will succeed.

General Purpose Machines

The other lesson pertains to education. The old paradigm was one-size-fits-all because only one size was available. It was an older, simpler, static world where you could learn a small set of skills and hoped to cope with the world for the rest of your life. The dynamic world of today requires constant learning and the acquisition of new skills. A useful analogy would be the distinction between a special purpose machine and a general purpose machine. A typewriter is a special purpose machine while a computer is a general purpose machine. Depending on what software you load, a computer can do a range of things – from guiding spaceships to controlling your microwave oven. People have to become the equivalent of general purpose machines. People must become capable of “loading the appropriate software” to handle any task they want done.

The education system of today churns out special purpose machines. To make it produce general purpose machines requires a few basic changes. First, it has to teach a set of very basic skills so well that everyone is literate and numerate. That is equivalent to designing a machine which has a complete set of machine instructions which it executes very efficiently and all the other tasks are just the execution of a long sequence of these basic operations. Once you know how to competently read, write, do arithmetic, and reason logically, you can pretty much learn how to do pretty much anything that the human mind is capable of.

That bit is the “teaching” bit of the educational system. Nothing else needs to be taught. The rest is entirely dependent on what the individual is interested in and capable of learning. Here the job of the educational system is to make accessible to the student a comprehensive information set – and NOT the entire public information – for the student to pick from, and in the sequence that he or she feels naturally inclined to, and internalize it. By allowing the student freedom to choose what he or she wants to internalize, it releases the information constraint (that is, the problem of knowing what the student knows) which otherwise is impossible to circumvent.

Development

The age of agriculture yielded to the age of industrialization. Agriculture did not go away. It just became sufficiently productive that it released labor that was absorbed in producing non-agricultural goods and services. The percentage share of agriculture declined – not the absolute amount of agricultural production. Wealth, standard of living, or whatever you call it increased in pace with the decline in direct employment in agriculture.

The industrial age is giving birth to the information age. Once again, it is not that the amount of goods produced by the industrial sector is itself declining. It is not. Indeed, it is increasing. But that increase is due primarily to an increase in productivity and hence it releases labor to the rising sector – the information sector. As the labor force increases in the information sector, the production and subsequent consumption of information is bound to increase.

In the agricultural age, those parts of the world which were the most productive agriculturally prospered. It largely depended on the endowment of natural resources and a bit of human capital. It was a simple world and the social order was commensurately simple. Not much investment in terms of human capital was required. Education was largely an informal affair.

In the industrial age, prosperity depended on industrial productivity, which in turn depended on a reasonably educated work force. Education had to be formalized and the requirements could be met with standardized schools. The public information was limited but sufficient to meet the needs of the industrial worker.

In the information age, prosperity depends on how efficiently the people can produce and consume information. It is critically dependent on a very highly educated labor force. Needless to say that agriculture and industries will continue to need labor as well and that that labor would not need to be highly educated. Conversely, if a population is very minimally educated, then it can only be engaged in agriculture; if the population is moderately educated, it can move up to manufacturing.

So at the highest level of abstraction we can reasonably say this. Prosperity in the world to come depends on how highly educated the population is. So those economies that are able to create the most effective and efficient educational system will count. The rest will be forever falling behind.

Most of India lives in the agricultural age because overall our educational system is only able to supply to that. A small part of India lives in the industrial age. That part is increasing but slowly because of the inability of the educational system to provide the human resources required. Less than one percent of India lives in the information age. To a first approximation, the Indian educational system does not create any human resources for the India to live in the information age.

This is a dismal assessment. But there is nothing in the laws of the universe that actually prevents the Indian educational system from creating what is needed for India to prosper. What is lacking is the understanding, the vision, and the will of the people and their leaders.

“Entrepreneurship creates immense wealth that permeates healthy economies. Entrepreneurship alone has the capacity to create innovations in education that no bureaucrat or centralized planning authority can ever hope to achieve. Yes, central control can control, but it cannot create.

“In an age where each of us is immensely ignorant relative to the sum total of human knowledge, the skills that the individual acquires over a lifetime of learning cannot be imparted by an educational system that was created for a different world. The resources for building that educational system are already there. All that society has to do is keep the state out of it so that private enterprise can do its job—which it invariably does. The role of the state is limited to light-handed regulation.

“Liberalization of the educational system from the political-bureaucratic nexus is absolutely necessary. Without economic freedom, we cannot expect the entrepreneurial innovation required to make the educational system in step with the dramatic changes that the future has in store. It would be profoundly ignorant to not liberalize education.

I keep in mind my own educational experience. I went to a school (same one from grade 1 to 11) and by all standards, had very limited information. I recall that we had fewer than a dozen textbooks for every grade. They were not massive tomes. They were fairly slim. My estimate is that the information that was contained in all those books would easily fit on a single DVD with room to spare.

My point is that it is not the humongous amount of information that is necessary for a quality education. It is sufficient to sit very quietly with a little bit of information and internalize it appropriately. A bit of uninterrupted time, a bit of good information, a bit of sweat and a bit of inquisitiveness helped most of us get educated. It may have been great if we had Macs and Digital Whiteboards and low-orbiting satellites and digital editing suits and quadruphonic surround sound and IPods and iPhones and Facebook and MySpace and YouTube and broadband internet connections. Or maybe not. I know that I would have ended up futzing around on the web and flunking basic arithmetic.

I could be mistaken. Perhaps some genetic mutation has occurred in the intervening years since I went to school and suddenly kids cannot learn unless they are immersed up to their necks in high-tech gizmos. Perhaps they have lost the ability to learn from internalizing a bit of information. Perhaps they have to be simultaneously SMSing their pals, surfing the web, downloading gazillion giga-bytes of information, creating their digital profiles on FaceBook, capturing hi-def video and editing them to actually learn the basics.

I just don’t know.

So there you are: the demands on your attention grows. But the supply of your attention is fairly limited. The price of your attention is naturally going to go up. The question then is whether you can increase your supply of attention, not for the demands of the commercial classes but for your own benefit.

Here’s the good news then. Yes, you can indeed increase the amount of attention that you have. The secret, discovered long ago in the sacred land of India, is meditation. It makes sense considering that India gave yoga to the world. Yoga — yoking of the mundane with the divine — has both mental and physical aspects. Meditation is mental exercise.

Some years ago, I learnt Vipassana[3], or Insight meditation. An American friend of mine, a logger by profession, was (and still is) big time into Vispassana and I took a 10-day course in Fresno, California. It is claimed that the technique goes back 2,500 years all the way to Gautama Buddha. Only now it is being discovered by the hard sciences that Vipassana meditation can actually increase your brain resources–something that was known by the practitioners thousands of years ago. Here’s the abstract of a report from PLoS Biology, a peer-reviewed open-access journal published by the Public Library of Science titled “Mental Training Affects Distribution of Limited Brain Resources” where they studied the efffect of Vipassana meditation. The conclusion was that their “study corroborates the idea that plasticity in brain and mental function exists throughout life, and illustrates the usefulness of systematic mental training in the study of the human mind.”

Here’s the abstract: “Meditation includes the mental training of attention, which involves the selection of goal-relevant information from the array of inputs that bombard our sensory systems. One of the major limitations of the attentional system concerns the ability to process two temporally close, task-relevant stimuli. When the second of two target stimuli is presented within a half second of the first one in a rapid sequence of events, it is often not detected. This so-called “attentional-blink” deficit is thought to result from competition between stimuli for limited attentional resources. We measured the effects of intense meditation on performance and scalp-recorded brain potentials in an attentional-blink task. We found that three months of intensive meditation reduced brain-resource allocation to the first target, enabling practitioners to more often detect the second target with no compromise in their ability to detect the first target. These findings demonstrate that meditative training can improve performance on a novel task that requires the trained attentional abilities.”

I find it astonishing that meditation is not widely taught in schools in India. I suppose that it will be only after American schools have incorporated meditation in their curricula, and after a good 50 years have passed, it will dawn on Macaulay’s children[4] that it is time for Indian children to learn meditation. This intermediate step of meditation first being taught in American schools is necessary because otherwise the so-called “secular” brigade will scream bloody murder saying that Hinduism is being taught in Indian schools. Oh the horror! And Muslims will threaten to not send their children to school if meditation is taught in Indian schools, as happened when some schools tried to incorporate the yoga exercise “Surya namaskar.”

If I ever get to run a school, the first thing I would do is incorporate meditation and yoga.

Links: Newsweek report on “Meditating your way to a better brain.”

The Daily Galaxy article on “Channeling Buddha –New Research Shows Meditation Improves Attention Span.

Notes: [1] “Had we but world enough, and time” is the first line of a metaphysical poem “To his coy mistress” by Andrew Marvell (1621-1678).

[2] Alex Iskold writes, ” It is no secret that we live in an information overload age. The explosion of new types of information online is a double-edged sword. We both enjoy and drown in news, blogs, podcasts, photos, videos and cool MySpace pages. And the problem is only going to get worse, as more and more people discover the new web.”

[3] The Vipassana course I took was taught by the institution headquartered in Igatpuri, a small town close to Mumbai. Shri S.N.Goenka is the founder of this school.

[4] See Subhash Kak’s short piece explaining Macaulay’s Children.

It is important to stress that the WAC is a superset of NAC. So theoretically if you have access to the WAC, you have access to the NAC. The novice needs access to the NAC while the expert needs access to the WAC. While the expert can identify the NAC from among the WAC, a novice with her limited understanding of the subject area is likely to get lost in the WAC.

To illustrate that point, suppose you do a Google search on “Prisoner’s Dilemma,” you get 539,000 results. (Footnote: When I did the search with “Prisoners Dilemma” — in essence misspelling the term, I got 280,000 results. This shows that minor variations in spelling can radically change the search results and therefore how difficult it is to search effectively on the web.) Somewhere on the 14th page of search results is the definitive introductory work which an expert will take only a few minutes to identify. A novice will have to be very lucky to identify that same work without spending days crawling through hundreds of pages.

To illustrate the distinction between the WAC and the NAC, consider this. First, an economics textbook such as “Microeconomic Theory” by Hal Varian. Many students in the process of learning the basics of micro theory need access to the text, and each student requires repeated access to the text over the period of study. That text is part of NAC. Contrast that with papers on microeconomic theory that are of interest to doctoral students of economics. These papers are accessed only occasionally and that too by not all doctoral students of economics. This sort of information is part of the WAC.

One way to state the distinction between the WAC and NAC is to note that the former consists of a very large number of pages, the average number of page hits on which are small; and the latter consist of a much smaller number of pages with very high average number of hits. Thus 50,000 pages of an economics journal will get an average hit rate of perhaps 1 per year (assume 500 researchers accessing about 100 pages a year), while 200 pages of the basic economics text book will get . . . Let’s do the numbers. The average student reads a text book about 10 times over the course of the term. If a particular school has 100 students reading that text book, that implies 1,000 accesses to the entire book. In other words, the average hit rate for the pages of the book is 1,000 during a term. If the text is used for five consecutive terms for different batches of students, then the content of the book is accessed 5,000 times.

Here are the two points from the concocted example above that need underlining: First, the NAC is accessed orders of magnitude more often than the WAC. Second, the NAC is orders of magnitude smaller than the WAC.

In the next bit, I would like to explore the implications of the above two facts. You have to know the NAC if you wish to make anything of the WAC.

Ever since I have set eyes on the internet, I have been distracted by it. And I thank the gods for the distractions because at least a bit of the path to enlightenment can be illuminated by the faint glow of a CRT.

The web mimics and reflects into the virtual world the most salient feature of the real worl–which is that every tiny bit of the real world is connected to every other bit. The correspondence betweeen the image and the reality is continually gaining fidelity and the day may not be far off when it would be hard to distinguish the image from the reality.

It was John Muir, the environmentalist and ecologist, who used to say that he cannot write about nature. He said that when he starts to write about something, he finds it is connected to something else and that something else is connected to some other things and so on to infinity. So if he attempted to comprehensibly describe something however small, he would fail miserably.

I have the same pleasant complaint with the virtual universe as John Muir did with the universe when he noted that “when we try to pick anything out by itself, we find it hitched to everything else in the universe.”

Connections. I am fascinated by connections. I used to sit glued to the TV watching James Burke’s BBC series called “Connections.” A journalist and historian of science, Burke is in a class all by himself. If you have an hour to spare for some delightful insights into the nature of innovation and the histroy of technology, listen to the man. And the miracle of it all is that you can do it from the comfort of your own home and at a time of your choosing, if you have an internet connection.

{to be continued.}

The larger the stock of information, the more expensive it is to search through it to locate the precise bit that is relevant at any particular instance. To make the task of searching more tractable, ordering the information in some fashion—called sorting—becomes paramount. Computer scientists have worked on the problem of sorting and searching for decades with phenomenally successful advancement in our understanding in this regard.

(Volume three of Donald Knuth’s magnum opus The Art of Computer Programming is devoted to Sorting and Searching. I did not get past the first volume on The Fundamental Algorithms, leave alone tackling the third volume in my graduate computer science courses.)

This is a continuation of my earlier piece on the age of superfluous information. I argue here that in post-industrial world and increasingly so in the future, sorting and searching through information will occupy the role that manufacturing did in the growth of the old economy.

The libraries of the world contain an ever-expanding stock of information, much of which is very rapidly being added to the humongous stock already existent on the world wide web. That stock is growing rapidly as the flow of information is turning into a flood as the internet spreads its tentacles into every nook and cranny of human activity. Billions of people today have access to information equivalent to hundreds of millions of books on the world wide web. Compare that to just a hundred years ago when the average human had access to half a dozen books worth of information at most. When taken to such extremes, quantitative change amounts to qualitative change. The world of information is not what it used to be. The challenges therefore are qualitatively different.

When the quantity supplied of any good is in excess of anything reasonably required or demanded, the variable of importance is quality. Basically a person’s information needs are really very simple. One can only read so much, listen to so much talk and music, watch so much video, and wish to know only so much about what is going on around in one’s neighborhood and in the world at large.

Here is the result of some simple arithmetic I did just now. I estimated the total stock of information available today. Then I divided it by the maximum rate at which information can be scanned by a human. The result: a person will take about 18 billion years to merely scan the information to exhaust the current store of information. At the current rate of increase of information (an accelerating rate, I might add), or flow, a person would require 44 additional years for every passing hour. Compare the 18 billion years to the current estimated remaining lifetime of the sun: a mere 5 billion years.

The bottom line is this: there is already so much information out there that even if no additional information were generated, each one of us could be occupied a little longer than forever to finish it. Information, as we well know, is a non-rival good. That is, my “consuming” a particular piece of information will not diminish the amount available to you. Compare this to a rival good such as food. Stock of food is enough to last the six billion humans for about 3 months. In other words, if we produce no additional food, all together humans would finish the stock in three months. Or, a single human can therefore finish this in 1.5 billion years. But it is not so in the case of information. Each of us would take the estimated 18 billion years to finish the information we already have before we ask for more.

Clearly, for an average human, about 0.00000000001 percent of the total information stock is more than enough. About 99.9999999999 percent of the available information is worthless. So how does one go about searching out the teaspoonful of useful information from the oceans of available information. That is the challenge and therein lie the opportunities. That is why firms like Google will make the big bucks. The opportunity is not so much in making information available but making the right information available.

Which brings me to the point which I started off with. Searching is only part of the story when it comes to information. The other part is sorting. If one can sort the information along some relevant dimension, then you have meaningful information. What is meaningful can only be defined in the context of the entity processing the information. From the same stock and flow of information, different entities define different subsets that are relevant and meaningful. This subset can be labeled private information as opposed to the vast store of public information. Private information is the top of the sorted list of public information. Internalizing the private information leads to what we can call a stock of knowledge associated with the individual.

It is useful at this point to remind ourselves of the distinction between information and knowledge. Information is a public good the stock of which is growing exponentially. Knowledge is a private good and its primary raw material is the private information which is a very vanishingly small subset of the available public information. Even though public information has no known bounds, there are limits to how much private information can processed by a human brain and thus there are limits to the acquisition of the private good we call knowledge.

Conflating knowledge and information is distressingly too common these days and so I would like to dwell on this distinction for a bit. Some say that today we have a knowledge economy. It is trivially true because it has always been a knowledge economy ever since humans evolved brains capable of processing information into knowledge and began using knowledge to organize and coordinate economic activities. What is novel is the unfathomably huge stock of information we have available today. What distinguishes one individual from another today is the capacity to figure out what is relevant information and to internalize it efficiently into knowledge. That capacity is one of the basic skills imparted by what we call education.

To summarize the story so far: from the vantage point of an individual, this is an age of superfluous information; only a tiny fraction is relevant and meaningful; searching through the information can be automated but efficiently sorting for relevance is a private skill; imparting that skill is a primary function of education.

Next time I will explore the role of education in an age of superfluous information.

Food, for instance, is a good that in excessive quantities is a bad as the success of the dieting industry so starkly demonstrates. Yet tens of millions poor people around the world dying of malnutrition and starvation every year is the horrible demonstration of the problem at the other extreme.

The same holds for information.

What brings all this to mind is the so-called information revolution occurring globally. Information is a good which is also subject to the law of negative marginal utility beyond the satiation point. Information overload can be as debilitating as too little. For much of the world now the problem is no longer a shortage of information but rather a surfeit.

All living beings acquire information from their surroundings for survival. Information needs vary depending on the nature of the being and the sense-organs have co-evolved to perform the required function. The more complex the entity, the more sophisticated the sense organs are. The sophistication has two dimensions: the bandwidth and the filter.

Higher bandwidth means that more information per unit of time is sensed by the entity. But the information gathered is filtered severely for relevance and only a small percentage is passed on to the processing unit (the brain) for internalization and for response. Therefore the filtering mechanism had to evolve in step with the evolution of high bandwidth sense organs.

Here is my conjecture. Higher intelligence is marked by possessing very high bandwidth sense organs (channel capacity), and a very sophisticated filtering system which rejects most of the inputs and processes only a very small portion of the total information received

We humans sense the world around us mainly through our eyes and ears. Most of the megabits per second of information we receive is rejected and only a minute fraction is processed by our brains. The filtering mechanisms developed over our evolutionary history and did so gradually. Even the brain has an inbuilt capacity to forget. There are a few who suffer from a pathological condition which does not allow them to forget anything that they have ever seen or heard.

Now we are faced with a world where information is being generated and accumulated at an exponentially increasing rate and we face the possibility of information overload that could overwhelm our capacity to filter and meaningfully process it.

Going back to the food analogy, for much of human history, we have been at the edge of starvation. Our bodies evolved a strategy of accumulating fat whenever it could. Now even though for many starvation is not a threat, their bodies continue to use the same strategy of averting starvation and it ends up as obesity. The external change has been too rapid for our bodies to evolve a different strategy. Therefore external agencies such as the dieting industry have evolved to protect us against the body’s internal mechanism.

I conjecture that the age of information dieting industry is upon us. The day is not far off when you would have to go on an information diet.

In the case of food, we do need an adequate amount of calories and that too the nutritional dimensions of the calories we consume matter. Sugars and fats have calories and we do need a sufficient amount of them but a diet of solely of sugars and fats is far from healthy. The information equivalent of sugars and fats is news. We do need news but if that is all we consume, we are likely to become information fat without being information healthy.

Allow me to speak personally, if you will. I have a very low threshold for news and information. In the US, my major source of news was the radio. In India I don’t have that luxury. I don’t have a TV at home. So much of the little news I get is from off the web. Random surfing occasionally is sufficient for me to get to know about the big events. Blogs are a rich source of information. I don’t read newspapers because I believe that anything that really importance which is reported in the papers, I will get to know eventually; and anything that is trivial (which is about 95 percent of the newspapers), I will not miss anyway.

It takes me a long time to process information into knowledge and understanding. I cannot read five books a week. There are books on my bookshelf which I have been reading for the last four or five years and I still have not read them all the way through. Not that you asked but just for the record they are: “A Theory of Justice” by John Rawls, “Thinking about Development” by Paul Streeten, “Contemporary Political Philosophy” by Will Kymlicka, just to name a few. I might read a page or two and then it would take me hours to comprehend what I had read. Then a few months later, upon re-reading the same pages, I realize that now I really understand. Only after a three or four passes, do I think that I fully internalized an idea. It is an excruciatingly slow process for me. Perhaps I am not alone in this. I read somewhere that Robert Solow requires three readings before he can comprehend an idea. And he is a Nobel prize winning economist.

Time-out for a joke. Two guys were talking. “So what are you going to do now that you are retiring?” “Well, I was thinking of finishing my book.” “You are? I didn’t know you were writing a book!” “Who said I am writing a book? I am reading one.”

I am almost as bad as that guy when it comes to reading heavy books. The light stuff such as “Freakonomics” I can read in a few hours. But enough of this digression. Back to the topic at hand. What is it going to be like to live in a world where information is cheap and over abundant? What sort of services will emerge which would help people cope? There is a challenge of managing information and therefore there are opportunities for firms that will help you reject information in an age of superfluous information, just as there was a challenge of bringing you information in an age where it was scarce.

Those questions I will ponder about the next time.

Post Script: The topic is continued here.

Related Post: The world is information fat and a followup.

Uday wrote in:

As usual, a thought provoking article that makes me periodically check this blog!

Now don’t go thinking that I bribed Uday into writing that.

I would like to question the validity/implications of 4, 6, and 7.!

For reference, here are those points:

Fun Fact #4: Regarding the quality of information online: as the quantity is increasing, the variance is increasing and the average is decreasing.

Fun Fact #6: The cost of identifying the information is going up.

Fun Fact #7: The cost, therefore, of obtaining knowledge has gone up.

As we all know search efficacy has been taken to new levels by Google; and Inktomi/Yahoo! and MSN are not slacking off either. I wonder if you are implicitly questioning the limitations of the ranking algorithms used by these companies. As the amount of information increases by leaps and bounds, the total quantum of search results for a particular phrase will keep increasing proportionally. But I would suppose the key challenge is to rank order the search result — after all, how many people go beyond the first few pages of any search result?

My claim in #4 is restricted to the average quality and the total quantity of information available, with the former coming down and the latter increasing exponentially. Naturally the good stuff is harder to find. Even if the search is aided by Google with their admittedly excellent page ranking algorithm, the results can be less satisfactory than if the search domain was restricted as it was earlier when there was less information overall. My unsubstantiated claim is that the combined effect on the average quality of information of increased volume (negative) and more efficient search engines (positive) is on the whole negative.

Put another way, would their ever be a real need to sift through information that is ranked beyond the top 50, or, would a person be better off refining the query to zero-in more effectively on what was sought. I believe it is the latter; hence the amount of information should not affect the quality of refined queries. But yes, refined queries will be increasingly needed to substitute coarse and aggregate ones.

I agree that a refined search would yield a more accurate result, of course. But that refining the search is exactly what I mean when I say that the search costs increases.

The explosion of consumer choice is a good thing, overall. But there are implicit costs associated with choosing from a large menu as opposed to a limited but excellent menu. The implicit costs include a more discerning consumer.

I contend that this is an “intellectual” cost. It will be simplified for the masses if leading edge companies can provide intuitive means to elicit query refinement. i.e., an important question is how the search companies could provide artifacts beyond the existing ones to allow users to phrase their queries better and obtain exactly what they seek. Google for one has done tremendously well on such aspects. Google suggest completes search phrases before they are completed (offering suggestions), the “:” prefixes are extremely handy and work very effectively, the seamless carry over of context to news, images, local etc. are so convenient. I don’t know if I am missing your point Atanu, but I reckon that bright engineers have hit upon very good metaphors to enable searching for exactly what you seek. I grant that these tools can be expanded upon, and are taken for granted by the programming savvy; if and how the larger population can effortlessly imbibe these skills depends on !
the user interface exposed and the rate of exposure (so as to not intimidate the consumer base).

I don’t think you are missing my point. We are merely working on different sets of assumptions. The object of a search is to find something that you by definition do not know. Therefore the best you can do is to define the boundaries of the search. Having defined the boundary, all else remaining the same, the denser the information set, the more numerous will be the result obtained.

Another question is whether people in general have communication skills to express exactly what they seek. It is very instructive to look at some of the results on Google Answers and look at what the experts searched on to provide the answer. People seem to be all over the spectrum when it comes to the ability to express their needs smartly and succinctly. As with most things in life, yet again, we realize the 80-20 rule J

In my conjectures in the piece, I was working with averages. My contention is that the sophistication of the average consumer may have not improved significantly, and that the variance is increased. Therefore the conjecture that on average, finding information is more costly now.

Finally, would you have any stats or thoughts on what proportion of the growth of the information is on largely “new” topics and how much represents “accretive” bloat? This is surely an ill-formed question because the definition of ‘new’ and ‘accretive’, in this context, are themselves fuzzy.

Excellent point. New topics would have lower aggregate information and therefore will have greater average quality and lower variance. They would not have time to have what you call accretive bloat. I think there may be some equivalent of Zipf’s Law (Rank-size distribution) which may shed some light on how the growth of information is distributed among topics that are new as opposed to old.

Imagine that each topic is ranked from the most recent to the most ancient. Associate with each topic the total amount of information available and a bit of regression analysis will yield the exact law.

[Disclaimer: I don't work for a search company, and may be demonstrating gross ignorance of progress in the area].

Hey, my ignorance will leave your ignorance in the dust.

If you come to think about it for a moment, what we really want is knowledge, not information. (Recall what the business school guru said: what people want is not a quarter-inch drill but rather a quarter-inch hole.) The good news is that there is a lot of information out there. The better news is that the cost of accessing that information has been dropping exponentially. But the bad news is that the cost of searching through the vast stock of information to satisfy your knowledge needs is increasing.

First a brief aside on the distinction between knowledge and information. People use the terms interchangeably but they must be distinguished if we wish to reason with some degree of clarity about our information-suffused modern society. A telephone book has information about names and numbers, but it does not ‘know’ telephone numbers. A human brain ‘knows’ a phone number, in contrast. Outside the human brain, it is information and organized within the structures of a human mind, it is knowledge. Dr Johnson appreciated the distinction very keenly. Information is what economists call a public good, while knowledge is a private good. This has important implications which need not detain us right now.

Back to the good news about information. Here are some fun facts about information.

Fun Fact #1: There is a heck a lot of information in the world today. The stock of information is stupendous. Naturally so because human activity primarily produces information. The increased production of information has been intensive and extensive.

Intensive because more of what we do is recorded, whether in government files, in private sector databases, or in your own record of your private life such as in blogs and photo albums, and so on. You have lots more bits of information associated with you than was associated with your ancestors.

Extensive because first there are more of us on the planet today, and second, because more of us are doing stuff that produces more information. A much larger percentage of our six billion human population is engaged in generating and processing information than before. Millions of researchers and scientists do stuff that generates truck-loads of information. Commercial, governmental, and personal information grows unbounded.

Fun Fact #2: A heck of a lot the available information is available online. Some of it is on the world wide web accessible through the internet, while much of it is in the deeper web not generally accessible to the average web surfer. As time goes on, a greater percentage of the total stock of information will be online.

Fun Fact #3: The stock of information is increasing exponentially. And consequently, the stock of online information is also increasing exponentially. Exponential increases are fairly dangerous things. This is a not-so-much fun fact we will come back to bite us. Flow (increase in the stock) of information is threatening to become a tsunami.

Fun Fact #4: Regarding the quality of information online: as the quantity is increasing, the variance is increasing and the average is decreasing.

Let’s just focus on books although we could as easily tell a similar story about movies, or books, or research papers, or photographs, or blogs, or usenet postings. A century ago, there were very few books published compared to today. Given the high cost barrier of publishing, only those works which had some enduring quality made the grade. Therefore the average quality of available printed matter was high. Today, there millions of titles are published because of both greater supply (more writers) and greater demand (more readers), and because the cost of publishing (relative to average incomes) has fallen. The quality of the average book, I believe, is lower than before.

It is my contention that the best book of today is better than the best book of yesterday, and that the worst book of today is worse than the worst book of yesterday. Just a hunch and I don’t have hard data to support this hunch.

Allow me to make a hand-waving argument about quality and quantity. Let’s take photographs. When I used an analog camera (print or slides), I used to take a lot fewer pictures than I do today with my digital camera. But I rejected a lot lower percentage of pictures in those analog days. Today, I throw away most of what I take with my digital camera but I end up with much higher quality “best pictures” than before. Taken as a whole, my digital pictures are on average “lower quality” because I click a lot of pictures with greater abandon today given the low average cost of each click.

Greater variance and lower average quality coupled with an immensely larger stock leads us to the bad news about today’s information age. But first, a few more fun facts.

Fun Fact #5: The cost of accessing information is going down.

Let’s just say “google.” Enter some keywords and you will get about four million hits, give or take a few million. Marginal cost to you: nearly zero (assuming that you have a connected computer at your disposal.)

Another way of putting this is to say that the “channel capacity” has increased. The information can flow through to you through a vast pipe if you need it.

Fun Fact #6: The cost of identifying the information is going up.

So you do get four millions hits in less than 0.4 seconds when you do that search on Google. But, unless you are very lucky, or have been very clever in specifying the search, it will take you a lot of time to sort through it all to find the information you need.

For any given stock of information, the lower the average cost of accessing the information, the higher the search cost for any specific required information. Here I would like to enunciate what I call the Information cost complementarity principle: for a given cost, quality and quantity are complementary. You can have high quantity but will have to put up with low quality; or you can have high quality but it would cost you. (Compare with the folk wisdom: “Good, fast, cheap: Choose two.”) Another way of putting it: the cost of searching and cost of sorting through the results of the search are complementary.

I model the principle after Neils Bohr, the father of quantum mechanics. He stressed the importance of the complementarity principle and held that it has wide applicability in areas far removed from physics. The principle says that knowledge of one aspect of a system precludes knowledge of certain other aspects of the system. (Ref: Steven Weinberg’s Dreams of a Final Theory, pg. 74.) Heisenberg’s uncertainty principle is an instance of the complementarity principle: absolute knowledge of a particle’s position (or momentum) precludes absolute knowledge of the particle’s momentum (or position.)

Fun Fact #7: The cost, therefore, of obtaining knowledge has gone up.

Recall that we really are not interested in information for its own sake; we are interested in knowledge. Being given four million hits in return for a search is about as helpful as being thrown both ends of a rope when one is drowning.

The claim that the cost of knowledge has gone up simultaneously with a dramatic decrease in the cost of information is clearly counter-intuitive. Counter-intuitive but consistent with the facts. Low quality information is cheap. That leads to what I call “information obesity” and is inconsistent with “knowledge health.”

For all of our evolutionary history, food was not easily available and so we have evolved to subsist on a low calorie diet. Suddenly (in evolutionary time-scales) in developed countries, calories are abundant because low quality high-calorie foods are cheap. High quality low-calorie foods are expensive. So the poor in rich nations such as the US suffer differentially more than the rich from obesity. This is in contrast to poor nations where the rich are obese. Obese people are calorie-rich but health-poor.

I contend that one can be information-rich and knowledge-poor. And further that in an information overloaded society, the poor people will be information-rich and knowledge-poor, and the rich people will be information-poor but knowledge-rich.

After all this talk, let me come to the point that I want to make: There is a big opportunity managing information overload. Create a filter which will let only the top quality information through and people will beat a path to your door. You may say that it is a Super Filter which filters out not just spam but low quality non-spam content as well. In the past, portals which gave you everything (when there was not very much of anything) were a big hit. Now (when there is much too much of everything) portals which give you access to absolutely selective exclusive stuff will make it big.

In one specific area — education — I have figured out how to manage the information overload problem. Perhaps we should talk about it later. Or perhaps not.

Follow-up Post: The Age of Superfluous Information and The World is Information Fat Followup.