NewScientist: A STATISTICAL method that picks out the most significant words in a book could help scholars decode ancient texts like the Voynich manuscript – or even messages from aliens.

Humans find it easy to identify the words that capture the theme of a text – for example, that “whale” is a key word in Moby Dick – but this is a difficult task for computers. Now Marcelo Montemurro, a systems biologist at the University of Manchester, UK, and colleagues have developed a method to identify word importance based on a branch of mathematics called information theory. “It seems that what we call semantics or meaning has a signature at the level of the statistics of words,” says Montemurro.

Simply counting the frequency of words in a text is not enough, as connective words such as “for” and “the” confuse the picture. Important words tend to clump in paragraphs and chapters that deal with the topic they relate to, but this only provides a crude guide, says Montemurro.

For a more detailed analysis, the team calculated the “entropy” of each word, a measure of how evenly distributed it is, in both the original text and in a scrambled version in which the words appeared in a random stream. From the difference between the two entropies multiplied by the frequency of the word, the team generated that word’s “information value” in the text. More

Despite the fact that Wired editor-in-chief Chris Anderson’s latest book, Free: The Future of a Radical Price, wasn’t released until this week, it has still managed to generate much pre-publication discussion about the future of the digital economy. Anderson found himself enmeshed in a pre-publication plagiarism scandal two weeks ago when the Virginia Quarterly Review found that some passages in the book directly matched Wikipedia entries. (Anderson quickly apologized, blaming inaccurate citing and overall carelessness.)

Then, of course, there’s the actual content of the book, which has been received by journalists and business-minded folks in decidedly polarizing ways. Malcolm Gladwell unleashed a scathing review of Free in last week’s New Yorker, scolding Anderson for adhering to the freeconomy as an “iron law” and writing, “The only iron law here is the one too obvious to write a book about, which is that the digital age has so transformed the ways in which things are made and sold that there are no iron laws.” (Plenty of responses followed.)

But for Anderson, Free is indeed the ultimate destiny of our economy. “Sooner or later every company is going to have to figure out how to use Free or compete with Free, one way or another,” he writes in the beginning of the book. This assertion will probably look depressingly familiar to journalists who’ve watched their traditional business models fall apart in the wild west of the web, where “free” is the gold standard.  Read more

I met Liz Lynch at Kevin Nations’ Big Ticket Blueprint event. What a sweet and wonderful person! She is interviewing me tonight on a conference call. I highly recommend her “Smart Networking” book, which you can read about at http://SmartNetworkingNow.com

Check it out!