Michael:

Now that we’ve been blogging for over a year, a lot of headlines I read bring to mind previous postings we’ve cranked out. I thought I’d highlight a one of these from the October 16 New York Times. Headlined “Digging for Nuggets of Wisdom” it discusses the increasing use of text-mining to search through enormous piles of documents looking for connections that no human can (efficiently) find. According to the story by Lisa Guernsey, which you can read here:

In most cases, text-mining software is built upon the foundations of data minining, which uses statistical analysis to pull information out of structured databases…But text mining…works on unstructured data…To make sense of what it is reading, the software uses algorithms to examine the context behind words. If someone is doing research on computer modeling, for example, it not only knows to discard documents about fashion models but can also extract important phrases, terms, names and locations.

In a May 31 post I wrote on Family Trees (which you can read here) I proposed using another piece of pattern-matching software deriving from biology to create maps showing the relatedness of various strings of data, and suggested that we could turn such software loose to create pedigrees for many ideas floating around in the cultural portions of the Web:

And although it would be more difficult to track the pedigree of works of fiction, I wondered if it would be possible to reduce stories, or at least their plots, to a standard alphabet of relationships between the characters. For example, Hamlet might be reduced to “father, son, step-father, mother, murder, revenge, madness, mistaken identity.” It would be interesting to see the family tree of Hamlet’s antecedents and its descendants. In fact, it would be interesting to equip “Google” with such a relatedness testing device and use it to create family trees for “memes” propogating themselves through cyberspace.

Well, it looks like with the advent of software like text-mining, even more powerful tools are becoming available to extract and document such cultural patterns from the Internet or other large digital storehouses of data.

I also wrote a post on Culture and Scale Free Networks (which you can read here) suggesting that the cultural community is organized as a sort of “scale-free network” of influential individuals, giving the cultural universe a combination of relative stability but a tendency to be vulnerable to sudden, massive shifts of taste as key “nodes” (people) drop out of the network. According to the New York Times article, text-mining software often comes with the ability to visually map the relationships spelled out within its universe of texts; it would seem that such a system might well be able to diagram the exact confines--names, job titles, organizations--of the cultural networks sustaining the prestige of say, given movements in art or literature.

Text Mining Relational Chart from New York Times Article

(It would be pretty wild to discover that, say, the true cultural linchpin in the Renaissance was actually, say, the head interior decorator of the Vatican and not Pope Julius. Hey, I can dream, can't I?)

So remember, for important ongoing stories, you can read about them here first. (Well, sort of.).

Cheers,

Friedrich

*grin* Is that like being fashion-foward? ;)

Posted by: Courtney on October 16, 2003 04:06 PM

From one point of view: we're born editors and trendspotters who do it better than the pros. From another: we've got a lousy sense of timing.

Posted by: Michael Blowhard on October 16, 2003 11:24 PM






Post a comment

Name:


Email Address:


URL:


Comments:


Remember your info?