Rensselaer Center for Open Source Software Funded Project Completes Data Mining Framework for Wikipedia

Troy, New York -- Saturday, December 13, 2008 -- An Undergraduate Research Project funded by the Rensselaer Center for Open Source Software completed a data mining framework for Wikipedia. The framework, which is free software and is released under GPL v3, makes it easier for programmers to write data mining programs for Wikipedia by pre-processing the raw text into a database that allows for faster searching and retrieval. Included in the framework is an application that retrieves articles on similar topics.

"Writing this framework was a lot harder than I had thought." said Yongqian Li, the student who worked on this Undergraduate Research Project. "The Wikipedia data-set is so large that using traditional methods, such as SQL databases, is impossible." The full text of the English version of Wikipedia, without revision history or contributor information, is about 60 GiB when uncompressed. "The file wouldn't even fit on my hard drive, so I had to use a streaming technique to process it bit by bit." Yongqian Li explained.

An additional difficulty was that the link structure of Wikipedia is very different from the rest of the Web. Most of the leading search engines, such as that of Google, analyze the link structure of the Web to make inferences about the contents of a page. "What totally surprised me when I started looking at Wikipedia more closely was that the links on Wikipedia are much more dense, probably because the syntax of coding a link is simpler in WikiText that HTML." Yongqian Li explained. WikiText is the format that Wikipedia is written in while HTML is the language of the rest of the Web. As a result, analyzing Wikipedia is harder since the code has to take into consideration that a link on a page may not actually be meaningful. "It turns out that augmenting more traditional link analysis techniques with insights derived from Bayesian statistics made everything work." explained Yongqian Li. Bayesian statistics is also employed by spam filters to analyze the text of email messages.


The Rensselaer Center for Open Source Software was funded by donations from Mr. Sean O'Sullivan, RPI Class of 1985. "We are all very grateful for this opportunity to learn and we get paid for it too! I hope that one day I will become wealthy enough to fund projects like this myself!" said Yongqian Li. At the research project's conclusion at the end of the fall semester Rensselaer Polytechnic Institute students were encouraged to participate during the Spring semester.

###