The first pass over the Wikipedia dump is done and here are some preliminary results:
* 7,278,279 articles, including stub articles that contains metadata on images.
* The SQLite database is 613 MiB
* It took about 3 days on a relatively slow computer
There needs to be two passes because of the need to resolve links. In the first pass, all articles are recognized, inserted into the database, and given a unique id. In the second pass, all links will be resolved as a pair of ids (plus metadata). Without the first pass the second pass would be more difficult.
The second pass will probably take even longer because there are lots of regexps involved. The code will be committed to svn soon.
Subscribe to:
Post Comments (Atom)
1 comment:
I hate to break it to you, but Wikipedia provides a tool that will do all the splitting and processing for you.
http://meta.wikimedia.org/wiki/Xml2sql
Breaks up the 7,00,000 article xml dump in about 45 minutes on a dual core macbook.
Post a Comment