Progress report

Today I have most of the parsing and database code done.

The program is written in Python and takes as input the compressed Wikipedia dumps. It will parse the XML and insert the parsed results into a SQL database.

It is more complicated that it seems, however, because some articles are nothing but redirects.

The way that I have it currently set up is that the SQL tables that deals with Wikipedia and the SQL tables that deals with the graph is separate. Article and Link objects have pointers that point to Vertex and Edge objects. That way the algorithm that runs on the graph will not need to have any knowledge about the specifics of Wikipedia.

The graph representation is not an adjacency list although the performance should be the same. It is an edge-set representation. However, since SQL databases do indexes, it is really a hashtable: looking up whether v and u are connected is O(1) and finding all edges from u is O(n). Thus, it has both the benefits of both adjacency lists and adjacency matrices.

Parsing Wikipedia is more complicated than it sounds since many people make syntax mistakes. There is still some bugs to work out, when I am done with it I will upload the code to Google Code. I will do that on Friday the latest.