Today I just finished implementing HITS. There is some bugs to work out: for one thing it is too slow, another problem is that it gives very bad results. I am still trying to figure out what is causing the problems.
One optimizations I have done is that instead of inserting each link into the db as a separate entry I now insert all of the links in an articles together under a single key, using serializing the objects. This cut around 19 GB from the database size as well as made parsing a lot faster.
Right now the graph structure is held in memory as a giant hashtable. Python's hashtables are pretty slow however, and either I will have to figure out more optimizations or I will rewrite the querying code in Java.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment