App plan

App plan

Init algorhytm

1. Create database if not exists
2. Insert pubdb infos (new objects, .store() method)

Main algorhytm, on submit search

1. Get search options from the GUI
   • author
   • title
   • onlyLocal ?
2. if (!onlyLocal)
   • Get db list (from local db)
   • For all dbs:
     • Submit form
     • do
       • Get HTML
       • Extract links to publication pages
       • Get next page
     • while (there's next page)
     • For all links
       • (We can skip links that are already processed earlier.)
       • Get HTML
       • Extract data (bibtex, authors, title, year)
       • Extract referencing publications list
         • Go through them, extract data (bibtex, authors, title, year)
       • Store results (to local db)
3. Query results from local db and show

Notes:

• We should detect if a result is already in the database - compare title, authors (?), and year.
• We will store the size of the downloaded HTML code, for some statistic on how much internet traffic is needed to crawl.
• The main cycle can be implemented with threads for each db, it may be more efficient, but maybe we should limit the number of threads running at a moment, so must build a scheduler mechanism.

Classes (at first sight...)

• GUI, Main:
  • Main (main window, UI)
  • PubWindow
  • (maybe an abstract MyWindow)
• Crawl
  • Main Crawler (control the whole search process)
  • DatabaseCrawler
  • ResultListCrawler
  • PubPageCrawler
  • Tools (requests via proxy, regex, html clean)
• Data (EntityClasses)
  • DatabaseInfo (informations of a pub. db.)
  • PublicationData (data of a publication)

Entity classes will have these methods:

• store() - stores the current object in the local database
• static getAll() - queries all the objects of the current type, and returns it in a List<> (we can do the proper joined queries here too)