[libimdb-commit] CVS: libimdb TODO,NONE,1.1 README,1.1,1.2

[Jerry V. <jve...@us...>]

Update of /cvsroot/libimdb/libimdb
In directory sc8-pr-cvs1:/tmp/cvs-serv16121

Modified Files:
	README 
Added Files:
	TODO 
Log Message:
moved to TODO, new README in place

--- NEW FILE: TODO ---

Short list to get things out.
 - build env
 - roll out c library
 - roll out beta java interface
 - simple web page and setup of cvs, mailing lists etc.
 - will want to use something like mpatrol to assess what
   the memory requirements are for different parts of the
   library.
 - need to version things.
   libimdb_getVersion();

   libimdb_getListFileInfo(); ??

 - is it easy to convert lfetch to a c program ?
   libimdb_updateListFile();

 - look at perl script to just get list files
 - look at perl script to get and merge list diff files.

Things to be done
-----------------

Since movie lables are the key to the database, maybe a temperary
(option) on-disk db database to hold the movie info and rest of
them return references to entries via a movie key. The user does
a second set of lookups when the movie detail is needed. Making
it optional means not paying the disk price and instead paying
the memory price. (may also just be a runtime optional though,
since we may want to use an in-memory db still - this could be
substituted for a non-db solution later).

Guess that means we may need to lookup movies by key, but also
generate a list of movie label based results when searching by
title. This may require a second pass over the movie database.

Need to take a closer look at all the list file to assess if
this is the best strategy.

Simpler scan actor db one at a time - callbacks get a single actors
  and multiple movie keys (and char info).

perl interface and callback using swig'd interface
 - build simple example and look for help via
   perl5 book and swig mailing list.

  - support callback or first/next... processing ?
    - first/next might be just simpler for perl
    - probably pretty easy to support both.

libjlog
  - main log_open should initialize default
    logtracker and logtee from env variables
    - do we want this to happen every log_open ?
      - yes
  - how do we define filters ?
    - file/func filters
    - in/out filters
    - regexp ?
    - config file

  - can mklog.pl generate/manage the list of
    modules supported ? Would require that
    enum gets initialized though.
    - one enum should alwyas be MOD_OKAY=0

  - completely separate tree for jvlog ?

  - can success/failure callback be combined
    with current callback mechanism ?

  - support success call from void function

  - LOGerror_e should become LOGvalues_e
    and include LOG_SUCCESS=0, enum values should be negative
    for error

  - how can we get gcc to check __attribute()__ stuff ?

nuke build environment and re-write.

 - use simple gmake makefiles that need to be
   massaged to work....

 - could have basic configure at the top.

 - swig generation should be separate target and
   not part of normal build.
    - change name of .javaswig file to not specify
      packaging, explicit move command and rmdir
      with warnings if files end up lying around

 - cvs ci should include java generated c etc.

 - can I or should I use sysinclude.h ?
   - probably don't need it... would need to test port to
     cygwin for compiling into a dll.

   - clean up in order - remove NATIVE_WINDOWS

Utility for downloading/managing ftp list files

What is this thing for ?
------------------------
- look at what Scott did for mysql loading.

- look at xmltv2db on sf for a basic example of
  putting data into jdbc database - now sure how useful this is.

- support gzip'd list files.
  - www.gzip.org/zlib
    - can probably just include it.

- need bigger picture plan... what is libimdb ?
  - for use with tv_imdb ?
    - maybe tv_imdb just uses a libimdb database
      - libimdb's own database could just be a
	series of text files in sorted key order.
	then use look to locate things.
      - perl interface kept simple, while libimdb
	could use binary search algorithm.
      - still doesn't require mysql and is platform
	independent.
      - what other db options are out there ? gdbm ?
      - nothing to say we can't use dynamic libimdb.so
        for searching, provide a generic interface
	that includes what you want for tv_imdb
        Then just have java and a perl interface.

    - guess a perl script is the easies to load
      things into mysql ?

    - xml output of db entry (probably useful for
      debugging as well).

    - xmlrpc ? what is it, is it useful here ?

  - linux and windows supported ?
  - load imdb data into hsqldb / mysqldb
    - how do we standardize on db tables ?
    - how do you look flexible ?
    - interface must support db clean, load
    - way in the future what can we do about doing
      minimal updates ? Can we only update the table
      entries that changed since the last fresh build ?
      Maybe the parse should support an update
      operation ? How do we identify things that have
      to be removed ? Can we ? Maybe intermediate db
      (flat sorted index) and a compare or refresh mysql
      utility.
  - query data 
  - do all of this from java and perl

- can I build a java centric database loader (int hsqldb ?)
  - what kind of progress bars are in java ?

  - JFreeChart to plot things ?
    - number of actors / movie
    - years of movies ?
    - present stats generated from loader ?
    - actors to actresses totals
    - movie ratings (user rated) against years

- what about a imdb server that has XML-RPC interfaces
  for queries ?
  perl use Soap::Lite package (www.soaplite.com)

Index: README
===================================================================
RCS file: /cvsroot/libimdb/libimdb/README,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** README	16 Jun 2003 04:18:37 -0000	1.1
--- README	16 Jun 2003 04:43:03 -0000	1.2
***************
*** 1,169 ****
  
! Short list to get things out.
!  - build env
!  - roll out c library
!  - roll out beta java interface
!  - simple web page and setup of cvs, mailing lists etc.
!  - will want to use something like mpatrol to assess what
!    the memory requirements are for different parts of the
!    library.
!  - need to version things.
!    libimdb_getVersion();
! 
!    libimdb_getListFileInfo(); ??
! 
!  - is it easy to convert lfetch to a c program ?
!    libimdb_updateListFile();
! 
!  - look at perl script to just get list files
!  - look at perl script to get and merge list diff files.
! 
! Things to be done
! -----------------
! 
! Since movie lables are the key to the database, maybe a temperary
! (option) on-disk db database to hold the movie info and rest of
! them return references to entries via a movie key. The user does
! a second set of lookups when the movie detail is needed. Making
! it optional means not paying the disk price and instead paying
! the memory price. (may also just be a runtime optional though,
! since we may want to use an in-memory db still - this could be
! substituted for a non-db solution later).
! 
! Guess that means we may need to lookup movies by key, but also
! generate a list of movie label based results when searching by
! title. This may require a second pass over the movie database.
! 
! Need to take a closer look at all the list file to assess if
! this is the best strategy.
! 
! Simpler scan actor db one at a time - callbacks get a single actors
!   and multiple movie keys (and char info).
! 
! perl interface and callback using swig'd interface
!  - build simple example and look for help via
!    perl5 book and swig mailing list.
! 
!   - support callback or first/next... processing ?
!     - first/next might be just simpler for perl
!     - probably pretty easy to support both.
! 
! libjlog
!   - main log_open should initialize default
!     logtracker and logtee from env variables
!     - do we want this to happen every log_open ?
!       - yes
!   - how do we define filters ?
!     - file/func filters
!     - in/out filters
!     - regexp ?
!     - config file
! 
!   - can mklog.pl generate/manage the list of
!     modules supported ? Would require that
!     enum gets initialized though.
!     - one enum should alwyas be MOD_OKAY=0
! 
!   - completely separate tree for jvlog ?
! 
!   - can success/failure callback be combined
!     with current callback mechanism ?
! 
!   - support success call from void function
! 
!   - LOGerror_e should become LOGvalues_e
!     and include LOG_SUCCESS=0, enum values should be negative
!     for error
! 
!   - how can we get gcc to check __attribute()__ stuff ?
! 
! nuke build environment and re-write.
! 
!  - use simple gmake makefiles that need to be
!    massaged to work....
! 
!  - could have basic configure at the top.
! 
!  - swig generation should be separate target and
!    not part of normal build.
!     - change name of .javaswig file to not specify
!       packaging, explicit move command and rmdir
!       with warnings if files end up lying around
! 
!  - cvs ci should include java generated c etc.
! 
!  - can I or should I use sysinclude.h ?
!    - probably don't need it... would need to test port to
!      cygwin for compiling into a dll.
! 
!    - can probably keep it if I get rid of Precise/Invidi
!      related bits (NATIVE_WINDOWS)
!      (switch if !defined(NATIVE_WINDOWS) to if defined(LINUX))
!      etc.
! 
! Utility for downloading/managing ftp list files
! 
! What is this thing for ?
! ------------------------
! - look at what Scott did for mysql loading.
! 
! - look at xmltv2db on sf for a basic example of
!   putting data into jdbc database - now sure how useful this is.
! 
! - support gzip'd list files.
!   - www.gzip.org/zlib
!     - can probably just include it.
! 
! - need bigger picture plan... what is libimdb ?
!   - for use with tv_imdb ?
!     - maybe tv_imdb just uses a libimdb database
!       - libimdb's own database could just be a
! 	series of text files in sorted key order.
! 	then use look to locate things.
!       - perl interface kept simple, while libimdb
! 	could use binary search algorithm.
!       - still doesn't require mysql and is platform
! 	independent.
!       - what other db options are out there ? gdbm ?
!       - nothing to say we can't use dynamic libimdb.so
!         for searching, provide a generic interface
! 	that includes what you want for tv_imdb
!         Then just have java and a perl interface.
! 
!     - guess a perl script is the easies to load
!       things into mysql ?
! 
!     - xml output of db entry (probably useful for
!       debugging as well).
! 
!     - xmlrpc ? what is it, is it useful here ?
! 
!   - linux and windows supported ?
!   - load imdb data into hsqldb / mysqldb
!     - how do we standardize on db tables ?
!     - how do you look flexible ?
!     - interface must support db clean, load
!     - way in the future what can we do about doing
!       minimal updates ? Can we only update the table
!       entries that changed since the last fresh build ?
!       Maybe the parse should support an update
!       operation ? How do we identify things that have
!       to be removed ? Can we ? Maybe intermediate db
!       (flat sorted index) and a compare or refresh mysql
!       utility.
!   - query data 
!   - do all of this from java and perl
  
! - can I build a java centric database loader (int hsqldb ?)
!   - what kind of progress bars are in java ?
  
!   - JFreeChart to plot things ?
!     - number of actors / movie
!     - years of movies ?
!     - present stats generated from loader ?
!     - actors to actresses totals
!     - movie ratings (user rated) against years
  
! - what about a imdb server that has XML-RPC interfaces
!   for queries ?
!   perl use Soap::Lite package (www.soaplite.com)
--- 1,19 ----
+ # $Id$
  
! The first this you need to know is this project is currently
! in pre-Alpha stages. The build environment is still under
! construction. You'll also probably be missing pieces like
! the correct version of swig etc. I'm currently developing
! under RH 9, so you'll have better luck if that's where
! you are.
  
! You'll need SWIG Version 1.3.19
  
! current status
!  - the jlog library works okay
!  - the parser library works (imdbDict.c will be changing alot)
!  - the java classes basically work
!  - the perl test code is completely broken.
!  - everything else is a crap shoot.
  
! jerry