[libimdb-commit] CVS: libimdb TODO,NONE,1.1 README,1.1,1.2
Status: Pre-Alpha
Brought to you by:
jveldhuis
|
From: Jerry V. <jve...@us...> - 2003-06-16 04:43:06
|
Update of /cvsroot/libimdb/libimdb
In directory sc8-pr-cvs1:/tmp/cvs-serv16121
Modified Files:
README
Added Files:
TODO
Log Message:
moved to TODO, new README in place
--- NEW FILE: TODO ---
Short list to get things out.
- build env
- roll out c library
- roll out beta java interface
- simple web page and setup of cvs, mailing lists etc.
- will want to use something like mpatrol to assess what
the memory requirements are for different parts of the
library.
- need to version things.
libimdb_getVersion();
libimdb_getListFileInfo(); ??
- is it easy to convert lfetch to a c program ?
libimdb_updateListFile();
- look at perl script to just get list files
- look at perl script to get and merge list diff files.
Things to be done
-----------------
Since movie lables are the key to the database, maybe a temperary
(option) on-disk db database to hold the movie info and rest of
them return references to entries via a movie key. The user does
a second set of lookups when the movie detail is needed. Making
it optional means not paying the disk price and instead paying
the memory price. (may also just be a runtime optional though,
since we may want to use an in-memory db still - this could be
substituted for a non-db solution later).
Guess that means we may need to lookup movies by key, but also
generate a list of movie label based results when searching by
title. This may require a second pass over the movie database.
Need to take a closer look at all the list file to assess if
this is the best strategy.
Simpler scan actor db one at a time - callbacks get a single actors
and multiple movie keys (and char info).
perl interface and callback using swig'd interface
- build simple example and look for help via
perl5 book and swig mailing list.
- support callback or first/next... processing ?
- first/next might be just simpler for perl
- probably pretty easy to support both.
libjlog
- main log_open should initialize default
logtracker and logtee from env variables
- do we want this to happen every log_open ?
- yes
- how do we define filters ?
- file/func filters
- in/out filters
- regexp ?
- config file
- can mklog.pl generate/manage the list of
modules supported ? Would require that
enum gets initialized though.
- one enum should alwyas be MOD_OKAY=0
- completely separate tree for jvlog ?
- can success/failure callback be combined
with current callback mechanism ?
- support success call from void function
- LOGerror_e should become LOGvalues_e
and include LOG_SUCCESS=0, enum values should be negative
for error
- how can we get gcc to check __attribute()__ stuff ?
nuke build environment and re-write.
- use simple gmake makefiles that need to be
massaged to work....
- could have basic configure at the top.
- swig generation should be separate target and
not part of normal build.
- change name of .javaswig file to not specify
packaging, explicit move command and rmdir
with warnings if files end up lying around
- cvs ci should include java generated c etc.
- can I or should I use sysinclude.h ?
- probably don't need it... would need to test port to
cygwin for compiling into a dll.
- clean up in order - remove NATIVE_WINDOWS
Utility for downloading/managing ftp list files
What is this thing for ?
------------------------
- look at what Scott did for mysql loading.
- look at xmltv2db on sf for a basic example of
putting data into jdbc database - now sure how useful this is.
- support gzip'd list files.
- www.gzip.org/zlib
- can probably just include it.
- need bigger picture plan... what is libimdb ?
- for use with tv_imdb ?
- maybe tv_imdb just uses a libimdb database
- libimdb's own database could just be a
series of text files in sorted key order.
then use look to locate things.
- perl interface kept simple, while libimdb
could use binary search algorithm.
- still doesn't require mysql and is platform
independent.
- what other db options are out there ? gdbm ?
- nothing to say we can't use dynamic libimdb.so
for searching, provide a generic interface
that includes what you want for tv_imdb
Then just have java and a perl interface.
- guess a perl script is the easies to load
things into mysql ?
- xml output of db entry (probably useful for
debugging as well).
- xmlrpc ? what is it, is it useful here ?
- linux and windows supported ?
- load imdb data into hsqldb / mysqldb
- how do we standardize on db tables ?
- how do you look flexible ?
- interface must support db clean, load
- way in the future what can we do about doing
minimal updates ? Can we only update the table
entries that changed since the last fresh build ?
Maybe the parse should support an update
operation ? How do we identify things that have
to be removed ? Can we ? Maybe intermediate db
(flat sorted index) and a compare or refresh mysql
utility.
- query data
- do all of this from java and perl
- can I build a java centric database loader (int hsqldb ?)
- what kind of progress bars are in java ?
- JFreeChart to plot things ?
- number of actors / movie
- years of movies ?
- present stats generated from loader ?
- actors to actresses totals
- movie ratings (user rated) against years
- what about a imdb server that has XML-RPC interfaces
for queries ?
perl use Soap::Lite package (www.soaplite.com)
Index: README
===================================================================
RCS file: /cvsroot/libimdb/libimdb/README,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** README 16 Jun 2003 04:18:37 -0000 1.1
--- README 16 Jun 2003 04:43:03 -0000 1.2
***************
*** 1,169 ****
! Short list to get things out.
! - build env
! - roll out c library
! - roll out beta java interface
! - simple web page and setup of cvs, mailing lists etc.
! - will want to use something like mpatrol to assess what
! the memory requirements are for different parts of the
! library.
! - need to version things.
! libimdb_getVersion();
!
! libimdb_getListFileInfo(); ??
!
! - is it easy to convert lfetch to a c program ?
! libimdb_updateListFile();
!
! - look at perl script to just get list files
! - look at perl script to get and merge list diff files.
!
! Things to be done
! -----------------
!
! Since movie lables are the key to the database, maybe a temperary
! (option) on-disk db database to hold the movie info and rest of
! them return references to entries via a movie key. The user does
! a second set of lookups when the movie detail is needed. Making
! it optional means not paying the disk price and instead paying
! the memory price. (may also just be a runtime optional though,
! since we may want to use an in-memory db still - this could be
! substituted for a non-db solution later).
!
! Guess that means we may need to lookup movies by key, but also
! generate a list of movie label based results when searching by
! title. This may require a second pass over the movie database.
!
! Need to take a closer look at all the list file to assess if
! this is the best strategy.
!
! Simpler scan actor db one at a time - callbacks get a single actors
! and multiple movie keys (and char info).
!
! perl interface and callback using swig'd interface
! - build simple example and look for help via
! perl5 book and swig mailing list.
!
! - support callback or first/next... processing ?
! - first/next might be just simpler for perl
! - probably pretty easy to support both.
!
! libjlog
! - main log_open should initialize default
! logtracker and logtee from env variables
! - do we want this to happen every log_open ?
! - yes
! - how do we define filters ?
! - file/func filters
! - in/out filters
! - regexp ?
! - config file
!
! - can mklog.pl generate/manage the list of
! modules supported ? Would require that
! enum gets initialized though.
! - one enum should alwyas be MOD_OKAY=0
!
! - completely separate tree for jvlog ?
!
! - can success/failure callback be combined
! with current callback mechanism ?
!
! - support success call from void function
!
! - LOGerror_e should become LOGvalues_e
! and include LOG_SUCCESS=0, enum values should be negative
! for error
!
! - how can we get gcc to check __attribute()__ stuff ?
!
! nuke build environment and re-write.
!
! - use simple gmake makefiles that need to be
! massaged to work....
!
! - could have basic configure at the top.
!
! - swig generation should be separate target and
! not part of normal build.
! - change name of .javaswig file to not specify
! packaging, explicit move command and rmdir
! with warnings if files end up lying around
!
! - cvs ci should include java generated c etc.
!
! - can I or should I use sysinclude.h ?
! - probably don't need it... would need to test port to
! cygwin for compiling into a dll.
!
! - can probably keep it if I get rid of Precise/Invidi
! related bits (NATIVE_WINDOWS)
! (switch if !defined(NATIVE_WINDOWS) to if defined(LINUX))
! etc.
!
! Utility for downloading/managing ftp list files
!
! What is this thing for ?
! ------------------------
! - look at what Scott did for mysql loading.
!
! - look at xmltv2db on sf for a basic example of
! putting data into jdbc database - now sure how useful this is.
!
! - support gzip'd list files.
! - www.gzip.org/zlib
! - can probably just include it.
!
! - need bigger picture plan... what is libimdb ?
! - for use with tv_imdb ?
! - maybe tv_imdb just uses a libimdb database
! - libimdb's own database could just be a
! series of text files in sorted key order.
! then use look to locate things.
! - perl interface kept simple, while libimdb
! could use binary search algorithm.
! - still doesn't require mysql and is platform
! independent.
! - what other db options are out there ? gdbm ?
! - nothing to say we can't use dynamic libimdb.so
! for searching, provide a generic interface
! that includes what you want for tv_imdb
! Then just have java and a perl interface.
!
! - guess a perl script is the easies to load
! things into mysql ?
!
! - xml output of db entry (probably useful for
! debugging as well).
!
! - xmlrpc ? what is it, is it useful here ?
!
! - linux and windows supported ?
! - load imdb data into hsqldb / mysqldb
! - how do we standardize on db tables ?
! - how do you look flexible ?
! - interface must support db clean, load
! - way in the future what can we do about doing
! minimal updates ? Can we only update the table
! entries that changed since the last fresh build ?
! Maybe the parse should support an update
! operation ? How do we identify things that have
! to be removed ? Can we ? Maybe intermediate db
! (flat sorted index) and a compare or refresh mysql
! utility.
! - query data
! - do all of this from java and perl
! - can I build a java centric database loader (int hsqldb ?)
! - what kind of progress bars are in java ?
! - JFreeChart to plot things ?
! - number of actors / movie
! - years of movies ?
! - present stats generated from loader ?
! - actors to actresses totals
! - movie ratings (user rated) against years
! - what about a imdb server that has XML-RPC interfaces
! for queries ?
! perl use Soap::Lite package (www.soaplite.com)
--- 1,19 ----
+ # $Id$
! The first this you need to know is this project is currently
! in pre-Alpha stages. The build environment is still under
! construction. You'll also probably be missing pieces like
! the correct version of swig etc. I'm currently developing
! under RH 9, so you'll have better luck if that's where
! you are.
! You'll need SWIG Version 1.3.19
! current status
! - the jlog library works okay
! - the parser library works (imdbDict.c will be changing alot)
! - the java classes basically work
! - the perl test code is completely broken.
! - everything else is a crap shoot.
! jerry
|