linkbat-devel — Discussion of development related issues

Re: [Linkbat-devel] Data format, XML to DB conversion

[Shanta M. <sh...@fo...>]

On Wed, 2002-11-20 at 12:01, James Mohr wrote:
> >
> > That is why I like to store my data in SQL. It allows me to manipulate
> > the data as I see fit. This includes exporting data to cvs files.
> 
> In principle I don't have a problem with that. However, it requires the 
> intermediate step to be SQL. If you have already inserted it into an SQL DB, 
> why bother exporting it to CSV? (unless you want to move it a server that has 
> no SQL DB) Although not necessarily efficient, I see a "friendlier" approach 
> to first parse the XML into CSV and then import that into SQL. 
> 

eXtropia puts things into the chosen DataSource. There is no need for
the XML interface to do that. 

> >
> > In my setup 0 change exempt one variable in the Global site setup.
> > $DataSource = "file"; or $DataSource = "MySQL";
> 
> What about the code itself? If a single file = MySQL in terms of being a data 
> source, I do not see how you could get around the problem of having to redo A 
> LOT of code. If a file equaled a MySQL Table, then I see a 1:1 relationship 
> between fields in MySQL and fields in the file, so there would be very litle 
> code change necessary. 

Long ago I added the falowing code to my application files

my @BASIC_DATASOURCE_CONFIG_PARAMS;
if ($site eq "file"){
 @BASIC_DATASOURCE_CONFIG_PARAMS = (    -TYPE                       =>
'File', 
    -FILE                       =>
"$APP_DATAFILES_DIRECTORY/$APP_NAME.dat",
    -FIELD_DELIMITER            => '|',
    -COMMENT_PREFIX             => '#',
    -CREATE_FILE_IF_NONE_EXISTS => 1,
    -FIELD_NAMES                => \@DATASOURCE_FIELD_NAMES,
    -KEY_FIELDS                 => ['record_id'],
    -FIELD_TYPES                => {
                                    record_id        => 'Autoincrement',
                                    datetime         => 
                                    [
                                     -TYPE  => "Date",
                                     -STORAGE => 'y-m-d H:M:S',
                                     -DISPLAY => 'y-m-d H:M:S',
                                    ],
                                   },
);
}
else{
	@BASIC_DATASOURCE_CONFIG_PARAMS = (
	        -TYPE         => 'DBI',
	        -DBI_DSN      =>  $DBI_DSN,
	        -TABLE        => 'linkbat_questions_tb',
	        -USERNAME     => $AUTH_MSQL_USER_NAME,
	        -PASSWORD     => $MySQLPW,
	        -FIELD_NAMES  => \@DATASOURCE_FIELD_NAMES,
	        -KEY_FIELDS   => ['username'],
	        -FIELD_TYPES  => {
	            record_id        => 'Autoincrement',
                    datetime         => 
                                    [
                                     -TYPE  => "Date",
                                     -STORAGE => 'y-m-d H:M:S',
                                     -DISPLAY => 'y-m-d H:M:S',
                                    ],
	        },
	);

In the Global setup the variables are defined as they are used my all
apps in the site only the table name needs to be changed in telling
eXtropia how the data is stored. cvs file or SQL. 

> 
> > > > Validation, field
> > > > contents, required fields and many other checking is already in
> > > > eXtropia one just has to tell it what you which to check for.  Don't
> > > > know if this is what you are referring to with XML validation as I know
> > > > little of XML.
> > >
> > > I'm refering to the actual XML files before they are imported into the
> > > database. If someone forgets a closing tag or something else, we should
> > > know about it before we try to read it into the database.
> >
> > Ok. but that is why one develops on a separate server. could be added to
> > eXtropia but XML in the form View would likely be easier
> 
> What about performance. An XML files is at least 10 times larger. Plus we have 
> the issues of the inter-relationships between the various KUs. To what extent 
> can this be handle by eXtropia when directly reading the XML files? 
> 
Not sure! As a Perl programmer I see XML as an more complex form of
HTML. I would not chose to use XML to directly access the underlying
data files but that is largely because I have not taken the time yet to
look into XML as a programing language. In this context I was thinking
more in the form of Javascripts ability to do checking in the browser. I
don't even know if XML can do that.

In eXtropia XML is placed in TTML files. This blends Perl into the
"View" that gets exported into the browser.

> Even if eXtropia could read the XML files directly and be able to present the 
> relationships correctly. I cannot see how that would **not** lock us into 
> eXtropia as the delivery mechanism. If in a CSV text file or SQL DB, then we 
> can do standard queries on the data. 
> 

In eXtropia the queries are done on ActionHandlers to separate the logic
you are developing from the look and feel of the site. For the most part
you just tell eXtropia what you want it make the quire to the DataSource
so you don't have write new code to switch from SQL to CSV 

Here is an example

    $cgi->param(
        -NAME  => 'raw_search',
        -VALUE => "project_code=='Requested'"
    );

There is a subroutine built into the system that forms a SQL quire and
sends it to MySQL or parses the CSV File for the records that match your
request. 

No rewriting of code to accommodate the change in DataSource. I you want
to duplicate the mechanics of doing this you will take years to get it
right. I don't have any desire to do that unless the default dose not do
what I want then I will hack the code, Isolate it from the default code
and call it as I need it. Selena, Gunther, and Peter are far better
programmers than I am. Gunther worked for Lincoln Stein. They all
program for banks using this application.


> 
> -------------------------------------------------------
> This sf.net email is sponsored by: 
> Battle your brains against the best in the Thawte Crypto 
> Challenge. Be the first to crack the code - register now: 
> http://www.gothawte.com/rd521.html
> _______________________________________________
> Linkbat-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linkbat-devel
-- 
Shanta McBain <sh...@fo...>

Re: [Linkbat-devel] Questions Table

[James M. <lin...@ji...>]

On Wednesday 20 November 2002 19:33, Shanta McBain wrote:
> Hi All
>
> I have created the Questions table here is the details contained in the
> Client news link on the left side bar this is only visible to
> Linkbat_admin. I also have not converted this to the Linkbat View yet so
> you will not get a left side bar or style sheets. (this is written with
> View from the web)

I have three links labled "Client news". None of them provide any questions. 
(At least nothing I have found.) 

>
> Added Questions table
>
> Please note new addition in admin add table on the left side bar. This
> link is to the table of questions. see details.
>
> The Question number no longer match the original table as the first
> field in the MySQL table was set to auto incriminate so the table was
> populated with a different set of numbers than the csv file. If this is
> critical I can redo it. turning of auto inc then turning it on after
> install.

The actual question numbers (i.e. question IDs) are not important. What is 
important is the included page ID. This provides the link back to the page 
where the this question came from. 

>
> Another note of importance is the category field . The drop list of
> entries are contained in a table and filled via a ActionHandler. This
> allows admin to add new categories by placing them in the table. I am
> currently updating this system and will add the changes soon. This will
> consist of an add form in the admin add links and a ActionHandler tuned
> to Linkbats needs.

I am missing the point of a form to input questions. I understand that this is 
something that eXtropia can do, but unless this mechanism can insert the 
question into a KU in the XML file and make the necessary references I am 
missing the point. 

regards,

jimmo

-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Questions Table

[Shanta M. <sh...@fo...>]

Hi All

I have created the Questions table here is the details contained in the
Client news link on the left side bar this is only visible to
Linkbat_admin. I also have not converted this to the Linkbat View yet so
you will not get a left side bar or style sheets. (this is written with
View from the web)

Added Questions table

Please note new addition in admin add table on the left side bar. This
link is to the table of questions. see details.

The Question number no longer match the original table as the first
field in the MySQL table was set to auto incriminate so the table was
populated with a different set of numbers than the csv file. If this is
critical I can redo it. turning of auto inc then turning it on after
install.

Another note of importance is the category field . The drop list of
entries are contained in a table and filled via a ActionHandler. This
allows admin to add new categories by placing them in the table. I am
currently updating this system and will add the changes soon. This will
consist of an add form in the admin add links and a ActionHandler tuned
to Linkbats needs.


Have Fun.


-- 
Shanta McBain <sh...@fo...>

Re: [Linkbat-devel] Data format, XML to DB conversion

[James M. <lin...@ji...>]

On Wednesday 20 November 2002 18:33, Shanta McBain wrote:
> On Wed, 2002-11-20 at 10:46, James Mohr wrote:
> > That's exactly it: "when a host that dose not have access to SQL" and
> > that's really the only reason.  A corollary is someone who does not have
> > the skill or desire to set up an SQL database, but that is less important
> > to me than someone who cannot set it up. However, it will be a while
> > before it is a "product" that we can package and make available on
> > SourceForge, etc. I just don't want to paint ourselves into a corner.
>
> That is why I like to store my data in SQL. It allows me to manipulate
> the data as I see fit. This includes exporting data to cvs files.

In principle I don't have a problem with that. However, it requires the 
intermediate step to be SQL. If you have already inserted it into an SQL DB, 
why bother exporting it to CSV? (unless you want to move it a server that has 
no SQL DB) Although not necessarily efficient, I see a "friendlier" approach 
to first parse the XML into CSV and then import that into SQL. 

> > > I will have the CSV s files into MySQL in a few hours. I have code to
> > > do this by simply importing the contents into the SQL Table.PHP but it
> > > works.  From that point on to switch the code from on DataSource to
> > > another takes a simple change in one variable in the site setup file.
> > > Export the contents of the SQL table to a | delimited file though and
> > > delimiter can be used.  and there you have switched.
> >
> > Does that  mean there would be little change in the eXtropia code to
> > reflect a CSV data source as opposed to SQL?
>
> In my setup 0 change exempt one variable in the Global site setup.
> $DataSource = "file"; or $DataSource = "MySQL";

What about the code itself? If a single file = MySQL in terms of being a data 
source, I do not see how you could get around the problem of having to redo A 
LOT of code. If a file equaled a MySQL Table, then I see a 1:1 relationship 
between fields in MySQL and fields in the file, so there would be very litle 
code change necessary. 


> > > Validation, field
> > > contents, required fields and many other checking is already in
> > > eXtropia one just has to tell it what you which to check for.  Don't
> > > know if this is what you are referring to with XML validation as I know
> > > little of XML.
> >
> > I'm refering to the actual XML files before they are imported into the
> > database. If someone forgets a closing tag or something else, we should
> > know about it before we try to read it into the database.
>
> Ok. but that is why one develops on a separate server. could be added to
> eXtropia but XML in the form View would likely be easier

What about performance. An XML files is at least 10 times larger. Plus we have 
the issues of the inter-relationships between the various KUs. To what extent 
can this be handle by eXtropia when directly reading the XML files?

Even if eXtropia could read the XML files directly and be able to present the 
relationships correctly. I cannot see how that would **not** lock us into 
eXtropia as the delivery mechanism. If in a CSV text file or SQL DB, then we 
can do standard queries on the data. 

I think I'll draw a diagram of what I was thinking about it and post it.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] The Linux Tutorial Addition

[<sh...@fo...>]

Add Event.
----------
  
subject                        Added Questions table
abstract                       Please note new addition in admin add table on the left side bar. This link is to the table of questions. see details
full_text                      The Question number no longer match the original table as the first field in the MySQL table was set to auto incrimate so the table was populated with a diffrent set of numbers than the csv file.

Another note of importance is the category field . The drop list of entries are contained in a table and filled via a ActionHandler. This alows the 


  
  
    
   

Re: [Linkbat-devel] Data format, XML to DB conversion

[Shanta M. <sh...@fo...>]

On Wed, 2002-11-20 at 10:46, James Mohr wrote:

> That's exactly it: "when a host that dose not have access to SQL" and that's 
> really the only reason.  A corollary is someone who does not have the skill 
> or desire to set up an SQL database, but that is less important to me than 
> someone who cannot set it up. However, it will be a while before it is a 
> "product" that we can package and make available on SourceForge, etc. I just 
> don't want to paint ourselves into a corner. 
> 

That is why I like to store my data in SQL. It allows me to manipulate
the data as I see fit. This includes exporting data to cvs files.

> >
> > I will have the CSV s files into MySQL in a few hours. I have code to do
> > this by simply importing the contents into the SQL Table.PHP but it
> > works.  From that point on to switch the code from on DataSource to
> > another takes a simple change in one variable in the site setup file.
> > Export the contents of the SQL table to a | delimited file though and
> > delimiter can be used.  and there you have switched. 
> 
> Does that  mean there would be little change in the eXtropia code to reflect a 
> CSV data source as opposed to SQL?
> 

In my setup 0 change exempt one variable in the Global site setup.
$DataSource = "file"; or $DataSource = "MySQL";

DataSources currently supported by eXtropia are

file, MySQL, Oracle and a few others but they are not important. 

> > Validation, field
> > contents, required fields and many other checking is already in eXtropia
> > one just has to tell it what you which to check for.  Don't know if this
> > is what you are referring to with XML validation as I know little of
> > XML.
> 
> I'm refering to the actual XML files before they are imported into the 
> database. If someone forgets a closing tag or something else, we should know 
> about it before we try to read it into the database. 
> 
Ok. but that is why one develops on a separate server. could be added to
eXtropia but XML in the form View would likely be easier

> > > Following are some thoughts and ideas about how the tables can be
> > > created. I have no particular affection for any of these ideas, so feel
> > > free to tell me they stink.
> > >
> > > The first thing we need to decided for the data model is how to store the
> > > KUs themselves. Since different KU types have different attributes, I do
> > > not see a way of storing them all in a single table. Instead, I would see
> > > one table for each KU type and then a central table that contains the
> > > unique KU ID and then the KU type. When making a cross reference from one
> > > KU to another, we would then go through this central table.
> > >
> > > Next is the inter-relationship between the KUs, which is a core aspect of
> > > linkbat. There is a potential that each KU (regardless of type) can
> > > reference every other type of KU. However, I cannot really imagine an
> > > index for each relationship. For example, the Concept KU has an index
> > > listing all of the Concept KUs and with pointers to all of the referenced
> > > Glossary KUs.  Then another index with the MoreInfo KUs. In my mind that
> > > would mean too many tables. If there are five KU types and each has a
> > > index for the relationship to the others, then we would have 20 tables
> > > (5x4). With 6 KU tables, 30 tables (6x5). Is my math right?
> > >
> > > However, the question is whether 20 or 30 indexes is "too many".  Does
> > > having seperate indexes for each relationship provide any advantages?
> > > Quicker access? If so, does it compensate for the extra work to manage
> > > 20-30 tables??
> >
> > The SQL table takes care of that for another reason to store all data in
> > SQL.You can assign indexes within each table.
> 
> I'm aware of that. However, these are not quite the same indexes that an RDBMS 
> system like MySQL would generate automatically. It's not a index of a single 
> table used to speed-up access, but rather a table containing the relationship 
> between the KUs. These would be something we would have to define and fill 
> ourselves I'm just curous about the performance issue of having 1 or 30 
> tables versus the administration of 30 tables. Is that really an issue. 
> 


> Based on the discusion following this part, I think only a single table will 
> be needed since MySQL should be able to handle the amount of data we are 
> dealing with.
> 

> -------------------------------------------------------
> This sf.net email is sponsored by: To learn the basics of securing 
> your web site with SSL, click here to get a FREE TRIAL of a Thawte 
> Server Certificate: http://www.gothawte.com/rd524.html
> _______________________________________________
> Linkbat-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linkbat-devel
-- 
Shanta McBain <sh...@fo...>

Re: [Linkbat-devel] Data format, XML to DB conversion

[James M. <lin...@ji...>]

On Wednesday 20 November 2002 17:24, Shanta McBain wrote:

<extropia stuff snipped>
> I am a bit at a loss as to what is so special about CSV text files. I
> see them only as a poor and limited (Hard to display code) form of data
> storage When data is stored in SQL there is no need for them at all
> except as an alternate storage system when a host that dose not have
> access to SQL. That being said I still use SQL quires to search the
> files for the records and contents of the records you what to display in
> the "View".

That's exactly it: "when a host that dose not have access to SQL" and that's 
really the only reason.  A corollary is someone who does not have the skill 
or desire to set up an SQL database, but that is less important to me than 
someone who cannot set it up. However, it will be a while before it is a 
"product" that we can package and make available on SourceForge, etc. I just 
don't want to paint ourselves into a corner. 

> > Rama has volunteered to work on the code to convert the XML files into
> > something the database can handle. Shanta is working on the "presentation
> > layer", so obviously you two will be working closely together. When the
> > code is done for the CSV->XML conversion, I would like Luu to start
> > working on the code to validate the XML files. That is, making sure the
> > right tags are there, tags match, etc.
>
> I will have the CSV s files into MySQL in a few hours. I have code to do
> this by simply importing the contents into the SQL Table.PHP but it
> works.  From that point on to switch the code from on DataSource to
> another takes a simple change in one variable in the site setup file.
> Export the contents of the SQL table to a | delimited file though and
> delimiter can be used.  and there you have switched. 

Does that  mean there would be little change in the eXtropia code to reflect a 
CSV data source as opposed to SQL?

> Validation, field
> contents, required fields and many other checking is already in eXtropia
> one just has to tell it what you which to check for.  Don't know if this
> is what you are referring to with XML validation as I know little of
> XML.

I'm refering to the actual XML files before they are imported into the 
database. If someone forgets a closing tag or something else, we should know 
about it before we try to read it into the database. 

> > Following are some thoughts and ideas about how the tables can be
> > created. I have no particular affection for any of these ideas, so feel
> > free to tell me they stink.
> >
> > The first thing we need to decided for the data model is how to store the
> > KUs themselves. Since different KU types have different attributes, I do
> > not see a way of storing them all in a single table. Instead, I would see
> > one table for each KU type and then a central table that contains the
> > unique KU ID and then the KU type. When making a cross reference from one
> > KU to another, we would then go through this central table.
> >
> > Next is the inter-relationship between the KUs, which is a core aspect of
> > linkbat. There is a potential that each KU (regardless of type) can
> > reference every other type of KU. However, I cannot really imagine an
> > index for each relationship. For example, the Concept KU has an index
> > listing all of the Concept KUs and with pointers to all of the referenced
> > Glossary KUs.  Then another index with the MoreInfo KUs. In my mind that
> > would mean too many tables. If there are five KU types and each has a
> > index for the relationship to the others, then we would have 20 tables
> > (5x4). With 6 KU tables, 30 tables (6x5). Is my math right?
> >
> > However, the question is whether 20 or 30 indexes is "too many".  Does
> > having seperate indexes for each relationship provide any advantages?
> > Quicker access? If so, does it compensate for the extra work to manage
> > 20-30 tables??
>
> The SQL table takes care of that for another reason to store all data in
> SQL.You can assign indexes within each table.

I'm aware of that. However, these are not quite the same indexes that an RDBMS 
system like MySQL would generate automatically. It's not a index of a single 
table used to speed-up access, but rather a table containing the relationship 
between the KUs. These would be something we would have to define and fill 
ourselves I'm just curous about the performance issue of having 1 or 30 
tables versus the administration of 30 tables. Is that really an issue. 

Based on the discusion following this part, I think only a single table will 
be needed since MySQL should be able to handle the amount of data we are 
dealing with.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Command KU database tables

[James M. <lin...@ji...>]

Hi All!

It just dawned on me that we might have a problem with the command KUs. Each 
command has a different number of options and examples. So I don't see a way 
of having one record per command. The way is see it is that we have a table 
for the options that has (at least three fields):

COMMAND_KU_ID:OPTION:ACTION

Where the OPTION is the text of the option and ACTION is a description of what 
the option does. We would also have a table for the examples:

COMMAND_KU_ID:TEXT:DESCRIPTION

Where TEXT is the syntax example and DESCRIPTION is a short description of 
that example.

Commands also have other attributes like a synopsis, description and syntax 
(look at any man-page for what I am talking about).  However,  these will be 
consistant for all Commands KU.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

Re: [Linkbat-devel] Data format, XML to DB conversion

[Shanta M. <sh...@fo...>]

I have made limited comments in line. Could everyone go to
http://webcthelpdesk.com/cgi-bin/Linkbat/linkbat.cgi?site=Linkbat

click the logon link under the logo on the left and create an account. I
will set the system to display so group normal will display all the
links. Until I get you all changed to Linkbat_admin. The script mails me
when an account is created.

To bring everyone up to speed on this. This site is created in a web
application delivery product known as eXtropia. The product takes care
of Sessions, Authentication, DataSource (works transparently with CVS or
SQL), Views, name space, and much much more. There is 15 or more years
of development here with the last five for banks. I have been developing
applications in this product now for 4 or 5 years. At first it was vary
much a blackbox but now it is a resource I would not be without. 

Remember that this is not necessarily the end delivery engine. James has
not at this time accepted this as the method of delivering the project
but rather a Demo on what eXtropia can do. This site is also a blend of
Linkbat and my admin services. It has tools in it that are not available
to any but my clients at this time. Project tracking, todo's logs, FAQ
etc. They are in the extra Admin link box. All these tool's contents are
in MySQL.
   
On Wed, 2002-11-20 at 07:02, James Mohr wrote:
> Hi All!
> 
> I think we have reached the point where we need to be talking about the actual 
> data/table structure in CSV text or a database. For now let's just work on 
> the database model, to make things simpler. However, we all need to keep in 
> mind that I eventually want to have the system work with CSV text files.
> 

I am a bit at a loss as to what is so special about CSV text files. I
see them only as a poor and limited (Hard to display code) form of data
storage When data is stored in SQL there is no need for them at all
except as an alternate storage system when a host that dose not have
access to SQL. That being said I still use SQL quires to search the
files for the records and contents of the records you what to display in
the "View". 

> Rama has volunteered to work on the code to convert the XML files into 
> something the database can handle. Shanta is working on the "presentation 
> layer", so obviously you two will be working closely together. When the code 
> is done for the CSV->XML conversion, I would like Luu to start working on the 
> code to validate the XML files. That is, making sure the right tags are 
> there, tags match, etc. 

I will have the CSV s files into MySQL in a few hours. I have code to do
this by simply importing the contents into the SQL Table.PHP but it
works.  From that point on to switch the code from on DataSource to
another takes a simple change in one variable in the site setup file.
Export the contents of the SQL table to a | delimited file though and
delimiter can be used.  and there you have switched. Validation, field
contents, required fields and many other checking is already in eXtropia
one just has to tell it what you which to check for. Don't know if this
is what you are referring to with XML validation as I know little of
XML.

> Following are some thoughts and ideas about how the tables can be created. I 
> have no particular affection for any of these ideas, so feel free to tell me 
> they stink.
> 
> The first thing we need to decided for the data model is how to store the KUs 
> themselves. Since different KU types have different attributes, I do not see 
> a way of storing them all in a single table. Instead, I would see one table 
> for each KU type and then a central table that contains the unique KU ID and 
> then the KU type. When making a cross reference from one KU to another, we 
> would then go through this central table. 
> 
> Next is the inter-relationship between the KUs, which is a core aspect of 
> linkbat. There is a potential that each KU (regardless of type) can reference 
> every other type of KU. However, I cannot really imagine an index for each 
> relationship. For example, the Concept KU has an index listing all of the 
> Concept KUs and with pointers to all of the referenced Glossary KUs.  Then 
> another index with the MoreInfo KUs. In my mind that would mean too many 
> tables. If there are five KU types and each has a index for the relationship 
> to the others, then we would have 20 tables (5x4). With 6 KU tables, 30 
> tables (6x5). Is my math right?
> 
> However, the question is whether 20 or 30 indexes is "too many".  Does having 
> seperate indexes for each relationship provide any advantages? Quicker 
> access? If so, does it compensate for the extra work to manage 20-30 tables??
> 

The SQL table takes care of that for another reason to store all data in
SQL.You can assign indexes within each table. 

> Alternatively, I could imagine one index that keeps tracks of the type of KUs, 
> as well:
> 
> UNIQUE_KU_ID:REFERENCED_TYPE:REFERENCED_KU_ID
> 
> UNIQUE_KU_ID - The ID of the current KU
> REFERENCED_TYPE - The type of KU that this KU is referencing
> REFERENCED_KU_ID - The ID of that KU that is being refereced. 
> 
> A query could be done sorted by UNIQUE_KU_ID  and then REFERENCED_TYPE. Once 
> we have that, the code to present a list of referenced KUs grouped by KU type 
> would be easy.  Currently we are only dealing with a few thousands KUs. 
> However, what about when we get to tens of thousands?? Hopefully MySQL should 
> be in a position to deal with that **few** records. So, I would not see it as 
> a problem to have all of the KUs referenced in a single table like that. 
> 
> Is the REFERENCED_TYPE even necessary here? We obviously need a table that 
> contains the attributes of each KU and a unique ID, so the REFERENCED_KU_ID 
> can be used to get the type through the central table that I mentioned above 
> and we have the KU type. On the other hand, to get a list of all referenced 
> KUs in the correct order we could have a query like this:
> 
> select REFERENCED_KU_ID from WHATEVER_TABLE where UNIQUE_KU_ID = 
> 'CURRENT_KU_ID" sort by REFERENCED_TYPE
> 
> This is a single table access and is obviously faster than mutliple access. 
> Therefore, I think this would more than make up for having an extra field in 
> the index. 
> 
> Since we can expect that new data files will have mixed KU types, I am 
> thinking that a single index which also contains the KU type would be more 
> efficient. Each time we read a new KU, it is inserted into that single index, 
> so we don't have to deal with mulitple indexes.  Comments?
> 
> Although not KUs in the true sense, I think we need an index by topic and 
> skill level (if we decide to implement that). This would be used in 
> generateing the technical FAQs or lists of all info on a particular topic. 
> 
> What about the reverse references. For example a list of all Concept KUs that 
> reference a particular Glossary KU. This is not the same as the list of 
> Glossary KUs that this Concept KU references. We click on a link that 
> displays a Glossary KU, with a link to all the KUs that it references. I 
> would also like links to all of the KUs that reference this glossary term. 
> Although it seems logical that each Concept KU that references a particular 
> Glossary KU should also be referenced by that Glossary KU, it is likely that 
> we will forget the reverse references when create the XML files. Think about 
> the glossary terms on the content pages. Each time we add a new page we have 
> to update the referenced glossary KUs to point back to this page.
> 
> I'm sure there is a lot more, but I should get this off to start the 
> discussion going.
> 
> Regards,
> 
> jimmo
> -- 
> ---------------------------------------
> "Be more concerned with your character than with your reputation. Your
> character is what you really are while your reputation is merely what others
> think you are." -- John Wooden
> ---------------------------------------
> Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info
> 
> 
> -------------------------------------------------------
> This sf.net email is sponsored by: To learn the basics of securing 
> your web site with SSL, click here to get a FREE TRIAL of a Thawte 
> Server Certificate: http://www.gothawte.com/rd524.html
> _______________________________________________
> Linkbat-devel mailing list
> Lin...@li...
> https://lists.sourceforge.net/lists/listinfo/linkbat-devel
-- 
Shanta McBain <sh...@fo...>

[Linkbat-devel] Data format, XML to DB conversion

[James M. <lin...@ji...>]

Hi All!

I think we have reached the point where we need to be talking about the actual 
data/table structure in CSV text or a database. For now let's just work on 
the database model, to make things simpler. However, we all need to keep in 
mind that I eventually want to have the system work with CSV text files.

Rama has volunteered to work on the code to convert the XML files into 
something the database can handle. Shanta is working on the "presentation 
layer", so obviously you two will be working closely together. When the code 
is done for the CSV->XML conversion, I would like Luu to start working on the 
code to validate the XML files. That is, making sure the right tags are 
there, tags match, etc. 

Following are some thoughts and ideas about how the tables can be created. I 
have no particular affection for any of these ideas, so feel free to tell me 
they stink.

The first thing we need to decided for the data model is how to store the KUs 
themselves. Since different KU types have different attributes, I do not see 
a way of storing them all in a single table. Instead, I would see one table 
for each KU type and then a central table that contains the unique KU ID and 
then the KU type. When making a cross reference from one KU to another, we 
would then go through this central table. 

Next is the inter-relationship between the KUs, which is a core aspect of 
linkbat. There is a potential that each KU (regardless of type) can reference 
every other type of KU. However, I cannot really imagine an index for each 
relationship. For example, the Concept KU has an index listing all of the 
Concept KUs and with pointers to all of the referenced Glossary KUs.  Then 
another index with the MoreInfo KUs. In my mind that would mean too many 
tables. If there are five KU types and each has a index for the relationship 
to the others, then we would have 20 tables (5x4). With 6 KU tables, 30 
tables (6x5). Is my math right?

However, the question is whether 20 or 30 indexes is "too many".  Does having 
seperate indexes for each relationship provide any advantages? Quicker 
access? If so, does it compensate for the extra work to manage 20-30 tables??

Alternatively, I could imagine one index that keeps tracks of the type of KUs, 
as well:

UNIQUE_KU_ID:REFERENCED_TYPE:REFERENCED_KU_ID

UNIQUE_KU_ID - The ID of the current KU
REFERENCED_TYPE - The type of KU that this KU is referencing
REFERENCED_KU_ID - The ID of that KU that is being refereced. 

A query could be done sorted by UNIQUE_KU_ID  and then REFERENCED_TYPE. Once 
we have that, the code to present a list of referenced KUs grouped by KU type 
would be easy.  Currently we are only dealing with a few thousands KUs. 
However, what about when we get to tens of thousands?? Hopefully MySQL should 
be in a position to deal with that **few** records. So, I would not see it as 
a problem to have all of the KUs referenced in a single table like that. 

Is the REFERENCED_TYPE even necessary here? We obviously need a table that 
contains the attributes of each KU and a unique ID, so the REFERENCED_KU_ID 
can be used to get the type through the central table that I mentioned above 
and we have the KU type. On the other hand, to get a list of all referenced 
KUs in the correct order we could have a query like this:

select REFERENCED_KU_ID from WHATEVER_TABLE where UNIQUE_KU_ID = 
'CURRENT_KU_ID" sort by REFERENCED_TYPE

This is a single table access and is obviously faster than mutliple access. 
Therefore, I think this would more than make up for having an extra field in 
the index. 

Since we can expect that new data files will have mixed KU types, I am 
thinking that a single index which also contains the KU type would be more 
efficient. Each time we read a new KU, it is inserted into that single index, 
so we don't have to deal with mulitple indexes.  Comments?

Although not KUs in the true sense, I think we need an index by topic and 
skill level (if we decide to implement that). This would be used in 
generateing the technical FAQs or lists of all info on a particular topic. 

What about the reverse references. For example a list of all Concept KUs that 
reference a particular Glossary KU. This is not the same as the list of 
Glossary KUs that this Concept KU references. We click on a link that 
displays a Glossary KU, with a link to all the KUs that it references. I 
would also like links to all of the KUs that reference this glossary term. 
Although it seems logical that each Concept KU that references a particular 
Glossary KU should also be referenced by that Glossary KU, it is likely that 
we will forget the reverse references when create the XML files. Think about 
the glossary terms on the content pages. Each time we add a new page we have 
to update the referenced glossary KUs to point back to this page.

I'm sure there is a lot more, but I should get this off to start the 
discussion going.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

Re: [Linkbat-devel] Locaton of data of FAQ's

[James M. <lin...@ji...>]

On Wednesday 20 November 2002 02:31, Shanta McBain wrote:
> Were is the data for FAQ's stored?

If you are referring to the site FAQ, then it is pure text. That is the page 
is all hardcoded. It started out as a simply introduction with just a couple 
of paragraphs and grew into its current form.

Currently there are no technical FAQs, but that will come out of the XML 
files.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

Re: [Linkbat-devel] Data File definitions

[James M. <lin...@ji...>]

On Wednesday 20 November 2002 02:30, Shanta McBain wrote:
> This line came from tyk.data.
>
> 1:3:Two different users can share the same program.:T::::
>
> my guess
>
> RecordID auto inc. | quiz_id | Question | Answer :?:? :etc
> Please give your current field names. I will create an access to the raw
> data (this will become the Question db) and put the content of this file
> into MySQL. This is obviously the quiz dataset.

The format is: 

QUESTION_ID:PAGE_ID:QUESTION_TEXT:CORRECT_ANSWER:WRONG_ANSWER1:..:WRONG_ANSWER4

If the correct answer is either T or F, the code knows that it is a True/False 
question and the other answer is obviously wrong. The page ID is the 
reference into the page.data file. 

The "real" version of the DB also needs to account for a "reason" field for 
each answer. 

> I have on hand a copy of WebCT's they have a well developed quiz system
> that is split into to parts.
>
> The actual quiz and the questions that it uses. It will be a good source
> of code snippets and ideas( not open licence so mostly ideas.) This
> product is a complex course delivery tool licenced to University's
> across the world. If you have taken an on line course from a major
> University it likly was delivered with WebCT software. But that another
> story.

The name sounds familiar, but I am not sure. I have seen a number of CBT 
systems that have some really nice features, one is obviously the user 
management. Another is proving a series of questions without revealing the 
answer until all questions are answered. Currently my system presents the 
correct answer immediately. However, that  is useful too. One idea for the 
future is the dynamic question generation based on what has been answered 
wrong up to that point (what's the correct term?). If they get a question 
wrong in a particular area they keep getting more in that area. 

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Locaton of data of FAQ's

[Shanta M. <sh...@fo...>]

Were is the data for FAQ's stored?

-- 
Shanta McBain <sh...@fo...>

[Linkbat-devel] Data File definitions

[Shanta M. <sh...@fo...>]

This line came from tyk.data.

1:3:Two different users can share the same program.:T::::

my guess

RecordID auto inc. | quiz_id | Question | Answer :?:? :etc
Please give your current field names. I will create an access to the raw
data (this will become the Question db) and put the content of this file
into MySQL. This is obviously the quiz dataset.

I have on hand a copy of WebCT's they have a well developed quiz system
that is split into to parts.

The actual quiz and the questions that it uses. It will be a good source
of code snippets and ideas( not open licence so mostly ideas.) This
product is a complex course delivery tool licenced to University's
across the world. If you have taken an on line course from a major
University it likly was delivered with WebCT software. But that another
story.

Thanks

-- 
Shanta McBain <sh...@fo...>

[Linkbat-devel] New data file (includes Topics)

[James M. <lin...@ji...>]

Hi All!

Here is a new version of  the dyk.data file (which is converted into Concept 
KUs). I have added 1-3 topics to each of them. These are the topics that I 
talked about in my last email.

Luu, please integrate this into the code. I will be working on adding topics 
to the other data files, but I would like to have input on the list of topics 
before I do all of them. With this new dyk.data file, you can do the code to 
create the <Topics> container and the <TopicRef> entries. The code should not 
be any different for the other data files.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Linkbat Knowledge Unit Topics

[James M. <lin...@ji...>]

Hi All!

I would like some input about standard topics. Here are the ones that I came 
up with (in no special order):

Networking
Operating System
Administration
Working with the System
Users
Hardware
Troubleshooting
Security
Linux-Windows Integration

These are "related" to the chapter headings within the tutorial but not 
identical.  Chapter headings tend to be more "wordy". Plus a single chapter 
can relate to multiple topics, plus a single topic could be spread across 
multiple chapters. I would like to have the list of topics as simple as 
possible, but cover the most important topics. 

I am pretty happy with this list, although "Linux-Windows Integration" 
**might** be a little too much. However, we must keep our audience in mind. 
There will be a lot of people who are moving from Windows to Linux and need 
information about integration, easing the transition, and so on. The term 
"integration" is not necessarily the best choice, so I am open to 
suggestions. 

Another one is "Working with the System". This refers to issue related to 
working with the system (what else), like shells, X-Windows, logging in, etc. 
This is not the same as "Users" which refers to topics related to user 
accounts (passwords, permissions, allowing remote access, etc.)

Please let me hear from you as soon as possible. (Please, be sure to send to 
the list and not just to me!)

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Re: question on moreinfo.data (Everyone please read)

[James M. <lin...@ji...>]

(Note this was sent to the list.)

Hey everyone, the conversion is almost done (well, at least the code for it). 
Thanks Luu! However, there are some important questions to answer NOW, before 
we continue. PLEASE, please, please read this and give me your input.

On Tuesday 19 November 2002 00:31, Luan Luu wrote:
> according the the moreinfo.data, the format:
> ID#:TYPE:DESCRIPTION:LOCATION
>
> the XML are
>
> <KnowledgeUnit>
> <Atrributes>
>   <Type sub-type="[TYPE]" location="[LOCATION]">MoreInfo</Type>
>   <Text>[DESCRIPTION]</Text>
> </Atrributes>
> </KnowledgeUnit>
>
> In the reference to the brackets, is the pointer the the type, location,
> and description like that?

Perfect. The only question is whether we should actually do it that way or 
not. That is, should the sub-type and location be attributes within the 
<Type> tag or should they be seperate tags, i.e. <SubType>  <Location>?

By gut feeling is that they should be attributes within the  <Type> tag. They 
not necessarily attributes of KU, but rather provide additional info for the 
type. 

Comments anyone? This needs to be answered before we continue. 

> is the url be absolute path with the http infront right?

Yes. You will note that in the data file, they just begin with // and not 
http://. This was because I made an unwise decision to use the colon (:) as 
the field seperator. However, the colon appears frequently in Linux 
(especially with URLs) so it became a problem. 

I was going to change to something like a pipe (|) which comes less 
frequently. Regardless, we will have a problem since the odds are that 
whatever character we use, it will appear in text somewhere in the data. 

Obviously this is not a problem when we import directly into a database. 
However, as I said to Shanta, I don't have a problem with importing the files 
directly into a database for the first release. However, eventually I want 
the system to be independant of the data source (CSV, database)  and 
independant of the presentation (eXtropia, other portal).  Therefore, we need 
to consider a new seperator.

Suggestions?

> Inside the tyk xml question tags, there is a topicRef tag, which is the 
> reference of the PAGE_ID.  So, do you want to put the Page_id in the 
> topicRef tags or the actual topic name in the page.data ?

The TopicRef is a references to a topic, such as Administration, Networking, 
Security, etc. This is just text. There are PageRef tags and these contains 
the page *name* from page.data. However, I am nop longer sure we should do it 
that way (see below). This needs to be changed in tykToXML.pl. 

Keep in mind that the questions will exist only within a KU. This KU will have 
a primary page so we automatically have the primary page for the question. 
However, the question could reference multiple topics. 

We have a problem with some of the questions where an angle bracket is one of 
the answer ("What symbol is used to "pipe" two commands together?") This 
means we have two angle brackets together (<< or >>) which could confuse the 
XML parser. There are only a few and we can change them by hand. We just need 
to be aware of them. 

Also watch the format of the answers, even for the T/F questions:

        <Correct>
                <Text>T</Text>
                <Reason>why this answer is correct</Reason>
        </Correct>

not just 

<Correct>T</Correct>

I think that as much as possible it is better to have the same format for all 
types of questions.  More than likely, "fill in the blank" type questions 
won't have an <Incorrect> answer, but I still want to have a <Reason> tag to 
provide an explanation why the answer is correct. 

However, since we do not yet have the reasons, I think you should simply leave  
as <Reason></Reason>. I think if we leave the text as "put your reason why is 
correct/incorrect. " we might forget to change it and then displaying that 
text would look silly. If the <Reason> tag is empty, we can just ignore it. 
OR we could simply not include the <Reason> tag at this point. What do you 
think?

With the Glossary KUs please create a <GlossaryTerms> container with the 
GlossaryRefs to the other terms. These are the numbers at the end of each 
line in glossary.data. They are the ID numbers of the other glossary terms. 
Therefore, instead of reading in each line from glossary.data and processing 
it, you will need to read it all at once and put it into an array, then parse 
that array.

EVERYONE PLEASE READ AND COMMENT:
Currently the glossary.idx file contains a list of pages that contain each 
glossary item. This is created by an external script and is **not** done when 
the glossary item is loaded. That would take way too much time. The question 
is whether we should have PageRefs within the Glossary KU.

Personally, I do not think so. We can create the index of  glossary-page_id 
along  with everything else. If we include the page ID/page name within the 
Glossary KU and add a new glossary item, then we would need to go looking for 
all of the pages that have that glossary term. Obviously we need to search 
for the pages to add the <Glossary> tag within the page. However, I just see 
it as unnecessary work to add PageRefs withing the Glossary KU since we can 
create the index by other, more efficient means.


EVERYONE PLEASE READ AND COMMENT:
It just hit me that we might be building a trap for ourselves. If we use the 
full path instead of the ID number, we will have problems if we ever rename 
the file, move it to a different directory, etc. I **expect** to be moving 
files to different directories real soon! I want to change the order of the 
files and their locations. As we get more content, I can imagine that we 
change locations again. 

I see three options:

Use the full-path as the PageRef:
- Easy to find/insert the reference we want
- Tracking down the actual page from a KU is easy
- In the display code we don't need to do a look up to display the page.
- PROBLEM: moving/renaming the file. Since the XML files are text, we can use 
sed/perl to make a global change.

Use just the page name without the path:
- Once the file name is defined, it is less likely that the page name will be 
changed. 
PROBLEM: We must have *completely unique* page names. We cannot have a "Known 
Problems" in the Network section AND in the Printing section. They must be 
named "Known Problems-Network"  and "Known Problems-Printing" (or something 
like that). 

Use the ID as the PageRef:
- Remains constant, independant of the actual name of the file.
- PROBLEM: Need to do a look-up to find the correct file. However, since 
page.data is current sorted by chapter/section, I have found that  it is not 
al that hard. For the existing moreinfo and DYK entries one PageRef can be 
inserted automatically. Still, if we want to include more PageRefs, we will 
hve to do it by hand and look up the ID, but we will have to look it up any 
way to get the full path. So whether we lookup and insert the page name or 
the page ID it's the same amount of work.

I still like the idea of using the full-path and NOT and ID number. You need 
to do a look-up anyway to find the ID or the correct text for the full path. 
Tracking down the original page from the XML file is straight forward. Making 
a change would be a simple matter of running a sed/perl script. We could even 
write it in advance and it becomes a part of our "utility" package:

rename_page.pl [-f filename] original_name new_name

It then scans all PageRefs in the named file and changes them accordingly. 

Using an ID number bothers me because makes the construct dependant on an 
external file or we are imposing a structure on it unnecessarily. Therefore, 
the knowledge base is not self-contained. 

EVERYONE PLEASE READ AND COMMENT:
We have a similar problem with the MoreInfoRefs for the Page KUs. Currently 
they are referenced by their ID number and Luu did the same thing in her 
code. However, once again, I am not happy with idea of using ID numbers 
instead of text. So, do we reference the text of the MoreInfo KUs??

I have pretty much decided to go through the existing data files and add up to 
three topics. I will add these to the **end** of each line for all of the 
data files. So, Luu, could you change the code to create a <Topics> container 
and <TopicRefs> for all of the data files? Note that I will probably not list 
three topics for everything. Therefore, the code will need to be smart enough 
to recognized this. Since you are probably asleep already I can work on it 
today and send you at least one file with the topics, so you will see the 
format.

Regards,

jimmo
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Re: questions (CSV to XML conversion)

[James M. <lin...@ji...>]

Hi Luan!

(for the rest of you, please take a look at this as well, I would like some 
feedback).

Wow! That was quick. 

First, note that I replied to the linkbat-devel mailing list. I think all of 
these discussions should be on the list.   The rest of the comments are 
below.

On Monday 18 November 2002 21:20, Luan Luu wrote:
> 1.  Do you want the code automatic create the output file, or we could
> manually in the command line to concatenate to new xml file?

I think you should just write to standard output. This gives us greater 
flexability to specify any file name we choose. 

> 2.  Only one top level is allowed, so, i put another <KUs> tag wrap around
> the <knowledgeUnit> tags. is that ok?

In the specification, I had defined the top-level as being <KnowledgeUnits> 
(with the 's' at the end). In general, I had it so that the container were 
all plurals like <KnowledgeUnits>  and <MoreInfos>. 

> check the first one out, see if it is correct. let me know. thanks.

So far that looks great!!! However, a couple of things. The  <PageRef> should 
contain the file name out of the page.data file. These numbers will more or 
less disappear once the conversion is made. Also, please put in an line 
between the KnowledgeUnits.

What I was thinking about was going through the original data files and add 
topics. It would be fairly straight forward to define a handful of topics 
(administration, network, security, users, hardware, etc.) and add them to 
the CSV files. That would mean an extra field to parse, but the code is 
pretty much written. In fact, I could add multiple topics and if the field is 
empty, just don't print a topic tag. How does that sound?

I am thinking that we could save a fair bit of time that way. If at least one 
(or maybe even 2 or 3) topics are already present and can be added 
automatically, then we don't need to do it by hand. Granted, sub-topics will 
probably need to be added, later. But we have saved some work. In the CVS 
state right now, it is alot easier to add information in bulk. 

Comments, anyone? Are there other references that we could add in bulk now?

Regards,

Jim
=================
NOTE: I snipped most of the stuff. This is just an example to see the results 
and the original file. 

 <KnowledgeUnit>
   <Attributes>
      <Type>Concept</Type>
      <Text>Linux can be started from any partition.</Text>
   </Attributes>
   <Pages>
      <PageRef primary="true">106</PageRef>
   </Pages>
   <Questions>
      
   </Questions>
 </KnowledgeUnit>
 <KnowledgeUnit>
   <Attributes>
      <Type>Concept</Type>
      <Text>Linux can combine multiple drives into a single RAID system, even 
if the drives are of different types.</Text>
   </Attributes>
   <Pages>
      <PageRef primary="true">127</PageRef>
   </Pages>
   <Questions>
      
   </Questions>
 </KnowledgeUnit>
 <KnowledgeUnit>
   <Attributes>
      <Type>Concept</Type>
      <Text>Unwanted cron output can be redirected just like any other 
command.</Text>
   </Attributes>
   <Pages>
      <PageRef primary="true">21</PageRef>
   </Pages>
   <Questions>
      
   </Questions>
 </KnowledgeUnit>
 <KnowledgeUnit>
   <Attributes>
      <Type>Concept</Type>
      <Text>Cron is started through an rc-script like most system daemons. 
</Text>
   </Attributes>
   <Pages>
      <PageRef primary="true">67</PageRef>
   </Pages>
   <Questions>
      
   </Questions>
 </KnowledgeUnit>
 <KnowledgeUnit>
   <Attributes>
      <Type>Concept</Type>
      <Text>Cron can be disabled through the /etc/rc.config file.</Text>
   </Attributes>
   <Pages>
      <PageRef primary="true">67</PageRef>
   </Pages>
   <Questions>


1:106:Linux can be started from any partition.
2:127:Linux can combine multiple drives into a single RAID system, even if 
the drives are of different types.
3:21:Unwanted cron output can be redirected just like any other command.
4:67:Cron is started through an rc-script like most system daemons.
5:67:Cron can be disabled through the /etc/rc.config file.

#!perl
#!/usr/bin/perl
#convert the dyk.data into xml data.


open(DATA,"<dyk.data");

$TYPE = "Concept"; #hardcoded.
$TEXT = "put your text here";
$PAGE_ID = "your Page id from the file.";

        #required, only one top level is allowed.
        print "<KUs>\n";
while (<DATA>){
        chomp;
        $READLINE=$_;
        ($ID,$PAGE_ID,$TEXT) = split(/:/,$READLINE);
        #print the output...
        print " <KnowledgeUnit>\n";
        print "   <Attributes>\n";
        print "      <Type>$TYPE</Type>\n";
        print "      <Text>$TEXT</Text>\n";
        print "   </Attributes>\n";
        print "   <Pages>\n";
        print "      <PageRef primary=\"true\">$PAGE_ID</PageRef>\n";
        print "   </Pages>\n";
        print "   <Questions>\n";
        print "      \n"; #blank for now, insert the questions later.
        print "   </Questions>\n";
        print " </KnowledgeUnit>\n";
}
        print "</KUs>\n";
close DATA;

[Linkbat-devel] Linkbat data model, content, etc.

[James M. <lin...@ji...>]

Hi All, 

this is the second half of a conversation I have been having with Shanta 
McBain who is working on the presentation of the data (among other things). 
She reminded me that we need to have these discussions on the list and not in 
private. 

I have created a linkbat-writers mailing list to discuss actual content. Most 
of the people on that list will not be all too interested in the technical 
details of the actual implementation and I don't think you developers are all 
that interested in what goes into which article. However, I'll let you be on 
both lists if you want. 

As another reminder, please, please, please send me comments, questions, etc 
about the data model as soon as possible. 

On Monday 18 November 2002 17:41, Shanta McBain wrote:
> > Here are the content pages for the tutorial. With the data files and code
> > that I sent last week, you should have enough to get a working demo
> > running using eXtropia.  The actual data won't change much and the format
> > of any CSV file should be *basically* the same. Even if the order of the
> > fields is different or there are more fields, the basic principles are
> > the same.
>
> I will over the course of the next few days convert your data to MySQL.
> Hope to have the FAQ done tonight.

Keep in mind that what is currently in the CSV files is only a small fraction 
of what will be there when we get all of the KUs defined.

> > So, where do you want to go from here? Do you want to be responsible for
> > the data extraction and presentation? Just the presentation? I have one
> > person (Luan) already working on the conversion from the existing CSV to
> > XML and two people who were interested in the coding from the other
> > direction. However, they have not responded since their original email. I
> > am going to ping them again to see if they are still interested. We also
> > have a couple of new editors and content writers.
>
> XML is a subset of eXtropia views. I will have to show her how to put
> her code into the views used to display the content of the MySQL tables.
> Once the cvs is converted there is no longer any need for them.
> Except if you what others to install the system on their server which I
> don't see a need for.

There is a  misunderstanding here and I am not sure on which side. Luan is 
currently working on the code to generate the XML file from the existing CSV 
files. The other two are planned to write the code to covert the XML to 
either new, expanded CSV or insert them into a database. Once the XML files 
are generated their will a need for CSV files, in general.  We need to 
provide the option of running the system without an SQL database or some 
other database other than MySQL. Although we can initially work toward this 
first public version being on MySQL, I don't want code written exclusively to 
import to MySQL.  
 
> > If you are doing the presentation component, could you also come up with
> > a decent CSS concept? What I did was a kludge and doesn't look good on
> > all systems, particularly MS Internet Exploder 5.x. With 6x it's okay,
> > but 5 gives me problems. We could tell people to upgrade, but I think it
> > is just as much a problem with the styles as it is with IE5.
>
> You will not have too much luck accommodating IE except to give them a
> complete view done just for them. This is one reason that I have avoided
> fancy dancy. My suggestion here is to code to HTML 4 get Bobby
> certification and encourage users to upgrade to a gecko based browser
> such as Netscape or the leading edge Mozilla by providing links to them.
> Possibly a page giving reason why they should abandon IE for a net
> standard browsers.

If we can get it looking decent on Mozilla-based, Konquerer, Opera and any 
thing else that runs on Linux, plus fairly decent on IE 5/6 then I am happy. 
We have people using IE visiting the site. As long as no functionality is 
missing and something looks goofy, then I am happy. We cannot afford to 
alienate people by saying the site only works on non-MS browsers. That's 
shooting ourselves in the foot.

> > I just what to re-emphasize that I don't want to turn this into an
> > eXtropia project. That is, a project to get linkbat work explicitely with
> > eXtropia. It should be completely independent of the front end we choose.
> > I understand your enthusiasm with the product, but as much as possible
> > the presentation layer of linkbat needs to be portable to any system.
>
> eXtropia is portable to any system that has Perl. The data can be in any
> file or SQL. So if your code can access the files or SQL then you can
> display the contents. eXtropia takes care of authentication, Sessions,
> encryption, DataAccess (file, MySQL, PosGress, Oracle etc), translation,
> Logging, to name a few. This is part of the base eXtropia distribution.

Not eXtropia has to be portable, but linkbat. eXtropia handling the data 
access concerns me in that we will be dependant on eXtropia. 
If eXtropia access an "api" (one we define or is provided by someone else) 
that goes out and gets the data from MySQL, CSV, or whatever, then there is 
less of an issue. However, eventually (not necessarily in the first 
iteration), I want the Data Access to be presentation independant or at least 
have the ability to make it so. Obviously if it is MySQL and someone know how 
the tables are defined they can access the data. However, I do not see that 
as the same thing as us providing the "api". 

> > It will eventually need to be made available for download from
> > SourceForge, so it needs to be in a form that people can install without
> > hounding us to death. On the other hand, is there any **requirement**
> > that says we need to make things  available throguh source forge.  Hmmmm.
> > Still, eventually I would like at least the code and data files format
> > available for download. I want to adher to the open source principles
> > while at the same time not giving aware the farm (just yet).
>
> eXtropia is already available for down load and is vary easy to install.

I was thinking more along the lines of our stuff getting available for 
download. What "our stuff" is and when it would be ready for download is 
still unclear. 

> They have their own support system so one can just pass eXtropia level
> questions to them or my HelpDesk system. The part that will be:
>
> HTMLViews spacific to us

How configurable are these? It is possible for users to change the color 
scheme, for example. 

What about the layout from and administration pespective? I want certain data 
presented in certain places. Take the Linux Tutorial as an example. It's a 
3x3 table with each cell performing a particular function (i.e. displaying 
certain data). 

> The Data Files. I can export the current table contents into a cvs file
> or SQL file for import into SQL.
>
> Any ActionHandlers that are unique to our project so far I don't see a
> need for them.

Please clarify for us (i.e. me ) what an ActionHandlers really does. If I 
click on a "More Info" link to see the more info related to the current page, 
wouldn't it be an ActionHandler that generates the new page? However, since 
we are dealing with "standard" queries a "standard" action handler could runn 
the query, something like this:

select * from moreinfo_table where page_id = '1234'

Is that what we are talking about?

Let's say I want a list of topics and clicking on one gives me a list of all 
KUs that reference this topic. (See Topic Tree on the linkbat concept page). 
How would that be realized? Would that also be a simply query?

> The setup files unique to the project.

What do they look like? How are they configured?

> I only code to open source though I do accept payment for modification
> of code but the code anyone can have and use.

Open source is one reason for doing it through SourceForge. Code is not the 
issue, but rather the data (i.e. the KUs). One reason I am concerned with 
releasing the data is that I see us opening up a whole can of worms to 
support raw data. If we have a complete "product" which includes instructions 
on how to configure eXtopia, perl code, XML files, etc I don't have a 
problem. I just don't want to deal with pieces.

> > The more I think about it, the more I like th
> > e idea of extra content at the
> > various levels of "membership". In addition to the online output of
> > commands,
>
> Easy to do in eXtropia.

Out of curiosity, can you define different levels? 

> > I have been planning to include "pseudo-animated" graphics on the site.
> > Using an IFRAME that refreshes itself at regular intervals, you can get
> > some nice looking graphics. Since there is nothing on the site that needs
> > 3D life-like animations, this works well. The images are pretty small
> > (typically < 50K) so I don't see bandwidth problem. As an alternative an
> > co-worker of mine showed me a trick to split up the graphics into small
> > chunks. Large white spaces are s single byte that is blown up to fill the
> > whole area. Probably not necessary, but something to consider. At any
> > rate, these animations would probably be something for those people who
> > have registered, whereas online command output would be for "paying"
> > members as they would need to download it to their local systems. Members
> > would still have the current updates and we could potentially have things
> > like the quizes only available for registered members since I plan to
> > keep track of the user's score, completed sections and so on. Hmmm. There
> > are many possibilities.
>
> This is part of the views.

And views can be user based, right? So a registed user could get the 
animation, where an anonymous person does not? Is that right?

> > Have you had a chance to go through the concept and data description? I
> > would really like some input as quickly as possible to finalize the
> > format. Luan will need to know what the final format is before she can
> > convert the CSV files. She is already working on the code (I think), but
> > is using the "suggestions" I provided.  Her code shouldn't change too
> > much, however I would like to get moving.
>
> No I have been a bit busy and took a few hours off for myself this
> weekend. I will get on them tonight. Get Luan in touch with me and move
> this conversation to the list. If you don't have one I can create a list
> for us on my system. I have access to my lists via Perl as well. Users
> can use a web form to sign up.

Please, please, please read the concept and data model. Also, if I don't hear 
back from the others who wanted to do the XML code, then I will work together 
with Luan on it. Assuming she wants to keep going. ;-)

> I will be back in touch. Check the site tomorrow night I should have the
> FAQ converted to MySQL by them.

Will do.

Regards,

Jim
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Linkbat Data Model

[James M. <lin...@ji...>]

Hi All!

Just a friendly remember that I would like (read: need) comments about the 
Linkbat data model. In reviewing the descriptions I have come up with several 
questions that need to be discussed and answered:

- Does this data model make sense?

- Are the other KU types that w should include?

- Are there any relationships that its missing? 

- Do we need to have an ID within the <KnowledgeUnit> tag?
Originally, it was intended to make sure each KU was unique and to use as a 
reference point. However, a unique ID can be generated when the XML files are 
parsed during the conversion to CSV or database.

- Do we need to include a definition for the Page KU within the same files as 
the other KUs?
Eventually we will be converting the HTML pages to XML and all of the 
necessary tags can be included in that file.  At this point, I think it would 
be a lot of work to convert the HTML files to XML and parse them. Since we 
can automate the creation of Page KUs, I think we should go ahead and have 
the Page KU definition seperate from the actual content for the time being. 
That is, we should include the definition with the other KUs **for now**.

-  How do we define KU sub-types?
For example, the "man-page" sub-type to the MoreInfo KU. 
Should we have a <Type> and <SubType> tag or is the sub-type an attribute of 
the Type tag: <Type sub-type="">? We also need to account for the specific 
location and other info. For other sites we need a location; for books, we 
need the ISBN, author and perhaps publisher? Should these be attributes of 
the <Type> tag or seperate tags?

- Should be include a <Language> tag?
Eventually we will have everything translated into other languages. We could 
create a special tag to identify the language the would allow us to have a 
single knowledge base in multiple language. Or we simply keep the languages 
seperate.

- Should be include a <SkillLevel> tag?
The content is getting to the point where is a lot of it might be overwhelming 
to newbies. Therefore, it would be nice to be able to configure the system to 
only display KUs that are appropriate for their skill level. This could be 
accomplished with a <SkillLevel> tag. Questions could also be marked with 
this tag and only questions appropriate for that skill level would be asked. 

- Do command KU need a <SeeAlso> tag?
Like the man-pages the point to more information, but that's the point of 
simply referencing another KU. If it is important enough to mention as a "See 
Also", then it should have a KU, right?

- When we create the Glossary KUs, should we include the page references?
Currently, I have a script that searches through the content pages and creates 
a list of all pages that contain each glossary term. This is more efficient 
than including the PageRef within the Glossary KU. I see no need and it it is 
one more place to forget something. For example, if you add a new content 
page we could forget to update the appropriate glossary KUs. Scanning the 
content pages seams more efficient.

MAYBE it would be necessary to create a "primary" page for the glossary 
entries, but I cannot convince myself that it is necessary.

Well, I am sure there are more questions to answer, but these are the ones 
that seem to be most pressing.

Regards,

Jim
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] RE: Linkbat Project

[Hal F. G. <hal...@ur...>]

-----Original Message-----
From: James Mohr [mailto:lin...@ji...] 
Sent: Wednesday, November 13, 2002 3:41 PM
To: lin...@li...
Cc: Suriya Narayanan M S; hal gottfried; John Purdy; Rama; A .; Russell
Cotten
Subject: Linkbat Project


I've got a couple of administrative issues to cover. I have not yet
added 
everyone to the developers list because not everyone has confirmed that
they 
want me to do it.  Check out the project page on SourceForge to see if
you 
are listed as a developer (http://sourceforge.net/projects/linkbat/). If
you 
are not listed and do want to be, let me know. I will then put you on
the 
developers mailing list, as well.  


		HG> You have not listed me but I would like to be listed
there
		    as well.


Most people do not have a public skill profile on SourceForge. That's
okay, 
but I would like to know what exprience and skills people have, plus
exactly 
what areas they want to work on. There is plenty of work in multiple
areas, 
so don't be shy! Check out the task list on SourceForge. Most are
related to 
content and not programming.

		HG> I will go fill that out for you today.


Also please let me know what style of management you want from me. Do
you 
expect me to hand out assignments to you or will you simply grab one? Do
you 
need regular reminders? I realize that we are all doing this when other 
projects permit and sometimes we loose track of time. I will fit myself
to 
your style. Since this is a team project, I really don't mind if you
tell me 
that you think something is wrong. I believe that we are headed in the
right 
direction, but please tell me before we drive over a cliff. 


		HG> I am working on quite a few projects so you can tell
me if you
		    would like what part you'd like me to work on, or I
can just grab
		    something.



You can send this kind of "personal" info to me directly, instead of the
list. 
However, technical/project related info, I would like to go to the list.
That 
way if someone else can/should comment on any topic, they can. 

Please give me an idea of you availability. I don't need the specific
number 
of hours you can work per week, but I would really like to know what
other 
projects you have going. Are you in school, working full time, whatever?

If can no longer contribute for whatever reason please let me know, as
soon as 
possible. If you have to take a break for whatever reason, please let me

know, as well. I know this is an all volunteer project and I am very
grateful 
for the help. However, communication is the only way this can be a
success, 
so please communicate. 


		HG> I am a corperate IT Trainer and I am currently on
the road
		    about 20 - 25 days a month but I can provide at
least 4-5 
		    hours per week, for this as I see it now. 



Also please let me know where you live. I am in Germany, so I am several
hours 
ahead of the people in North America. Aside from simply being curious,
it 
helps me to know when you will be reading email, etc. If I send a
message 
early in the morning (before I go to work) , some of you might still be 
awake. If I send it later in the morning, I will expect to wait a while 
before you even read it.


		HG> I live in Pittsburgh USA, I used to live in Germany
though.


Up to this point, I have been encouraging visitors to support the
tutorial by 
making a donation to the Jessie Arbogast Medical Fund. Since we are a
team, I 
think we should all have an equal say in how to distribute any
donations.  At 
this point, I do **not** what to soliticate donations to pay us.  Note
the 
phrase "at this point". I don't want to count our chickens before they
hatch. 
However, I think we can have a "care-ware" site, where we encourage
donations 
to needy causes. Each person can make a suggestion and we can create a 
"donations page" or whatever to point people to the right place. Also,
if the 
organization your want  to support has a banner ad that we can use, it
is 
really easy to add it to the system.  There are a number already up. 

Now for the technical stuff. As soon as possible, please, please read
both the 
Concept and Detailed KU Description on the linkbat site 
(http://linkbat.sourceforge.net). I want to finalize this as soon as 
possible. One nice thing about storing in XML is that we can more easily

change the structure, but I would like to have as many questions
answered and 
decisions made as possible before we continue. I would like to start the

hardcore work no later than the beginning of next week.  

One person had expressed interest in helping, but could not shake the
idea of 
linkbat as being a huge Linux glossary. It ain't! Although I am not
trying 
for the ultimate knowledge base with artificial intelligence that can
answer 
every question you input, I feel this data model can simplify access the

knowledge necessary to get the answer (if not providing the answer
directly).  
Also keep in mind that a knowledge base is useless without content and
we can 
always use content. So, if there are content related tasks that you
could do 
and need a break from coding or whatever, **please** grab one. Also, the

tutorial can always use some editorial review (spellchecking, etc). Even
if 
it is a single correction, it make the tutorial a little better. 

Once we have the data structure we can start moving the data from the
current 
CSV into XML. While that is happening we can also work on the code to do
the 
conversion back into CSV as well as into an database (i.e. SQL).  The
reason 
for the conversion back into CSV (as mentioned in the linkbat concept)
is 
speed when online. XML is ten times larger and takes that much larger to

read. Plus the indexes need to be created and there is no way we can do
them 
in real-time. (Or does anyone see that differently?)

Three people have said they could help doing the conversion. That might
be 
overkill, but doing a quick conversation means we have the XML files for
the 
coding. That means moving sooner to other tasks. 

Once I get people's comments I will "finalize" the model and create a
DTD. I 
don't think it is necessary to wait to convert the data to XML before
the DTD 
is finished. The DTD is just a stylized description of things we already

decided upon. However for someone providing us content in the future, it

would be useful.

I guess that's it for now. I am sure I will have more to say real soon. 

Best Regards,

Jim
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what
others think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info

[Linkbat-devel] Linkbat Project

[James M. <lin...@ji...>]

Hi All!

First, I want to welcome you all to the team. I really appreaciate your 
willingness to help. I am excited about this project and I am sure that it 
will be a success. Considering how successul the Linux Tutorial has been 
already, it can only get better. 

I've got a couple of administrative issues to cover. I have not yet added 
everyone to the developers list because not everyone has confirmed that they 
want me to do it.  Check out the project page on SourceForge to see if you 
are listed as a developer (http://sourceforge.net/projects/linkbat/). If you 
are not listed and do want to be, let me know. I will then put you on the 
developers mailing list, as well.  

Most people do not have a public skill profile on SourceForge. That's okay, 
but I would like to know what exprience and skills people have, plus exactly 
what areas they want to work on. There is plenty of work in multiple areas, 
so don't be shy! Check out the task list on SourceForge. Most are related to 
content and not programming.

Also please let me know what style of management you want from me. Do you 
expect me to hand out assignments to you or will you simply grab one? Do you 
need regular reminders? I realize that we are all doing this when other 
projects permit and sometimes we loose track of time. I will fit myself to 
your style. Since this is a team project, I really don't mind if you tell me 
that you think something is wrong. I believe that we are headed in the right 
direction, but please tell me before we drive over a cliff. 

You can send this kind of "personal" info to me directly, instead of the list. 
However, technical/project related info, I would like to go to the list. That 
way if someone else can/should comment on any topic, they can. 

Please give me an idea of you availability. I don't need the specific number 
of hours you can work per week, but I would really like to know what other 
projects you have going. Are you in school, working full time, whatever? 
If can no longer contribute for whatever reason please let me know, as soon as 
possible. If you have to take a break for whatever reason, please let me 
know, as well. I know this is an all volunteer project and I am very grateful 
for the help. However, communication is the only way this can be a success, 
so please communicate. 

Also please let me know where you live. I am in Germany, so I am several hours 
ahead of the people in North America. Aside from simply being curious, it 
helps me to know when you will be reading email, etc. If I send a message 
early in the morning (before I go to work) , some of you might still be 
awake. If I send it later in the morning, I will expect to wait a while 
before you even read it.

Up to this point, I have been encouraging visitors to support the tutorial by 
making a donation to the Jessie Arbogast Medical Fund. Since we are a team, I 
think we should all have an equal say in how to distribute any donations.  At 
this point, I do **not** what to soliticate donations to pay us.  Note the 
phrase "at this point". I don't want to count our chickens before they hatch. 
However, I think we can have a "care-ware" site, where we encourage donations 
to needy causes. Each person can make a suggestion and we can create a 
"donations page" or whatever to point people to the right place. Also, if the 
organization your want  to support has a banner ad that we can use, it is 
really easy to add it to the system.  There are a number already up. 

Now for the technical stuff. As soon as possible, please, please read both the 
Concept and Detailed KU Description on the linkbat site 
(http://linkbat.sourceforge.net). I want to finalize this as soon as 
possible. One nice thing about storing in XML is that we can more easily 
change the structure, but I would like to have as many questions answered and 
decisions made as possible before we continue. I would like to start the 
hardcore work no later than the beginning of next week.  

One person had expressed interest in helping, but could not shake the idea of 
linkbat as being a huge Linux glossary. It ain't! Although I am not trying 
for the ultimate knowledge base with artificial intelligence that can answer 
every question you input, I feel this data model can simplify access the 
knowledge necessary to get the answer (if not providing the answer directly).  
Also keep in mind that a knowledge base is useless without content and we can 
always use content. So, if there are content related tasks that you could do 
and need a break from coding or whatever, **please** grab one. Also, the 
tutorial can always use some editorial review (spellchecking, etc). Even if 
it is a single correction, it make the tutorial a little better. 

Once we have the data structure we can start moving the data from the current 
CSV into XML. While that is happening we can also work on the code to do the 
conversion back into CSV as well as into an database (i.e. SQL).  The reason 
for the conversion back into CSV (as mentioned in the linkbat concept) is 
speed when online. XML is ten times larger and takes that much larger to 
read. Plus the indexes need to be created and there is no way we can do them 
in real-time. (Or does anyone see that differently?)

Three people have said they could help doing the conversion. That might be 
overkill, but doing a quick conversation means we have the XML files for the 
coding. That means moving sooner to other tasks. 

Once I get people's comments I will "finalize" the model and create a DTD. I 
don't think it is necessary to wait to convert the data to XML before the DTD 
is finished. The DTD is just a stylized description of things we already 
decided upon. However for someone providing us content in the future, it 
would be useful.

I guess that's it for now. I am sure I will have more to say real soon. 

Best Regards,

Jim
-- 
---------------------------------------
"Be more concerned with your character than with your reputation. Your
character is what you really are while your reputation is merely what others
think you are." -- John Wooden
---------------------------------------
Be sure to visit the Linux Tutorial:  http://www.linux-tutorial.info