mdbtools-dev

Re: [mdb-dev] Problems with database.

[David M. <mdb...@dm...>]

On 24 Apr 2003, Marc Deichmann wrote:

> Hi!
> 
> I'm using the mdb-export tool to export tables out of an database into
> csv files. So far mdb-export do what it should do.
> 
> My problem is, I found out that in a few csv files data is missing. The
> header row is there, but there is no data below it.  I can read the 
> data in Access.
> The thing that disturbs me is, that only the data of a few (but
> important ;( ) columns is missing. The rest of the csv file is complete.
> 
> My workaround at the moment is, to export the problematic tables with
> Access to csv files and then use them together with the other csv files.
> 
> Could someone help me?

I currently have a large-ish combined patch against mdbtools-0.5rc2.tar.gz 
which fixes a number of bugs.

It's attached.

To apply it, save the attachment to a file, untar the original source,
change directory into the mdbtools-0.5rc2 directory and type:

patch -p1 < mdbtools-combined.patch

Then build and install as usual.

I actually have 18 separate patches against mdbtools, but the attachment 
is my work-in-progress combined patch.

If you do try this, let me know if it does/doesn't fix your problems.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] Problems with database.

[Marc D. <mar...@gm...>]

Hi!

I'm using the mdb-export tool to export tables out of an database into
csv files. So far mdb-export do what it should do.

My problem is, I found out that in a few csv files data is missing. The
header row is there, but there is no data below it.  I can read the=20
data in Access.
The thing that disturbs me is, that only the data of a few (but
important ;( ) columns is missing. The rest of the csv file is complete.

My workaround at the moment is, to export the problematic tables with
Access to csv files and then use them together with the other csv files.

Could someone help me?

Bye,
Marc...

--=20
 -> mailto:mar...@gm... <-> http://www.creative-sadness.de <-

[mdb-dev] Empty mdb ?

[Sven H. <sve...@we...>]

Hi,

I have to extract some data from mdb files. However, "mdb-export" generates 
the correct table colum headings, followed by 38 empty table lines. A hexdump 
of the file shows that there seems to be some data.
Is this some "feature" of the mdb file (saved, but already deleted data) or a 
bug/non-implementation of the mdbtools ?
I am using mdbtools 0.4-1. The mdb file is JET version 3.

Kind Regards,
Sven Heithecker

-- 
Sven Heithecker     sve...@we...
Pestalozzistr 6     Fax 0531-3489815
38114 Braunschweig  Tel 0531-336580

Re: [mdb-dev] [PATCH] optimize the scan for next data page

[David M. <mdb...@dm...>]

On Sat, 12 Apr 2003, Brian Bruns wrote:

> David
> 
> i'm not intentionally ignoring your patches. i'm just real busy these 
> days.  The patches look good.  Would you like CVS perms to check them in 
> yourself?

I could certainly check them in if you want, but I'm not really going to
be spending any more time than necessary on mdbtools (I think :-).  I just
needed to get the export working to suck the data into an oracle database,
and that is working great now.  If anything comes up in the process, I'll
try to fix it of course.

So either way is fine.

By the way, I was looking at a project called 'sqlite' which seems to be a 
complete sql engine built on a standalone file interface.  Without really 
having looked at the code, it may be possible to wedge the mdbtools 
low-level data interface below that sql parser/planner interface.  It 
already supports almost the entire ansi-92 sql standard.  That would 
remove the need to create both the data layer and the sql language layer.

Just an thought.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] Re: CSV export does not correctly output " characters

[Jochen R. <Joc...@db...>]

Hi Frederik,

I requested a similar feature a while ago - a little more general, with the 
possibility to give a certain quote string, like "" or \" or something else.

Jochen
-- 
Jochen Riehm
DBRent GmbH
Joc...@db...

[mdb-dev] CSV export does not correctly output " characters

[Frederik H. <fh...@pa...>]

Hi,

I have used gmdb2 to export an Access table to a CSV file, but now in
processing this file with Perl, I noticed a small problem. When the
"-character occurs in a field, this should be exported as "" in the CSV
file, while it currently is exported as ".

Little example:
When the contents of the field read
This is a "test"
this should be exported as
"This is a ""test"""

Frederik Himpe

Re: [mdb-dev] [PATCH] optimize the scan for next data page

[Brian B. <ca...@ai...>]

David

i'm not intentionally ignoring your patches. i'm just real busy these 
days.  The patches look good.  Would you like CVS perms to check them in 
yourself?

Brian

On Fri, 11 Apr 2003, David Mansfield wrote:

> 
> The scan for the next data page for a table starts again from the 
> beginning each time.  For really large tables this causes a noticable slow 
> down as we get further and further into the data.
> 
> This patch uses the table->cur_phys_pg variable to initialize the starting 
> point of the scan to exactly the place we left off last time.  This saves 
> about 30% cpu time for some exports here.
> 
> David
> 
> 

[mdb-dev] mde data base files...

[Scott C. <sca...@ga...>]

Any tips or tricks for mde database files?  Exported data to .mdb file but
can't get forms, suggestions?..

Tnxs,

[mdb-dev] [PATCH] optimize the scan for next data page

[David M. <mdb...@dm...>]

The scan for the next data page for a table starts again from the 
beginning each time.  For really large tables this causes a noticable slow 
down as we get further and further into the data.

This patch uses the table->cur_phys_pg variable to initialize the starting 
point of the scan to exactly the place we left off last time.  This saves 
about 30% cpu time for some exports here.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] preserve the row_start variable in Jet4 databases

[David M. <mdb...@dm...>]

In Jet3 there is something I don't understand called 'jumps'.  In Jet4, 
it's apparently not necessary.  However, some of the code for it was still 
being executed unconditionally.  This caused the row_start value to be 
mangled for larger row sizes, which in turn caused all of the variable 
length fields to get random data.

This patch conditionalizes one part of the code, and preserves the
row_start variable.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] fix handling of sparse usage_map

[David M. <mdb...@dm...>]

On really large databases, some of the usage_map entries can be empty.  
This fixes the code to add the appropriate value to the pgnum before 
continuing to look for a non-empty value.

Without this, the page numbers used were getting out of wack for the first 
non-empty 0x05 page.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] col variable unitialized when -H specified

[David M. <mdb...@dm...>]

This fixes a bug which caused the wrong column spec, or an uninitialized
column spec, to be used in the export program.

The col variable needs to be initialized for the first column.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] add option to remove invalid characters from schema export

[David M. <mdb...@dm...>]

Many databases don't want funny characters (space, quote etc.) in the 
table and column names.  I added a -S option to mdb-schema to turn on 
'sanitization' of the names.  A sanitized name contains only alpha/numeric 
and '_'.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] fix bug in row handling for deleted rows

[David M. <mdb...@dm...>]

The bug that the 05-.. patch was trying to fix is fixed here.  When the 
0th row in a data page is deleted, it's offset is given as 0x1000 (meaning 
that it's empty).  However, the mask that was being used to remove the 
delflag and lookupflag (0x0FFF) was turning this into 0.  This caused the 
length calculation for the next row to get screwed up.

Fixing the mask to 0x1FFF seems to fix the problem. 

Note: I didn't change the ole or memo code, because I couldn't test it.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] not all databases support length for datatype in schema

[David M. <mdb...@dm...>]

In oracle, the Date argument doesn't accept a length argument.  
mdb-schema, however, creates the following in the create statement:

...
my_date_col         DATE(8)
...

And oracle barfs on this.  I added a feature to the backend specification 
which controls whether the datatype takes a length argument.  Prefixing 
the datatype name with '-' turns off the length in the column-spec 
generation. 

It's a bit of a hack, I'll admit.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] add a 'namespace' feature to mdb-schema

[David M. <mdb...@dm...>]

I found this useful. YMMV.  I added the -N option to mdb-schema which 
allows the user to specify a prefix which will be prepended to all of the 
table names.  This enables the import of a schema into another existing 
schema without causing any naming conflicts with existing tables in the 
destination schema.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] fix bounds checking for double type

[David M. <mdb...@dm...>]

This simple patch fixes the bounds checking when extracting a double from 
a data page.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] Re: [PATCH] enable code to correctly find end-of-row

[David M. <mdb...@dm...>]

>       The cause was the mdb_find_end_of_row code was not using the 
> intelligent algorithm (it was ifdef'ed out) and was using a bogus value 
> when there were 'lookupflag'ed rows in the table.

> This patch enables the already existing code, and enables it to compile 

Ok.  I mangled this patch before sending it out.  Which is just as well 
because it was bogus.  Please ignore the 05- patch (and 04- never 
existed!).

I now have a complete schema and export of every table working from my 
Jet4 MDB file.  I will send the rest of the necessary patches under 
separate cover.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] How can I get some code examples in C

[<jl...@it...>]

How can I get some code samples in C or a programming guide with the 
mdb-tools functions ?

[mdb-dev] [PATCH] enable code to correctly find end-of-row

[David M. <mdb...@dm...>]

Hi Brian,

Here's one more.  

I ran valgrind on mdbtools and it found accesses outside of the malloced 
region.  The cause was the mdb_find_end_of_row code was not using the 
intelligent algorithm (it was ifdef'ed out) and was using a bogus value 
when there were 'lookupflag'ed rows in the table.

This patch enables the already existing code, and enables it to compile 
;-)

Apply with -p1.

David

P.S. All of the patches are against 0.5rc2
P.P.S Valgrind has found some memory leaks, which doesn't matter much for 
me, but it could matter for the gui-based code which is expected to run 
for an arbitrary amount of time (unlike the mdb-schema/export which is a 
one-pass and exit type of thing).

-- 
/==============================\
| David Mansfield              |
| mdb...@db...       |
\==============================/

[mdb-dev] [PATCH] handle column names spanning page boundary

[David M. <mdb...@dm...>]

Hi again,

This last patch of the set fixes extraction of the column names when the 
name crosses a page boundary.  A few errors in the Jet4 section caused 
column name parsing to go bonkers as soon as the boundary was crossed.

Additionally, the final null-termination:

pcol->name[name_sz]='\0';

which should have been:

pcol->name[name_sz/2]='\0';

was overshooting the name buffer (since the name_sz is in unicode bytes) 
and destroying a random field in the column buffer.

This was causing the Unknown 0x00 type problem which I reported, and was 
reported by someone else one the list.

This fixes all of the problems I've had so far.

Apply with -p1.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@dm...       |
\==============================/

[mdb-dev] [PATCH] be very careful about accessing beyond page

[David M. <mdb...@dm...>]

Hi again,

In my attempts to track down an elusive bug, I thought maybe the accesses 
in mdb_read_columns may be at fault.

There is already a fair amount of checking there, but not every access was 
prefaced with a read_pg_if...  Also, the fields were being accessed out of 
order, so an access causing advance to the next page could screw up the 
access to a prior field.

The attached patch fixes up these calls, and changes some of the 
hard-coded offsets to use the fmt structure.

My only question is the prec & scale assignments.  Are these used?

Anyway, the patch does no harm here, and seems right.

Apply with -p1.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@dm...       |
\==============================/

[mdb-dev] Re: [PATCH] handle no-rows case in table (now with patch)

[David M. <mdb...@dm...>]

Oops.  Forgot to include the patch ;-)

-- 
/==============================\
| David Mansfield              |
| mdb...@dm...       |
\==============================/

[mdb-dev] [PATCH] handle no-rows case in table

[David M. <mdb...@dm...>]

Hi Brian,

It seems as though the table->num_rows variable contains junk.  It is 
always 1625 (or thereabouts) for every table in my Jet4 database.

In any case, the mdb_get_relations was seg-faulting because it was 
accessing a bogus page.  It was accessing a bogus page because the 
allocation map specifies no pages at all.  In this case, the initial 
mdb_read_next_dpg is returning 0, but the code ignores this and uses the 
data in mdb->pg_bug as if it were valid.

The attached patch fixes this, and cures the segfault (also reported on 
the list) in the mdb-schema relations section.

Apply with -p1.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@dm...       |
\==============================/

[mdb-dev] [PATCH] fix return of local variable

[David M. <mdb...@dm...>]

Hi,

The attached patch fixes a potential (probably harmless) bug when
encountering an unknown type.  I was encountering an unknown type before I
fixed a different bug (patch to follow).

Apply with -p1.

David

-- 
/==============================\
| David Mansfield              |
| mdb...@dm...       |
\==============================/