mysqlfs-general — General discussion list for MySQLfs users and developers.

Re: [Mysqlfs-general] question about mysqlfs

[Michal L. <mi...@lo...>]

Daniel Gonzalez Schiller wrote:

> but my OS is Debian unstable... and I cannot find the pthreads package...
> also i think debian uses other system than pthreads...
> 
> configure: error: Please install pthreads library first.
> 
> Am I right? i think not... but... how can i compile your fusemysqlfs on 
> debian?

Apparently there's a bug in the configure script that fails pthreads 
check in case you don't have mysql development package installed.
I.e. it looks like you don't have pthreads library while in fact it's 
missing mysql library that triggers it.

Do "aptitude install libfuse-dev libmysqlclient-dev" and then run the 
configure script again. It should now work.

Michal

Re: [Mysqlfs-general] why fsck should not be run every mount

[Michal L. <mi...@lo...>]

Tony Shadwick wrote:
 >> [ MySQL fs isn't meant to be a clustered filesystem. ]
> FYI, that was my primary hope when I first started trying to use 
> MySQL-fs. :)  Unfortunately on freebsd it won't ever do the inserts. :P 
>   I tried...
> 
> So how much work are we talking to put in sane file locking?

I have no idea how to do it in master-master DB replication setup. You 
may need a way to communicate directly between the daemons (tcp or 
something) to handle remote locking, because the DB nodes aren't 
synchronous.

In master-slave replication or with single DB without replication you 
could use a db table locking or transactions or something like that. In 
this case it probably isn't much work. Well, depending on developer's 
experience ;-)

BTW Please use bottom-posting on this list, thanks.

Michal

Re: [Mysqlfs-general] why fsck should not be run every mount

[Tony S. <tsh...@os...>]

FYI, that was my primary hope when I first started trying to use 
MySQL-fs. :)  Unfortunately on freebsd it won't ever do the inserts. :P 
  I tried...

So how much work are we talking to put in sane file locking?


Michal Ludvig wrote:
> Smart Weblications GmbH, Florian Wiessner wrote:
>> hi list,
>>
>>
>> i think we should make a binary fsck.mysqlfs without
>> having mysqlfs to do fsck everytime a fs is mounted.
>>
>> why this? because if you attend to use mysqlfs with mysql5 ndb you are
>> running into trouble.
>>
>> if some other servers are still using the fs via mysql-nbd, there
>> is a risk of destroying data when the fsck is run, when there are
>> still active nodes running using the fs.
>>
>> so we should build fsck.mysqlfs in a seperate binary.
>>
>>
>> what do you think?
> 
> I think that you shouldn't have mysqlfs mounted concurrently from 
> different clients at all. IIRC There is some locking that synchronizes 
> threads of the same process, but this locking doesn't span across 
> different processes / hosts.
> 
> MySQL fs isn't meant to be a clustered filesystem. Well, maybe later, 
> but not now.
> 
> Michal
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Mysqlfs-general mailing list
> Mys...@li...
> https://lists.sourceforge.net/lists/listinfo/mysqlfs-general

Re: [Mysqlfs-general] why fsck should not be run every mount

[Michal L. <mi...@lo...>]

Smart Weblications GmbH, Florian Wiessner wrote:
> hi list,
> 
> 
> i think we should make a binary fsck.mysqlfs without
> having mysqlfs to do fsck everytime a fs is mounted.
> 
> why this? because if you attend to use mysqlfs with mysql5 ndb you are
> running into trouble.
> 
> if some other servers are still using the fs via mysql-nbd, there
> is a risk of destroying data when the fsck is run, when there are
> still active nodes running using the fs.
> 
> so we should build fsck.mysqlfs in a seperate binary.
> 
> 
> what do you think?

I think that you shouldn't have mysqlfs mounted concurrently from 
different clients at all. IIRC There is some locking that synchronizes 
threads of the same process, but this locking doesn't span across 
different processes / hosts.

MySQL fs isn't meant to be a clustered filesystem. Well, maybe later, 
but not now.

Michal

[Mysqlfs-general] why fsck should not be run every mount

[Smart W. G. F. W. <f.w...@sm...>]

hi list,


i think we should make a binary fsck.mysqlfs without
having mysqlfs to do fsck everytime a fs is mounted.

why this? because if you attend to use mysqlfs with mysql5 ndb you are
running into trouble.

if some other servers are still using the fs via mysql-nbd, there
is a risk of destroying data when the fsck is run, when there are
still active nodes running using the fs.

so we should build fsck.mysqlfs in a seperate binary.


what do you think?
-- 

Mit freundlichen Grüßen,


Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 700 762 789 32 - 0,12 EUR/Min*
fax.: +49 700 762 789 32 - 0,12 EUR/Min*
24/7: +49 900 311 886 00 - 1,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz

Re: [Mysqlfs-general] Cannot mount Filesystem

[Smart W. G. F. W. <f.w...@sm...>]

Andrew Rose schrieb:
> It could be a problem with your fsck code?  I've used mysqlfs quite
> extensively and not come across this in the production code, but have
> in my own additions.
>

I don't think it is a problem with my code, i have patched like this:


diff -Naur query.c.orig query.c
--- query.c.orig        2007-08-29 04:53:00.000000000 +0200
+++ query.c     2007-08-29 16:28:43.000000000 +0200
@@ -920,5 +920,124 @@
  int query_fsck(MYSQL *mysql)
  {
      // See TODO file for what should be here...
+    printf("Starting fsck\n");
+
+    // 1. delete inodes with deleted==1
+    int ret;
+    int ret2;
+    int result;
+    char sql[SQL_MAX];
+    printf("Stage 1...\n");
+    snprintf(sql, SQL_MAX,
+             "DELETE from inodes WHERE inodes.deleted = 1");
+
+    log_printf(LOG_D_SQL, "sql=%s\n", sql);
+
+    ret = mysql_query(mysql, sql);
+    if(ret){
+        log_printf(LOG_ERROR, "Error: mysql_query()\n");
+        log_printf(LOG_ERROR, "mysql_error: %s\n", mysql_error(mysql));
+        return -EIO;
+    }
+    snprintf(sql, SQL_MAX,
+             "OPTIMIZE TABLE inodes;");
+
+    log_printf(LOG_D_SQL, "sql=%s\n", sql);
+
+    ret = mysql_query(mysql, sql);
+    if(ret){
+        log_printf(LOG_ERROR, "Error: mysql_query()\n");
+        log_printf(LOG_ERROR, "mysql_error: %s\n", mysql_error(mysql));
+        return -EIO;
+    }
+
+    // 2. - delete direntries without corresponding inode
+    printf("Stage 2...\n");
+    // not sure if delete from tree where not inode in (select inode 
from inodes) will work!
+    //
+    snprintf(sql, SQL_MAX, "delete from tree where not inode in (select 
inode from inodes);");
+
+    log_printf(LOG_D_SQL, "sql=%s\n", sql);
+
+    ret = mysql_query(mysql, sql);
+    if(ret){
+        log_printf(LOG_ERROR, "Error: mysql_query()\n");
+        log_printf(LOG_ERROR, "mysql_error: %s\n", mysql_error(mysql));
+        return -EIO;
+    }
+
+
+
+    // 3. set inuse=0 for all inodes
+    printf("Stage 3...\n");
+    snprintf(sql, SQL_MAX, "UPDATE inodes SET inuse=0;");
+
+    log_printf(LOG_D_SQL, "sql=%s\n", sql);
+
+    ret = mysql_query(mysql, sql);
+    if(ret){
+        log_printf(LOG_ERROR, "Error: mysql_query()\n");
+        log_printf(LOG_ERROR, "mysql_error: %s\n", mysql_error(mysql));
+        return -EIO;
+    }
+
+
+    // 4. delete data without existing inode
+    printf("Stage 4...\n");
+    snprintf(sql, SQL_MAX, "delete from data_blocks where inode not 
inodes.inode;");
+
+    log_printf(LOG_D_SQL, "sql=%s\n", sql);
+
+    ret = mysql_query(mysql, sql);
+    if(ret){
+        log_printf(LOG_ERROR, "Error: mysql_query()\n");
+        log_printf(LOG_ERROR, "mysql_error: %s\n", mysql_error(mysql));
+        return -EIO;
+    }
+
+
+    // 5. synchronize inodes.size=data.LENGTH(data)
+    printf("Stage 5...\n");
+    long inode;
+    long size;
+
+    snprintf(sql, SQL_MAX, "select inode, sum(OCTET_LENGTH(data)) as 
size from data_blocks group by inode");
+
+    log_printf(LOG_D_SQL, "sql=%s\n", sql);
+
+    ret = mysql_query(mysql, sql);
+
+    MYSQL_RES* myresult;
+    MYSQL_ROW row;
+
+    myresult = mysql_store_result(mysql);
+
+    while ((row = mysql_fetch_row(myresult)) != NULL) {
+      inode = *(row[0]);
+      size  = *(row[1]);
+      snprintf(sql, SQL_MAX, "update inodes set size=%ld where 
inode=%ld;", size, inode);
+      log_printf(LOG_D_SQL, "sql=%s\n", sql);
+      result = mysql_query(mysql, sql);
+
+/*      if (myresult) { // something has gone wrong.. delete datablocks...
+
+        snprintf(sql, SQL_MAX, "delete from inodes where inode=%ld;", 
inode);
+        log_printf(LOG_D_SQL, "sql=%s\n", sql);
+        ret2 = mysql_query(mysql, sql);
+
+      }
+*/ // skip this for now!
+
+    }
+
+    if(ret){
+        log_printf(LOG_ERROR, "Error: mysql_query()\n");
+        log_printf(LOG_ERROR, "mysql_error: %s\n", mysql_error(mysql));
+        return -EIO;
+    }
+    mysql_free_result(myresult);
+    printf("fsck done!\n");
      return 0;
+
+
  }


so normaly this shouldn't affect mounting mysqlfs.

-- 

Mit freundlichen Grüßen,


Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 700 762 789 32 - 0,12 EUR/Min*
fax.: +49 700 762 789 32 - 0,12 EUR/Min*
24/7: +49 900 311 886 00 - 1,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz

Re: [Mysqlfs-general] Cannot mount Filesystem

[Andrew R. <ros...@gm...>]

It could be a problem with your fsck code?  I've used mysqlfs quite
extensively and not come across this in the production code, but have
in my own additions.

Andrew

On 29/08/2007, Smart Weblications GmbH,  Florian Wiessner
<f.w...@sm...> wrote:
> Hello,
>
>
> i have some problem mounting mysqlfs:
>
> ./mysqlfs -ouser=3Droottest -opassword=3Dxxxxxx-ohostname=3Dlocalhost
> -odatabase=3Dmysqlfs /mnt
> 2007-08-29 15:56:45 5194 Error: mysql_query()
> 2007-08-29 15:56:45 5194 mysql_error: Commands out of sync; you can't
> run this command now
> 2007-08-29 15:56:45 5194 Error: pool_init() failed
>
> what am i doing wrong.
>
>
> - i also have written the fsck for mysqlfs, where can i submit my code?
>
>
> --
>
> Mit freundlichen Gr=FC=DFen,
>
>
> Smart Weblications GmbH
> Martinsberger Str. 1
> D-95119 Naila
>
> fon.: +49 700 762 789 32 - 0,12 EUR/Min*
> fax.: +49 700 762 789 32 - 0,12 EUR/Min*
> 24/7: +49 900 311 886 00 - 1,99 EUR/Min*
> http://www.smart-weblications.de
>
> --
> Sitz der Gesellschaft: Naila
> Gesch=E4ftsf=FChrer: Florian Wiessner
> HRB-Nr.: HRB 3840 Amtsgericht Hof
> *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> Mysqlfs-general mailing list
> Mys...@li...
> https://lists.sourceforge.net/lists/listinfo/mysqlfs-general
>

[Mysqlfs-general] Cannot mount Filesystem

[Smart W. G. F. W. <f.w...@sm...>]

Hello,


i have some problem mounting mysqlfs:

./mysqlfs -ouser=roottest -opassword=xxxxxx-ohostname=localhost
-odatabase=mysqlfs /mnt
2007-08-29 15:56:45 5194 Error: mysql_query()
2007-08-29 15:56:45 5194 mysql_error: Commands out of sync; you can't
run this command now
2007-08-29 15:56:45 5194 Error: pool_init() failed

what am i doing wrong.


- i also have written the fsck for mysqlfs, where can i submit my code?


-- 

Mit freundlichen Grüßen,


Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 700 762 789 32 - 0,12 EUR/Min*
fax.: +49 700 762 789 32 - 0,12 EUR/Min*
24/7: +49 900 311 886 00 - 1,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz

Re: [Mysqlfs-general] mysqlfs patch

[Stef B. <st...@bo...>]

On Wednesday 22 August 2007 10:50:32 Andrew Rose wrote:

> > I would like to reply to you. I 'm not a programmer, but as far as I can see
> > multithreading is also important. I wanted to use mysqlfs as a backup, but failed beacuse of the
> > lack of multithreading.
> 
> Correct me if I'm wrong but I was under the impression FUSE handled
> the threading.  All that mysqlfs needs to do is maintain the mysql
> connections (the pool) and keep them thread safe.  The locking (inode
> and data_blocks) as far as I can picture it should be done on the
> database level.
Now I'm do not have a lot of experience (I'm systemenigineer, not a C/C++ programmer)
with multithreading, but when I look to the code of for example sshfs or fusesmb, threading 
is done in the modules as well.(look for pthread_mutex_lock).
> 
> Does anyone have any information on how FUSE handles inode creation?
> The problem with multi-master inode clashing as far as I can see it is
> when two servers (or more) create a file and pick the same inode (for
> a file that they regard as local) that the clash occurs.

I'm sorry. I cannot help you here. Did you look into the source?

Stef Bon

Re: [Mysqlfs-general] mysqlfs patch

[Andrew R. <ros...@gm...>]

On 22/08/07, Stef Bon <st...@bo...> wrote:
> On Wednesday 22 August 2007 04:58:02 Michal Ludvig wrote:
> > Andrew Rose wrote:
> > > On 21/08/07, Michal Ludvig <mi...@lo...> wrote:
>
> >
> > I still wonder what scenario leads to the duplicate keys and subsequent
> > replication failure. Do you write to the same mysqlfs on both hosts?
> > Even if you create a file on "db1" and another file on "db2" they should
> > have assigned different inode numbers. Perhaps if you write to a single
> > file on both DBs it the INSERT on line 558 of query.c may fail. Hmm...
> > Try changing it to REPLACE, ie "REPLACE INTO data_blocks ...". But I'm
> > afraid you may end up with a data corruption if blindly replacing blocks
> > like this. But your original solution that simply didn't do the write on
> > detected conflict wasn't much more corruption proof either. In both
> > cases you'd lose a write that you expected to go through.
> >
> > I personally wouldn't run it in master-master setup, it's a bit
> > complicated to synchronize them, not a simple "one-liner" solution.
> >
> > Michal
> >
> >
> I would like to reply to you. I 'm not a programmer, but as far as I can see
> multithreading is also important. I wanted to use mysqlfs as a backup, but failed beacuse of the
> lack of multithreading.

Correct me if I'm wrong but I was under the impression FUSE handled
the threading.  All that mysqlfs needs to do is maintain the mysql
connections (the pool) and keep them thread safe.  The locking (inode
and data_blocks) as far as I can picture it should be done on the
database level.

Does anyone have any information on how FUSE handles inode creation?
The problem with multi-master inode clashing as far as I can see it is
when two servers (or more) create a file and pick the same inode (for
a file that they regard as local) that the clash occurs.

Andrew

Re: [Mysqlfs-general] mysqlfs patch

[Michal L. <mi...@lo...>]

Stef Bon wrote:

> I would like to reply to you. I 'm not a programmer, but as far as I can see
> multithreading is also important. 

It is, I agree. I'm happy to check and eventually accept any relevant 
patches fixing any problems you observe :-)

Michal

Re: [Mysqlfs-general] mysqlfs patch

[Stef B. <st...@bo...>]

On Wednesday 22 August 2007 04:58:02 Michal Ludvig wrote:
> Andrew Rose wrote:
> > On 21/08/07, Michal Ludvig <mi...@lo...> wrote:

> 
> I still wonder what scenario leads to the duplicate keys and subsequent 
> replication failure. Do you write to the same mysqlfs on both hosts? 
> Even if you create a file on "db1" and another file on "db2" they should 
> have assigned different inode numbers. Perhaps if you write to a single 
> file on both DBs it the INSERT on line 558 of query.c may fail. Hmm... 
> Try changing it to REPLACE, ie "REPLACE INTO data_blocks ...". But I'm 
> afraid you may end up with a data corruption if blindly replacing blocks 
> like this. But your original solution that simply didn't do the write on 
> detected conflict wasn't much more corruption proof either. In both 
> cases you'd lose a write that you expected to go through.
> 
> I personally wouldn't run it in master-master setup, it's a bit 
> complicated to synchronize them, not a simple "one-liner" solution.
> 
> Michal
> 
> 
I would like to reply to you. I 'm not a programmer, but as far as I can see
multithreading is also important. I wanted to use mysqlfs as a backup, but failed beacuse of the
lack of multithreading.

Stef

Re: [Mysqlfs-general] mysqlfs patch

[Michal L. <mi...@lo...>]

Andrew Rose wrote:
> On 21/08/07, Michal Ludvig <mi...@lo...> wrote:
> 
>> Not in the immediate future, sorry. I don't do any active development on
>> mysqlfs anymore.
> 
> So if your not actively developing mysqfs anymore, has anyone else
> picked or is going to pick it up that you know of?

As far as I know it just sits there on SourceForge and that's it.

It was an interesting experience to take over the codebase and put it 
into shape but as I don't use it myself I don't have much interest in 
spending time on it.

>> I suggest you reconfigure the DB instead of crossing fingers ;-)
> 
> Sorry about my reaction in the last post BTW, I know you where trying
> to be helpful. Your right though, I really shouldn't cross my fingers.

I still wonder what scenario leads to the duplicate keys and subsequent 
replication failure. Do you write to the same mysqlfs on both hosts? 
Even if you create a file on "db1" and another file on "db2" they should 
have assigned different inode numbers. Perhaps if you write to a single 
file on both DBs it the INSERT on line 558 of query.c may fail. Hmm... 
Try changing it to REPLACE, ie "REPLACE INTO data_blocks ...". But I'm 
afraid you may end up with a data corruption if blindly replacing blocks 
like this. But your original solution that simply didn't do the write on 
detected conflict wasn't much more corruption proof either. In both 
cases you'd lose a write that you expected to go through.

I personally wouldn't run it in master-master setup, it's a bit 
complicated to synchronize them, not a simple "one-liner" solution.

Michal

Re: [Mysqlfs-general] mysqlfs patch

[Andrew R. <ros...@gm...>]

On 21/08/07, Michal Ludvig <mi...@lo...> wrote:

> Not in the immediate future, sorry. I don't do any active development on
> mysqlfs anymore.

So if your not actively developing mysqfs anymore, has anyone else
picked or is going to pick it up that you know of?

> I suggest you reconfigure the DB instead of crossing fingers ;-)

Sorry about my reaction in the last post BTW, I know you where trying
to be helpful. Your right though, I really shouldn't cross my fingers.

Andrew

Re: [Mysqlfs-general] mysqlfs patch

[Andrew R. <ros...@gm...>]

On 21/08/07, Michal Ludvig <mi...@lo...> wrote:
> Andrew Rose wrote:
>
> > I am currently running a chained master->master setup. It is a common
> > MySQL replication problem with master->master setups, and crops up
> > when the PK's are not auto incrementing.
>
> So it isn't MySQL error but an improper MySQL configuration ;-)

Actually it is a problem with the inode field in data_nodes being a
primary key, and NOT auto_increment'ing.  Not a lot you can do about
that, but certainly not a "improper MySQL configuration" thank you
very much.

Andrew

Re: [Mysqlfs-general] mysqlfs patch

[Michal L. <mi...@lo...>]

Andrew Rose wrote:

> I am currently running a chained master->master setup. It is a common
> MySQL replication problem with master->master setups, and crops up
> when the PK's are not auto incrementing.

So it isn't MySQL error but an improper MySQL configuration ;-)

In case of master-master you have to ensure that each server generates
different keys, not that apps will somehow take care of it. Common
solution is to make one master generate sequence 1,3,5,... and the
second master 2,4,6,... In the case you run more than two masters you'll
obviously increase the stepping to accommodate them all.

Check out this page for details:
http://dev.mysql.com/doc/refman/5.0/en/replication-auto-increment.html

> I figured there would be a problem with threading.  Have you thought
> about maybe providing some shared memory locking functionality? ;)

Not in the immediate future, sorry. I don't do any active development on
mysqlfs anymore.

> Many thanks for the help.  For now I think I'll keep my fingers
> crossed and hope that what I have done has lowered the odds enough
> that I will rarely, if ever, see a problem, considering there is
> minimal writes going on to my mounted filesystems anyway.

I suggest you reconfigure the DB instead of crossing fingers ;-)

Michal

[Mysqlfs-general] Furio Azzolini is out of the office.

[<fur...@fi...>]

I will be out of the office starting  11/08/2007 and will not return until
03/09/2007.

I will respond to your message when I return.


_______________

The information transmitted is intended for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

Re: [Mysqlfs-general] mysqlfs patch

[Andrew R. <ros...@gm...>]

On 21/08/07, Michal Ludvig <mi...@lo...> wrote:
> Hi Andrew,
>
> > I've recently created a patch for mysqlfs.  The idea of the patch is
> > to check for the existence of an entry by key in the data_blocks table
> > before attempting to write (query_write_one_block).  The reason? to
> > stop mysqlfs killing MySQL replication when attempting to insert an
> > data_block with the same inode/seq as one that already exists.
>
> Interesting. What version of MySQL are you running on both master and
> slave? Replication should never break when you attempt to insert a
> record with a primary key that already exists. In fact *any* user action
> or query should never break MySQL replication. If it does in your case
> and if it's reproducible I'd consider it a MySQL server bug worth filing
> a bugreport.
>

I am currently running a chained master->master setup. It is a common
MySQL replication problem with master->master setups, and crops up
when the PK's are not auto incrementing.

> > Now I'm not that familiar with the mysqlfs code, and was hoping
> > someone with more knowledge than myself could let me know if I've gone
> > in the right direction.
> >
> > http://dev.iris.ac/~ody/mysqlfs-x86_64/mysqlfs-0.4.0-rc1.patch
>
> Problem is that your query_inode_key_exists() and the subsequent insert
> are two independent transactions. I.e. even if your check returns false
> the inode can appear in another thread before the first thread decides
> to run the INSERT. You'd have to make it atomic to be safe, which in
> case of MYISAM tables means locking the whole table and therefore
> resulting to a worse performance.

I figured there would be a problem with threading.  Have you thought
about maybe providing some shared memory locking functionality? ;)

Many thanks for the help.  For now I think I'll keep my fingers
crossed and hope that what I have done has lowered the odds enough
that I will rarely, if ever, see a problem, considering there is
minimal writes going on to my mounted filesystems anyway.

cheers
Andrew

Re: [Mysqlfs-general] mysqlfs patch

[Michal L. <mi...@lo...>]

Hi Andrew,

> I've recently created a patch for mysqlfs.  The idea of the patch is
> to check for the existence of an entry by key in the data_blocks table
> before attempting to write (query_write_one_block).  The reason? to
> stop mysqlfs killing MySQL replication when attempting to insert an
> data_block with the same inode/seq as one that already exists.

Interesting. What version of MySQL are you running on both master and
slave? Replication should never break when you attempt to insert a
record with a primary key that already exists. In fact *any* user action
or query should never break MySQL replication. If it does in your case
and if it's reproducible I'd consider it a MySQL server bug worth filing
a bugreport.

> Now I'm not that familiar with the mysqlfs code, and was hoping
> someone with more knowledge than myself could let me know if I've gone
> in the right direction.
> 
> http://dev.iris.ac/~ody/mysqlfs-x86_64/mysqlfs-0.4.0-rc1.patch

Problem is that your query_inode_key_exists() and the subsequent insert
are two independent transactions. I.e. even if your check returns false
the inode can appear in another thread before the first thread decides
to run the INSERT. You'd have to make it atomic to be safe, which in
case of MYISAM tables means locking the whole table and therefore
resulting to a worse performance.

Anyway you shouldn't experience this problem even with the current code.
I suspect MySQL. What does SHOW SLAVE STATUS\G say when it breaks?

Michal

[Mysqlfs-general] mysqlfs patch

[Andrew R. <ros...@gm...>]

Hi all

 I've recently created a patch for mysqlfs.  The idea of the patch is
to check for the existence of an entry by key in the data_blocks table
before attempting to write (query_write_one_block).  The reason? to
stop mysqlfs killing MySQL replication when attempting to insert an
data_block with the same inode/seq as one that already exists.

Now I'm not that familiar with the mysqlfs code, and was hoping
someone with more knowledge than myself could let me know if I've gone
in the right direction.

http://dev.iris.ac/~ody/mysqlfs-x86_64/mysqlfs-0.4.0-rc1.patch

many thanks
Andrew

Re: [Mysqlfs-general] Trouble building on FreeBSD 6 using Fuse from Ports

[Tony S. <tsh...@os...>]

Yes and yes.  Followed your instructions to the hilt.  I've even tried 
user root instead of user mysqlfs.  Same outcome.

Michal Ludvig wrote:
> Tony Shadwick wrote:
>> K, I feel like I'm not making progress here. :(  With log debugging on, 
>> here's what happens when I go to run mkdir testdir:
>>
>> 2007-04-14 11:09:21 1338 mysqlfs_getattr("/testdir")
>> 2007-04-14 11:09:21 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
>> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testdir'
>> 2007-04-14 11:09:21 1338 mysqlfs_mkdir("/testdir", 040755)
>> 2007-04-14 11:09:21 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
>> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testdir'
>>
>> and touch testfile
>>
>> 2007-04-14 11:11:29 1338 mysqlfs_getattr("/testfile")
>> 2007-04-14 11:11:29 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
>> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testfile'
>> 2007-04-14 11:11:29 1338 mysqlfs_getattr("/testfile")
>> 2007-04-14 11:11:29 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
>> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testfile'
>> 2007-04-14 11:11:29 1338 mysqlfs_mknod("/testfile", 100644): file
>> 2007-04-14 11:11:29 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
>> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testfile'
>>
>> The console responds the same both ways - no such file or directory.
> 
> It doesn't do any INSERTs - weird. Have you imported the DB schema into 
> database 'mysqlfs'? Does user 'mysqlfs' have enough rights to read/write it?
> 
> Michal
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Mysqlfs-general mailing list
> Mys...@li...
> https://lists.sourceforge.net/lists/listinfo/mysqlfs-general

Re: [Mysqlfs-general] Trouble building on FreeBSD 6 using Fuse from Ports

[Tony S. <tsh...@os...>]

mysql -u <user> --password=<pass> <db_name> -e 'show table status;' | 
awk '{sum=sum+$7+$9;} END {print sum/1024/1024}'

where <user>, <pass> and <db_name> is your account and database.

returns the size of (sum of) columns Data_length + Index_length in MB

That came from mysql's website.   So far as total space available, 
that's trickier.  Unless you've built-in some sort of limit for a 
particular database, it would be ~= to the amount of space left of the 
filesystem that the database resides on *or* ~= to the space available 
left in the ndb cluster.  Good question either way though. :\

Michal Ludvig wrote:
> Tony Shadwick wrote:
> 
>> manager# df -h
>> Filesystem     Size    Used   Avail Capacity  Mounted on
>> /dev/ad0s1a    496M     76M    380M    17%    /
>> devfs          1.0K    1.0K      0B   100%    /dev
>> /dev/ad0s1e    496M     20K    456M     0%    /tmp
>> /dev/ad0s1f     28G    1.9G     24G     7%    /usr
>> /dev/ad0s1d    2.9G     39M    2.6G     1%    /var
>> devfs          1.0K    1.0K      0B   100%    /var/named/dev
>> /dev/fuse1       0B      0B      0B   100%    /mysqlfs
>>
>> Um...capacity 100%?
> 
> I have no idea how to get an amount of free space from MySQL. I could 
> indeed count the Used space quite easily but not the free space. I'm not 
> sure that MySQL can report it somehow. Any ideas?
> 
> Michal
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Mysqlfs-general mailing list
> Mys...@li...
> https://lists.sourceforge.net/lists/listinfo/mysqlfs-general

Re: [Mysqlfs-general] Trouble building on FreeBSD 6 using Fuse from Ports

[Michal L. <mi...@lo...>]

Tony Shadwick wrote:
> K, I feel like I'm not making progress here. :(  With log debugging on, 
> here's what happens when I go to run mkdir testdir:
> 
> 2007-04-14 11:09:21 1338 mysqlfs_getattr("/testdir")
> 2007-04-14 11:09:21 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testdir'
> 2007-04-14 11:09:21 1338 mysqlfs_mkdir("/testdir", 040755)
> 2007-04-14 11:09:21 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testdir'
> 
> and touch testfile
> 
> 2007-04-14 11:11:29 1338 mysqlfs_getattr("/testfile")
> 2007-04-14 11:11:29 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testfile'
> 2007-04-14 11:11:29 1338 mysqlfs_getattr("/testfile")
> 2007-04-14 11:11:29 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testfile'
> 2007-04-14 11:11:29 1338 mysqlfs_mknod("/testfile", 100644): file
> 2007-04-14 11:11:29 1338 sql=SELECT t1.inode, t1.name, t1.parent, 
>   (SELECT COUNT(inode) FROM tree AS t2 WHERE t2.inode=t1.inode) 
>         AS nlinks FROM tree AS t0 LEFT JOIN tree AS t1 ON t0.inode = 
> t1.parent WHERE t0.parent IS NULL AND t1.name = 'testfile'
> 
> The console responds the same both ways - no such file or directory.

It doesn't do any INSERTs - weird. Have you imported the DB schema into 
database 'mysqlfs'? Does user 'mysqlfs' have enough rights to read/write it?

Michal

Re: [Mysqlfs-general] Trouble building on FreeBSD 6 using Fuse from Ports

[Michal L. <mi...@lo...>]

Tony Shadwick wrote:

> manager# df -h
> Filesystem     Size    Used   Avail Capacity  Mounted on
> /dev/ad0s1a    496M     76M    380M    17%    /
> devfs          1.0K    1.0K      0B   100%    /dev
> /dev/ad0s1e    496M     20K    456M     0%    /tmp
> /dev/ad0s1f     28G    1.9G     24G     7%    /usr
> /dev/ad0s1d    2.9G     39M    2.6G     1%    /var
> devfs          1.0K    1.0K      0B   100%    /var/named/dev
> /dev/fuse1       0B      0B      0B   100%    /mysqlfs
> 
> Um...capacity 100%?

I have no idea how to get an amount of free space from MySQL. I could 
indeed count the Used space quite easily but not the free space. I'm not 
sure that MySQL can report it somehow. Any ideas?

Michal

Re: [Mysqlfs-general] Trouble building on FreeBSD 6 using Fuse from Ports

[Tony S. <tsh...@os...>]

 > gcc -v
Using built-in specs.
Configured with: FreeBSD/i386 system compiler
Thread model: posix
gcc version 3.4.6 [FreeBSD] 20060305



Michal Ludvig wrote:
> Tony Shadwick wrote:
>>>> query.c: In function `query_utime':
>>>> query.c:439: warning: long int format, time_t arg (arg 4)
>>>> query.c:439: warning: long int format, time_t arg (arg 5)
>>>> query.c:439: warning: long int format, time_t arg (arg 4)
>>>> query.c:439: warning: long int format, time_t arg (arg 5)
>>>> gmake[1]: *** [query.o] Error 1
>>>> gmake[1]: Leaving directory `/usr/local/src/mysqlfs-0.4.0-rc1'
>>>> gmake: *** [all] Error 2
>>>>
>>>> The code section it's complaining about is below.  The last line is line 
>>>> 439.
> 
> What GCC version have you got on your FreeBSD?
> 
>> Nevermind, I got it.  Your code appears to be fine, it's just the 
>> default CFLAGS includes -Wall -Werror.  I removed the -Wall and it built 
>> fine.  Weirdness.  Is this fixable?
> 
> You should have removed -Werror instead (this switch turns warnings into 
> errors) and keep -Wall to see all warnings. Some might be useful. I'll 
> remove -Werror from default CFLAGS in future versions.
> 
> Michal
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by DB2 Express
> Download DB2 Express C - the FREE version of DB2 express and take
> control of your XML. No limits. Just data. Click to get it now.
> http://sourceforge.net/powerbar/db2/
> _______________________________________________
> Mysqlfs-general mailing list
> Mys...@li...
> https://lists.sourceforge.net/lists/listinfo/mysqlfs-general