mysqlfs-general

[Mysqlfs-general] query_inode_full: how does it work?

[Stef B. <st...@bo...>]

Hello,

now I can check the flow of what happens (by activating debugging). It's 
much more clear now.

But when I look at query_inode_full, I do not understand what happens 
there at all.
The purpose of this function is to translate a path into a inode, right? 
But it does much more things.
It "returns" a table nlinks, the inode, the parent. (and more?)

Can someone explain why this is? It runs complicated queries....

Thanks in advance,

Stef Bon

Re: [Mysqlfs-general] query_inode_full: how does it work?

[Stef B. <st...@bo...>]

Stef Bon wrote:
> Hello,
>
> now I can check the flow of what happens (by activating debugging). It's 
> much more clear now.
>
> But when I look at query_inode_full, I do not understand what happens 
> there at all.
> The purpose of this function is to translate a path into a inode, right? 
> But it does much more things.
> It "returns" a table nlinks, the inode, the parent. (and more?)
>
> Can someone explain why this is? It runs complicated queries....
>   

Ok,

now I've read it a couple of times, I understand it more or less.

A (long) query is constructed to determine the inode, with the help of a 
LEFT JOIN.

This query does grow bigger as the path is longer (deeper).

I'm looking into the mysql syntax now. Any hints about it welcome.

Stef Bon

Re: [Mysqlfs-general] query_inode_full: how does it work?

[Smart W. G. F. W. <f.w...@sm...>]

Hi,

Stef Bon schrieb:

>>
>> now I can check the flow of what happens (by activating debugging). It's 
>> much more clear now.
>>
>> But when I look at query_inode_full, I do not understand what happens 
>> there at all.
>> The purpose of this function is to translate a path into a inode, right? 
>> But it does much more things.
>> It "returns" a table nlinks, the inode, the parent. (and more?)
>>
>> Can someone explain why this is? It runs complicated queries....
>>   
> 
> Ok,
> 
> now I've read it a couple of times, I understand it more or less.
> 
> A (long) query is constructed to determine the inode, with the help of a 
> LEFT JOIN.
> 
> This query does grow bigger as the path is longer (deeper).
> 
> I'm looking into the mysql syntax now. Any hints about it welcome.
> 

Yes, the query grows bigger, and it is executed every time a lstat() or 
lstat64() syscall happens - which makes performance really bad on 
ndbcluster with distributed clients on different locations. We should 
cache it somewhere (either in mysqlfs_getattr, or as cache somewhere in 
query_inode_full.

the cache must be cleared for the inode, if write/rename or chown/chmod 
is executed so that query_inode_full always returns proper values.

i thought about that inode-cache - it shouldn't be more than an array 
containing values which has to be checked before the query is run.

as for the locking-problem, what about creating a table metadata to 
store information on which inode is currently locked? metadata could 
also be stored with ndbcluster-engine, and thus we would easily be able 
to implement file locking.

-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 700 762 789 32 - 0,12 EUR/Min*
fax.: +49 700 762 789 32 - 0,12 EUR/Min*
24/7: +49 900 311 886 00 - 1,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz

Re: [Mysqlfs-general] query_inode_full: how does it work?

[Stef B. <st...@bo...>]

Smart Weblications GmbH, Florian Wiessner wrote:
> Hi,
>
> Stef Bon schrieb:
>
>   
>> This query does grow bigger as the path is longer (deeper).
>>
>>     
> Yes, the query grows bigger, and it is executed every time a lstat() or 
> lstat64() syscall happens - which makes performance really bad on 
> ndbcluster with distributed clients on different locations. We should 
> cache it somewhere (either in mysqlfs_getattr, or as cache somewhere in 
> query_inode_full.
>
> the cache must be cleared for the inode, if write/rename or chown/chmod 
> is executed so that query_inode_full always returns proper values.
>
> i thought about that inode-cache - it shouldn't be more than an array 
> containing values which has to be checked before the query is run.
>   
Yes, a cache should will things up. The construction now does a lot of 
lookups and never uses
data it has found before. For example, if you want to read a file 
/a/b/c/d/testfile, it will check you've permissions to
that file, and walk the path to it:
check /
check /a
check /a/b
check /a/b/c
check /a/b/c/d
check /a/b/c/d/testfile

to determine the inode of say /a/b/c, it does not use the information it 
just found before.

So a cache is a smart thing to do.
There are other fuse fs's using a cache(sshfs, fusesmb). My idea is to 
look first there how it is solved there.
My first impression is that fuse offers possibilities to use a cache, 
but I'm not sure.


> as for the locking-problem, what about creating a table metadata to 
> store information on which inode is currently locked? metadata could 
> also be stored with ndbcluster-engine, and thus we would easily be able 
> to implement file locking.
>
>   
Why not create an extra field in the inodes table, with a information 
about the locking?
An extra table is making things extra complicated, I think.

Stef Bon

Re: [Mysqlfs-general] query_inode_full: how does it work?

[Smart W. G. F. W. <f.w...@sm...>]

Hi,

Stef Bon schrieb:

>> Yes, the query grows bigger, and it is executed every time a lstat() or 
>> lstat64() syscall happens - which makes performance really bad on 
>> ndbcluster with distributed clients on different locations. We should 
>> cache it somewhere (either in mysqlfs_getattr, or as cache somewhere in 
>> query_inode_full.
>>
>> the cache must be cleared for the inode, if write/rename or chown/chmod 
>> is executed so that query_inode_full always returns proper values.
>>
>> i thought about that inode-cache - it shouldn't be more than an array 
>> containing values which has to be checked before the query is run.
>>   
> Yes, a cache should will things up. The construction now does a lot of 
> lookups and never uses
> data it has found before. For example, if you want to read a file 
> /a/b/c/d/testfile, it will check you've permissions to
> that file, and walk the path to it:
> check /
[check...]
> check /a/b/c/d/testfile
> 
> to determine the inode of say /a/b/c, it does not use the information it 
> just found before.
> 
> So a cache is a smart thing to do.
> There are other fuse fs's using a cache(sshfs, fusesmb). My idea is to 
> look first there how it is solved there.
> My first impression is that fuse offers possibilities to use a cache, 
> but I'm not sure.
> 
>> as for the locking-problem, what about creating a table metadata to 
>> store information on which inode is currently locked? metadata could 
>> also be stored with ndbcluster-engine, and thus we would easily be able 
>> to implement file locking.
>>
>>   
> Why not create an extra field in the inodes table, with a information 
> about the locking?
> An extra table is making things extra complicated, I think.
> 

ok, i'll look at other fuse-fs's how cache is implemented there.

one extra field in the inode table is sufficient if mysqlfs is not run 
on a cluster, but i don't know if it would be smarter to store this kind 
of metainfo in a seperate table "metadata" - e.g. which mysql-host has 
locked which inode etc.

but i think we should first implement the cache, then we can solve the 
locking problem...



-- 

Mit freundlichen Grüßen,

Florian Wiessner

Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 700 762 789 32 - 0,12 EUR/Min*
fax.: +49 700 762 789 32 - 0,12 EUR/Min*
24/7: +49 900 311 886 00 - 1,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz