ATA over Ethernet Tools

The Ubuntu driver probably only works with a maximum of PAGE_SIZE
data per AoE packet.  The driver on the CORAID website can have
more than PAGE_SIZE bytes in each AoE packet.

-- 
  Ed

Hello Ed,

I was now able to install the driver. But still the number of requested 
sectors is not higher than 8.
Do you have any other idea?

Best regards
Philip


Ed Cashin schrieb:
> Hi.  Did you mean to take the conversation off the mailing list?
> 
> Anyway, the most common reason for an invalid module error at load
> time is some mis-match between the kernel sources you're using and
> the kernel you're running.
> 
> If you check dmesg in such a case, the kernel will usually tell
> you exactly what doesn't match.  Usually it's a detail like the
> compiler version, SMP support in the kernel, etc.
> 
> You might have already seen it, but we have an Ubuntu page on
> the CORAID support site.
> 
>   http://support.coraid.com/support/linux/distros/ubuntu.html
> 
> Please let me know (or support@...) if that information
> needs updating.
>

On Fri May 22 07:19:45 EDT 2009, philip.kirchhoff@... wrote:
> 
> Hello Ed,
> 
> I was now able to install the driver. But still the number of requested 
> sectors is not higher than 8.
> Do you have any other idea?

Which driver did you install and which is running?  You can tell by
examining the output of aoe-version.

If it's version 71, and you only see 4 KiB payloads (you can see the
payload size in the output of today's aoe-stat), then the driver is
probably choosing to use a smaller size because either the local
interface is something like 4200, or else the target's advertised
buffer count is eight.

Those are the two inputs to the decision the aoe driver uses to
determine payload size.

-- 
  Ed

Seems like it is not a problem of the driver.

I run a benchmark with hdparm and it seems like the tool is just 
requesting the data in 4k blocks. Is this possible?

With normal file operations I can see packets with 16 sectors.

Ed Cashin schrieb:
> On Fri May 22 07:19:45 EDT 2009, philip.kirchhoff@... wrote:
>> Hello Ed,
>>
>> I was now able to install the driver. But still the number of requested 
>> sectors is not higher than 8.
>> Do you have any other idea?
> 
> Which driver did you install and which is running?  You can tell by
> examining the output of aoe-version.
> 
> If it's version 71, and you only see 4 KiB payloads (you can see the
> payload size in the output of today's aoe-stat), then the driver is
> probably choosing to use a smaller size because either the local
> interface is something like 4200, or else the target's advertised
> buffer count is eight.
> 
> Those are the two inputs to the decision the aoe driver uses to
> determine payload size.
>

I mean to say :

vbladed 0 1 eth0 /dev/loop0

doesn't work in my case to detach the exported device.


On Tue, May 19, 2009 at 2:09 PM, er krishna <erkrishna@...> wrote:

> Dear All,
>
>
> Can anybody tell me how to detach exported block device through
> vblade/vbladed command. For ex: ./vblade 0 0 eth0 /dev/hda6 is the command
> to export the block device over network. What is the exact command to
> remove/detach this block device  from network.
>
> What I did to do this, I killed all the processes after doing ps -eaf |
> grep vblade, it worked but didn't remove entry from client side. Still
> aoe-stat & ls -l /dev/etherd shows an entry of corresponding e0.0 exported
> device, which is not exported now. Can Anybody help me here, It seems I am
> missing something.
>
>
> Thanks & Best Regards,
> Krishna
>

Yes, I  thought like that & used aoe-flush; but it didn't work as per the
expectation. Again I am repeating my steps for clearification:

On server side:

1) To  Export : vblade 0 0 eth0 /dev/loop0
2) To  detach : vbladed 0 0 eth0 /dev/loop0


ps -eaf | grep vblade still shows the entry against vblade.

On client side:
I did following things to detach it, see the snippet & please correct me :

[root@... ~]#
[root@... ~]# aoe-stat
      e0.0         0.209GB   eth0 up
      e0.6         0.209GB   eth0 up
[root@... ~]#
[root@... ~]#
[root@... ~]# aoe-flush -a
aoe-flush: /dev/etherd/flush does not exist or is not writeable.
[root@... ~]#
[root@... ~]# aoe-flush /dev/etherd/e0.0
aoe-flush: /dev/etherd/flush does not exist or is not writeable.
[root@... ~]#
[root@... ~]# ls -l /dev/etherd/
total 0
brw-r----- 1 root disk 152,  0 May 19 17:26 e0.0
brw-r----- 1 root disk 152, 96 May 19 17:26 e0.6
[root@... ~]#
[root@... ~]#
[root@... ~]#
[root@... ~]# rmmod aoe
[root@... ~]# modprobe aoe
[root@... ~]# aoe-stat
      e0.0         0.209GB   eth0 up
      e0.6         0.209GB   eth0 up
[root@... ~]#
[root@... ~]#

So, How can I get my vblade stop from server side, so that it removes aoe
device from client as well as from server end.


Thanks,
Krishna




On Tue, May 19, 2009 at 5:00 PM, Ed Cashin <ecashin@...> wrote:

> The Linux aoe driver remembers targets it discovers until you
> tell it to forget them by using the aoe-flush command.  The
> man page for aoe-flush should have the details.
>
> --
>   Ed
>

I don't understand why you mention "vbladed" and "detach" together.
The vbladed is a wrapper for vblade that launches it as a daemon.

Your aoe-flush command cannot work without a /dev/etherd/flush
device file in place.  Check the Linux Support Page at CORAID
for distro-specific tips, where you'll find RHEL/CentOS tips
that should help udev create the device nodes.

-- 
  Ed

Oh.  And as for the vblade process itself, just kill it like any other
userland process.

(And like any userland process, you'd send a SIGTERM, not a SIGKILL,
so that the process has a chance to do any necessary cleanup on exit.)

-- 
  Ed

Dear ED,

Thanks for reply and sorry for all the unclear mails from myside. Actaully
my requiremnt is to stop vblade service from server side ( or to remove the
exported device ), Please tell weather its possible with vblade and if it is
then how ? Any specific command or url will be most welcome.



On Tue, May 19, 2009 at 6:20 PM, Ed Cashin <ecashin@...> wrote:

> I don't understand why you mention "vbladed" and "detach" together.
> The vbladed is a wrapper for vblade that launches it as a daemon.

 daemonize

Perhaps, I have misundrstood vblade man page, sorry for it.


The  vbladed  script can be used to daemonize the vblade process, detaching
it from your terminal and sending its output to the system  logs.


>
> Your aoe-flush command cannot work without a /dev/etherd/flush
> device file in place.  Check the Linux Support Page at CORAID
> for distro-specific tips, where you'll find RHEL/CentOS tips
> that should help udev create the device nodes.



Thanks for the suggestion. I am trying this.



>
>
> --
>   Ed



Best Regards,
Krishna

Hi,

On Tue, 2009-05-19 at 18:30 +0530, er krishna wrote:
> Dear ED, 
> 
> Thanks for reply and sorry for all the unclear mails from myside.
> Actaully my requiremnt is to stop vblade service from server side ( or
> to remove the exported device ), Please tell weather its possible with
> vblade and if it is then how ? Any specific command or url will be
> most welcome.
> 
> 

Stop vblade/vbladed on the storage server.

Run aoe-flush on the nodes accessing the storage.

This can easily be wrapped in automation scripts, as facilitated by
things like SSH.

Cheers,
--Tim

> Stop vblade/vbladed on the storage server.

man kill

There is no way to stop vblade with vblade.  I suspect you're
wanting something akin to the init.d scripts to start/stop
a program.  vblade doesn't do that.  Use the kill command
to stop the process.

Sam

On Tue, May 19, 2009 at 6:40 PM, Tim Post <echo@...> wrote:

> Hi,
>
> On Tue, 2009-05-19 at 18:30 +0530, er krishna wrote:
> > Dear ED,
> >
> > Thanks for reply and sorry for all the unclear mails from myside.
> > Actaully my requiremnt is to stop vblade service from server side ( or
> > to remove the exported device ), Please tell weather its possible with
> > vblade and if it is then how ? Any specific command or url will be
> > most welcome.
> >
> >
>
> Stop vblade/vbladed on the storage server.


How ? Currently I am doing ps-eaf | grep vblade ; & then I kill all the
process; is there any other way ?



>
>
> Run aoe-flush on the nodes accessing the storage.


ok, so aoe-flush is necessary; Currently I am doing rmmod aoe ; modprobe aoe
; aoe-stat  & results are as per exptation. But aoe-flush seems better way.



>
> This can easily be wrapped in automation scripts, as facilitated by
> things like SSH.



Yes, I exactly want to do this from server end only. I can do script. Please
correct following steps if there is any better way.


1) On server ps -eaf | grep vblade;  find all the processes & kill those by
kill -9.

2) Run ssh command & login on each machine ( w/o passwd ) ; then execute
aoe-flush command on each node on the network.








>
>
> Cheers,
> --Tim
>
>

Dear Sam,

Thanks for the mail. I am doing this only.

On Tue, May 19, 2009 at 8:06 PM, Sam Hopkins <sah@...> wrote:

> > Stop vblade/vbladed on the storage server.
>
> man kill
>
> There is no way to stop vblade with vblade.  I suspect you're
> wanting something akin to the init.d scripts to start/stop
> a program.


Yes i was looking something like that.



>  vblade doesn't do that.


vbladectl should be an utility which should do this. I will try to make one;
( Actually for specific work which I am trying its necessary to do this )



> Use the kill command
> to stop the process.
>
> Sam
>
>

Hi,

On Wed, 2009-05-20 at 09:40 +0530, er krishna wrote:

>         
>         Stop vblade/vbladed on the storage server.
> 
> How ? Currently I am doing ps-eaf | grep vblade ; & then I kill all
> the process; is there any other way ?

You could try the command pidof vblade. This will print all vblade PID's
to stdout.

so kill `pidof vblade` may be easier for you, however it is not
discriminate. In other words, you may want to stop just one vblade, not
all of them. Currently there is no easy way to keep track of which PID
is exporting what.

I have some patches that I'll send later this week that 'properizes'
vblade for use as a daemon. It then becomes easier to manage. The
changes are kind of invasive, so I'm guessing they can just live in
contrib/.


> ok, so aoe-flush is necessary; Currently I am doing rmmod aoe ;
> modprobe aoe ; aoe-stat  & results are as per exptation. But aoe-flush
> seems better way.

Yes.

> 1) On server ps -eaf | grep vblade;  find all the processes & kill
> those by kill -9.

Don't use -9, its not needed. Let vblade / vbladed handle the signals as
they are designed to do (i.e. clean up tasks, etc).

> 2) Run ssh command & login on each machine ( w/o passwd ) ; then
> execute aoe-flush command on each node on the network.

Yes, that is the simplest way. Puppet as others have mentioned would
also do the job.

Cheers,
--Tim

Dear Tim,

Thanks for the info and valuable suggestions.

On Wed, May 20, 2009 at 10:18 AM, Tim Post <echo@...> wrote:

> Hi,
>
> On Wed, 2009-05-20 at 09:40 +0530, er krishna wrote:
>
> >
> >         Stop vblade/vbladed on the storage server.
> >
> > How ? Currently I am doing ps-eaf | grep vblade ; & then I kill all
> > the process; is there any other way ?
>
> You could try the command pidof vblade. This will print all vblade PID's
> to stdout.
>
> so kill `pidof vblade` may be easier for you, however it is not
> discriminate. In other words, you may want to stop just one vblade, not
> all of them. Currently there is no easy way to keep track of which PID
> is exporting what.
>
> I have some patches that I'll send later this week that 'properizes'
> vblade for use as a daemon. It then becomes easier to manage. The
> changes are kind of invasive, so I'm guessing they can just live in
> contrib/.



>>>> What about a new dameon, if we create vbladectl which will keep track
of all the exported device based on  interface & path ( /dev/loop0 or eth0 )
release the attached resource or add new resource ? I think we can have a
look for this implemenatation.

Ed, does it seems valuable or funny. A patch will be welcome if I try for
this implementation. Just suggest.




>
>
>
> > ok, so aoe-flush is necessary; Currently I am doing rmmod aoe ;
> > modprobe aoe ; aoe-stat  & results are as per exptation. But aoe-flush
> > seems better way.
>
> Yes.
>
> > 1) On server ps -eaf | grep vblade;  find all the processes & kill
> > those by kill -9.
>
> Don't use -9, its not needed. Let vblade / vbladed handle the signals as
> they are designed to do (i.e. clean up tasks, etc).
>
> > 2) Run ssh command & login on each machine ( w/o passwd ) ; then
> > execute aoe-flush command on each node on the network.
>
> Yes, that is the simplest way. Puppet as others have mentioned would
> also do the job.
>
> Cheers,
> --Tim
>
>
>
>

On Wed, 2009-05-20 at 11:57 +0530, er krishna wrote:


> >>>> What about a new dameon, if we create vbladectl which will keep
> track of all the exported device based on  interface & path
> ( /dev/loop0 or eth0 ) release the attached resource or add new
> resource ? I think we can have a look for this implemenatation. 
> 
> Ed, does it seems valuable or funny. A patch will be welcome if I try
> for this implementation. Just suggest.

What I was going to do was add some more options to vblade, i.e. -d /
--daemon. vbladed would then become a symbolic link to vblade, and if
vblade is run as vbladed, --daemon is automatically assumed, everything
just goes to syslog.

The other change I will finish and submit is a simple ini style
configuration file, something like this:

[daemon]
loglevel = info
; %S is always expanded to shelf, %s always expands to slot
lockfile = /var/run/vblade/blade-%S-%s.pid

[/dev/volgroup/foo]
shelf = xx
slot = xx
eth = xx
directio = true / false
on_boot = true

[/home/foo/file.img]
...
...
...

I don't think a wire protocol (as vbladectl would suggest) is really
needed, all each vbladed process has to do is lock
via /var/run/vblade/{shelf}.{slot} , which would contain its process ID.

Similarly, vbladectl can just parse the configuration file.. i.e

vbladectl stop /dev/volgroup/foo
vbladectl restart /dev/volgroup/foo

vbladectl show would have the task of validating whatever is
in /var/run/vblade, and could easily correlate each export to the pid of
the blade handling it.

I wasn't going to get any more complicated than that, as that alone is
enough for anyone to write custom rc scripts / tools.

Really, in fact, all of that can be accomplished without touching vblade
itself. It just seemed a little cleaner to just modify it vs making a
bunch of wrappers to work around it.

This approach would also keep from breaking existing use of vblade /
vbladed.

Cheers,
--Tim

I've been using something like this with a small patch to vblade (see  
attached diff).
To sum it up, there is an added option of '-p' to specify a file to  
write the pid and ppid to.

Another part is the vblade-skeleton used as an init script in /etc/ 
init.d/vblade-resourcename.  Just copy it to a new file and update the  
variables ( probably needs an update to work on redhat style systems  
using chkconfig ).

Example... so to start and stop a vblade with resource name e1.0:
  /etc/init.d/vblade-e1.0 start
  /etc/init.d/vblade-e1.0 stop

or for SIGKILL
  /etc/init.d/vblade-e1.0 stop-force

Hope thats useful.

--
Matth Ingersoll

On Tue, May 19, 2009 at 6:20 PM, Ed Cashin <ecashin@...> wrote:

> I don't understand why you mention "vbladed" and "detach" together.
> The vbladed is a wrapper for vblade that launches it as a daemon.
>
> Your aoe-flush command cannot work without a /dev/etherd/flush
> device file in place.  Check the Linux Support Page at CORAID
> for distro-specific tips, where you'll find RHEL/CentOS tips
> that should help udev create the device nodes.



Ed, Can you please send me the exact URL for this. I have everything working
on fedora ( means ls -l /dev/etherd was working fine with each entry
corrected ) but on CENTOS it is not working correctly.....

I mean there is no entry for /dev/etherd/flush or anything except following
one:

[root@... aoetools-30]#
[root@... aoetools-30]# ls -l /dev/etherd/
total 0
brw-r----- 1 root disk 152, 48 May 21 16:34 e0.3
[root@... aoetools-30]#
[root@... aoetools-30]#


Please help by suggesting how to get udev customize for this .


Krishna




>
>
> --
>   Ed
>

It has done by following command :

[root@... ~]# cp /root/Desktop/aoe6-71/linux/Documentation/aoe/udev.txt
/etc/udev/rules.d/60-aoe.rules





On Tue, May 19, 2009 at 2:09 PM, er krishna <erkrishna@...> wrote:

> Dear All,
>
>
> Can anybody tell me how to detach exported block device through
> vblade/vbladed command. For ex: ./vblade 0 0 eth0 /dev/hda6 is the command
> to export the block device over network. What is the exact command to
> remove/detach this block device  from network.
>
> What I did to do this, I killed all the processes after doing ps -eaf |
> grep vblade, it worked but didn't remove entry from client side. Still
> aoe-stat & ls -l /dev/etherd shows an entry of corresponding e0.0 exported
> device, which is not exported now. Can Anybody help me here, It seems I am
> missing something.
>
>
> Thanks & Best Regards,
> Krishna
>

Hello Mark,

Am Freitag, den 08.05.2009, 14:21 +0100 schrieb Mark Sutton:
> On 20 Apr 2009, at 11:12, er krishna wrote:
> 
> > I am trying to setup NFS failover and its recovery via GFS (using  
> > CMAN) tool. Of course, I have created a gfs file system over my  
> > logical volume ( basically its on exported block devices ). I have  
> > three nodes centos, centos1 and centos2. I am attaching my  
> > cluster.conf file for further reference. Everything seems fine, but  
> > I am not able to migrate my NFS services when I poweroff my first  
> > node. Anybody has any idea about it ?
> >
> 
> A quick glance at your cluster.conf suggests you're not telling cman  
> how many votes to require for quorum. I'm not sure what the behaviour  
> of cman is in these conditions but it might well be setting  
> expected_votes to the total number of nodes in your cluster ie 3. So  
> if one node went down then the cluster would lose quorum and any  
> cluster operations including failover will fail.
> 
> <cman/>
> 
> For a 3 node cluster I guess expected_votes would be best at 2, so  
> maybe try changing your cman line to this:
> 
> <cman expected_votes="2" />

cman does the right thing by default so there is no need to specify it.

In this case it also wouldn't make any difference at all:

quorum =  floor( expected_votes / 2 ) + 1

2 in booth cases.

> And that might fix it... Of course this thread is a little old now,  
> you might have already fixed the problem...
> 
> Hope this helps,
> 
> Mark

Marc