Re: [E-devel] e5: Call for VM

[Carsten H. (T. R. <ra...@ra...>]

On Wed, 6 Feb 2013 20:19:56 +0100 Bertrand Jacquin <be...@me...> said:

> Hi,
> 
> D'ar lun 04 a viz C'hwevrer 2013 e 07 eur 29, « Carsten Haitzler » he deus
> skrivet :
> > On Wed, 30 Jan 2013 12:32:23 +0100 Bertrand Jacquin <be...@me...>
> > said:
> > 
> > while this would be nice... we don't even have our "home turf" vm's all set
> > up yet.
> > 
> > 1. we need to drop lvm. qcow+file images. it just makes migrating vms
> > elsehwere a lot harder if you put them on lvm requiring extra levels of
> > special setup to use them.
> 
> I do agree that qcow can be more easy, but here is the trick. We have
> two cases : VM migration, VM duplication for tests purpose.
> 
> ** VM duplication :
> 
> Using libvirt to migrate need one of Fibre Channel, NFS, GFS or
> iSCSI between the two hosts. We don't have it. I asked to OSUOSL what is
> their internal process so maybe we can use it in the future.

eh? i literally have just copied a qemu disk image from machine a to machine
b... and brought up qemu on it and it works out-of-the-box, zero changes,
migrating etc. it was "scp" and presto. thats why "nfs" for homedirs i think is
bad - it's not self-contained in the image file - you have to transprot your
entire setup across which is complex and not self-contained. that's why i think
lvm just complicates things too. with simple disk files we can easily do delta
snapshot files to transport "current state relative to a baseline" and have
the baseline backed up and stored too. u can just bring it up with qemu cmdline
by hand. dont HAVE to use libvirt. it's trivial to do. you can use libvirt too.

qemu-img create -f qcow2 -b base.img delta.img

personally i'm perfectly happy with shutting down the kvm/qemu image once per
day and doing the above to gen a new daily delta img and backing that up (eg
scp it home - i can do this trivially - i have the bandwidth and space. we can
also do it via scping over to e3/e4 too etc.). osuosl plan on bringing up a big
nfs storage system sometime for backups and data and we can use that too when
it appears. :)

and yes - i know qemu can do LIVE snapshots without shutdown.

http://wiki.qemu.org/Features/Snapshots

also possible. :)

> For the moment there is nothing, so here is the process :
> 
>  = Using LVM :
> 
>   pv /dev/vg-data/e5-template1-sys | ssh osuosl dd of=/somewhere/e5-t1-sys.img
> 	<repeat for each disk>
> 
>  = Using qcow :
> 
> 	scp /somewhere/e5-template1-sys.img osuosl:/somewhere
> 	<repeat for each disk>
> 
> So here it's the same and not so much complicated for one way or one
> other.

the files are more obvious. if we need bigger base images it's a simple
re-generate of img file. lvm is more involved to expand image sizes. :)

> ** Dump to test purpose
> 
> That's already what we did with Tom and Dan needed a fresh VM :
> 
>   pv /dev/vg-data/e5-template1-sys | xz -f | dd
> of=/root/e5-template1-sys.img.xz
> 
> In all case, the source file is just different, so it's not higly more
> complicated, it's the same.
> 
> Using FS over FS in not really efficace, plus, you need to care of the
> multiple sync that append on the VM FS, on the host FS, on the host
> RAID, on the host disks. Using LVM, there is one less layer.

this isnt a "mission critical" system. we will have backups. if we loose half a
day of stuff because of a disk failure.. so be it.. we lose it. go back to last
snapshot/backup. with git we have a massively distributed backup/copy of all
the src which is the most important thing we have, so we are good imho.

> Also, all resize feature etc are still available.
> 
> I do think performance is a criteria as we can observe on e2 for now. We
> should avoid to reproduce a such situation at the very begenning to not
> be surprise in some times.

i am sitting here with a totally unoptimized ubuntu 12.04 image in qemu. i
actually don't NOTICE a slowdown compared the to the 12.04 host when it comes
to I/O. in fact... its FASTER than the host - by a wide margin, when it comes
to boot for example. once cached it boots in about 7 seconds all the way to a
gui (e17+xorg+etc. with all the ubuntu overhead behind it too). the host takes
something like 20-30sec to boot... you currently say that it takes about 5 sec
to boot your gentoo vm's.. thats without e17+xorg+the rest ubuntu slaps on top.
this is a simple core i7 desktop with only 3.5gb of available ram (32bit and
your regular spinning hdd... i see no issues here.

what we need to avoid is things like mysql that hammer the disk with syncs. svn
+apache are a lethal combination easily consuming 2gb of ram when someone
checks out our svn tree via http. and trac in general just loves to consume
cycles. remember e2 is a 6+ year old machine, with a raid controller on the
blink, 2g of ram and a fairly old cpu... e5 is beefy on a bad day. 48g of ram
is one really nice disk cache.

> Also, yes we have a lot of RAM and CPU on e5, that's really fine. But
> I/O are absolutely not better with a lot of RAM or CPU.

that's why i'd rather not FORCE it to be I/O - let it go to ram. this is not
"mission critical". if we "lose a transaction" - too bad. someone's bug report
disappears. for git - this is not a problem as its a distributed system with
every client having backups. the person who did the push has the data - we
didn't lose it. we can push it again in this rare event. we can actually
minimize this simply by running fsync on the host every now and again to ensure
syncs to real media - but dont force this per transaction.

i understand your point of view. it's one of "this is a mission critical system
and we must not lose anything ever".  but we can afford that - thus we can make
different trade-offs. ease of administration. obviousness. ease of migration of
vm's, ease of setting up new things ad-hoc when needed and simplicity in
general are worth more imho. the experience of 6+ years of e.org tell me to
"keep it simple". what happens when you get run over by a bus? we don't have
full-time admins following your every setup and reading manuals. they look at
it once and then use it. in 3 years something happens to you (the bus, or you
get bored of e, or are just on a long holiday for 2 months?) and someone else
needs to "fix things"... if what they find is incredibly complex, tied together
deeply and they haven't spent day in and out adminning the box, they can't
figure out what to do and eventually just ad-hoc make a mess trying to get
things back up at all. you may be back after your holiday only to find things
are now messed up, or we spend longer "down" than needed etc. SIMPLICITY trumps
performance here. GIT solves the most important side of data integrity by its
very nature. the rest is "nice to have, but not critical". if we have to roll
back 1/2 a day or a day of data... we'll live just fine, unless this is
happening every few days.. and then we have a major serious problem.

> > 2. nfs homedirs are a no-go. homedirs must/should be local to the vm fs
> > image. making them nfs makes them non-portable. the whole POINT of using
> > qemu/kvm in the first place is, that if we get overloaded, or if e5 goes
> > down, or whatever we can TRIVALLY just move any vm over to osuosl's vm
> > cluster system and re-point dns and presto... it works. this is WHY we want
> > a SIMPLE setup. not complex. SIMPLE.
> 
> Yep ! NFS is here to speed up thing for now, it's temporary and will not
> stay. Never wanted to.

cool! :) again as above - speed.. don't care. :) the qemu -fsdev/virtfs option
is more "sane" imho if we want to have arbitrary expansion space outside the fs
image - this means we can have base fs img + delta + dir of "extra
files" (tar/rsync that dir as needed)

> > 3. firewall needs to go - NAT is enough.
> 
> NAT is not firewalling, hope we agree on this.

i know.. but it provides protection in that "you can't get in through the nat
unless something explicitly forwards a port". getting in is a reality we will
have to live with unless we want to ask for an IP for every VM... and that's
asking a lot of osu to go vie us so many IP's.

> > firewall just wastes peoples time
> > debugging services.
> 
> We should not have to debug anything when thing are up, when we deploy
> new since I agree, it's normal. And this does not happend so often.

ask daniel. he already spent hours figuring out a problem that was the fw
blocking OUTGOING traffic. we JUST need a NAT. we all know the NAT will provide
incoming connections unless we explicitly route them. we "expect" outgoing
connections to "just work" (tm) with nothing in the way.

> > in all the years e1/e2 have been up we have nver needed or
> > uses a fw.
> 
> And for 4 years, MySQL has been opened to whole internet. This just
> prevent mistakes.

we're behind a nat... that already does all the work we need. the host system
(e5) has nothing running anyway, so doesn't matter.

> > if someone "breaks in" and then can "get data out"... WHO CARES...
> > we arent hiding secret data! this is not a paranoid corporate system.
> 
> I can be really more paranoid than that ;)

:) sure. just saying - that we don't need to go this far. :)

> > if they
> > broke in and start modifying our git repos and then people download src with
> > trojans in it.. thats bad.. but a fw wont stop that... and tbh... we havent
> > had this problem before (i know there is a 1st time), and if we do. thats
> > WHY we have vm filesystem images and qcoq deltas/backups
> 
> I also do FS backup with delta on my own.

:)

> > we can just pull out of cold
> > storage. with qcow we can keep fs DELTAS offline "at home" trivially by just
> > scping a qcow snapshot (or series) and simply upload any compromised one and
> > restart from there. enough devs have home storage to keep enough of these
> > images copied and safe in case this happens.
> 
> > :)
> > 
> > > Hi,
> > > 
> > > For e5 buildbot/whatever we will need different OS and architecture to
> > > test. So if some volunteer are free to build some, we can gladly host
> > > them.
> > > 
> > > Needed OS :
> > > 
> > >  - Windows XP x86
> > >  - Windows XP x86_64
> > >  - Windows Vista x86_64
> > >  - Windows 7 x86_64
> > >  - Windows 8 x86_64
> > > 
> > >  (maybe more declination are needed ?)
> > > 
> > >  - Mac OS X Snow Leopard
> > >  - Mac OS X Lion
> > >  - Mac OS X Mountain Lion
> > > 
> > >  - FreeBSD 8
> > >  - FreeBSD 9
> > > 
> > >  - NetBSD 4.0 (still supported)
> > >  - NetBSD 6.0 (still supported)
> > >  - NetBSD 6.0
> > > 
> > >  - OpenBSD 5.2
> > > 
> > >  - ?? Other, and maybe more declination on previous list
> > > 
> > > Here are characteristics :
> > > 
> > >  - More the system is light, faster we can transfer it
> > >  - dd if=/dev/zero of=/device at first
> > >  - Make the image available as a disk gzip/xz format (as main content will
> > >    be unused space defined with zeros previously)
> > >  - Use virtIO for network, disk and console
> > >  - kernel parameters:
> > >     panic=5 clocksource=kvm-clock console=tty0 console=ttyS0,115200
> > >  - 1 network interface with DHCP
> > >  - syslog everything to host: log.e5 (I can send you a syslog-ng conf if
> > >    necessery)
> > >  - smtp relay: smtp.e5 (I can send you a exim config)
> > >  - minimal install
> > >  - watchdog (wdd prefered http://linux.exosec.net/watchdog/daemon/)
> > >  - necessary packages
> > >     - snmpd
> > >     - acpid
> > > 
> > > Thanks,
> > > 
> > > -- 
> > > Beber
> > 
> > 
> > -- 
> > ------------- Codito, ergo sum - "I code, therefore I am" --------------
> > The Rasterman (Carsten Haitzler)    ra...@ra...
> > 
> 
> -- 
> Beber


-- 
------------- Codito, ergo sum - "I code, therefore I am" --------------
The Rasterman (Carsten Haitzler)    ra...@ra...