backuppc-users

Re: [BackupPC-users] very slow backup speed

[Josh M. <jo...@wo...>]

> rsync is saving bandwidth at the cost of processing overhead doesnt mean that rsync
> is just for this purpose, it is a way to think about it. Dont be so single minded.
>
> As a matter of fact the rsync page says 'rsync is an open source
> utility that provides fast incremental file transfer.' and I want speed which
> coincides with the statement in rsync web pages.
>  
>   
Just something that needs pointing out here, is that rsync is only a 
"fast incremental file transfer" program when transferring large files 
over a slow link. The rsync protocol reads the files at both ends, 
compares them blocks at a time with checksums, then writes out a whole 
new copy of each file. For a slow link this is great, as the data 
transferred is a little bigger than the differences in the file, so a 
1Gb file with 1 byte changed at the end only has to transfer that 1 byte 
plus all the checksums etc. For a fast link this is actually slower, as 
both ends have to read the file, compute checksums, communicate them to 
each other, transfer chunks of the file down the line, then the 
destination server has to write out the file after applying the 
difference patch. For fast links it is much faster for the system to 
read at one end and write at the other, transferring large chunks 
without using any cpu power. And for completely new or completely 
changed files (think a gzip'd file) this actually means rsync is slower 
and transfers more data!

So in the case where is says it's a fast incremental file transfer, it's 
still saying you're saving bandwidth (read that as time on a slow link 
and therefore is faster) at the expense of processing overhead.

Just had to clear that misunderstanding up.

Josh.

P.S. If you're using rsync over ssh then make sure you use the blowfish 
cipher - it's much much faster than the default. I personally use rsyncd 
as it's on a trusted network and don't want the encryption overhead of ssh.

Re: [BackupPC-users] very slow backup speed

[Les M. <le...@fu...>]

Josh Marshall wrote:

> Just something that needs pointing out here, is that rsync is only a 
> "fast incremental file transfer" program when transferring large files 
> over a slow link.

Or - in the incremental case - when you skip files that match in length 
and timestamp and there aren't many changes.

-- 
   Les Mikesell
    le...@fu...

Re: [BackupPC-users] very slow backup speed

[Holger P. <wb...@pa...>]

Hi,

Evren Yurtesen wrote on 30.03.2007 at 01:05:48 [Re: [BackupPC-users] very slow backup speed]:
> Right away when you enable 'tar' backuppc would be misconfigured by your 
> definition, should tar support be removed? Tar can not check checksums 
> anyway (plus has other flaws where it can miss newer/changed files)
> [...]
> There has been a misunderstanding here. I was only talking about backuppc 
> comparing checksums for deciding if the file should be backed up or not and 
> it is not impossible to disable this, it is disabled when you use tar for 
> example.

yep, there have been several misunderstandings here. Most of us have
noticed, I believe.

1.) Rsync checksums are not a feature of BackupPC. They are a feature of the
    rsync protocol.
2.) Tar full backups do *not* miss files.

And with that, my troll-sitting quota is used up. See you.

Regards,
Holger

Re: [BackupPC-users] very slow backup speed

[David R. <dr...@gm...>]

On 3/29/07, Evren Yurtesen <yur...@is...> wrote:
> I didnt blame anybody, just said BackupPC is working slow and it was working
> slow, very slow indeed. checksum-seeds option seems to be doing it's trick though.

How long are full and incremental backups taking now?

> I am thankful to people who wrote suggestions here in this forum, I tried all of
> those suggestions one by one. I think that shows that I took them seriously even
> though some of them looked like long shots. Eventually one of the suggestions
> seems to be working.

You only tried 2 things. Mounting the backup partition async and
turning on checksum-seeds. Are you going to the 2 others? (Add memory
and try tar instead of rsync)

-Dave

Re: [BackupPC-users] very slow backup speed

[Holger P. <wb...@pa...>]

Hi,

Evren Yurtesen wrote on 29.03.2007 at 17:04:05 [Re: [BackupPC-users] very slow backup speed]:
> Holger Parplies wrote:
> > "Most backup programs" take "do a full backup" to mean "read *everything*
> > over the network and write it to backup medium". BackupPC modifies this to
> > mean "read *everything* over the network and store *unchanged* contents
> > efficiently by using hardlinks" while taking the notion of "unchanged" very
> > seriously. If previous backups have become corrupted for some reason, that
> > is no good reason to feel fine with storing unchanged contents in unchangedly
> > [corrupted form]
> 
> I am willing to take that risk, who are you to disagree? I am not asking that
> checksum checking should be removed. I am just asking that people who want to 
> take that risk should be able to disable it.

well, go ahead and implement it. If you want other people to do it for you,
you'll at least have to convince them that your idea is not plain braindead.
Isn't that obvious? And you'll have to accept others making a clear
statement like:

I wouldn't want to use backup software that can easily be misconfigured to
make compromises concerning data security in favour of speed.

You've made a convincing demonstration that it can already easily be
misconfigured to be slow, and you've also shown that some people would
try dubious things to speed it up, and, finally, you've shown that people
would blame the author for their mistakes. That should compell the author
to implement what you want, shouldn't it?

> Sure the way it works now fits to your usage doesnt mean that new features
> shouldnt be added.

I'm not saying it fits my usage. I'm saying the concepts make sense. And
you're not asking for new features, you're complaining that the impossible
has not been achieved yet. Right. You're only asking for lowering bandwidth
requirements at no extra cost. I wonder why nobody has thought of that
before you.

Regards,
Holger

Re: [BackupPC-users] very slow backup speed

[Evren Y. <yur...@is...>]

Holger Parplies wrote:

> Hi,
> 
> Evren Yurtesen wrote on 29.03.2007 at 17:04:05 [Re: [BackupPC-users] very slow backup speed]:
> 
>>Holger Parplies wrote:
>>
>>>"Most backup programs" take "do a full backup" to mean "read *everything*
>>>over the network and write it to backup medium". BackupPC modifies this to
>>>mean "read *everything* over the network and store *unchanged* contents
>>>efficiently by using hardlinks" while taking the notion of "unchanged" very
>>>seriously. If previous backups have become corrupted for some reason, that
>>>is no good reason to feel fine with storing unchanged contents in unchangedly
>>>[corrupted form]
>>
>>I am willing to take that risk, who are you to disagree? I am not asking that
>>checksum checking should be removed. I am just asking that people who want to 
>>take that risk should be able to disable it.
> 
> 
> well, go ahead and implement it. If you want other people to do it for you,
> you'll at least have to convince them that your idea is not plain braindead.
> Isn't that obvious? And you'll have to accept others making a clear
> statement like:
> 
> I wouldn't want to use backup software that can easily be misconfigured to
> make compromises concerning data security in favour of speed.


You are missing something here please read below (but you might want to stop 
using backuppc :D)

Right away when you enable 'tar' backuppc would be misconfigured by your 
definition, should tar support be removed? Tar can not check checksums anyway 
(plus has other flaws where it can miss newer/changed files)

> You've made a convincing demonstration that it can already easily be
> misconfigured to be slow, and you've also shown that some people would
> try dubious things to speed it up, and, finally, you've shown that people

I guess everybody who is using 'tar' have been doing 'dubious things' :)

> would blame the author for their mistakes. That should compell the author
> to implement what you want, shouldn't it?

I didnt blame anybody, just said BackupPC is working slow and it was working 
slow, very slow indeed. checksum-seeds option seems to be doing it's trick though.

I am thankful to people who wrote suggestions here in this forum, I tried all of 
those suggestions one by one. I think that shows that I took them seriously even 
though some of them looked like long shots. Eventually one of the suggestions 
seems to be working.

Not as fast as I would like but it is way better now and within acceptable 
levels (0.2mbyts/sec was really not acceptable).

>>Sure the way it works now fits to your usage doesnt mean that new features
>>shouldnt be added.
> 
> 
> I'm not saying it fits my usage. I'm saying the concepts make sense. And
> you're not asking for new features, you're complaining that the impossible
> has not been achieved yet. Right. You're only asking for lowering bandwidth

There has been a misunderstanding here. I was only talking about backuppc 
comparing checksums for deciding if the file should be backed up or not and it 
is not impossible to disable this, it is disabled when you use tar for example.

I have nothing against it using checksums when storing the file. (which would be 
impossible to disable)

> requirements at no extra cost. I wonder why nobody has thought of that
> before you.

Actually on the contrary, what I suggest might or might not increase/decrease 
bandwidth requirements depending on the situation while making backups 
(especially full backups) faster and use less cpu as there wont be checksum 
checks done for each and every file. That is why people are suggesting to use 
tar to get better speed even though it is flawed.

Even when this is disabled, the checksums are compared when any file with 
non-matching attribes is found. So it should be possible to find if there is a 
corruption in backed up data. After all when you enable checksum caching only 1% 
of the checksums in the backed up data is re-checked (by default) (this setting 
is what speeded up things for me, if you are using it, you must disable it for 
better file checks)

Now that it is ok to suggest people to use tar even though it has the same flaw 
and more but making something better than tar but perhaps slight worse than 
current rsync implementation in catching changed files is not a good idea? How come?

 From BackupPC Info:
-------------
For SMB and tar, BackupPC uses the modification time (mtime) to determine which 
files have changed since the last lower-level backup. That mean SMB and tar 
incrementals are not able to detect deleted files, renamed files or new files 
whose modification time is prior to the last lower-level backup.
-------------

Thanks,
Evren

Re: [BackupPC-users] very slow backup speed

[Evren Y. <yur...@is...>]

David Rees wrote:

> On 3/29/07, Evren Yurtesen <yur...@is...> wrote:
> 
>>I didnt blame anybody, just said BackupPC is working slow and it was working
>>slow, very slow indeed. checksum-seeds option seems to be doing it's trick though.
> 
> 
> How long are full and incremental backups taking now?

In one machine it went down from 900 minutes to 175 minutes. I expect better performance
when more memory is added (today or tomorrow they will add it) and I dont think all
files had checksums cached when this full was ran.

  Totals  	 Existing Files  	 New Files
Backup# 	Type 	#Files 	Size/MB 	MB/sec 	#Files 	Size/MB 	#Files 	Size/MB
245  	 full  	 280030  	 7570.4  	 0.14  	 274205  	 6797.6  	 10578  	 776.3
252  	 full  	 283960  	 8020.8  	 0.76  	 276665  	 6959.3  	 12232  	 1065.0

 	 Existing Files  	 New Files
Backup# 	Type 	Comp Level 	Size/MB 	Comp/MB 	Comp 	Size/MB 	Comp/MB 	Comp
245  	 full  	 9  	 6797.6  	 3868.9  	 43.1%  	 776.3  	 368.7  	 52.5%
252  	 full  	 9  	 6959.3  	 4056.9  	 41.7%  	 1065.0  	 539.0  	 49.4%

> 
>>I am thankful to people who wrote suggestions here in this forum, I tried all of
>>those suggestions one by one. I think that shows that I took them seriously even
>>though some of them looked like long shots. Eventually one of the suggestions
>>seems to be working.
> 
> 
> You only tried 2 things. Mounting the backup partition async and
> turning on checksum-seeds. Are you going to the 2 others? (Add memory
> and try tar instead of rsync)

The memory will be added. As I mentioned before the machine is at a remote location
and the guys there should add it.

I could try tar for testing purposes if you like? I think rsync will be sufficiently
fast enough. I am guessing that with checksum-seeds the difference shouldnt be so much
tar probably transfers much more data in full backups? Rsync can be faster perhaps if
ignore-times was removed when taking full backups. I am thinking of removing ignore-times
option from full backups with rsync and see how much it effects for seeing the difference.

> -Dave
> 
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> BackupPC-users mailing list
> Bac...@li...
> https://lists.sourceforge.net/lists/listinfo/backuppc-users
> http://backuppc.sourceforge.net/

Re: [BackupPC-users] very slow backup speed

[David R. <dr...@gm...>]

On 3/30/07, Evren Yurtesen <yur...@is...> wrote:
> David Rees wrote:
> > How long are full and incremental backups taking now?
>
> In one machine it went down from 900 minutes to 175 minutes. I expect better
> performance
> when more memory is added (today or tomorrow they will add it) and I dont think all
> files had checksums cached when this full was ran.

Wow, that is a huge difference! I didn't expect performance to
increase that much, apparently the checksum caching is really reducing
the number of disk IOPs.

> I could try tar for testing purposes if you like? I think rsync will be sufficiently
> fast enough. I am guessing that with checksum-seeds the difference shouldnt be so
> much
> tar probably transfers much more data in full backups? Rsync can be faster perhaps if
> ignore-times was removed when taking full backups. I am thinking of removing
> ignore-times
> option from full backups with rsync and see how much it effects for seeing the difference.

Tar is definitely worth a shot if it's short comings for incremental
backups are acceptable and network bandwidth isn't an issue.

Removing rsync ignore-times may also be an option if the reduction in
possible data integrity is acceptible.

-Dave

Re: [BackupPC-users] very slow backup speed

[David R. <dr...@gm...>]

On 3/27/07, Evren Yurtesen <yur...@is...> wrote:
> Yes it is terrible. I get much better performance if I do the tar option
> with the same files. As a matter of fact I was using a smaller script
> for taking backups earlier. (which I still use on some servers) and
> transfer files over NFS. It works way faster, especially incremental
> backups take 5-10 minutes compared to 400 minutes with backuppc

So have you tried using tar for backups using BackupPC?

I've said it before, there is something wrong with your setup because
on my server incrementals also only take 5-10 minutes as you'd expect.
And my server isn't that different than yours disk-wise, just RAID1
instead of no raid, it's even the exact same disk.

On 3/27/07, Evren Yurtesen <yur...@is...> wrote:
> David Rees wrote:
> That is true, full backups take about 500-600 minutes and incrementals
> take 200-300minutes.

Is that from all 4 clients being backed up? Are all similar in having
~150k files and ~2GB data to back up?

> > Your transfer rates are at least 5x slower than his and 10x slower
> > than what most people here are getting. There is something wrong
> > specifically with your setup. I suspect that if your backups were
> > going 5x faster you'd be happy and 10x faster you'd be very happy.
>
> I would be happy if backups were made with the same speed as other
> backup programs can do.

What other backup programs are you comparing BackupPC to?

> I am using rsync to sync 2GB of files from one machine to another and
> the number of files is about 100k and the operation takes 30seconds to 1
> minute in average.

Is this using the same backup server and client in your earlier data example?

> > Are you using the --checksum-seed option? That will significantly
> > speed up (the 3rd and later) backups as well.
>
> No, it requires a patched rsync.

You must have a very old version of rsync on your machine.
--checksum-seed has been available for quite some time in the official
rsync tree. It significantly improves rsync performance, I recommend
that you installed a recent rsync if possible. You need rsync 2.6.3
(released 30 Sep 2004) or later.

> > Are you also sure the backup server isn't swapping as well? Can you
> > get us some good logs from vmstat or similar during a backup?
>
> I can tell that it is not swapping because the disk where the swaps are
> idle while taking backup. ad0 is the disk with the swap. If there was
> such problem then it would be reading like crazy from swap. I have given
> this information before.

The information you provided before was just about impossible to read.
Please provide additional data in a readable format.

> The machine is working fine. I was using another backup program which
> was working way faster to backup the same machines. So I dont think that
> I have a hardware problem or such.

OK, so it is using the same machines?

> Earlier I sent the disk stats when the backup was working and I have
> pointed out that the disk with swap is not working at all while taking
> backup. We can easily conclude that swapping is not an issue.

Can you provide stats during the entire backup?

> > On all my production machines I make sure sysstat is running and then
> > run `sar -A` the minute before midnight and pipe the output to email
> > for analysis should the need arise.
>
> sar -A ?

That provides detailed system statistics collected at specified intervals.

Here's a sample from my backuppc server which has 3 disks while
backups are being run. The backuppc partition uses 2 disks in RAID1
exactly the same as yours. The other disk is the system disk (also
7200rpm ATA).
Hardware/OS: AthlonXP 2000+ 1GB RAM, Fedora Core 6. A max of 3 backups
will run at a time on this machine. This same data can be generated
using sar -b.

23:00:02          tps      rtps      wtps   bread/s   bwrtn/s
23:10:02       639.17    219.11    420.06   5440.01   8864.10
23:20:03       757.28    215.81    541.46   5422.98  11696.18
23:30:04       497.98    211.21    286.78   3367.85   5903.83
23:40:04       984.20    322.85    661.35   7436.95  14135.63
23:50:03       619.68    391.74    227.93  10355.83   5160.97

You can see how much higher this system's IO/s seems to be that yours
from the only performance data you sent us.

On 3/28/07, Evren Yurtesen <yur...@is...> wrote:
> Adam Goryachev wrote:
> > anyway, could you please elaborate on the method that your 'other'
> > backup solution used, ie, if your other solution used rsync, and you are
> > using rsync with backuppc, then that might be helpful to know. If you
> > used tar before, and now use rsync, that would obviously make a difference.
>
> I have already said it earlier, the other backup was taking backup over
> NFS. I also have machines were for example 2GB data with 100k files is
> synced to another using rsync, it takes 30-60 seconds to go through 100k
> files if there are not so many files to sync.

Same machines or different machines? Please be very exact when you
post information.

> I disagree, I have given all the information requested from me. I have
> tried different things like mounting filesystems with async even though
> I know that it is not the problem just to make you guys happy. Now you
> say that my attitude is not helping the resolution? What makes you say so?

The information you have been giving has been sparse and confusing. I
still feel like I am pulling teeth.

> Other people have been using it with dual-processor systems with 2-3GB
> of memory and raid setups. I would hardly consider this 'similar'
> conditions. BackupPC is very inefficent at handling files so you guys
> have to use very fast hardware solutions to speed it up. So you are
> covering up the problem from another side and say that there is no problem.

Others have posted in the same thread saying that they have much
slower hardware which runs faster than yours. One of my servers
appears to be similar to yours but it has 4x the memory and you still
haven't said what CPU is in your backuppc server.

BTW, I think BackupPC is very efficient at handling files.

> > I'd suggest you use whatever tools are available under your chosen OS (I
> > think it is freebsd), to measure:
> > 1) CPU utilisation on both the client being backed up, and the backuppc
> > server
>
> I have already given this information the machines are almost idle.

If you are using rsync, you should see a good spike in client IO when
backups start. You had not given the information before now.

> > 3) Disk utilisation/bandwidth on both client and server
>
> I have sent this information also. I didnt send it on client actually
> but server is almost idle diskwise. The main disk load is on the server.

The data you sent for your server is lacking. More detail is needed.

> > Also, if you are using tar as your transfer method, then I'd suggest you
> > try a 'manual' backup something like this (note, this command line is
> > wrong, but you will need to adjust to get the right one).
>
> This is a good idea, I will try it later and return back to you. (even
> though I am using rsync)

Good.

> > Of course, if you are using rsync, then do the same test with rsync
> > data, but I'm sure it has already been said that you should use tar for
> > your case with lots of small files.
> >
> > Just my 0.02c worth....
>
> I will do the test with both actually, I will copy all files from the
> client to server using both rsync and tar and return back the results.

Good. Please provide detailed information of your tests that is easy
to read. Also provide detailed system load information as well.

You earlier said you were going to increase memory on the server, are
you still planning on doing that?

Perhaps it would be helpful if you summarized all your data in one email.

What types of files are being backed up? I wonder if there is
something about them which is triggering this performance issue...

http://backuppc.sourceforge.net/faq/BackupPC.html#some_design_issues

If a lot of files are ending up under the same pool directory, that
could definitely trigger some bad behavior, but given what we've seen
of your workload, I would expect a large number of identical files to
be backed up.

What is the largest directory under your pool/cpool directory? Are
there any directories which have files with a lot of underscores in
them?

-Dave

Re: [BackupPC-users] very slow backup speed

[Les M. <les...@gm...>]

Carl Wilhelm Soderstrom wrote:

>>> I would really like to see hard drives made to be more reliable, rather than
>>> just bigger.
>> I'm not sure that can be improved enough to matter.  A failure of an 
>> inexpensive part once in five years isn't a big deal other than the side 
>> effects it might cause, and unless they can be 100% reliable (probably 
>> impossible) you have to be prepared for those side effects anyway.
> 
> on a small scale, perhaps. 
> when you get to an organization the size of (TBS|Hotmail|Yahoo)tho; it may
> make sense to spend a little more money in order to spend less labor
> swapping parts.

I think before you get to that that scale you'd be network booting 
diskless and mostly disposable motherboards.  You still have to deal 
with the drives somewhere, but using larger ones instead of large 
numbers of them.

> the market seems to agree with you tho... which is a pretty good indictment
> of central planning notions (i.e. Me) vs. what really is efficient.

I find complete-system redundancy with commodity boxes to be cheaper and 
more efficient than using military strength parts that cost 10x as much 
and fail half as often.  In our operations the most common failure is in 
software anyway since we try to push out new capabilities as fast as 
possible so I like lots of redundancy and the ability to run multiple 
versions of things concurrently.  I could see where that would be 
different for operations where everything had to be in a single 
database, though.

>>  > I want reliable storage.
>> Run mirrored hot-swap drives. If one dies, replace it at some convenient 
>> time, sync it up and keep going.  I have one machine last booted in 
>> mid-2003 that has had a couple of it's drives replaced that way.
> 
> Your point is well-taken. I do keep in mind tho that I've seen multiple
> drives fail simultaneously or in rapid succession, and that the process of
> replacing drives costs time and effort.

Yes, your building might catch fire or be flooded too. Those are 
probably more likely than multiple cheap drives failing simultaneously 
from some cause that wouldn't also kill better drives. You need offsite 
backups to cover these unlikely events anyway.

> It is not as trivial as you might
> think, once you factor in the time to detect the failure, find the
> replacement (if you can), replace the drive (which may involve removing the
> old drive from the carrier and replacing it with the new one), and make sure
> it's working properly. In an ideal world it's 5 minutes of work, in the real
> world I usually expect to lose an hour or two dealing with a failed drive.

The real killer is the time planning out the replacement operation so 
the only time wasted is the one person who does the work.  In the 
whole-system scenario where the systems are load balanced anyway, 
yanking a machine out isn't anything special and you don't even need the 
hot-swap drive case.

 > This can be accounted as several hundred dollars of lost
 > productivity in many cases; so it's worthwhile to spend more money
 > on a better-quality drive sometimes.

In practice I don't see any relationship between price and reliability. 
  I'm inclined to think that they all come off the same production line 
these days.

-- 
   Les Mikesell
    le...@fu...

[BackupPC-users] drive quality (was: Re: very slow backup speed)

[Carl W. S. <ch...@re...>]

On 03/27 02:49 , Les Mikesell wrote:
> I find complete-system redundancy with commodity boxes to be cheaper and 
> more efficient than using military strength parts that cost 10x as much 
> and fail half as often.  

That's Google's solution, and it works for them. Nothing fancy... rebooting
servers means a dude with a broomstick walking around pushing power buttons.

> The real killer is the time planning out the replacement operation so 
> the only time wasted is the one person who does the work.  In the 
> whole-system scenario where the systems are load balanced anyway, 
> yanking a machine out isn't anything special and you don't even need the 
> hot-swap drive case.

it's a nice ideal circumstance. :)
In practice I've found that clustering systems often increases
administrative headaches exponentially. There's a lot more that *can* go
wrong and therefore *does* go wrong with a cluster. There's a place for
them, like everything else... but they are far from a panacea.

> In practice I don't see any relationship between price and reliability. 
>   I'm inclined to think that they all come off the same production line 
> these days.

some are still better than others... how much better, I don't know. I've
seen some really expensive hardware go belly-up at times too.

Here's what I did with some of the aforementioned bad & expensive hardware:
http://www.redchrome.org/shooting_computers/shooting_computers_index.html
(the Sparc 10, specifically)

-- 
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

Re: [BackupPC-users] very slow backup speed

[Les M. <le...@fu...>]

Carl Wilhelm Soderstrom wrote:

> I would really like to see hard drives made to be more reliable, rather than
> just bigger.

I'm not sure that can be improved enough to matter.  A failure of an 
inexpensive part once in five years isn't a big deal other than the side 
effects it might cause, and unless they can be 100% reliable (probably 
impossible) you have to be prepared for those side effects anyway.

 > I want reliable storage.

Run mirrored hot-swap drives. If one dies, replace it at some convenient 
time, sync it up and keep going.  I have one machine last booted in 
mid-2003 that has had a couple of it's drives replaced that way.

-- 
   Les Mikesell
    les...@gm...

Re: [BackupPC-users] very slow backup speed

[Carl W. S. <ch...@re...>]

On 03/27 01:40 , Les Mikesell wrote:
> Carl Wilhelm Soderstrom wrote:
> > I would really like to see hard drives made to be more reliable, rather than
> > just bigger.
> I'm not sure that can be improved enough to matter.  A failure of an 
> inexpensive part once in five years isn't a big deal other than the side 
> effects it might cause, and unless they can be 100% reliable (probably 
> impossible) you have to be prepared for those side effects anyway.

on a small scale, perhaps. 
when you get to an organization the size of (TBS|Hotmail|Yahoo)tho; it may
make sense to spend a little more money in order to spend less labor
swapping parts.

the market seems to agree with you tho... which is a pretty good indictment
of central planning notions (i.e. Me) vs. what really is efficient.

>  > I want reliable storage.
> Run mirrored hot-swap drives. If one dies, replace it at some convenient 
> time, sync it up and keep going.  I have one machine last booted in 
> mid-2003 that has had a couple of it's drives replaced that way.

Your point is well-taken. I do keep in mind tho that I've seen multiple
drives fail simultaneously or in rapid succession, and that the process of
replacing drives costs time and effort. It is not as trivial as you might
think, once you factor in the time to detect the failure, find the
replacement (if you can), replace the drive (which may involve removing the
old drive from the carrier and replacing it with the new one), and make sure
it's working properly. In an ideal world it's 5 minutes of work, in the real
world I usually expect to lose an hour or two dealing with a failed drive.
This can be accounted as several hundred dollars of lost productivity in
many cases; so it's worthwhile to spend more money on a better-quality drive
sometimes.

-- 
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com

Re: [BackupPC-users] very slow backup speed

[John P. <jp...@cl...>]

Evren Yurtesen wrote:
>
>
> I know that the bottleneck is the disk. I am using a single ide disk to 
> take the backups, only 4 machines and 2 backups running at a time(if I 
> am not remembering wrong).
>
> I see that it is possible to use raid to solve this problem to some 
> extent but the real solution is to change backuppc in such way that it 
> wont use so much disk operations.
>
>   


 From what I can tell the issue is that each file requires a hard link - 
depending on your file system metadata like directory entries, had links 
etc get treated differently that regular data - on a BSD ufs2 system 
metadata updates are typically synchronous, that is the system doesn't 
return until the write has made it to the disk.   This is good for 
reliability but really bad for performance since it prevents out of 
order writes which can save a lot of disk activity.   

Changing backuppc would be decidedly non-trivial - eyeballing it to hack 
in a real database to store the relationship between pool and individual 
files would touch almost just about every part of the system.

What filesystem are you using and have you turned off atime - I found 
that makes a big difference.

John

Re: [BackupPC-users] very slow backup speed

[Evren Y. <yur...@is...>]

John Pettitt wrote:
> Evren Yurtesen wrote:
>>
>>
>> I know that the bottleneck is the disk. I am using a single ide disk 
>> to take the backups, only 4 machines and 2 backups running at a 
>> time(if I am not remembering wrong).
>>
>> I see that it is possible to use raid to solve this problem to some 
>> extent but the real solution is to change backuppc in such way that it 
>> wont use so much disk operations.
>>
>>   
> 
> 
>  From what I can tell the issue is that each file requires a hard link - 
> depending on your file system metadata like directory entries, had links 
> etc get treated differently that regular data - on a BSD ufs2 system 
> metadata updates are typically synchronous, that is the system doesn't 
> return until the write has made it to the disk.   This is good for 
> reliability but really bad for performance since it prevents out of 
> order writes which can save a lot of disk activity.  
> Changing backuppc would be decidedly non-trivial - eyeballing it to hack 
> in a real database to store the relationship between pool and individual 
> files would touch almost just about every part of the system.
> 
> What filesystem are you using and have you turned off atime - I found 
> that makes a big difference.
> 
> John

I have noatime, I will try bumping up the memory and hope that the 
caching will help. I will let you know if it helps.

Thanks,
Evren

Re: [BackupPC-users] very slow backup speed

[Les M. <les...@gm...>]

John Pettitt wrote:

> Changing backuppc would be decidedly non-trivial - eyeballing it to hack 
> in a real database to store the relationship between pool and individual 
> files would touch almost just about every part of the system.

And there's not much reason to think that a database could do this with 
atomic updates any better than the filesystem it sits on.

-- 
   Les Mikesell
    les...@gm...

Re: [BackupPC-users] very slow backup speed

[Jason H. <jh...@st...>]

Evren Yurtesen wrote:
>> And, you could consider buying a faster drive, or one with a larger 
>> buffer.  Some IDE drives have pathetically small buffers and slow 
>> rotation rates.  That makes for a greater need for seeking, and worse 
>> seek performance.
>
> Well this is a seagate barracuda 7200rpm drive with 8mb cache ST3250824A
> http://www.seagate.com/support/disc/manuals/ata/100389997c.pdf
>
> Perhaps it is not the maximum amount of cache one can have on a drive 
> but it is not that bad really.

That drive should be more than adequate.  Mine is a 5400rpm 2mb buffer 
clunker.  Works fine. 

Are you running anything else on the backup server, besides BackupPC?  
What OS?  What filesystem?  How many files total?

> I read your posts about wifi etc. on forum. The processor is not the 
> problem however adding memory probably might help bufferwise. I think 
> this idea can actually work.:) thanks! I am seeing swapping problems 
> but the disk the swap is on is almost idle. The backup drive is 
> working all the time.

Hmm.  That's a separate disk, not a separate partition of the same disk, 
right?  If it's just a separate partition, I'm not sure how well the OS 
will be able to allocate wait states to logical devices sharing the same 
physical media... in other words, what looks like waiting on ad2 may be 
waiting on ad0.  Someone more familiar with device drivers and linux 
internals would have chime in here.  I'm not an expert.

>
> I have to say that slow performance with BackupPC is a known problem. 
> I have heard it from several other people who are using BackupPC and 
> it is the #1 reason of changing to another backup program from what I 
> hear.
>
> Things must improve on this area.
>

I did quite a lot of research and found only one other program that was 
near my needs, and it was substantially slower due to encryption 
overhead, and didn't have a central pool to combine backup data.  I may 
have missed an app out there, though.  What are these people switching 
to, if you don't mind?

Re: what must improve is more people helping Craig.  He's doing it all 
for free.  I think if it's important enough to have fixed, it's 
important enough to pay for.  Or dive into the code and start making 
those changes.  It is open source, after all.

My $.02,
JH

Re: [BackupPC-users] very slow backup speed

[Evren Y. <yur...@is...>]

Jason Hughes wrote:
> Evren Yurtesen wrote:
>>> And, you could consider buying a faster drive, or one with a larger 
>>> buffer.  Some IDE drives have pathetically small buffers and slow 
>>> rotation rates.  That makes for a greater need for seeking, and worse 
>>> seek performance.
>>
>> Well this is a seagate barracuda 7200rpm drive with 8mb cache ST3250824A
>> http://www.seagate.com/support/disc/manuals/ata/100389997c.pdf
>>
>> Perhaps it is not the maximum amount of cache one can have on a drive 
>> but it is not that bad really.
> 
> That drive should be more than adequate.  Mine is a 5400rpm 2mb buffer 
> clunker.  Works fine.
> Are you running anything else on the backup server, besides BackupPC?  
> What OS?  What filesystem?  How many files total?

FreeBSD, UFS2+softupdates, noatime.

There are 4 hosts that have been backed up, for a total of:

     * 16 full backups of total size 72.16GB (prior to pooling and 
compression),
     * 24 incr backups of total size 13.45GB (prior to pooling and 
compression).

# Pool is 17.08GB comprising 760528 files and 4369 directories (as of 
3/27 05:54),
# Pool hashing gives 38 repeated files with longest chain 6,
# Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54),
# Pool file system was recently at 10% (3/27 07:16), today's max is 10% 
(3/27 01:00) and yesterday's max was 10%.

  Host  	 User  	 #Full  	 Full Age (days)  	 Full Size (GB)  	 Speed 
(MB/s)  	 #Incr  	 Incr Age (days)  	 Last Backup (days)  	 State  	 
Last attempt
host1	 		4 	5.4 	3.88 	0.22 	6 	0.4 	0.4 	idle 	idle
host2	 		4 	5.4 	2.10 	0.06 	6 	0.4 	0.4 	idle 	idle
host3	 		4 	5.4 	7.57 	0.14 	6 	0.4 	0.4 	idle 	idle
host4	 		4 	5.4 	5.56 	0.10 	6 	0.4 	0.4 	idle 	idle


>> I read your posts about wifi etc. on forum. The processor is not the 
>> problem however adding memory probably might help bufferwise. I think 
>> this idea can actually work.:) thanks! I am seeing swapping problems 
>> but the disk the swap is on is almost idle. The backup drive is 
>> working all the time.
> 
> Hmm.  That's a separate disk, not a separate partition of the same disk, 
> right?  If it's just a separate partition, I'm not sure how well the OS 
> will be able to allocate wait states to logical devices sharing the same 
> physical media... in other words, what looks like waiting on ad2 may be 
> waiting on ad0.  Someone more familiar with device drivers and linux 
> internals would have chime in here.  I'm not an expert.

It is a separate disk. The disk is on FreeBSD not Linux. They are not 
waiting for each other, they can be used simultaneously.

>>
>> I have to say that slow performance with BackupPC is a known problem. 
>> I have heard it from several other people who are using BackupPC and 
>> it is the #1 reason of changing to another backup program from what I 
>> hear.
>>
>> Things must improve on this area.
>>
> 
> I did quite a lot of research and found only one other program that was 
> near my needs, and it was substantially slower due to encryption 
> overhead, and didn't have a central pool to combine backup data.  I may 
> have missed an app out there, though.  What are these people switching 
> to, if you don't mind?
> 
> Re: what must improve is more people helping Craig.  He's doing it all 
> for free.  I think if it's important enough to have fixed, it's 
> important enough to pay for.  Or dive into the code and start making 
> those changes.  It is open source, after all.

I think we are already helping already by discussing the issue. Even if 
we wanted to pay, there is nothing to pay yet, as there is no agreed 
solution to this slowness.

Thanks,
Evren

Re: [BackupPC-users] very slow backup speed

[Jason H. <jh...@st...>]

Evren Yurtesen wrote:
> Jason Hughes wrote:
>> That drive should be more than adequate.  Mine is a 5400rpm 2mb 
>> buffer clunker.  Works fine.
>> Are you running anything else on the backup server, besides 
>> BackupPC?  What OS?  What filesystem?  How many files total?
>
> FreeBSD, UFS2+softupdates, noatime.
>
> There are 4 hosts that have been backed up, for a total of:
>
>     * 16 full backups of total size 72.16GB (prior to pooling and 
> compression),
>     * 24 incr backups of total size 13.45GB (prior to pooling and 
> compression).
>
> # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 
> 3/27 05:54),
> # Pool hashing gives 38 repeated files with longest chain 6,
> # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54),
> # Pool file system was recently at 10% (3/27 07:16), today's max is 
> 10% (3/27 01:00) and yesterday's max was 10%.
>
>  Host       User       #Full       Full Age (days)       Full Size 
> (GB)       Speed (MB/s)       #Incr       Incr Age (days)       Last 
> Backup (days)       State       Last attempt
> host1             4     5.4     3.88     0.22     6     0.4     
> 0.4     idle     idle
> host2             4     5.4     2.10     0.06     6     0.4     
> 0.4     idle     idle
> host3             4     5.4     7.57     0.14     6     0.4     
> 0.4     idle     idle
> host4             4     5.4     5.56     0.10     6     0.4     
> 0.4     idle     idle
>

Hmm.  This is a tiny backup setup, even smaller than mine.  However, it 
appears that the average size of your file is only 22KB, which is quite 
small.  For comparison sake, this is from my own server:
    Pool is 172.91GB comprising 217311 files and 4369 directories (as of 
3/26 01:08),

The fact that you have tons of little files will probably give 
significantly higher overhead when doing file-oriented work, simply 
because the inodes must be fetched for each file before seeking to the 
file itself.  If we assume no files are shared between hosts (very 
conservative), and you have an 8ms access time, you will have 190132 
files per host and two seeks per file, neglecting actual i/o time, gives 
you 50 minutes.  Just to seek them all.  If you have a high degree of 
sharing, it can be up to 4x worse.  Realize, the same number of seeks 
must be made on the server as well as the client.

Are you sure you need to be backing up everything that you're putting 
across the network?  Maybe excluding some useless directories, maybe 
temp files or logs that haven't been cleaned up?  Perhaps you can 
archive big chunks of it with a cron job?

I'd start looking for ways to cut down the number of files, because the 
overhead of per-file accesses are probably eating you alive.  I'm also 
no expert on UFS2 or FreeBSD, so it may be worthwhile to research its 
behavior with hard links and small files.

JH

Re: [BackupPC-users] very slow backup speed

[brien d. <bri...@cg...>]

Jason Hughes wrote:
> Evren Yurtesen wrote:
>   
>> Jason Hughes wrote:
>>     
>>> That drive should be more than adequate.  Mine is a 5400rpm 2mb 
>>> buffer clunker.  Works fine.
>>> Are you running anything else on the backup server, besides 
>>> BackupPC?  What OS?  What filesystem?  How many files total?
>>>       
>> FreeBSD, UFS2+softupdates, noatime.
>>
>> There are 4 hosts that have been backed up, for a total of:
>>
>>     * 16 full backups of total size 72.16GB (prior to pooling and 
>> compression),
>>     * 24 incr backups of total size 13.45GB (prior to pooling and 
>> compression).
>>
>> # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 
>> 3/27 05:54),
>> # Pool hashing gives 38 repeated files with longest chain 6,
>> # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54),
>> # Pool file system was recently at 10% (3/27 07:16), today's max is 
>> 10% (3/27 01:00) and yesterday's max was 10%.
>>
>>  Host       User       #Full       Full Age (days)       Full Size 
>> (GB)       Speed (MB/s)       #Incr       Incr Age (days)       Last 
>> Backup (days)       State       Last attempt
>> host1             4     5.4     3.88     0.22     6     0.4     
>> 0.4     idle     idle
>> host2             4     5.4     2.10     0.06     6     0.4     
>> 0.4     idle     idle
>> host3             4     5.4     7.57     0.14     6     0.4     
>> 0.4     idle     idle
>> host4             4     5.4     5.56     0.10     6     0.4     
>> 0.4     idle     idle
>>
>>     
>
> Hmm.  This is a tiny backup setup, even smaller than mine.  However, it 
> appears that the average size of your file is only 22KB, which is quite 
> small.  For comparison sake, this is from my own server:
>     Pool is 172.91GB comprising 217311 files and 4369 directories (as of 
> 3/26 01:08),
>
> The fact that you have tons of little files will probably give 
> significantly higher overhead when doing file-oriented work, simply 
> because the inodes must be fetched for each file before seeking to the 
> file itself.  If we assume no files are shared between hosts (very 
> conservative), and you have an 8ms access time, you will have 190132 
> files per host and two seeks per file, neglecting actual i/o time, gives 
> you 50 minutes.  Just to seek them all.  If you have a high degree of 
> sharing, it can be up to 4x worse.  Realize, the same number of seeks 
> must be made on the server as well as the client.
>
> Are you sure you need to be backing up everything that you're putting 
> across the network?  Maybe excluding some useless directories, maybe 
> temp files or logs that haven't been cleaned up?  Perhaps you can 
> archive big chunks of it with a cron job?
>
> I'd start looking for ways to cut down the number of files, because the 
> overhead of per-file accesses are probably eating you alive.  I'm also 
> no expert on UFS2 or FreeBSD, so it may be worthwhile to research its 
> behavior with hard links and small files.
>
> JH
>
>   
For what it's worth, I have a server that backs up 8.6 million files  
averaging 10k in size from one host.  It takes a full 10 hours for a 
full backup via tar over NFS ( 2.40MB/s for 87GB). CPU usage is low, 
around 10-20%, however IOwait is a pretty steady 25%.

Server info:
HP DL380 G4
debian sarge
dual processor 3.2ghz xeon
2GB Ram
5x10k rpm scsi disks, raid5
128MB battery backed cache (50/50 r/w)
ext3 filesystems

brien

Re: [BackupPC-users] very slow backup speed

[Evren Y. <yur...@is...>]

brien dieterle wrote:
> Jason Hughes wrote:
>> Evren Yurtesen wrote:
>>   
>>> Jason Hughes wrote:
>>>     
>>>> That drive should be more than adequate.  Mine is a 5400rpm 2mb 
>>>> buffer clunker.  Works fine.
>>>> Are you running anything else on the backup server, besides 
>>>> BackupPC?  What OS?  What filesystem?  How many files total?
>>>>       
>>> FreeBSD, UFS2+softupdates, noatime.
>>>
>>> There are 4 hosts that have been backed up, for a total of:
>>>
>>>     * 16 full backups of total size 72.16GB (prior to pooling and 
>>> compression),
>>>     * 24 incr backups of total size 13.45GB (prior to pooling and 
>>> compression).
>>>
>>> # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 
>>> 3/27 05:54),
>>> # Pool hashing gives 38 repeated files with longest chain 6,
>>> # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54),
>>> # Pool file system was recently at 10% (3/27 07:16), today's max is 
>>> 10% (3/27 01:00) and yesterday's max was 10%.
>>>
>>>  Host       User       #Full       Full Age (days)       Full Size 
>>> (GB)       Speed (MB/s)       #Incr       Incr Age (days)       Last 
>>> Backup (days)       State       Last attempt
>>> host1             4     5.4     3.88     0.22     6     0.4     
>>> 0.4     idle     idle
>>> host2             4     5.4     2.10     0.06     6     0.4     
>>> 0.4     idle     idle
>>> host3             4     5.4     7.57     0.14     6     0.4     
>>> 0.4     idle     idle
>>> host4             4     5.4     5.56     0.10     6     0.4     
>>> 0.4     idle     idle
>>>
>>>     
>> Hmm.  This is a tiny backup setup, even smaller than mine.  However, it 
>> appears that the average size of your file is only 22KB, which is quite 
>> small.  For comparison sake, this is from my own server:
>>     Pool is 172.91GB comprising 217311 files and 4369 directories (as of 
>> 3/26 01:08),
>>
>> The fact that you have tons of little files will probably give 
>> significantly higher overhead when doing file-oriented work, simply 
>> because the inodes must be fetched for each file before seeking to the 
>> file itself.  If we assume no files are shared between hosts (very 
>> conservative), and you have an 8ms access time, you will have 190132 
>> files per host and two seeks per file, neglecting actual i/o time, gives 
>> you 50 minutes.  Just to seek them all.  If you have a high degree of 
>> sharing, it can be up to 4x worse.  Realize, the same number of seeks 
>> must be made on the server as well as the client.
>>
>> Are you sure you need to be backing up everything that you're putting 
>> across the network?  Maybe excluding some useless directories, maybe 
>> temp files or logs that haven't been cleaned up?  Perhaps you can 
>> archive big chunks of it with a cron job?
>>
>> I'd start looking for ways to cut down the number of files, because the 
>> overhead of per-file accesses are probably eating you alive.  I'm also 
>> no expert on UFS2 or FreeBSD, so it may be worthwhile to research its 
>> behavior with hard links and small files.
>>
>> JH
>>
>>   
> For what it's worth, I have a server that backs up 8.6 million files  
> averaging 10k in size from one host.  It takes a full 10 hours for a 
> full backup via tar over NFS ( 2.40MB/s for 87GB). CPU usage is low, 
> around 10-20%, however IOwait is a pretty steady 25%.
> 
> Server info:
> HP DL380 G4
> debian sarge
> dual processor 3.2ghz xeon
> 2GB Ram
> 5x10k rpm scsi disks, raid5
> 128MB battery backed cache (50/50 r/w)
> ext3 filesystems
> 
> brien

You are distributing the reads and writes on 5 disks here. Dont you 
think that 2.40mb/s is a little bit slow compared to the horsepower you 
have?

If you scale it down to my system, I think 1/10 performance is normal...

Thanks,
Evren

Re: [BackupPC-users] very slow backup speed

[Les M. <le...@fu...>]

Evren Yurtesen wrote:

>> Server info:
>> HP DL380 G4
>> debian sarge
>> dual processor 3.2ghz xeon
>> 2GB Ram
>> 5x10k rpm scsi disks, raid5
>> 128MB battery backed cache (50/50 r/w)
>> ext3 filesystems
>>
>> brien
> 
> You are distributing the reads and writes on 5 disks here. Dont you 
> think that 2.40mb/s is a little bit slow compared to the horsepower you 
> have?

Raid5 doesn't distribute disk activity - it puts the drives in lockstep 
and is slower than a single drive, especially on small writes where it 
has to do extra reads to re-compute parity on the existing data.

> If you scale it down to my system, I think 1/10 performance is normal...

Its the RAM and disk cache that makes this system work.

-- 
   Les Mikesell
    les...@gm...

Re: [BackupPC-users] very slow backup speed

[Les M. <le...@fu...>]

Evren Yurtesen wrote:
>>
>> Raid5 doesn't distribute disk activity - it puts the drives in 
>> lockstep and is slower than a single drive, especially on small writes 
>> where it has to do extra reads to re-compute parity on the existing data.
> 
> I am confused, when a write is done the data is distributed in the disks 
> depending on the stripe size you are using. When you start reading the 
> file, you are reading from 5 different disks. So you get way better 
> performance for sure on reads.

The stripe effect only comes into play on files large enough to span 
them and not at all for directory/inode accesses which is most of what 
you are doing. Meanwhile you have another head tied up checking the 
parity and for writes of less than a block you have to read the existing 
contents before the write to re-compute the parity.

-- 
   Les Mikesell
    le...@fu...

Re: [BackupPC-users] very slow backup speed

[Evren Y. <yur...@is...>]

Les Mikesell wrote:
> Evren Yurtesen wrote:
> 
>>> Server info:
>>> HP DL380 G4
>>> debian sarge
>>> dual processor 3.2ghz xeon
>>> 2GB Ram
>>> 5x10k rpm scsi disks, raid5
>>> 128MB battery backed cache (50/50 r/w)
>>> ext3 filesystems
>>>
>>> brien
>>
>> You are distributing the reads and writes on 5 disks here. Dont you 
>> think that 2.40mb/s is a little bit slow compared to the horsepower 
>> you have?
> 
> Raid5 doesn't distribute disk activity - it puts the drives in lockstep 
> and is slower than a single drive, especially on small writes where it 
> has to do extra reads to re-compute parity on the existing data.

I am confused, when a write is done the data is distributed in the disks 
depending on the stripe size you are using. When you start reading the 
file, you are reading from 5 different disks. So you get way better 
performance for sure on reads.

Agreed, writes might be slower depending on the raid controller, amount 
of cache it has etc. but my problem here is the reads. In my machine 
BackupPC server is working slow even when it is backing up a very small 
amount of files.

>> If you scale it down to my system, I think 1/10 performance is normal...
> 
> Its the RAM and disk cache that makes this system work.
> 

Agreed, I will add more RAM and report the results.

Thanks,
Evren

Re: [BackupPC-users] very slow backup speed

[Craig B. <cba...@us...>]

Evren writes:

>   Host  	 User  	 #Full  	 Full Age (days)  	 Full Size (GB)  	 Speed=20
> (MB/s)  	 #Incr  	 Incr Age (days)  	 Last Backup (days)  	 State  	=20
> Last attempt
> host1	 		4 	5.4 	3.88 	0.22 	6 	0.4 	0.4 	idle 	idle
> host2	 		4 	5.4 	2.10 	0.06 	6 	0.4 	0.4 	idle 	idle
> host3	 		4 	5.4 	7.57 	0.14 	6 	0.4 	0.4 	idle 	idle
> host4	 		4 	5.4 	5.56 	0.10 	6 	0.4 	0.4 	idle 	idle

These xfer rates are certainly very low.

I don't think I caught which XferMethod you are using.  Also,
what is your BackupPC version?  If you are using 3.0, what is
the value of $Conf{IncrLevels}?

Craig