BackupPC

and 31.21 GB instead of 31214312331231 bytes!!!!!!!!!!!!!!
Ability to email a list of files backed up per host "wood" me kool too.....

I know, I know......


On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio <flavio@...:

> Hello people,
>
> I hope developers will read this and take it into account if possible: I
> am not used to consider minutes in decimal format (like 36.8 minutes).
> Would it be possible to convert that data into time format (like
> 36m48sec) and extend the same thing to hours (not anymore 242.8minutes,
> but instead 4hours 2mins 48sec)?
>
> Thanks,
> Flavio Boniforti
>
> PIRAMIDE INFORMATICA SAGL
> Via Ballerini 21
> 6600 Locarno
> Switzerland
> Phone: +41 91 751 68 81
> Fax: +41 91 751 69 14
> URL: http://www.piramide.ch
> E-mail: flavio@...
>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables
> unlimited royalty-free distribution of the report engine
> for externally facing server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>

Hi,

Bharat Mistry wrote on 2009-05-15 15:57:57 +0100 [Re: [BackupPC-users] [SUGGESTION] "Duration/mins" not in decimal format]:
> and 31.21 GB instead of 31214312331231 bytes!!!!!!!!!!!!!!

("." instead of "!!!!!!!!!!!!!!", too? :)

> Ability to email a list of files backed up per host "wood" me kool too.....

thank you for making this point (though I don't suppose you *wanted* to make
it).

> On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio <flavio@...:
> 
> > I am not used to consider minutes in decimal format (like 36.8 minutes).

I don't think you are supposed to. The point of the web page, as I understand
it, is to give you a rough idea of what is going on. Seeing a list of figures
36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or
whatever) stand out much more than if you obfuscate it into 4h2m48s. A list
like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an
impression of how the times are developping than 36m6s, 36m12s, ... would.
If you want more than a rough idea (or rather, if you have better use of your
time than staring at the statistics), you'll try to automatically process the
numbers, and parsing "4h2m48s" into something you can calculate with will be
nothing less than a nuisance (and, yes, if you're backuppc on the BackupPC
server, you can parse the backups files, but if you only have HTTP access to
one client's host page, you can't). Similarly, you won't enjoy the reduced
precision of 31.21 GB. If you are generating an email summary, for instance,
you can always convert the numbers to whatever format you want, and it's still
easier to convert 242.8 minutes than to translate "4hours 2mins 48secs" to a
different language or shift the whitespace around to match your taste.

> > Would it be possible to convert that data into time format (like
> > 36m48sec) and extend the same thing to hours (not anymore 242.8minutes,
> > but instead 4hours 2mins 48sec)?

Yes, but who's the target audience? Are you saying you *need* to know more
than "my full backups take somewhere between 4 and 5 hours"? If it's less than
4h17m25s it's ok, but if it's more, you'll need to speed it up somehow?

As for the seconds, I'd argue to rather drop them(*). They're almost certainly 
below the exactness of the measurement (well, yes, the backup *did* take 36
minutes and 13.7 seconds, but that the next backup took 36 minutes and 50
seconds probably tells you more about the state of the machines and the link
at that time than about the backup itself in relation to the other one).
Measure with a micrometer, mark with chalk, cut with an axe.

Regards,
Holger

(*) Well, no, keep them. They don't hurt as long as they're just decimal
    minutes :).

I agree totally with Holger. It is much easier to convert raw data to less
precise and/or more humanly readable formats than vica-versa. Plus,
the data is usually something you only look at when troubleshooting or
analyzing performance, not something that is of interest to the casual
user.

If you want an alternative more lay person format, I'm sure you could
rather easily develop another web page that suits your purposes and
contribute it back to the project. In fact, if you start by thinking
of your audience and purpose, the ideal summary page for the lay
person is likely to have many other changes beyond just time and
storage formats.

Holger Parplies wrote at about 21:18:54 +0200 on Friday, May 15, 2009:
 > Hi,
 > 
 > Bharat Mistry wrote on 2009-05-15 15:57:57 +0100 [Re: [BackupPC-users] [SUGGESTION] "Duration/mins" not in decimal format]:
 > > and 31.21 GB instead of 31214312331231 bytes!!!!!!!!!!!!!!
 > 
 > ("." instead of "!!!!!!!!!!!!!!", too? :)
 > 
 > > Ability to email a list of files backed up per host "wood" me kool too.....
 > 
 > thank you for making this point (though I don't suppose you *wanted* to make
 > it).
 > 
 > > On Fri, May 15, 2009 at 7:39 AM, Boniforti Flavio <flavio@...:
 > > 
 > > > I am not used to consider minutes in decimal format (like 36.8 minutes).
 > 
 > I don't think you are supposed to. The point of the web page, as I understand
 > it, is to give you a rough idea of what is going on. Seeing a list of figures
 > 36.8, 37.1, 35.9, 36.4, 242.8, 37.3 ... makes the full backup (or problem, or
 > whatever) stand out much more than if you obfuscate it into 4h2m48s. A list
 > like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... gives you much more of an
 > impression of how the times are developping than 36m6s, 36m12s, ... would.
 > If you want more than a rough idea (or rather, if you have better use of your
 > time than staring at the statistics), you'll try to automatically process the
 > numbers, and parsing "4h2m48s" into something you can calculate with will be
 > nothing less than a nuisance (and, yes, if you're backuppc on the BackupPC
 > server, you can parse the backups files, but if you only have HTTP access to
 > one client's host page, you can't). Similarly, you won't enjoy the reduced
 > precision of 31.21 GB. If you are generating an email summary, for instance,
 > you can always convert the numbers to whatever format you want, and it's still
 > easier to convert 242.8 minutes than to translate "4hours 2mins 48secs" to a
 > different language or shift the whitespace around to match your taste.
 > 
 > > > Would it be possible to convert that data into time format (like
 > > > 36m48sec) and extend the same thing to hours (not anymore 242.8minutes,
 > > > but instead 4hours 2mins 48sec)?
 > 
 > Yes, but who's the target audience? Are you saying you *need* to know more
 > than "my full backups take somewhere between 4 and 5 hours"? If it's less than
 > 4h17m25s it's ok, but if it's more, you'll need to speed it up somehow?
 > 
 > As for the seconds, I'd argue to rather drop them(*). They're almost certainly 
 > below the exactness of the measurement (well, yes, the backup *did* take 36
 > minutes and 13.7 seconds, but that the next backup took 36 minutes and 50
 > seconds probably tells you more about the state of the machines and the link
 > at that time than about the backup itself in relation to the other one).
 > Measure with a micrometer, mark with chalk, cut with an axe.
 > 
 > Regards,
 > Holger
 > 
 > (*) Well, no, keep them. They don't hurt as long as they're just decimal
 >     minutes :).
 > 
 > ------------------------------------------------------------------------------
 > Crystal Reports - New Free Runtime and 30 Day Trial
 > Check out the new simplified licensing option that enables 
 > unlimited royalty-free distribution of the report engine 
 > for externally facing server and web deployment. 
 > http://p.sf.net/sfu/businessobjects
 > _______________________________________________
 > BackupPC-users mailing list
 > BackupPC-users@...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    http://backuppc.wiki.sourceforge.net
 > Project: http://backuppc.sourceforge.net/
 >

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeffrey J. Kosowsky wrote:
> I agree totally with Holger. It is much easier to convert raw data to less
> precise and/or more humanly readable formats than vica-versa. Plus,
> the data is usually something you only look at when troubleshooting or
> analyzing performance, not something that is of interest to the casual
> user.
>
> If you want an alternative more lay person format, I'm sure you could
> rather easily develop another web page that suits your purposes and
> contribute it back to the project. In fact, if you start by thinking
> of your audience and purpose, the ideal summary page for the lay
> person is likely to have many other changes beyond just time and
> storage formats.
Exactly, perhaps the a better (but more work involved) solution would
be to create a new page which shows a nice pretty graph of the various
numbers instead of stacks of numbers in great big tables...

In fact, I've got one installation og backuppc which now has over 800
backups of each host, and it can take quite a while to view the host
pages. As time goes on, and the number of backups increases, this will
just get worse, again, it would be nice to simply have a few graphs
(or a limit to the number of backups displayed (maybe last 20 backups
etc).

However, while we can all make suggestions of improvements, if we
cared enough we would provide a patch, and if other people cared
enough, it would be included into backuppc..... I don't have the time,
and I don't care enough about the above, and so I haven't *done*
anything about it.

So thanks for a great product, it is good enough for me to not need to
put loads of work into it, and I can just use it to do useful things.

Regards,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoOA5gACgkQGyoxogrTyiV2jgCeOCgRWvzTWWBgp730qBc7oY1K
TKIAoJpLuHbMZbOe9Ymw6mFUh/n+JR2c
=nOiV
-----END PGP SIGNATURE-----

> > > I am not used to consider minutes in decimal format (like 
> 36.8 minutes).
> 
> I don't think you are supposed to. The point of the web page, 
> as I understand it, is to give you a rough idea of what is 
> going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 
> 242.8, 37.3 ... makes the full backup (or problem, or
> whatever) stand out much more than if you obfuscate it into 
> 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... 
> gives you much more of an impression of how the times are 
> developping than 36m6s, 36m12s, ... would.

It's simply about "human format": humans do not speak about 36.8
minutes. But as you also pointed out: if I'd cut the seconds, nothing
would change in such an important way: 36.8 or 36 is the same thing
because I'm not interested into "seconds"...

The main goal I'm seeking is to get *real* values of time and
transferred amount of data (which I adressed in my other thread).

Regards,
F.

> If you want an alternative more lay person format, I'm sure 
> you could rather easily develop another web page that suits 
> your purposes and contribute it back to the project. In fact, 
> if you start by thinking of your audience and purpose, the 
> ideal summary page for the lay person is likely to have many 
> other changes beyond just time and storage formats.

Of course would I be doing this, once I have the technical skills to
achieve my goals. At this point in time, where I'm not enough skilled
for creating a "customized" web page for my view, I'm just asking how to
make it...

Cheers,
F.

> Exactly, perhaps the a better (but more work involved) 
> solution would be to create a new page which shows a nice 
> pretty graph of the various numbers instead of stacks of 
> numbers in great big tables...

Well, if somebody is willing to cooperate with me to do it, I'll be
willing to learn and contribute.
A *very nice* add-on would be having graphs for each host, not only for
the whole pool. I'd really love to see ho my single hosts evolve in
relation to GBs occupied. Also, I'd like to see how transfer rates may
be varying...

Regards,
F.

Boniforti Flavio wrote:
>> Exactly, perhaps the a better (but more work involved) 
>> solution would be to create a new page which shows a nice 
>> pretty graph of the various numbers instead of stacks of 
>> numbers in great big tables...
> 
> Well, if somebody is willing to cooperate with me to do it, I'll be
> willing to learn and contribute.
> A *very nice* add-on would be having graphs for each host, not only for
> the whole pool. I'd really love to see ho my single hosts evolve in
> relation to GBs occupied. Also, I'd like to see how transfer rates may
> be varying...

I guess you could track the transfer times and sizes for each 
host/share, but there is a philosophical/practical issue in tracking the 
storage space since it is pooled and there is no handy way to tell 
which, if any, other hosts have links to a common file.  In terms of 
real space consumed, all of your targets can have multiple copies of 
some large file and it will barely take any more room than one single 
copy on one host.

-- 
   Les Mikesell
    lesmikesell@...

Boniforti Flavio wrote:
>>>> I am not used to consider minutes in decimal format (like 
>> 36.8 minutes).
>>
>> I don't think you are supposed to. The point of the web page, 
>> as I understand it, is to give you a rough idea of what is 
>> going on. Seeing a list of figures 36.8, 37.1, 35.9, 36.4, 
>> 242.8, 37.3 ... makes the full backup (or problem, or
>> whatever) stand out much more than if you obfuscate it into 
>> 4h2m48s. A list like 36.1, 36.1, 36.2, 36.3, 36.7, 36.8 ... 
>> gives you much more of an impression of how the times are 
>> developping than 36m6s, 36m12s, ... would.
> 
> It's simply about "human format": humans do not speak about 36.8
> minutes.

_Except_ when they are trying to compare different amounts of time, when 
the discrete jumps in days/hours/minutes/seconds become inconvenient to 
normal math operations.


-- 
   Les Mikesell
    lesmikesell@...

> I guess you could track the transfer times and sizes for each 
> host/share, but there is a philosophical/practical issue in 
> tracking the storage space since it is pooled and there is no 
> handy way to tell which, if any, other hosts have links to a 
> common file.  In terms of real space consumed, all of your 
> targets can have multiple copies of some large file and it 
> will barely take any more room than one single copy on one host.

Yes, I was indeed thinking about the same thing too: because of
"pooling" it's somehow unprecise defining the effective space used.
Therefore I will be assuming that the data showing in the "Host Summary"
is the space consumed by each single host.

But is there any way to know how much data has been transferred from the
start of the backup process until its end? 

Thanks,
F.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Les Mikesell wrote:
> I guess you could track the transfer times and sizes for each 
> host/share, but there is a philosophical/practical issue in tracking the 
> storage space since it is pooled and there is no handy way to tell 
> which, if any, other hosts have links to a common file.  In terms of 
> real space consumed, all of your targets can have multiple copies of 
> some large file and it will barely take any more room than one single 
> copy on one host.

Consider that from a cost perspective, it is imprecise to charge
internet usage (ie downloads) which includes data downloaded from the
proxy server, since that data may (or may not) have been downloaded by
multiple people, and therefore has minimal real cost for the 'copies'.

However, we still *do* charge based on the traffic on the interface to
the users connection... Regardless of what pooling, or optimisation
might be done in the network, cloud, backuppc system, there is still an
'argument' that says the user is responsible for the total amount of
space consumed by a single host regardless of whether any other host
also shares the same space....

In any case, I think what our colleague is asking for is how much
transit bandwidth did a host consume during the backup process, and this
has nothing at all to do with pooling. The bad news is, (AFAIK) that
this data is not collected within backuppc, and would need a different
implementation for each transfer method. The best suggestion I could
make would be to measure this at the network interface of your backuppc
host. ie, the simplest method to track bandwidth consumption for rsyncd
transfers is to add an iptables allow rule for traffic to your client
host on port 873 (or whatever the correct ip + port is)...

Though, it would be nice if these stats could be collected by backuppc,
and stored in some clearly defined file, whereby some other tool could
easily collect the data and present it in whatever format is desired...

PS, for clarity, I would also like to see the bandwidth consumption, and
a statistic for the amount of disk space a backup consumes without
regard to any shared files with other hosts... Though I think this last
one can be done with "du -sm /var/lib/backuppc/pc/host/124"...

Regards,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoRbvcACgkQGyoxogrTyiWXZQCgqQq3sZ8oIC44QtGYc6XXP3Z/
SVEAn220PHng6Uvb9oj3NFg6BhUmsD0W
=ewyG
-----END PGP SIGNATURE-----

> In any case, I think what our colleague is asking for is how 
> much transit bandwidth did a host consume during the backup 
> process, and this has nothing at all to do with pooling. The 

Indeed, that's what I want to know: the really transferred bytes for
that host-to-host connection (from the remote host to my backuppc
server). Nothing concerning the backuppc pool...

> bad news is, (AFAIK) that this data is not collected within 
> backuppc, and would need a different implementation for each 
> transfer method. The best suggestion I could make would be to 
> measure this at the network interface of your backuppc host. 
> ie, the simplest method to track bandwidth consumption for 
> rsyncd transfers is to add an iptables allow rule for traffic 
> to your client host on port 873 (or whatever the correct ip + 
> port is)...

Well, if I'd achieve something like this, I guess I would be collecting
*every single bit* (also commands issued and stuff like that).
For my purpose, it would be sufficient to know that I transferred a
total amount of 14 files, which sum up to 1034896 bytes. I tried to look
at the log files, but I don't understand how to distinguish between the
words like "pool", "same", "skip", "create"... If I could assume that
all the "create" ones are really transferred bytes, I'd be summing them
up with a simple bash script. But what about "pool" ones (I just imagine
that "skip" and "same" are really NO TRANSFERS)?

> Though, it would be nice if these stats could be collected by 
> backuppc, and stored in some clearly defined file, whereby 
> some other tool could easily collect the data and present it 
> in whatever format is desired...

Regards,
F.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Boniforti Flavio wrote:
>> bad news is, (AFAIK) that this data is not collected within 
>> backuppc, and would need a different implementation for each 
>> transfer method. The best suggestion I could make would be to 
>> measure this at the network interface of your backuppc host. 
>> ie, the simplest method to track bandwidth consumption for 
>> rsyncd transfers is to add an iptables allow rule for traffic 
>> to your client host on port 873 (or whatever the correct ip + 
>> port is)...
> 
> Well, if I'd achieve something like this, I guess I would be collecting
> *every single bit* (also commands issued and stuff like that).
> For my purpose, it would be sufficient to know that I transferred a
> total amount of 14 files, which sum up to 1034896 bytes. I tried to look
> at the log files, but I don't understand how to distinguish between the
> words like "pool", "same", "skip", "create"... If I could assume that
> all the "create" ones are really transferred bytes, I'd be summing them
> up with a simple bash script. But what about "pool" ones (I just imagine
> that "skip" and "same" are really NO TRANSFERS)?

Also, when using rsyncd, although the file is different, the entire file
may not have been transferred. ie, a 100MB file might only transfer 5MB
which was modified from yesterday's backup....

In addition, when using rsync+ssh, if you are using compression, then
again, the actual data over the network will be less than the size of
the files transferred (probably also correct for tar+ssh with
compression, etc).

So all of these are estimations, and whether backuppc can internally
even know what the correct settings are/should be, is possibly (probably
as I think about it) impossible.

A simply pre/post script which sets up the iptables entry, and then
records the result and deletes the entry would probably be the quickest
solution for a couple of simple bash scripts.....

Still, this doesn't address the meanings of the various bits of data
which *are* kept by backuppc. Did someone manage to find the
documentation on the meaning of the data recorded in the backups files?

Regards,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoRgAoACgkQGyoxogrTyiXCKwCcD9eR1JNCzOvnQE9JVdxYNcyZ
FXQAn14DU0ediRfcfAXd6Ka7xoTb+8ze
=QOCY
-----END PGP SIGNATURE-----

Boniforti Flavio wrote:
>> I guess you could track the transfer times and sizes for each 
>> host/share, but there is a philosophical/practical issue in 
>> tracking the storage space since it is pooled and there is no 
>> handy way to tell which, if any, other hosts have links to a 
>> common file.  In terms of real space consumed, all of your 
>> targets can have multiple copies of some large file and it 
>> will barely take any more room than one single copy on one host.
> 
> Yes, I was indeed thinking about the same thing too: because of
> "pooling" it's somehow unprecise defining the effective space used.
> Therefore I will be assuming that the data showing in the "Host Summary"
> is the space consumed by each single host.
> 
> But is there any way to know how much data has been transferred from the
> start of the backup process until its end? 

I don't know about the accuracy of the numbers, but I always assumed 
that the duration (*60)times the MB/sec for the same backup run would 
give me the bytes transferred.  But if you are looking for bottlenecks 
you might need to note that rsync incrementals take quite a bit of 
wall-clock time even though they may transfer a tiny amount of data.

For non-rsync xfers, the transfer size should be the same as the file 
size, with everything except the 'new' files discarded after the 
transfer and replace will pool links.

In all cases, this would be skewed if you add the '-C' (compression) 
option to the ssh command since that would happen before backuppc sees 
the data.


-- 
   Les Mikesell
     lesmikesell@...

Il 18.05.09 17:34, "Adam Goryachev" <mailinglists@...> ha
scritto:

> So all of these are estimations, and whether backuppc can internally
> even know what the correct settings are/should be, is possibly (probably
> as I think about it) impossible.

You are quite right... I didn't want to put that stuff in our thread, but it
*has* to be mentioned that rsync + ssh (and YES I am using -C compression!)
does meanfully change the amount of data transferred.
 
> A simply pre/post script which sets up the iptables entry, and then
> records the result and deletes the entry would probably be the quickest
> solution for a couple of simple bash scripts.....

Do you already have some sort of practical suggestion? I'm not used to use
iptables, I stopped playing with that piece of software some 6 years ago :-/
But indeed, I think that would be the only way to do some sort of
"accounting" or "billing" of bandwidth usage...
 
> Still, this doesn't address the meanings of the various bits of data
> which *are* kept by backuppc. Did someone manage to find the
> documentation on the meaning of the data recorded in the backups files?

If you talk about the same things I am clueless about (skip, create, pool,
same, and so on), then I'd be *too* interested in some precise definitions.

Regards,
F.

Il 18.05.09 18:14, "Les Mikesell" <lesmikesell@...> ha scritto:

> In all cases, this would be skewed if you add the '-C' (compression)
> option to the ssh command since that would happen before backuppc sees
> the data.

And that's my case :-/
Do you also think that the best approach is the iptables one?

Boniforti Flavio wrote:
> 
> 
> Il 18.05.09 18:14, "Les Mikesell" <lesmikesell@...> ha scritto:
> 
>> In all cases, this would be skewed if you add the '-C' (compression)
>> option to the ssh command since that would happen before backuppc sees
>> the data.
> 
> And that's my case :-/
> Do you also think that the best approach is the iptables one?

I think the best approach is to have enough bandwidth that you don't 
need to worry about it - or at least can get by with tracking it at the 
interface or switch port level with an snmp tool like cacti or opennms. 
  If you need to count bytes between machines, ntop might work but it is 
a fairly big load on the server.  It is a handy thing to have around 
when you suspect some machines have viruses or just want to see the 
bandwidth hogs - http://www.ntop.org.

-- 
   Les Mikesell
    lesmikesell@...

Hi,

Boniforti Flavio wrote on 2009-05-18 22:14:50 +0200 [Re: [BackupPC-users] [SUGGESTION] "Duration/mins" not in decimal format]:
> Il 18.05.09 17:34, "Adam Goryachev" <mailinglists@...> ha
> scritto:
> [...]

the question I believe should have been asked long ago is: What problem are
you trying to solve? Is there even one, or are you just trying to make
use/sense of numbers that are displayed somewhere?

Depending on whether you want to entertain bored users, bill your clients
for bandwidth, or conduct scientific measurements, the answers are likely to
be vastly different.

> You are quite right... I didn't want to put that stuff in our thread, but it
> *has* to be mentioned that rsync + ssh (and YES I am using -C compression!)
> does meanfully change the amount of data transferred.

Depending on what you want, that may or may not be relevant. You can just as
well bill your clients for your disk wear (and CPU usage), regardless of the
bandwidth savings ssh compression and rsync algorithm gain you. In fact,
bandwidth savings will probably not mean that you can do more concurrent
backups.

> > A simply pre/post script which sets up the iptables entry, and then
> > records the result and deletes the entry would probably be the quickest
> > solution for a couple of simple bash scripts.....
> 
> Do you already have some sort of practical suggestion?

iptables -I INPUT -s client_addr -d backuppc_server_addr -p tcp --sport 22

(and delete it with the same rule with "-D" instead of "-I"). Supposing you
only have one concurrent backup to one host and no other ssh usage. You might
prefer to count outgoing traffic (well, no, but maybe incoming + outgoing).
Note that the rule has no target - it's only for accounting.

> > Still, this doesn't address the meanings of the various bits of data
> > which *are* kept by backuppc. Did someone manage to find the
> > documentation on the meaning of the data recorded in the backups files?

Use the force, read the source :)

> If you talk about the same things I am clueless about (skip, create, pool,
> same, and so on), then I'd be *too* interested in some precise definitions.

I can give you some imprecise ones.

same - rsync determined that file matches and does not need to be transferred
pool - transferred file matched one already in the pool
create - transferred file did not match any existing pool file

To be honest, I think they're all rather self-explaining. I didn't do any
research on the answers, it's just what figures from observation. In
particular, on my "import backup from local source" quest, I was really
worried when I got lots of "pool" lines on the first remote backup where it
should have read "same". That explained rather well why the backup was taking
ages.

But, again, this is a *completely* different topic from counting bytes with
iptables. You should first find out what you want, then ask for help finding a
solution. Discussing all possible problems might be something people enjoy
that have an abundance of free time. These people should spend more time on
the BackupPC wiki ;-).

Regards,
Holger

> Depending on whether you want to entertain bored users, bill 
> your clients for bandwidth, or conduct scientific 
> measurements, the answers are likely to be vastly different.

I need to give my customers statistics about how much data has been
transferred each month, which means summing up day-by-day the
transferred amount of data.
For the HDD space usage, I simply rely on what my "Host Summary" page is
telling me (Full Size).

[cut]

> > Do you already have some sort of practical suggestion?
> 
> iptables -I INPUT -s client_addr -d backuppc_server_addr -p 
> tcp --sport 22
> 
> (and delete it with the same rule with "-D" instead of "-I"). 
> Supposing you only have one concurrent backup to one host and 
> no other ssh usage. You might prefer to count outgoing 
> traffic (well, no, but maybe incoming + outgoing).
> Note that the rule has no target - it's only for accounting.

Well, I'm having concurrent backups, but they use different TCP ports,
thus I can "--sport 8873" and "--sport 8874" and so on for my clients.

> > If you talk about the same things I am clueless about 
> (skip, create, 
> > pool, same, and so on), then I'd be *too* interested in 
> some precise definitions.
> 
> I can give you some imprecise ones.
> 
> same - rsync determined that file matches and does not need 
> to be transferred pool - transferred file matched one already 
> in the pool create - transferred file did not match any 
> existing pool file

What I interpreted was that "same" and "skip" have the same meaning:
file is not getting transferred. Why then using *two* words to define a
seamingly identical behaviour?
The "create" was clear to me, but the "pool" one not so clear: I had in
"Incr Backup 16" a "create" statement for a big file (2.7GB). The
subsequent backup (Incr Backup 17) the same file (which has *not* been
changed in any way, because it's a static ZIP file that was added before
backup 16) was indicated as "pool": would this be meaning that the file
had been transferred again and only after being transferred, BackupPC
recognized that it was already in the pool (and matching)? This is the
part I don't understand actually...

> To be honest, I think they're all rather self-explaining. I 
> didn't do any research on the answers, it's just what figures 
> from observation. In particular, on my "import backup from 
> local source" quest, I was really worried when I got lots of 
> "pool" lines on the first remote backup where it should have 
> read "same". That explained rather well why the backup was 
> taking ages.

So your case was the same as mine above? Are you saying that "the backup
was taking ages" because it was re-transferring your data?

Thanks for helping,
F.

Hi,

Boniforti Flavio wrote on 2009-05-19 08:53:31 +0200 [RE: [BackupPC-users] [SUGGESTION] "Duration/mins" not in decimal format]:
> > Depending on whether you want to entertain bored users, bill 
> > your clients for bandwidth, or conduct scientific 
> > measurements, the answers are likely to be vastly different.
> 
> I need to give my customers statistics about how much data has been
> transferred each month, which means summing up day-by-day the
> transferred amount of data.

so it's the "entertain bored users" case :-). Your definition leaves room for
interpretation. For instance, if that number is more readily available, you
could sum up the uncompressed data streams. On the other hand, you could leave
iptables accounting rules in place all the time and just read out the counters
(and zero them) once a month (assuming your BackupPC server doesn't reboot).

Aside from that, you will probably be counting in GB, not in bytes.

> > > Do you already have some sort of practical suggestion?
> > 
> > iptables -I INPUT -s client_addr -d backuppc_server_addr -p 
> > tcp --sport 22
> > [...]
> Well, I'm having concurrent backups, but they use different TCP ports,
> thus I can "--sport 8873" and "--sport 8874" and so on for my clients.

Even better. Those ports will not be used for maintainance, I suppose? Even if
so, I guess counting that traffic wouldn't strictly be wrong ...

> What I interpreted was that "same" and "skip" have the same meaning:
> file is not getting transferred. Why then using *two* words to define a
> seamingly identical behaviour?

I can't actually find "skip" in my XferLOGs. Probably because it only appears
with logLevel >= 2 (at least for rsync). Strange. You always seem to find
issues which, when looking at the code, disappear. There is one issue though:
your logLevel is set too high (unless you are actually tracking a problem,
which, in my experience, is not the case). I bet an 'ls -l' of your pc/
directories doesn't show as nicely which backups are full and which are
incremental. With mine it's really obvious from the XferLOG files.

skip - unchanged file skipped in incremental
same - file that would normally have been transferred (full backup or
       attribute change) turns out to be identical to reference file,
       no transfer needed

You obviously won't get "same" for tar/smb backups (nor will you get "skip",
because the files are skipped by the sender without notice to BackupPC).

> The "create" was clear to me, but the "pool" one not so clear: I had in
> "Incr Backup 16" a "create" statement for a big file (2.7GB). The
> subsequent backup (Incr Backup 17) the same file (which has *not* been
> changed in any way, because it's a static ZIP file that was added before
> backup 16) was indicated as "pool": would this be meaning that the file
> had been transferred again and only after being transferred, BackupPC
> recognized that it was already in the pool (and matching)? This is the
> part I don't understand actually...

By default, that is normal. Read about how incremental backups work, in
particular, which backups they are based on.

There are, of course, cases where this would not be normal, but I've run out
of motivation for going into details about hypothetical problems.

> > [...] In particular, on my "import backup from 
> > local source" quest, I was really worried when I got lots of 
> > "pool" lines on the first remote backup where it should have 
> > read "same". That explained rather well why the backup was 
> > taking ages.
> 
> So your case was the same as mine above?

No.

> Are you saying that "the backup
> was taking ages" because it was re-transferring your data?

Yes.

Regards,
Holger

Hallo Holger,

> > I need to give my customers statistics about how much data has been 
> > transferred each month, which means summing up day-by-day the 
> > transferred amount of data.
> 
> so it's the "entertain bored users" case :-). Your definition 
> leaves room for interpretation. For instance, if that number 
> is more readily available, you could sum up the uncompressed 
> data streams. On the other hand, you could leave iptables 
> accounting rules in place all the time and just read out the 
> counters (and zero them) once a month (assuming your BackupPC 
> server doesn't reboot).

I *really* think I'll be collecting iptables data...

> > Well, I'm having concurrent backups, but they use different 
> TCP ports, 
> > thus I can "--sport 8873" and "--sport 8874" and so on for 
> my clients.
> 
> Even better. Those ports will not be used for maintainance, I 
> suppose? Even if so, I guess counting that traffic wouldn't 
> strictly be wrong ...

Nothing else than ssh-tunnelled BackupPC dumps.

> > What I interpreted was that "same" and "skip" have the same meaning:
> > file is not getting transferred. Why then using *two* words 
> to define 
> > a seamingly identical behaviour?
> 
> I can't actually find "skip" in my XferLOGs. Probably because 
> it only appears with logLevel >= 2 (at least for rsync). 

Indeed I'm using XferLogLevel = 2
Skip actually only appears in "Incr" XFerLogs, which it seems then to be
substituted by "same" in "Full" backups...

> Strange. You always seem to find issues which, when looking 
> at the code, disappear. There is one issue though:
> your logLevel is set too high (unless you are actually 
> tracking a problem, which, in my experience, is not the 
> case). I bet an 'ls -l' of your pc/ directories doesn't show 
> as nicely which backups are full and which are incremental. 
> With mine it's really obvious from the XferLOG files.

I actually put XferLogLevel back to 1, let's see what will happen...

> By default, that is normal. Read about how incremental 
> backups work, in particular, which backups they are based on.

Will read about it in the BackupPC Documentation

> > Are you saying that "the backup
> > was taking ages" because it was re-transferring your data?
> 
> Yes.

OK, still a bit confused... Will eventually come back on this issue
later on...

Regards and thanks for your time.

F.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Boniforti Flavio wrote:
> Hallo Holger,
> 
>>> I need to give my customers statistics about how much data has been 
>>> transferred each month, which means summing up day-by-day the 
>>> transferred amount of data.
>> so it's the "entertain bored users" case :-). Your definition 
>> leaves room for interpretation. For instance, if that number 
>> is more readily available, you could sum up the uncompressed 
>> data streams. On the other hand, you could leave iptables 
>> accounting rules in place all the time and just read out the 
>> counters (and zero them) once a month (assuming your BackupPC 
>> server doesn't reboot).
> 
> I *really* think I'll be collecting iptables data...
> 
>>> Well, I'm having concurrent backups, but they use different 
>> TCP ports, 
>>> thus I can "--sport 8873" and "--sport 8874" and so on for 
>> my clients.
>>
>> Even better. Those ports will not be used for maintainance, I 
>> suppose? Even if so, I guess counting that traffic wouldn't 
>> strictly be wrong ...
> 
> Nothing else than ssh-tunnelled BackupPC dumps.
> 
>>> What I interpreted was that "same" and "skip" have the same meaning:
>>> file is not getting transferred. Why then using *two* words 
>> to define 
>>> a seamingly identical behaviour?
>> I can't actually find "skip" in my XferLOGs. Probably because 
>> it only appears with logLevel >= 2 (at least for rsync). 
> 
>>> Are you saying that "the backup
>>> was taking ages" because it was re-transferring your data?
>> Yes.
> 
> OK, still a bit confused... Will eventually come back on this issue
> later on...

- From my reading of the mailing list, and usage of backuppc, I think it
works something like this:

A full backup (level 0) will backup all files, using rsync, only
modified portions of existing files, and new files are transferred.

An incremental backup (level 1 - 9) will backup all new files, and
modified files since the last backup of a lower level.

In older versions, (I think 2.x) there was only one "level" of
incremental backups, hence all incremental backups would re-transfer all
modified/new data compared to the last full backup. Even if the file has
not changed since the last incremental backup. This would explain what
you saw, the file was transferred again for each incremental backup
(slow backup times) but after transfer, backuppc decided a matching file
already existed in the pool, and so it was discarded and linked to the
pool file.

In newer versions, it is possible to configure (optional) different
levels of backup (1 - 9). Thus, if you set IncrLevels = [1,2,3,4] and
you have four incrementals between each full backup, then you will never
re-transfer a file which rsync could know has not changed since the most
recent backup unless the filename/path has changed. However, this
increases the cpu of the backuppc because it needs to merge the full
backup plus up to 3 incremental backups to complete the fourth incremental.

I think you should find more definitive (and correct) documentation in
the changelog or documentation if you search for IncrLevel/IncrLevels
for where this feature was introduced.

I also recall (but could be wrong) that the next version of backuppc
would also (possibly) apply the same logic for a second backup of the
SAME level. In effect, IncrLevels = [1,2,3,4] would become the same as
IncrLevels = [1,1,1,1] (which is the same as IncrLevels = [1]).

To me, especially with rsync/rsyncd backups, this seemed to make a lot
of sense, though I can't comment on the effects this might have if using
smb/tar/ftp/etc...

So, in short, this is why I currently use IncrLevels = [1,2,3] and do
full backups after every 3rd incremental.... If anyone thinks the above
is grossly incorrect, please feel free to correct me before someone
follows my totally wrong random ramblings... If anyone wants to
implement or follow the above, I strongly suggest you find corroborating
evidence elsewhere first.

Regards,
Adam
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkoS0bwACgkQGyoxogrTyiXhsACguJxxmVd7pkX2zlHWmxgUfgA8
OjAAnAhNxHMZx6fOG8R2kaNpZmTixPmR
=vkZT
-----END PGP SIGNATURE-----

Boniforti Flavio wrote:
>> Depending on whether you want to entertain bored users, bill 
>> your clients for bandwidth, or conduct scientific 
>> measurements, the answers are likely to be vastly different.
> 
> I need to give my customers statistics about how much data has been
> transferred each month, which means summing up day-by-day the
> transferred amount of data.
> For the HDD space usage, I simply rely on what my "Host Summary" page is
> telling me (Full Size).
> 
> [cut]
> 
>>> Do you already have some sort of practical suggestion?
>> iptables -I INPUT -s client_addr -d backuppc_server_addr -p 
>> tcp --sport 22
>>
>> (and delete it with the same rule with "-D" instead of "-I"). 
>> Supposing you only have one concurrent backup to one host and 
>> no other ssh usage. You might prefer to count outgoing 
>> traffic (well, no, but maybe incoming + outgoing).
>> Note that the rule has no target - it's only for accounting.
> 
> Well, I'm having concurrent backups, but they use different TCP ports,
> thus I can "--sport 8873" and "--sport 8874" and so on for my clients.

If you get this working, let us know how far off it is from the values 
shown for duration and transfer rate (and a 10-25% allowance for ssh 
compression).

-- 
   Les Mikesell
    lesmikesell@...

I installed BackupPC from the Ubuntu repositories several days ago with
the intent of getting it set up and running as soon as a HDD came in.
Well last night I was reading thru some of the doc and took a look at
the control webpage and much to my surprise saw that a host was there
(localhost) and had done something... but also had an error about it's
name not matching netbios name.  I changed it's name and deleted the
backup files under /var/lib/backuppc/pc/localhost.

After some more reading I change 5 things from the default install:

1) changed default method to rsync
2) changed logs kept from 14 to 9
3) changed the host name "localhost" to "c17-desktop"
4) checked the box override box called "FixedIPNetBiosNameCheck"
5) changed blackout period to be 7.5 thru 18.5

Then requested a full backup on c17.

It returned a new error on the host summary page:
c17-desktop skip 0 0.00 0   idle backup failed (Unable to read 4 bytes)

This is the machine that BackupPC was installed on.  I guess getting
that machine backed up is as good as anyplace to start.

I also, via the web interface attempted to exclude some directories and
file types which ended up creating /etc/backuppc/c17-desktop.pl that
contains:

$Conf{FixedIPNetBiosNameCheck} = '0';
$Conf{BackupFilesExclude} = {
  '*.old' => [],
  '/var/lib/backuppc' => [],
  '/tmp' => [],
  '/lost+found' => [],
  '/proc' => [],
  '/cdrom' => [],
  '/mnt' => []
};

After saving these change I attempted another full backup and got the
same error: backup failed (Unable to read 4 bytes)

So, questions:

1) Is it advisable/legit to use rsync on the local machine backing up
itself?

2) Do my directory and file exclusions make any sense for backing up the
machine that BackupPC is on?

3) How can I find out more about this error I'm getting?

PS:  HDD just arrived so I'll get the RAID 1 set up tonight or tomorrow
and figure out how to point BackupPC to that.... think I read there's a
topDIR variable someplace.

I'm guessing the error is a ssh problem from this:

full backup started for directory /
Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
Xfer PIDs are now 1896
Read EOF: Connection reset by peer
Tried again: got 0 bytes
Done: 0 files, 0 bytes
Got fatal error during xfer (Unable to read 4 bytes)
Backup aborted (Unable to read 4 bytes)
Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)

I'm ASSUMING it doesn't like ssh'ing into itself??



On Wed, 2009-05-20 at 21:56 -0500, Skip Guenter wrote:
> I installed BackupPC from the Ubuntu repositories several days ago with
> the intent of getting it set up and running as soon as a HDD came in.
> Well last night I was reading thru some of the doc and took a look at
> the control webpage and much to my surprise saw that a host was there
> (localhost) and had done something... but also had an error about it's
> name not matching netbios name.  I changed it's name and deleted the
> backup files under /var/lib/backuppc/pc/localhost.
> 
> After some more reading I change 5 things from the default install:
> 
> 1) changed default method to rsync
> 2) changed logs kept from 14 to 9
> 3) changed the host name "localhost" to "c17-desktop"
> 4) checked the box override box called "FixedIPNetBiosNameCheck"
> 5) changed blackout period to be 7.5 thru 18.5
> 
> Then requested a full backup on c17.
> 
> It returned a new error on the host summary page:
> c17-desktop skip 0 0.00 0   idle backup failed (Unable to read 4 bytes)
> 
> This is the machine that BackupPC was installed on.  I guess getting
> that machine backed up is as good as anyplace to start.
> 
> I also, via the web interface attempted to exclude some directories and
> file types which ended up creating /etc/backuppc/c17-desktop.pl that
> contains:
> 
> $Conf{FixedIPNetBiosNameCheck} = '0';
> $Conf{BackupFilesExclude} = {
>   '*.old' => [],
>   '/var/lib/backuppc' => [],
>   '/tmp' => [],
>   '/lost+found' => [],
>   '/proc' => [],
>   '/cdrom' => [],
>   '/mnt' => []
> };
> 
> After saving these change I attempted another full backup and got the
> same error: backup failed (Unable to read 4 bytes)
> 
> So, questions:
> 
> 1) Is it advisable/legit to use rsync on the local machine backing up
> itself?
> 
> 2) Do my directory and file exclusions make any sense for backing up the
> machine that BackupPC is on?
> 
> 3) How can I find out more about this error I'm getting?
> 
> PS:  HDD just arrived so I'll get the RAID 1 set up tonight or tomorrow
> and figure out how to point BackupPC to that.... think I read there's a
> topDIR variable someplace.
> 
> 
> 
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, & 
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/

Skip Guenter wrote:
> I'm guessing the error is a ssh problem from this:
> 
> full backup started for directory /
> Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
> Xfer PIDs are now 1896
> Read EOF: Connection reset by peer
> Tried again: got 0 bytes
> Done: 0 files, 0 bytes
> Got fatal error during xfer (Unable to read 4 bytes)
> Backup aborted (Unable to read 4 bytes)
> Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)
> 
> I'm ASSUMING it doesn't like ssh'ing into itself??
>

There's nothing inherently different about sshing to the local host. 
Did you set up the ssh keys needed to make it work without passwords?

-- 
   Les Mikesell
    lesmikesell@...

Duh, no.  Thanx much.


On Wed, 2009-05-20 at 23:29 -0500, Les Mikesell wrote:
> Skip Guenter wrote:
> > I'm guessing the error is a ssh problem from this:
> > 
> > full backup started for directory /
> > Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
> > Xfer PIDs are now 1896
> > Read EOF: Connection reset by peer
> > Tried again: got 0 bytes
> > Done: 0 files, 0 bytes
> > Got fatal error during xfer (Unable to read 4 bytes)
> > Backup aborted (Unable to read 4 bytes)
> > Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)
> > 
> > I'm ASSUMING it doesn't like ssh'ing into itself??
> >
> 
> There's nothing inherently different about sshing to the local host. 
> Did you set up the ssh keys needed to make it work without passwords?
>

Skip Guenter wrote at about 03:25:53 -0500 on Thursday, May 21, 2009:
 > Duh, no.  Thanx much.

I think you may be confusing "no passwords" with no security. I assume
the respondent (whom you answered remarkably rudely) meant did you set
ssh up to use passkeys rather than passwords. As long as you keep the
passkey secure, the system security will be infinitely stronger using
RSA/DSA passkeys than a simple plaintext password.

 > 
 > 
 > On Wed, 2009-05-20 at 23:29 -0500, Les Mikesell wrote:
 > > Skip Guenter wrote:
 > > > I'm guessing the error is a ssh problem from this:
 > > > 
 > > > full backup started for directory /
 > > > Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
 > > > Xfer PIDs are now 1896
 > > > Read EOF: Connection reset by peer
 > > > Tried again: got 0 bytes
 > > > Done: 0 files, 0 bytes
 > > > Got fatal error during xfer (Unable to read 4 bytes)
 > > > Backup aborted (Unable to read 4 bytes)
 > > > Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)
 > > > 
 > > > I'm ASSUMING it doesn't like ssh'ing into itself??
 > > >
 > > 
 > > There's nothing inherently different about sshing to the local host. 
 > > Did you set up the ssh keys needed to make it work without passwords?
 > > 
 > 
 > 
 > ------------------------------------------------------------------------------
 > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
 > is a gathering of tech-side developers & brand creativity professionals. Meet
 > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
 > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
 > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
 > _______________________________________________
 > BackupPC-users mailing list
 > BackupPC-users@...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    http://backuppc.wiki.sourceforge.net
 > Project: http://backuppc.sourceforge.net/
 >

May I ask, who was rude to who... I'm not seeing it?

I hope you did not misinterpret my "Duh, no" about my own ignorance in
response to Les's question "Did you set up the ssh keys needed to make
it work without passwords?" but I suspect so.

Skip


On Thu, 2009-05-21 at 09:25 -0400, Jeffrey J. Kosowsky wrote:
> Skip Guenter wrote at about 03:25:53 -0500 on Thursday, May 21, 2009:
>  > Duh, no.  Thanx much.
> 
> I think you may be confusing "no passwords" with no security. I assume
> the respondent (whom you answered remarkably rudely) meant did you set
> ssh up to use passkeys rather than passwords. As long as you keep the
> passkey secure, the system security will be infinitely stronger using
> RSA/DSA passkeys than a simple plaintext password.
> 
>  > 
>  > 
>  > On Wed, 2009-05-20 at 23:29 -0500, Les Mikesell wrote:
>  > > Skip Guenter wrote:
>  > > > I'm guessing the error is a ssh problem from this:
>  > > > 
>  > > > full backup started for directory /
>  > > > Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
>  > > > Xfer PIDs are now 1896
>  > > > Read EOF: Connection reset by peer
>  > > > Tried again: got 0 bytes
>  > > > Done: 0 files, 0 bytes
>  > > > Got fatal error during xfer (Unable to read 4 bytes)
>  > > > Backup aborted (Unable to read 4 bytes)
>  > > > Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)
>  > > > 
>  > > > I'm ASSUMING it doesn't like ssh'ing into itself??
>  > > >
>  > > 
>  > > There's nothing inherently different about sshing to the local host. 
>  > > Did you set up the ssh keys needed to make it work without passwords?
>  > > 
>  > 
>  > 
>  > ------------------------------------------------------------------------------
>  > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>  > is a gathering of tech-side developers & brand creativity professionals. Meet
>  > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
>  > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
>  > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
>  > _______________________________________________
>  > BackupPC-users mailing list
>  > BackupPC-users@...
>  > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>  > Wiki:    http://backuppc.wiki.sourceforge.net
>  > Project: http://backuppc.sourceforge.net/
>  > 
> 
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, & 
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/

Sorry. I interpreted "Duh" as in being that was obvious. 'No' as in I
would not do that. 
Skip Guenter wrote at about 14:55:44 -0500 on Thursday, May 21, 2009:
 > May I ask, who was rude to who... I'm not seeing it?
 > 
 > I hope you did not misinterpret my "Duh, no" about my own ignorance in
 > response to Les's question "Did you set up the ssh keys needed to make
 > it work without passwords?" but I suspect so.
 > 
 > Skip
 > 
 > 
 > On Thu, 2009-05-21 at 09:25 -0400, Jeffrey J. Kosowsky wrote:
 > > Skip Guenter wrote at about 03:25:53 -0500 on Thursday, May 21, 2009:
 > >  > Duh, no.  Thanx much.
 > > 
 > > I think you may be confusing "no passwords" with no security. I assume
 > > the respondent (whom you answered remarkably rudely) meant did you set
 > > ssh up to use passkeys rather than passwords. As long as you keep the
 > > passkey secure, the system security will be infinitely stronger using
 > > RSA/DSA passkeys than a simple plaintext password.
 > > 
 > >  > 
 > >  > 
 > >  > On Wed, 2009-05-20 at 23:29 -0500, Les Mikesell wrote:
 > >  > > Skip Guenter wrote:
 > >  > > > I'm guessing the error is a ssh problem from this:
 > >  > > > 
 > >  > > > full backup started for directory /
 > >  > > > Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
 > >  > > > Xfer PIDs are now 1896
 > >  > > > Read EOF: Connection reset by peer
 > >  > > > Tried again: got 0 bytes
 > >  > > > Done: 0 files, 0 bytes
 > >  > > > Got fatal error during xfer (Unable to read 4 bytes)
 > >  > > > Backup aborted (Unable to read 4 bytes)
 > >  > > > Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)
 > >  > > > 
 > >  > > > I'm ASSUMING it doesn't like ssh'ing into itself??
 > >  > > >
 > >  > > 
 > >  > > There's nothing inherently different about sshing to the local host. 
 > >  > > Did you set up the ssh keys needed to make it work without passwords?
 > >  > > 
 > >  > 
 > >  > 
 > >  > ------------------------------------------------------------------------------
 > >  > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
 > >  > is a gathering of tech-side developers & brand creativity professionals. Meet
 > >  > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
 > >  > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
 > >  > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
 > >  > _______________________________________________
 > >  > BackupPC-users mailing list
 > >  > BackupPC-users@...
 > >  > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > >  > Wiki:    http://backuppc.wiki.sourceforge.net
 > >  > Project: http://backuppc.sourceforge.net/
 > >  > 
 > > 
 > > ------------------------------------------------------------------------------
 > > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
 > > is a gathering of tech-side developers & brand creativity professionals. Meet
 > > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
 > > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
 > > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
 > > _______________________________________________
 > > BackupPC-users mailing list
 > > BackupPC-users@...
 > > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > > Wiki:    http://backuppc.wiki.sourceforge.net
 > > Project: http://backuppc.sourceforge.net/
 > 
 > 
 > ------------------------------------------------------------------------------
 > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
 > is a gathering of tech-side developers & brand creativity professionals. Meet
 > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
 > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
 > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
 > _______________________________________________
 > BackupPC-users mailing list
 > BackupPC-users@...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    http://backuppc.wiki.sourceforge.net
 > Project: http://backuppc.sourceforge.net/
 >

I have been looking at this and back to a how-to I was trying to follow.
For the case of backing up the localhost (c17-desktop) & trying to
exclude the /var/lib/backuppc directory where c17-desktop is the
BackupPC server...

I can not follow where to copy ssh keys to/from.  Does anybody have time
to baby step me thru this?  

When I do: "sudo su backuppc"
followed by: "ssh -x -l backuppc c17-desktop date"
i get: "ssh: connect to host c17-desktop port 22: Connection refused"
Which I believe ya'll are telling me is because I don't have ssh-keys
set up correctly.

I do have user backuppc set up to be able to run rsync w/o a password
via: "sudo visudo" 
which shows: 
# User privilege specification
root     ALL=(ALL) ALL
backuppc ALL=NOPASSWD: /user/bin/rsync

Thanx in advance, Skip


On Wed, 2009-05-20 at 21:56 -0500, Skip Guenter wrote:
> I installed BackupPC from the Ubuntu repositories several days ago with
> the intent of getting it set up and running as soon as a HDD came in.
> Well last night I was reading thru some of the doc and took a look at
> the control webpage and much to my surprise saw that a host was there
> (localhost) and had done something... but also had an error about it's
> name not matching netbios name.  I changed it's name and deleted the
> backup files under /var/lib/backuppc/pc/localhost.
> 
> After some more reading I change 5 things from the default install:
> 
> 1) changed default method to rsync
> 2) changed logs kept from 14 to 9
> 3) changed the host name "localhost" to "c17-desktop"
> 4) checked the box override box called "FixedIPNetBiosNameCheck"
> 5) changed blackout period to be 7.5 thru 18.5
> 
> Then requested a full backup on c17.
> 
> It returned a new error on the host summary page:
> c17-desktop skip 0 0.00 0   idle backup failed (Unable to read 4 bytes)
> 
> This is the machine that BackupPC was installed on.  I guess getting
> that machine backed up is as good as anyplace to start.
> 
> I also, via the web interface attempted to exclude some directories and
> file types which ended up creating /etc/backuppc/c17-desktop.pl that
> contains:
> 
> $Conf{FixedIPNetBiosNameCheck} = '0';
> $Conf{BackupFilesExclude} = {
>   '*.old' => [],
>   '/var/lib/backuppc' => [],
>   '/tmp' => [],
>   '/lost+found' => [],
>   '/proc' => [],
>   '/cdrom' => [],
>   '/mnt' => []
> };
> 
> After saving these change I attempted another full backup and got the
> same error: backup failed (Unable to read 4 bytes)
> 
> So, questions:
> 
> 1) Is it advisable/legit to use rsync on the local machine backing up
> itself?
> 
> 2) Do my directory and file exclusions make any sense for backing up the
> machine that BackupPC is on?
> 
> 3) How can I find out more about this error I'm getting?
> 
> PS:  HDD just arrived so I'll get the RAID 1 set up tonight or tomorrow
> and figure out how to point BackupPC to that.... think I read there's a
> topDIR variable someplace.
>

No problem and thanx for the explanation. I see it now, but that was not
my intent.  It was "Duh" obvious to all but me and "no" I had not set it
up.  I'll keep a better eye on this going forward. 


On Thu, 2009-05-21 at 16:01 -0400, Jeffrey J. Kosowsky wrote:
> Sorry. I interpreted "Duh" as in being that was obvious. 'No' as in I
> would not do that. 
> Skip Guenter wrote at about 14:55:44 -0500 on Thursday, May 21, 2009:
>  > May I ask, who was rude to who... I'm not seeing it?
>  > 
>  > I hope you did not misinterpret my "Duh, no" about my own ignorance in
>  > response to Les's question "Did you set up the ssh keys needed to make
>  > it work without passwords?" but I suspect so.
>  > 
>  > Skip
>  > 
>  > 
>  > On Thu, 2009-05-21 at 09:25 -0400, Jeffrey J. Kosowsky wrote:
>  > > Skip Guenter wrote at about 03:25:53 -0500 on Thursday, May 21, 2009:
>  > >  > Duh, no.  Thanx much.
>  > > 
>  > > I think you may be confusing "no passwords" with no security. I assume
>  > > the respondent (whom you answered remarkably rudely) meant did you set
>  > > ssh up to use passkeys rather than passwords. As long as you keep the
>  > > passkey secure, the system security will be infinitely stronger using
>  > > RSA/DSA passkeys than a simple plaintext password.
>  > > 
>  > >  > 
>  > >  > 
>  > >  > On Wed, 2009-05-20 at 23:29 -0500, Les Mikesell wrote:
>  > >  > > Skip Guenter wrote:
>  > >  > > > I'm guessing the error is a ssh problem from this:
>  > >  > > > 
>  > >  > > > full backup started for directory /
>  > >  > > > Running: /usr/bin/ssh -q -x -l root c17-desktop /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
>  > >  > > > Xfer PIDs are now 1896
>  > >  > > > Read EOF: Connection reset by peer
>  > >  > > > Tried again: got 0 bytes
>  > >  > > > Done: 0 files, 0 bytes
>  > >  > > > Got fatal error during xfer (Unable to read 4 bytes)
>  > >  > > > Backup aborted (Unable to read 4 bytes)
>  > >  > > > Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)
>  > >  > > > 
>  > >  > > > I'm ASSUMING it doesn't like ssh'ing into itself??
>  > >  > > >
>  > >  > > 
>  > >  > > There's nothing inherently different about sshing to the local host. 
>  > >  > > Did you set up the ssh keys needed to make it work without passwords?
>  > >  > > 
>  > >  > 
>  > >  > 
>  > >  > ------------------------------------------------------------------------------
>  > >  > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>  > >  > is a gathering of tech-side developers & brand creativity professionals. Meet
>  > >  > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
>  > >  > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
>  > >  > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
>  > >  > _______________________________________________
>  > >  > BackupPC-users mailing list
>  > >  > BackupPC-users@...
>  > >  > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>  > >  > Wiki:    http://backuppc.wiki.sourceforge.net
>  > >  > Project: http://backuppc.sourceforge.net/
>  > >  > 
>  > > 
>  > > ------------------------------------------------------------------------------
>  > > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>  > > is a gathering of tech-side developers & brand creativity professionals. Meet
>  > > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
>  > > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
>  > > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
>  > > _______________________________________________
>  > > BackupPC-users mailing list
>  > > BackupPC-users@...
>  > > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>  > > Wiki:    http://backuppc.wiki.sourceforge.net
>  > > Project: http://backuppc.sourceforge.net/
>  > 
>  > 
>  > ------------------------------------------------------------------------------
>  > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>  > is a gathering of tech-side developers & brand creativity professionals. Meet
>  > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
>  > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
>  > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
>  > _______________________________________________
>  > BackupPC-users mailing list
>  > BackupPC-users@...
>  > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
>  > Wiki:    http://backuppc.wiki.sourceforge.net
>  > Project: http://backuppc.sourceforge.net/
>  > 
> 
> ------------------------------------------------------------------------------
> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
> is a gathering of tech-side developers & brand creativity professionals. Meet
> the minds behind Google Creative Lab, Visual Complexity, Processing, & 
> iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
> Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@...
> List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:    http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/

Skip - take a look at this how-to for centos.  Aside from setting up 
repos the sudo and ssh key tutorials should apply:
http://wiki.centos.org/HowTos/BackupPC

Skip Guenter wrote:
> I have been looking at this and back to a how-to I was trying to follow.
> For the case of backing up the localhost (c17-desktop) & trying to
> exclude the /var/lib/backuppc directory where c17-desktop is the
> BackupPC server...
>
> I can not follow where to copy ssh keys to/from.  Does anybody have time
> to baby step me thru this?  
>
> When I do: "sudo su backuppc"
> followed by: "ssh -x -l backuppc c17-desktop date"
> i get: "ssh: connect to host c17-desktop port 22: Connection refused"
> Which I believe ya'll are telling me is because I don't have ssh-keys
> set up correctly.
>
> I do have user backuppc set up to be able to run rsync w/o a password
> via: "sudo visudo" 
> which shows: 
> # User privilege specification
> root     ALL=(ALL) ALL
> backuppc ALL=NOPASSWD: /user/bin/rsync
>
> Thanx in advance, Skip

/
/

Skip Guenter wrote:
> 
> When I do: "sudo su backuppc"
> followed by: "ssh -x -l backuppc c17-desktop date"
> i get: "ssh: connect to host c17-desktop port 22: Connection refused"
> Which I believe ya'll are telling me is because I don't have ssh-keys
> set up correctly.

No, you haven't gotten that far.  Connection refused means you don't 
have a listening sshd running or a firewall is blocking port 22.  If 
your keys aren't right you'll normally just get a password prompt.

> I do have user backuppc set up to be able to run rsync w/o a password
> via: "sudo visudo" 
> which shows: 
> # User privilege specification
> root     ALL=(ALL) ALL
> backuppc ALL=NOPASSWD: /user/bin/rsync

You can skip ssh for local backups if you use another way to get root 
access.

-- 
   Les Mikesell
    lesmikesell@...

Les Mikesell wrote at about 16:30:35 -0500 on Thursday, May 21, 2009:
 > Skip Guenter wrote:
 > > 
 > > When I do: "sudo su backuppc"
 > > followed by: "ssh -x -l backuppc c17-desktop date"
 > > i get: "ssh: connect to host c17-desktop port 22: Connection refused"
 > > Which I believe ya'll are telling me is because I don't have ssh-keys
 > > set up correctly.
 > 
 > No, you haven't gotten that far.  Connection refused means you don't 
 > have a listening sshd running or a firewall is blocking port 22.  If 
 > your keys aren't right you'll normally just get a password prompt.
 > 
 > > I do have user backuppc set up to be able to run rsync w/o a password
 > > via: "sudo visudo" 
 > > which shows: 
 > > # User privilege specification
 > > root     ALL=(ALL) ALL
 > > backuppc ALL=NOPASSWD: /user/bin/rsync
 > 
 > You can skip ssh for local backups if you use another way to get root 
 > access.
 > 

I similarly don't use ssh on my local server for two reasons:
1. ssh adds unnecessary encryption (and depending on your
   configuration also compression) overhead that is not necessary on your
   local system where there is no remote link traffic. Why add an
   unnecessary tunnel?

2. giving ssh root-level access (necessary to read/write all files)
   *seems* more dangerous than letting user 'backuppc' run
   'rsync' as root. Now I know that technically if you can read/write
   a file with rsync then you can easily get root access but it just
   seems tighter and neater that way to limit unnecessary
   privileges. Also, technically you only need the ability for 'rsync'
   to *read* files as root as long as you are doing just backups (and
   not restores).

   I believe the following sudoer line restricts the rsync mode to
   sender so that you are only giving rsync root permission to read
   (not write) files. 

   backuppc        ALL=NOPASSWD: /usr/bin/rsync --server --sender *


 > -- 
 >    Les Mikesell
 >     lesmikesell@...
 > 
 > 
 > 
 > 
 > ------------------------------------------------------------------------------
 > Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
 > is a gathering of tech-side developers & brand creativity professionals. Meet
 > the minds behind Google Creative Lab, Visual Complexity, Processing, & 
 > iPhoneDevCamp asthey present alongside digital heavyweights like Barbarian
 > Group, R/GA, & Big Spaceship. http://www.creativitycat.com 
 > _______________________________________________
 > BackupPC-users mailing list
 > BackupPC-users@...
 > List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:    http://backuppc.wiki.sourceforge.net
 > Project: http://backuppc.sourceforge.net/
 >

Hi,

Les Mikesell wrote on 2009-05-21 16:30:35 -0500 [Re: [BackupPC-users] Noob Question Time... backup up localhost]:
> Skip Guenter wrote:
> > [...]
> > I do have user backuppc set up to be able to run rsync w/o a password
> > via: "sudo visudo" 
> > which shows: 
> > # User privilege specification
> > root     ALL=(ALL) ALL
> > backuppc ALL=NOPASSWD: /user/bin/rsync

that should almost definitely be "/usr/bin/rsync", not "/user/bin/rsync".

> You can skip ssh for local backups if you use another way to get root 
> access.

Meaning, for local backups of the backuppc server, you can just set

	$Conf{RsyncClientCmd} = '/usr/bin/sudo $rsyncPath $argList';

(note that it's "$argList" in this case, *not* "$argList+" as with ssh).
If you also want to enable automatic restores, you need to set
RsyncClientRestoreCmd (same value, I believe), too, otherwise you can
set it to undef:

	$Conf{RsyncClientRestoreCmd} = undef;

If you are *not* enabling restores, you might want to limit sudo priviledges
for backuppc like this:

	backuppc ALL=NOPASSWD: /usr/bin/rsync --server --sender *

(this would prevent anyone gaining access to the backuppc account from
overwriting arbitrary files - for whatever it's worth). You can include
more of the command (I have '/usr/bin/rsync --server --sender --numeric-ids
--perms --owner --group -D --links --times --block-size=2048 --recursive *'), 
but unless you can also include the directory to back up, that doesn't gain
you much, and if you do, you'll need two entries - one for full backups
including the "--ignore-times" argument and one without (for multiple shares
this gets a bit awkward).

But just "/usr/bin/rsync" should work fine to start with. You can always start
fine tuning once it's working.

Regards,
Holger

> > Well, I'm having concurrent backups, but they use different 
> TCP ports, 
> > thus I can "--sport 8873" and "--sport 8874" and so on for 
> my clients.
> 
> If you get this working, let us know how far off it is from 
> the values shown for duration and transfer rate (and a 10-25% 
> allowance for ssh compression).

Will let know, gotta try it out.

F.