diaser-devel Mailing List for DIASER
Cloud storage class, open source software.
Status: Beta
Brought to you by:
dbrasher28
Menu
▾
▴
diaser-devel — Development
You can subscribe to this list here.
| 2009 |
Jan
|
Feb
|
Mar
|
Apr
(3) |
May
(7) |
Jun
(2) |
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|---|
|
From: Damian B. <db...@ec...> - 2009-06-02 13:00:55
|
Cloud crystal formation can be achieved as previously stated by using a different account name and staggering or alternating the transfer times (phases) or lowering the LMB - lowest maximum bandwidth between nodes. Where more technical work is needed remains to be seen. Damian Brasher wrote: > Thinking about cloud technology and DIASER and talking on the local LUG > I have though of a couple of terms that may be useful: > > Extended private clouds to encapsulate 'private storage clouds'... > > DIASER will make the user a private storage cloud, after core features > have matured the use of extended number of nodes or ability to share > disk space with another user (currently simply use a different account > name and change bandwidth use LMB accordingly). > > http://www.hants.lug.org.uk/lurker/message/20090531.215952.2a86af2a.en.html > > Damian > > |
|
From: Damian B. <db...@ec...> - 2009-06-02 11:51:13
|
Thinking about cloud technology and DIASER and talking on the local LUG I have though of a couple of terms that may be useful: Extended private clouds to encapsulate 'private storage clouds'... DIASER will make the user a private storage cloud, after core features have matured the use of extended number of nodes or ability to share disk space with another user (currently simply use a different account name and change bandwidth use LMB accordingly). http://www.hants.lug.org.uk/lurker/message/20090531.215952.2a86af2a.en.html Damian -- Damian Brasher BSc (Open) RHCE MBCS db...@ec... tel: 02380 592888 Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-05-22 14:43:17
|
Fully deployable means: a) DIASER storage structure is stable b) the system is able to take real backup data c) upgradable d) secure e) extend-able f) but limited control, migration, reporting, data retrieve and management features -- Damian Brasher BSc (Open) RHCE MBCS db...@ec... tel: 02380 592888 Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-05-22 14:15:29
|
projected release life cycle - I will pad out the details soon current alpha beginning July 2009 - beta beginning August 2009 - beta2 End September 2009 - beta3 ? beta-rc -- Damian Brasher BSc (Open) RHCE MBCS db...@ec... tel: 02380 592888 Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-05-22 10:54:32
|
Sheet1 Damian Brasher 30th July 2007 www.diap.org.uk DRAFT PROOF OF CONCEPT not for production use. Data flow providing 30 days backup over 3 separate physical sites Three copies of every diff spanning 30 days spanning three different physical locations at any one time apart from first two days of operation Released under http://www.gnu.org/licenses/gpl.txt 1 st 30 days Subsequent 30 day cycles Rdiff Operation 12:00:00 AM Rdiff Operation 04:00:00 AM Directory Dir code Round/(Day(s)) A/D15 –> B/D15 Days stored Time B/D15 –> C/D15 Days stored Time Days 15 D15 1 – (1, 6, 11, 16, 21, 26) 1 – (1, 6, 11, 16, 21, 26) D15 + 0 diff 1 0 D15 + 0 diff 1 0 plus A B C 1 – (2, 7, 12, 17, 22, 27) 1 – (2, 7, 12, 17, 22, 27) 1 diff 2 0 1 diff 2 0 plus 1 – (3, 8, 13, 18, 23, 28) 2 diff 3 0 2 diff 3 0 plus 1 – (4, 9, 14, 19, 24, 27) 1 – (4, 9, 14, 19, 24, 27) 3 diff 4 0 3 diff 4 0 plus 1 – (5, 10, 15, 20, 25, 30) 1 – (5, 10, 15, 20, 25, 30) 4 diff 5 0 4 diff 5 0 plus C/D15 > A/D610 A/D610 > B/D610 Days 610 D610 2 – (6, 11, 16, 21, 26) 2 – (1, 6, 11, 16, 21, 26) D610 + 0 diff 6 0 D610 + 0 diff 6 0 plus A B 2 – (7, 12, 17, 22, 27) 2 – (2, 7, 12, 17, 22, 27) 1 diff 7 0 1 diff 7 0 plus 2 – (8, 13, 18, 23, 28) 2 – (3, 8, 13, 18, 23, 28) 2 diff 8 0 2 diff 8 0 plus 2 – (9, 14, 19, 24, 27) 2 – (4, 9, 14, 19, 24, 27) 3 diff 9 0 3 diff 9 0 plus 2 – (10, 15, 20, 25, 30) 2 – (5, 10, 15, 20, 25, 30) 4 diff 10 0 4 diff 10 0 plus B/D610 > C/D1115 C/D1115 > A/D1115 Days 1115 D1115 3 – (11, 16, 21, 26) 3 – (1, 6, 11, 16, 21, 26) D1115 + 0 diff 11 0 D1115 + 0 diff 11 0 plus C A 3 – (12, 17, 22, 27) 3 – (2, 7, 12, 17, 22, 27) 1 diff 12 0 1 diff 12 0 plus 3 – (13, 18, 23, 28) 3 – (3, 8, 13, 18, 23, 28) 2 diff 13 0 2 diff 13 0 plus 3 – (14, 19, 24, 27) 3 – (4, 9, 14, 19, 24, 27) 3 diff 14 0 3 diff 14 0 plus 3 – (15, 20, 25, 30) 3 – (5, 10, 15, 20, 25, 30) 4 diff 15 0 4 diff 15 0 plus A/D1115 –> B/D1620 B/D1620 –> C/D1620 Days 1620 D1620 4 – (16, 21, 26) 4 – (1, 6, 11, 16, 21, 26) D1620 + 0 diff 16 0 D1620 + 0 diff 16 0 plus B C 4 – (17, 22, 27) 4 – (2, 7, 12, 17, 22, 27) 1 diff 17 0 1 diff 17 0 plus 4 – (18, 23, 28) 4 – (3, 8, 13, 18, 23, 28) 2 diff 18 0 2 diff 18 0 plus 4 – (19, 24, 27) 4 – (4, 9, 14, 19, 24, 27) 3 diff 19 0 3 diff 19 0 plus 4 – (20, 25, 30) 4 – (5, 10, 15, 20, 25, 30) 4 diff 20 0 4 diff 20 0 plus C/D1620> A/D2125 A/D2125 > B/D2125 Days 2125 D2125 5 – (21, 26) 5 – (1, 6, 11, 16, 21, 26) D2125 + 0 diff 21 0 D2125 + 0 diff 21 0 plus A B 5 – (22, 27) 5 – (2, 7, 12, 17, 22, 27) 1 diff 22 0 1 diff 22 0 plus 5 – (23, 28) 5 – (3, 8, 13, 18, 23, 28) 2 diff 23 0 2 diff 23 0 plus 5 – (24, 27) 5 – (4, 9, 14, 19, 24, 27) 3 diff 24 0 3 diff 24 0 plus 5 – (25, 30) 5 – (5, 10, 15, 20, 25, 30) 4 diff 25 0 4 diff 25 0 plus B/D2125 > C/D2630 C/D2630 > A/D2630 Days 2630 D2630 6 – 26 6 – (1, 6, 11, 16, 21, 26) D2630 + 0 diff 26 0 D2630 + 0 diff 26 0 plus C A 6 – 27 6 – (2, 7, 12, 17, 22, 27) 1 diff 27 0 1 diff 27 0 plus 6 – 28 6 – (3, 8, 13, 18, 23, 28) 2 diff 28 0 2 diff 28 0 plus 6 – 29 6 – (4, 9, 14, 19, 24, 27) 3 diff 29 0 3 diff 29 0 plus 6 – 30 6 – (5, 10, 15, 20, 25, 30) 4 diff 30 0 4 diff 30 0 plus 6 operations per day 3 at time (0) and 3 at time (0 plus). Each node has 4 simultaneous transfers at time (0) and time (0 plus) Max D15 = min bw over 4 hours between any two nodes ABCA... / 4 Time (0) could be midnight and time (0 plus) could be 4 am (Test) Damian Brasher 30th July 2007 www.diap.org.uk Page 1 Sheet1 Rdiff Operation Diff clr. op. Time n any two nodes ABCA... / 4 -- Damian Brasher BSc (Open) RHCE MBCS db...@ec... tel: 02380 592888 Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-05-22 10:33:56
|
To further collaborator, developer and user understanding of the project I have made available publicly searchable project memory and thinking generated between Dec 2006 and Dec 2008. Paper designs, drawings and plans presented as high quality photographs (2048x1530) viewable in 640x480. An edited version of a closed development mailing list in pdf format. http://www.diap.org.uk/background_ip/ The contents of the Mailing List are in this post as well as available here. http://www.diap.org.uk/files/diap_project_thinking.pdf This is an edited version ... (to remove unnecessary email addresses and phone numbers and for brevity focus on architectural decision relevant material) of the closed DIAP mailing list run between Dec 2006 and March 20th 2009. Date: Fri, 8 Dec 2006 22:28:43 -0000 (GMT) Subject: test test >From ...sher@int... Sat Dec 09 01:22:22 2006 Dear all This private mailing list is to provide information about the current progress of interlinux community archive project. ... There have been many updates on the website www.interlinux.co.uk and now a feature where you can change your password to one that suits you as well as hugely improved security, which is what it is all about. As ever the ftp address is still ftp:// and if you have forgotten your username or password please email me at ...sher@int.... You can use Internet Explorer to upload and download files or an ftp client such as FileZilla. Details on the website. Damian >From ...sher@int... Fri May 18 19:21:43 2007 The project has made some more progress. The details are outlined in the new page http://www.interlinux.co.uk/page02.html including defined aims and brief as well as some additional testing with a third virtual server in London's Telehouse North. Interlinux website has also had an overhaul. ... >From ...sher@int... Sun May 20 10:45:42 2007 Dear Interlinux ftp users. As ftp access has been static for a long time I am loathed to change the service however I need tighten security by only allowing ftp and password change access to know ip addresses and ip address ranges. Please could you forward me you ip address if you still want to use the project so I can add you to the list. I you don't have a static ip address then access via FTPS / SFTP or WebDav will be available as soon as I can - this may need a server re-build. Damian Brasher. >From ...sher@int... Sun Oct 07 18:49:46 2007 Dear users / testers and interested parties, I have now shifted activity for the interlinux archive project to a downloaded product designed to help me and the wider Open Source community and described at www.diap.org.uk. This means that users need to retrieve their data and I will be closing down the user ftp access in four weeks time. Please can you inform me when you have cleared out files. There are a couple of family members that will continue to ftp access but this will be a reduced ad-hoc personal service. Thanks you all very much for helping and providing feedback over the last three years, without your help I could not have written the first version of the DIAP package which is in alpha phase and a working utility. To provide a product is a natural progression to the development and a much safer and less resource intensive method to communicate the ideas the project has let unfold. This mailing list is being turned over to DIAP development and interested parties if you wish to unsubscribe please use this link: Damian -- www.diap.org.uk "X between a data stick / cddvdrw / tape and shared drive" personal e: ...sher@int... work e: ...sher@ec... websites www.diap.org.uk www.interlinux.co.uk 000000000 All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Tue Oct 16 18:50:19 2007 ... DIAP version 0.2.0 released - much increased automation of setup scripts, smoother install of prototype available. www.diap.org.uk personal e: ...sher@int... work e: ...sher@ec... websites www.diap.org.uk www.interlinux.co.uk 000000000 All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Fri Oct 19 13:45:08 2007 Dear users / testers and observers, I have updated the main README.pdf and described the seven days as just a containment for prototyping as well as another optimization that will enable the goals outlined to be achievable. The prototype is now described as a partially working prototype and not designed to mislead. I intend to leave as is now for a couple of weeks to let things "ackle" :) www.diap.org.uk Damian Aiming to protect mission critical data: 'You can't afford to lose...' All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Sun Nov 11 15:27:22 2007 Dear users / testers and observers, Here is a summary of download statistics for the Interlinux and DIAP Open Source offerings:- ############################################################ # Report: dbrasher # Report Name: Downloads # Date Range: 01/05/2007 - 11/11/2007 ############################################################ /files/user_ctrl_scripts.tar 296 (LDAP login kit) /files/web_template.zip 49 (Website template) /files/README.pdf 40 (DIAP readme) /files/dsi-0.1-alpha-src.tar 21 (Space invader clone C source) /files/diap_v020.tar 20 (Latest DIAP kit - released 16th Oct 07) /files/diap_v01.tar 12 /HantsLugTrainingBoF.pdf 10 /files/dsi-0.1-win-bin.zip 9 /files/diap_v011.tar 8 /files/diap_v012.tar 5 DIAP now has a freshmeat entry:- http://freshmeat.net/projects/distributedinternetarchiveprotocol/ www.diap.org.uk Damian Aiming to protect mission critical data: 'You can't afford to lose...' All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Fri Dec 07 19:12:09 2007 ... DIAP releases have been written in bash script, I am investigating other languages and their future viability, there is a strong chance the project will continue to be written in bash however thoughts on the following languages would be welcome especially for: *Portability across platforms *Network handling capabilities *Handling of native Linux tools *Web interface for monitoring *Finding other programmers to help with the project Listed in order of additional learning curve - for me:) Only: Bash, Perl, Php, Java, Python or Forth Or perhaps a combination? Eg. if someone wants to commit to helping out on a particular area of the project and they know Perl well then I would be happy to use Perl for some or part of the project. Damian >From ...sher@int... Fri Feb 01 12:44:34 2008 Dear List members It's been a busy few months and there are plenty of developments that will improve DIAP testing platform and development: *The DIAP trademark registration has been listed in the online UK IP journal and unless the is opposition then the registration will be complete by the summer. http://webdb2.patent.gov.uk/tm/t-journal/t-tmj/journals/6719/domestic/t-tmj-domestic-6719.html *ECS have allowed me to use some webspace at http://www.... which I forward from www.diap.org.uk. *PIPEX ADSL line I use has been regraded and is now 6Mbs DS and 588Kbs US. *I have a new VPS (Virtual Private Server) provided a low cost by the ECS postmaster with a huge 111G hard drive and high bandwidth allowance as it resides within the JANET network. I have now have three servers across the Internet for 3 node DIAP deployment and testing. *I am waiting for all my DOMAIN names to be transferred from www.amenworld.com to http://www.blacknight.ie/ to save money and improve flexibility. *The project will continue to be written in bash as bash had good prototyping qualities and then gradually re-write parts in Perl. *I attended 2007 IBM Hursley Lecture: The Many Uses of Images in Healthcare and was interested in the secure distribution of medical images between consultants. http://www.soton.ac.uk/mediacentre/news/2007/oct/07_128.shtml *I am attending, for my day job, this Seminar http://www.ukuug.org/events/seminars/fab/ Unix 'Files and Backup' however It will be useful for DIAP development and good back grounding. *So next job is to improve the DIAP documentation and then continue with testing and development. Damian www.diap.org.uk Aiming to protect mission critical data: 'You can't afford to lose...' >From ...sher@int... Mon Mar 03 08:03:14 2008 Dear List members I gave an informal lightning talk at the UKUUG (UK Unix Users Group) Files and Backup seminar introducing DIAP. The talk was at the end of the two day seminar held in London, attended with a colleague from ECS and speakers included Red Hat, Sun, Novell, Cambridge University amongst others. My talk as well received, only 3-4 minutes, a couple of nice rounds of applause:) and a book for my trouble as I volunteered this talk. I also gleaned plenty of very useful information from the seminar. Day two - voice recordings and slides have not been uploaded to the UKUUG website yet so I have uploaded the slides to www.diap.org.uk. When day two slides and voice recordings have been made available I'll link to them. Slides (pdf):- http://www.diap.ecs.soton.ac.uk/DIAP_TALK.pdf UKUUG FAB:- http://www.ukuug.org/events/seminars/fab/programme/ The DIAP strapline / mission statement has now been changed to include the mechanical view of what DIAP does, a 'virtual auto-changer'. DIAP is attempting to achieve what large, hugely expensive mechanical tape library auto-changers do. Damian www.diap.org.uk The virtual auto-changer (Aiming to protect mission critical data: 'You can't afford to lose...' ) >From ...sher@int... Tue Mar 11 17:54:47 2008 ... I have just realised that using 'The Virtual auto-changer' in my strap line / mission statement is not the most accurate description of DIAP's intentions, the term is also already in use, however the term 'Hyper-virtual autochanger' is not and is in fact a more implicit than the former. Hyper implying a greater level of virtual or another layer of abstraction built into the tool which is truer in this case than a 'Virtual auto-changer' which still involves tape drives in any case. I think it is worth having a statement like this to encapsulate the essence of what the prototype tool is trying to do. So I am using: "Hyper-virtual auto-changer... and so on Previously:- "The Virtual auto-changer... and so on Damian >From ...sher@int... Tue Mar 11 18:13:05 2008 Dear List members I have just realised that using 'The Virtual auto-changer' in my strap line / mission statement is not the most accurate description of DIAP's intentions, the term is also already in use, however the term 'Hyper-virtual autochanger' is not and is in fact a more implicit than the former. Hyper implying a greater level of virtual or another layer of abstraction built into the tool which is truer in this case than a 'Virtual auto-changer' which still involves tape drives in any case. I think it is worth having a statement like this to encapsulate the essence of what the prototype tool is trying to do. So I am using: "Hyper-virtual auto-changer... and so on Previously:- "The Virtual auto-changer... and so on Damian >From ...sher@int... Wed Mar 19 13:06:44 2008 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear List Members The latest, Easter Break, completed tasks for DIAP (c/o. Interlinux LTD):- * UK Trade mark, DIAP, ownership transfer request to Interlinux LTD from a personal application has been sent, form TM16. I have checked with the IPO and they have confirmed that no opposition to the mark has been filed, currently published since January 18th 2008 in the UK trade mark journal, leaving under 30 days before the Trade Mark DIAP is live. * Clarification on www.diap.org.uk that all DIAP assets, (IP, Domains, hardware, drawn plans) belong to Interlinux LTD. * Clarification on www.diap.org.uk that all work has been undertaken so far has been in my spare time... * The above to ensure that in the future a) the project is accountable in terms of Tax b) To protect past and future earnings for share holders, including myself. c) To protect contributors to the project so that their work is recognised in the correct manner. d) If there was to be partial or full project acquisition by another organisation then their acquisition is fully accountable. * All Domain information has been verified and updated with Nominet including ownership, Registrars, Registration and TAG TAG is BLACKNIGHT-IE, this information is currently propagating through DNS as I write. *Blacknight web-hosting purchased, diap.org.uk domain is now residing with Blacknight as well as interlinux.co.uk. Temporary space with ECS in reserve but not used. * Email MX forwarding has also been adjusted to accommodate the domain ISP and host changes. * The latest download statistics available on www.interlinux.co.uk front page. * All project electronic and paper records files and checked, including accounts, certificates of ownership and registration, company records, paper plans, receipts and so on for easy access and retrieval. * 'Cite them right' the essential guide to referencing and plagiarism purchased. JISC Plagiarism Advisary Service. * Recent systems logs recording Interlinux office system and network activity archives created. Now for a bit of a rest before continuing with technical development:) Damian - -- personal e: ...sher@int... websites www.diap.org.uk www.interlinux.co.uk 000000000 All mail scanned by clam-av http://www.clamav.net/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFH4Q/jOJ/AxqomAe0RApIhAJ9PkaBEKfYXcpwgOYZkjM3kE7bdrwCfSYns Gs2WD/G4Yk0djeiYx6u1zJw= =MX4q -----END PGP SIGNATURE----- >From ...sher@int... Thu Mar 20 00:27:21 2008 Hi List, Before I settle down and holiday for the rest of the Easter break I have introduced this paragraph to the DIAP website to explain a trend I have only now been able to test and propose, it's not rocket science but adds more weight to the DIAP proposal:- "Observing my email archive, at 272MBytes, having never deleted an email permanently and the file, /mail, has been in use for 4 years. During this time my available xDSL line Bandwidth has increased, 2004 500MBits/sec to 1GBit/sec, 2007 1GBit/sec to 6GBits/Sec this is approx. 150% yearly increase whereas my mailbox has increased yearly by approx. 50%. It is this difference which DIAP attempts to use classing my email record as 'mission critical'. Other record types will increase at different rates but probably less than the average yearly bandwidth increase. This idea needs expanding but forms the foundation for the usefulness of DIAP, describing a DIAP rule of thumb." I will slice up my mail box by year to calculate the exact yearly increase, 50% is probably close. The bottom line is that it can only get better:) Damian >From ...sher@int... Sun Mar 23 13:21:26 2008 http://www.diap.org.uk has been completely revamped with menus etc. I am hoping this is much easier on the eye than the previous page and better organised. This is in readiness for the next main release which is hoped to be a working prototype, I am currently preparing it for code with the release planned for Wednesday. Off to peel some veggies:) Damian >From ...sher@int... Tue Mar 25 17:31:04 2008 Hi List, *version 0.3.0 alpha released. *latest refined README.pdf worked example uploaded to theory section. *website refined after recent update. The latest version is a theoretically working prototype that should be a useful utility to an experienced Linux/Unix Sys Admin. As the release is fresh full testing will take a ... cover the 30 day cycle it invokes. Damian >From ...sher@int... Thu Mar 27 21:34:59 2008 Hi List, *version 0.3.1 alpha released. 27/03/08 - Fixed trivial error that stopped data transfer, due to incorrect node label in crontabs 27/03/08 - source files not removed for first month, tab_a.sh --remove-source-files deleted, ensures a FULL backup remains on node A for recovery. 27/03/08 - Test dev switches to speed up cycle to hourly in config. Crontabs 3 extra variables. Damian >From ...sher@int... Fri Mar 28 17:39:56 2008 Hi List, Last of the big missing pieces for the project is the diagrammatic (JPEG not ASCI;)overview. Can be viewed here: http://www.diap.org.uk/diagram.html Also an improved title page background image based on the Mancala game - One of my favourites, (Not a replacement to the Logo - that's stays of course). Damian >From ...sher@int... Mon Mar 31 18:41:00 2008 Hi List, *version 0.3.2 alpha released. *Change Log available here: http://www.diap.org.uk/change_log.html *Website has been updated and the front page is less 'wordy' with an image insert. DVTL has been clarified as an emerging result of this research. *Freshmeat has been updated and the new release will be posted on the front page tonight or tomorrow. *Internet activity, downloads statistics, hits visits and page views can be viewed here http://www.diap.org.uk/awstats/ (as you can see traffic is high for just a week using this website and new Blacknight server! Damian >From ...sher@int... Tue Apr 01 10:19:21 2008 Hi List, Just wanted to let you all know that the prototype is *working* in development mode and reflected so on the website. Tests at home are running successfully. I am giving a short talk on Saturday about DIAP to Hants Lug then taking a break from development for a couple of weeks, monitoring community feedback before pursuing the next release. Damian >From ...sher@int... Wed Apr 02 21:47:33 2008 Hi List, I am really chuffed to say that DIAP has been formally 'boiler plated' and the Internet Draft is now in the IETF (Internet Engineering Task Force) database. It should be available for search at http://www.rfc-editor.org/index.html soon. If not I have a copy on diap.org.uk. http://www.diap.org.uk/protocol.html There is plenty of work to turn the ID into an RFC (the full IETF standard, request for comments) but this does mean the 'DIAP Engine' is closer to becoming an *industry standard*. This is the confirmation:- -------------------------------------------------------------------- A new version of I-D, draft-brasher-diap-00.txt has been successfuly submitted by Damian Brasher and posted to the IETF repository. Filename: draft-brasher-diap Revision: 00 Title: Distributed Internet Archive Protocol (DIAP) Creation_date: 2008-04-02 WG ID: Independent Submission Number_of_pages: 9 Abstract: A de-centralised, self-contained and managed storage protocol. A system to provide strong storage fail over by using existing resources over networks distributing vital data evenly. Rapid deployment and high redundancy for small to medium organisations as well as individuals. Designed to reduce dependency on tape backup systems. The protocol also has implications for long term archiving. By classifying data vitality values the limitations in physical space due to bandwidth constrictions can be overcome and the usefulness of DIAP maximised. The IETF Secretariat. Damian >From ...sher@int... Tue Apr 08 20:10:11 2008 Hi List, I presented another short talk for Hampshire Lug group Saturday (after a longer talk about another Open Source backup solution widely in use). It was more in depth than last time and there were some in depth and challenging question afterwards - I believe the answers I gave were complete and felt the audience were left with some though provoking material. The talks were not recorded as the camera man was away early, new born ... The slides (much updated from the UKUUG talk) are available from Hants lug Wiki: http://www.hantslug.org.uk/cgi-bin/wiki.pl?TechTalks I have also updated the credits on the About page of the website, if any contributer thinks these should adjusted please let me know off list. http://www.diap.org.uk/about.html Damian ...sher@int... >From ...sher@int... Tue Apr 15 17:43:25 2008 Hi List, just wanted to make readers and for the record aware of 4 focus areas: 1) To market DIAP to *Accounts* an area that will be explored and advertised is the *potential reduction* of a target orgaisations insurance premiums in relation to data protection. Other marketing threads to be investigated. 2) DIAP Trademark will be fully registered Sat 19th, so Monday the 21st for sake of argument. 3) IETF working group (WG) needs to be decided on (which hanger to reside in), there are many to target but this is required if DIAP is to follow a *standards track* to RFC rather than become an experimental RFC. I am investigating possible WG's and will contact the technical advisers of each to ensure a best fit. Currently DIAP-ID is metaphorically on the taxi/runway waiting to be sent to the right hanger/terminal. 4) Coding in Perl is likely to commence in June; use of SVN through Source Forge will be setup before hand at the same time as real world deployment testing. NB: Font size on website has been reduced slightly. Damian >From ...sher@int... Fri Apr 25 15:16:06 2008 Progress has been made in several areas: *DIAP (R) is now assigned in full No. 2466480 and visible on the website and on the DIAP Logo. *My line Manager and other senior staff at ECS - Southampton University have given me permission to apply for a ... month development grant with Joint Information Systems Committee (JISC) (1-1/2 days per week outside my normal role) subject to the below: *The above bid is subject to general agreement and clarification of IP (Intellectual Property) rights pre (background IP), during (foreground IP) and post funding (Exploitation rights). *I have discussed IPR with several people with the correct knowledge and have a meeting with the University Enterprise and Innovation Centre next week as well as a meeting with the ECS enterprise officer. ... ... *Systems will benefit from the Community product located at www.diap.org.uk as will the Open Source Community and Interlinux LTD as a backup solution. ... *interlinux.co.uk points to www.diap.org.uk. *Coding will now continue in Perl with release 0.4 alpha. Damian >From ...sher@int... Thu May 15 18:14:46 2008 Hi List members There are a number of items to announce; * The funding proposal to JISC has not yet been written as I am waiting from CEI (University Center of Enterprise and Innovation) to provide me with a 'Heads of Terms' agreement. I spoke to the adviser Monday and something should be with me soon. This includes and IPR and License evaluation. * The website has been worked on some more, the graphics and wording now include philosophy behind the latest architecture. The site has been streamlined to stay focused and the main logo has a new strap-line 'robust archive system'. http://www.diap.org.uk * The architecture has been improved to accommodate the above philosophy and released under FDL. You can read the philosophy on the front page of the site. * The mathematics of the system are now very simple and this is good news as the algorithm that has developed is doing it's job. http://www.diap.org.uk/theory.html * The website viewing statistics page is now private but I will send a stats report out later this month. * I have ordered a PO (Post Office Box) for Interlinux LTD to keep my home address private for all further online documentation, Companies House and other correspondence. * I intend to release a final Alpha prototype to accommodate the recent architectural enhancements, ver.0.3.3 in bash script then it will be Perl for the next DAIP alpha release series (0.4.0). * Finally I have been investigating De-Duplication as an extra feature of DIAP. DeDuping has enormous potential to increase the efficiency of DIAP. Damian -- personal e: ...sher@int... work e: ...sher@ec... websites www.diap.org.uk www.interlinux.co.ukk 000000000 All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Thu May 15 18:40:52 2008 Hi List members I think it is worth adding that the potential *financial gains* made when using DIAP have also been outlined on the front page http://www.diap.org.uk This is future proofing but also relevant I believe, there is a strong chance that DIAP may use a dual or more license model, in other words those that can afford to use the production product will pay for the privilege (i.e. very large organisations) of not having a restricted GPL type license but those that cannot afford to (Charities, Universities and Schools) can without penalty, apart from being legally obliged to make their improvements available to the rest of the community, use the GPL'ed version. Open Source and Open standards are a way of developing strong security standards based on the principle that many eyes make strong security possible. While DIAP is not a cryptographic algorithm I feel the principle still holds true for backup software. Here is a quote from the Open Source Watch UK:- [*] "Trade-secret cryptographic algorithms have a long history of being both weak and leaked to the public. The tendency to weakness is due to the fact that the only known reliable way to determine the strength of cryptographic algorithm is to have a large number of cryptographers examine it which is not possible to do while keeping it a trade-secret." [*] http://www.oss-watch.ac.uk/resources/security.xml Damian -- personal e: ...sher@int... work e: ...sher@ec... websites www.diap.org.uk www.interlinux.co.ukk 000000000 All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Fri May 16 20:53:31 2008 Hi List members Privacy measures have been ... ... Better to be safe than sorry I think:-) PO Box is now registered and will be used for:- Interlinux Ltd PO Box 1623 SOUTHAMPTON SO15 9AE ... Damian >From ...sher@int... Mon May 19 06:04:11 2008 Hi List members 18/05/08 version 0.3.3 Pre-alpha released. 18/05/08 Refined architecture, worksheet amended. 18/05/08 Diagram updated. 18/05/08 Refined architecture, worksheet uploaded. All available from http://www.diap.org.uk Damian -- personal e: ...sher@int... work e: ...sher@ec... websites www.diap.org.uk www.interlinux.co.ukk 000000000 All mail scanned by clam-av http://www.clamav.net/ >From ...sher@int... Thu May 22 20:35:25 2008 ... March: 177 Unique visits, more than one page - non webcrawler or bot April: 139 May: 175 Total non google bot type (human) hits: 5972 Downloads of 0.3.3 (26) since Monday morning. Freshmeat 2,341 record hits and 607 main website click through. DIAP has been registered anonymously to other Linux script watch sites (thank you however that was:) including http://www.linuxlinks.com/. Damian >From ...sher@int... Mon Jun 02 21:33:57 2008 Hi List (Survey designed for Hants Lug forward to DIAP (R) list members. Following on from 'Big home backups' thread and the continued DIAP (R) effort to which I will continue to feedback into the OSS communities - I invite Hants Lug members to complete a 4 question, generic and anonymous survey; IP's are not stored and more than one survey per IP is allowed. http://www.surveymonkey.com/s.aspx?sm=YgorpQ5jv9FNNfBw_2fsnu0Q_3d_3d password: ********* Identifying what data is important to you as an individual or your organisation is *hard* and [organisation (individual) specific] therefore the answer needs a some thought. Depending on 'value' placed on data and the 'value' if the data was completely lost; financial cost of recovery, time, damage or emotional cost or a mixture of these are the consequence. For the sake of this survey I assume you decide the value of data and it's relative importance. The survey may help you to rationalise your own backup requirements. You will need to read question 3 carefully and may need a pen and paper to follow through. http://www.surveymonkey.com/s.aspx?sm=YgorpQ5jv9FNNfBw_2fsnu0Q_3d_3d password: ******* Thanks for your time and I will post and analysis of the results in a 7-14 days or as soon as I have a good data set. Please forward this link to whoever you think may be interested including other lists. -- Damian L Brasher http://www.diap.org.uk >From ...sher@int... Thu Jun 19 19:52:46 2008 I have now received the voice recording from the lightning talk I gave on the 20th February 08 in London at the UKUUG Files and Backup Seminar. About 4 minutes and 4MB to download here is the link for the mp3:- http://www.diap.org.uk/files/talks/ukuugfab_diap_lightning_talk_20th_feb.mp3 The is talk pre-recent developments but projects the essence of DIAP. Perl learning ongoing and I expect the first alpha-development version to be ready by the end of the summer. Damian -- personal e: ...sher@int... website http://www.diap.org.uk 000000000 >From ...sher@int... Sun Jul 06 21:02:44 2008 Version 02 of the ID for IETF has been released. http://www.ietf.org/internet-drafts/draft-brasher-diap-02.txt Apart from some minor adjustments to the acknowledgments to bring then inline with the website: http://www.diap.org.uk/about.html. ... Damian personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Fri Aug 15 13:30:23 2008 The summer period has been filled with holidays and a bit of rest however I have a few pieces of news: *I now have a straightforward generic sentence to describe what DIAP does: "Quick and low-cost way to make an environment more robust by backing up data in multiple places." ... Damian -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Wed Aug 20 12:15:50 2008 Pilot survey results: Following on from a survey I posted back in June, see below, designed to help make decisions about data vitality and importance to individuals and organisations, as well as find out a little bit more about the relationship between the importance of data types and their size, I have some results to publish. The survey was exclusive to Hants Lug and deliberately kept low key and has been a very useful exercise even though the number of participants has been relatively small - but very good for such a small readership, quality not quantity. Results analysis:- *So from just this small pilot survey I can deduce qualitatively with reasonable certainly that importance of data, subjective to the individual or organisation, does not depend heavily on the data type. *That importance of data in relation to file size is loosely inversely proportional. So the most important files are generally the smallest in size. This is very encouraging information for the DIAP (R) project. 6 Participants: Question 1) Rate FIVE of these DATA TYPES if they were lost completely how best describes the effect to your users (and or yourself) organisation or home occupants. Results: *total damage - cost crippling - traumatic 2 [participants specified] Documents 1 Presentations 1 Photographs 1 Email boxes on a server 2 Code repository 1 Website code *massive damage high cost - devastating 1 Spreadsheets 1 .txt files 1 Email boxes on a server *major damage very costly - extremely upsetting 1 pdf documents 1 Photographs *significant damage - significant cost very upsetting 1 Spreadsheets 2 Accounting Software data 1 Photographs 1 server configuration files 1 music files 1 Virtual machine images *damage expensive - annoying 1 Documents 1 pdf documents 1 MS .pst file 1 Email boxes on a server 1 music files 1 ISO images 1 Virtual machine images 1 Code repository Question 2) Size the FIVE choices you made in the previous question Results: 1-100 MB 8 items >100 MB - 500 MB 1 item >500 MB - 1 GB 8 items >1 GB - 10 GB 8 items >10 GB - 50 GB 0 items >50 GB - 100 GB 2 items > 100 GB 1 item I thanked those that took part and those that provided feedback and constructive criticism also thanks to HL ML readers for living with the thread. I will now design a new survey with some sort of incentive and float this to a much larger audience. The results I have are enough to incorporate a DIAP (R) decision matrix on the project website. -------------snip June 08------------------------------------------------- > Identifying what data is important to you as an individual or your > organisation is *hard* and [orgaisation (individual) specific] therefore > the answer needs a some thought. Depending on 'value' placed on data and > the 'value' if the data was completely lost; financial cost of recovery, > time, damage or emotional cost or a mixture of these are the consequence. -- Damian L Brasher http://www.diap.org.uk - Quick and low-cost way to make an environment more robust by backing up data in multiple places. Damian -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Thu Oct 16 23:35:59 2008 I have deployed some SourceForge facilities to assist with development: SVN - Subversion for version control and consistent source repository. Checkout for read only non-members: All one line:- svn co https://diap.svn.sourceforge.net/svnroot/diap/ diap More SVN information: http://www.diap.org.uk/download.html Additional mailing lists at: (This list will remain as it is for the time being) ... More information: http://www.diap.org.uk/lists.html Damian -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Mon Oct 27 11:37:06 2008 Regarding collaboration between Interlinux LTD, and it's sole product, DIAP(R), and recent attempts to build a funding initiative and collaboration with OMII-UK within ECS, Southampton Univertsity. After meeting with the collaboration support manager from the Universities Research & Innovation Services and revisiting the Heads of Terms document, originally drafted early 2008, designed to allow funding collaboration to take place using a formal in-licence on receipt of appropriate funds. Interlinux LTD board decided with a 100% majority to proceed and accept the revised terms. This effectively means a ... funding application bid can be submitted. Damian -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Fri Oct 31 17:18:35 2008 1) New introduction title. ... 1) Advanced backup volume management is the new introduction title of the DIAP project. http://www.diap.org.uk. Advanced backup volume management is a technically accurate and meaningful to the average systems administrator, a subset of the target market, I believe. The distributed nature of the system is still and will remain as crucial function of the project and is reflected so in the acronym and trademark DIAP(R). 2) After a recent meeting with an academic on behalf of academics interested in archiving within the IAM group we clarified areas of interest then placed them into distinct subsets of the much broader area of repository preservation environments, archiving, backup and distributed storage. DIAP(R) clearly exists in distributed backup and archiving space where the academics within IAM are heavily monitoring and researching the area of distributed storage working with several companies, academics and organisations in the field. Distributed storage is this sense is aimed towards a different but possibly overlapping market than that of DIAP. Convergence in our interests could occur in the future at the abstract management layer of policy making. DIAP is a package and structure in the sense that a product/system is being designed and built to satisfy the immediate needs of the problem binding native tools and existing technologies. We discussed why DIAP is not re-inventing the wheel and why it is is a hybrid machine. We will meet again at the end of November to ensure our funding proposals are complementary ... DIAP then deals with compression, encryption, error checking, long term, robust and secure management and retrieval of the volumes and subsequently data objects within them. Damian -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Wed Nov 05 17:43:06 2008 I have created an improved diagrammatic overview of DIAP - the previous was out of date. This version says a lot more and provides the reader with essential information as well as being prettier and anti-aliased. http://www.diap.org.uk/diagram.html or http://diap.svn.sourceforge.net/viewvc/diap/diap/docs/overview.jpg?revision=18&view=markup -- Damian. >From ...sher@int... Fri Nov 21 21:43:13 2008 DIAP website has a new logo design, many thanks to Stuart Mackenzie (Step Son) the 'Mac Man' for his work on this, a big improvement on the previous but a variation on the original design nonetheless. http://www.diap.org.uk -- Damian >From ...sher@int... Tue Dec 02 21:05:33 2008 Hi List, IETF contribution draft-brasher-diap-03.txt, has been released. This, importantly, includes the architecture reworking completed earlier in May this year. http://www.diap.org.uk/protocol.html There is plenty of work to turn the ID into an RFC, nothing is set in stone yet but the structures and design is solidifying (RFC, the full IETF standard, request for comments) but this does mean the 'DIAP Engine' is closer to becoming standard. Here is the IETF link: http://www.ietf.org/internet-drafts/draft-brasher-diap-03.txt This is the confirmation: -------------------------------------------------------------------- A new version of I-D, draft-brasher-diap-03.txt has been successfuly submitted by Damian Brasher and posted to the IETF repository. Filename: draft-brasher-diap Revision: 03 Title: Distributed Internet Archive Protocol (DIAP) Creation_date: 2008-12-02 WG ID: Independent Submission Number_of_pages: 12 -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Fri Dec 05 16:45:22 2008 The process to obtain government funding to further the technical development of DIAP has commenced. Intellectual Property sharing provisional agreements have been accepted by all collaborative partners. I will now cease (public) development during the application bid process to avoid muddying the waters, to keep the goal posts in the same place for the executives and all partners involved. I'll inform this list of the result of this funding bid around January/February 09. Have a great Xmas and New Year. -- Damian >From ...sher@int... Fri Dec 26 16:41:59 2008 In preparation for submission of a JISC grant funding application I have updated the project credits - if there are any one mentioned... *Damian Brasher* RHCE BSc (Maths-IT) MBCS - Engineering, research, design, code and author of Internet Draft for IETF. DIAP is developed in my free time, I am a full time systems administrator on a ... term contract to OMII-UK at ECS Southampton University. Damian is also studying at systems architect grade with Red Hat GLS. *Marisa* (wife) MSc-Professional Studies, and many thanks for fantastic support. Marisa works as a skills development leader for Learning Curve. *Myles McClelland* MSc-Management FBCS - Part sponsor, involved with early conceptual design, project management advice and valuable technical reasoning as well as physical server maintenance, data storage and transfer tests. ... *Richard Brasher* Financial Advice. Richard works for a large multi-national. *Stuart* - (BTec Design) DIAP B.Tree Logo. *Stephen Pelc* Small business consulting. Stephen runs and owns MPEForth. *Other individuals* Thanks to a number of individuals from various groups for their advice and constructive criticism. Assets and Intellectual property Are owned, managed and registered in the name of Interlinux LTD in turn owned by the named partners above apart from Stuart - Trading services can be put in place when necessary. This is a part-time project and will remain so for the foreseeable future. Delivery is and will remain Open Source. - http://www.diap.org.uk/about.html - personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Thu Jan 01 21:55:35 2009 Developing this distributed project, DIAP, in conjunction with the OMII-UK http://www.omii.ac.uk brand here is a fuller explanation for the record; Please forward to associates or colleagues if relevant. DIAP's association with OMII-UK will help mitigate unforeseen large and minor technical difficulties and therefore reduce the overall project risk. DIAP technical development will share some of the problems distributed software writers and maintainers face, importantly; openness, network agility, deployment in heterogeneous environments, using network links of varying latencies therefore unpredictable failures in the network. Association specifically means access to technical resources both electronic and human. OMII-UK is an organisation with deep understanding of the nature of and difficulties faced by, open distributed systems - DIAP aims to be transparent, open and scalable. OMII-UK are also experienced maintainers of Open Source software and adhere to Open Source development methodology, including licence management and integration, which will be of value to DIAP development. Security within distributed computing can be very difficult and another area where OMII-UK has been successful in overcoming specific problems for the GRID and eScience communities. DIAP is a generic project is a sense of the intended user base but to spend some development time as a GRID/eScience satellite project is taking advantage of the distributed computing mindset, the specific benefits of doing so are outlined above, that OMII-UK has fostered over the years. Even if DIAP in years to come does not retain strong links with OMII-UK. Damian -- personal e: ...sher@int... website www.diap.org.uk 000000000 >From ...sher@int... Wed Feb 11 12:02:24 2009 Submission of a funding proposal for six months part-time (day time) DIAP development beginning in April and ending in September for JISC has been approved by Southampton University (ECS) and successfully submitted last night. I expect the result at the end of March. -- Damian >From ...sher@int... Fri Mar 20 11:04:49 2009 I'm very proud and grateful to have the opportunity to further technical development of DIAP(DIASER-Distributed Internet Archive System for Educational Repositories) with a small team within (OMII-UK)ECS for the next seven months. ... My role within OMII-UK will be split between my normal duties as a systems administrator. The collaboration will continue the endeavour to deliver a valuable product to the relevant OSS communities by November. This list is *now closed* and I invite you to join any of these lists: http://www.diap.org.uk/lists.html www.diap.org.uk Thanks you for your readership and look forward to your continued involvement as the project matures. Damian Brasher -- Damian Brasher BSc (Open) RHCE MBCS db...@ec... tel: 02380 592888 Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-05-11 12:41:12
|
*Changes:* Security has been greatly improved. The installation will ask for a new diaser password, which the user should note down. Root passwords for the nodes are requested but not stored... to disk. No passwords are written to a configuration file or transferred in plaintext or to nodes. In the future when adjustments are made, either when updating settings or upgrading passwords, passwords will be requested as required in the same way as described above. All this does mean I will need to generate certs then transfer to fix subroutines, run key copy and transfer_key_copy. -- Damian Brasher BSc (Open) RHCE MBCS Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-05-08 12:43:59
|
Enough of the core functionality is present within this release so a move away from alpha-dev to alpha has been made. The ability to choose how many years of operation combined with generation on the fly of the core 'hyper virtual auto-changer' functionality, this can be (re)generated quickly. This allows comprehensive adjustments to be made at any time post installation. The two core functionality algorithms based on DIAP are now in place and working together to give trial users and developers full control over operation. Now to move on to the remainder of the tasks in the development roadmap. http://diaser.svn.sourceforge.net/viewvc/diaser/ROADMAP_DEV -- Damian Brasher http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-04-28 15:51:37
|
I want cron to call a perl script located in each directory which will act as specified by user defined settings rather then try to populate years of cron jobs useing crontab -e - now is the time to do this:) Damian Brasher wrote: > This is the current task, taken from the roadmap > http://diaser.svn.sourceforge.net/viewvc/diaser/ROADMAP_DEV > > 4) The tab*.pl files currently set the timing and bandwidth of each copy > between nodes. The cron jobs are hard coded at the moment and this needs > to be > changed. I propose a single subroutine is written called gen_crons that will > generate three tab_*.pl files according to user requirements then copy these > new files to the nodes and then run them to overwrite the old cron > definitions. > This function can then be utilised by modify and sub configure, and > maybe other subs. > > Damian > > -- Damian Brasher BSc (Open) RHCE MBCS db...@ec... Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
|
From: Damian B. <db...@ec...> - 2009-04-28 10:46:38
|
This is the current task, taken from the roadmap http://diaser.svn.sourceforge.net/viewvc/diaser/ROADMAP_DEV 4) The tab*.pl files currently set the timing and bandwidth of each copy between nodes. The cron jobs are hard coded at the moment and this needs to be changed. I propose a single subroutine is written called gen_crons that will generate three tab_*.pl files according to user requirements then copy these new files to the nodes and then run them to overwrite the old cron definitions. This function can then be utilised by modify and sub configure, and maybe other subs. Damian -- Damian Brasher BSc (Open) RHCE MBCS Systems Admin/Prog OMII-UK ECS Southampton University http://www.diaser.org.uk - working together to make long term digital archives more accessible |
2 messages has been excluded from this view by a project administrator.