Re: [Exist-open] File2Exist is open source and available

[Palmer, E. <ep...@ri...>]

Casey

Files2Exist is pretty simple. It scans a folder (and subfolders if configured to do so) and looks for files that match a pattern (next version).  When a file is detected F2E records the file size. After detecting a file, F2E waits a certain amount of time (can be configured) and looks at the file size again.  When the file size stops changing and has stopped for a cetain amount of time then it moves the file. This prevents partial file moves. Almost all the timers are configurable.

F2E runs 24x7 but could be scheduled via cron or whatever.

The next version of F2E is coded but not fully tested yet:
1) should work in MS windows
2) will allow regular expression file matching and exclude others
3) will transfer binary files

Currently F2E does not maintain file state info across file moves so for SVN that would have to be added. We would need a way to trigger a scan of the SVN repo or do it via a timer and then we would have to be able to compare the revision number or an sha1 value with the last file copied.  

We do have a plugin strategy that uses dynamically loaded JavaBeans for file transforms. That could be extended to have a JavaBean that records state info in a data store and then compares the file info with the last copy.

Turning off file delete (move) and making this a copy for an svn update would be easy.

If you want to help figure out how to detect an svn revision change then I would consider adding this ability in my spare time. If others think this is useful that would be even better.

One alternative would be to have a working directory that you manually update with an svn update command. F2E could detect that and move the files (not the .svn files).  This could be implmemented ASAP with the next version of F2E in testing now. That would be an easy place to start and then we add the svn trigger next into F2E so that it could be automated.

Let me know what you think....

Eric Palmer
University of Richmond
________________________________________
From: Casey Jordan [cas...@jo...]
Sent: Friday, November 25, 2011 12:43 PM
To: Palmer, Eric
Cc: exi...@li...; Levy, Matthew; McMillian, Althea; Gravely, Phillip; Areki, Alem
Subject: Re: [Exist-open] File2Exist is open source and available

Eric,

What method does F2E use to detect when files are moved? Is it just run on a scheduler? Or does it detect when files change etc.

I am looking for a solution that updates various collections in eXist from our svn repository. However I don't really want to move files, just detect when they have changed and copy them into the database.

I was hoping F2E could help with this.

Thanks,

Casey

On Wed, Oct 12, 2011 at 10:08 PM, Palmer, Eric <ep...@ri...<mailto:ep...@ri...>> wrote:
Exist-db community,

This has taken longer than I ever wanted but Files2Exist is now uploaded to http://code.google.com/p/files-2-exist-db/

There are pre-built binaries and source code with a maven pom.

F2E is a little middle-ware command line java application that implements the EAI file transfer pattern. It serves one purpose, watch for files in folders and when the arrive move them to eXst-db.  We have used F2E for more than a year at the U of Richmond. Our CMS can't publish xml directly to eXist-db so we put this in the middle. But over time we have coded java apps that go to 3rd party systems and get xml formatted data that we need to put into eXist-db. So now we have applications that can focus 100% on the tasks of getting 3rd party data and writing the file as xml.  F2E detects these files and writes them to eXist-db.

Since I last discussed this on this forum, we have added a plugin strategy where you can write a java class outside of F2E that implements the file transform pattern.  We are using this to cross reference our CMS asset ID with a more human readable article ID.  But you can do a lot more with File Transforms.

If you know a little java, adding file transforms to a F2E instance is easy and does not require a rebuild of F2E.

We have not testing this on windows. It runs on Mac OSX, and Linux. It should run on Windows.

The documentation at http://code.google.com/p/files-2-exist-db/ is a little rough and will be enhanced overtime.

We would love your feedback.

Eric Palmer
Web Services
University of Richmond
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense.
http://p.sf.net/sfu/splunk-d2d-oct
_______________________________________________
Exist-open mailing list
Exi...@li...<mailto:Exi...@li...>
https://lists.sourceforge.net/lists/listinfo/exist-open



--
--
Casey Jordan
easyDITA a product of Jorsek LLC
"CaseyDJordan" on LinkedIn, Twitter & Facebook
(585) 348 7399
easydita.com<http://easydita.com>


This message is intended only for the use of the Addressee(s) and may
contain information that is privileged, confidential, and/or exempt from
disclosure under applicable law.  If you are not the intended recipient,
please be advised that any disclosure  copying, distribution, or use of
the information contained herein is prohibited.  If you have received
this communication in error, please destroy all copies of the message,
whether in electronic or hard copy format, as well as attachments, and
immediately contact the sender by replying to this e-mail or by phone.
Thank you.