From: Gerrit E.G. H. <Ger...@be...> - 2007-08-07 18:37:13
|
Paolo wrote: > On Mon, Aug 06, 2007 at 09:31:29PM +0200, Gerrit E.G. Hobbelt wrote: > >> - start each file with a versioned header (I'll come back to that later) >> > > that's well established for Fidelis' OSBF > I saw. It's just that I'm looking for a rather more generic solution, which is copy&paste-able when anyone (probably Bill) feels like adding other classifiers to crm114. Say some sort of 'file format/coding practice' thing: rip if off the other classifiers and just add your own classifier constant (so no fancy footwork with index [0] in the data arrays itself or anything like that). >> a) import all acceptable data, or >> > > there's a catch, as the original arch on which to do the export 1st might > not be avail anymore ... > Heh :-) That's where I refer to the legalese in there: 'sorry sir, it's _forward_ compatible as of this release' ;-) The whole point is that I'm trying to get at a mechanism which clearly identifies the data, both in type and version, so that we can develop a 'sure fire' and sane conversion. This while keeping in mind that design/devel/test time is a rather limited resource, so the 'management decision' may well turn out to be to forego the availability of a complete 'conversion' for specific versions (and that may include crm file versions predating this versioning mechanism). Right now, as I see it, you can't provide hard guarantees that conversions will work (and I suspect that, given my goal with crm114, I'll need that sort of thing), as you have several classifiers and software versions, while there's no way to tell them apart in a _guaranteed_ manner: all one can go on is some version info (OSBF et al) and a bit of heuristics. And 'it may work' isn't an option for me when I'm going to employ crm114, so I like to be able to _specifically_ test (and thus support) crm software versions and classifiers. Longwinded paragraphs cut short: I want to end up with a chart which tells me: "You've got crm114 release X and are using classifier C, well, we do support a 'full data transfer' for the current crm114 release." and maybe an additional (sub-)chart which says: "And incidentally, when you have crm114 running on system S, you can also _share_ that classifier's data on system type T using our import/export-based sync system." These charts have three ticks in each cell of their matrices: (a) may work (a.k.a. there's code for this in there) + (b) tested, a.k.a. we got word it works + (c) supported, a.k.a. you may bother us / complain when it isn't working. No tick in your cell on those charts means: you're SOL. Time for a retraining and ditching of the old files, probably. This would solve the problem of the ever lasting questions: can I keep my files or should I start from scratch? For folks that cannot retrain as they go, this 'charted' approach will provide them with a clear decision chart: can/should I upgrade, or shouldn't I? >> b) report the incompatibility and hence the need to 'recreate/relearn' >> the files. >> > > ... and b) might not always be an option. > See above. I'm well aware of that. I'm driving at a mechanism which allows everyone to clearly see when and what can/has been done. That includes you (J.R. User) helping the crm114 team by adding export/import support for those situations where the chart says 'not available' while you need that sort of thing. That also includes collecting and archiving feedback on [user] test results: did their transfer/upgrade work out ok? It's added work, but the benefit is that the upgrade process (and the decision to upgrade) can be fully automated in the end: for unmanned systems: only upgrade when our locally used version + classifier has a tested (and supported?) data migration path towards this new crm114 upgrade release. > yep, but I'd consider a bug (which might be just a TODO) a convertion > util/function which is unable to properly convert our own stuff from arch1 > to arch2, both ways, whatever arch* are. > Such converters won't be exactly trivial (byte swapping, aligning, padding, > etc) but feasable. > That's where the limited design/devel resourcing comes into play: I don't mind if the 'standard' decision is NOT to support/provide a data conversion path. It's understandable that we do so as we don't have an unlimited supply of dev power. But when we do choose to provide a conversion path it's clearly identifiable. (someone may need it and help Bill, you and the others by putting in the dev effort there, just like I'm reviving the Win32 port and adding error checking and stuff along the way) And, BTW, I've been writing that sort of cross-platform stuff more often. It gets a bit wicked when you need to convert VAX/VMS Fortran floating point values to PC/x86 IEEE format, for instance. ;-)) Otherwise, it's just really careful coding and a bit of proper up-front thinking. And then keeping a lookout for register/word-size issues (e.g. 32- vs. 64-bit) throughout the crm implementation, which is the hard part. Padding, endianess, etc. can be handled rather easily: define a 'special struct' with all the basic types in there and load it with a special byte sequence: that gives you endianess and alignment for all basic types. Floating point values need a bit of a special treatment when you travel outside the IEEE realm, but that's doable too. Not trivial, though, indeed. >> The binary format header will include these information items (at least): >> >> - the crm version used to create the file >> - the platform (integer size and format(endianess), floating point size >> and format, structure alignment, etc.) >> - the classifier used to create the file >> - the data content type (some classifiers use multiple files) >> - space for future expansion (this is a research tool too: allow folks >> to add their own stuff which may not fit the header items above) >> > > +file-format version and, since there'll be plenty of space, plain-text > file-format blurb and summary file-stats, so that head -x css would be > just fine to report the relevant things. > Brilliant idea! Hadn't thought about the 'head -x', but I _like_ it. I was thinking about maybe 1 or 2 Kbytes reserved for the header anyway, so, yes, plenty of space of a little informational text up front. A few Kbytes won't hurt. +file-format: yes. In case we find the format needs to be changed again (hopefully not before 2038 ;-) ) Another very good point. >> The approach includes the existence of an export/import tool to convert >> the data to/from a cross-platform portable text format, where >> > > that's the current CSV inter-format, though the converter should be able > to do it at once binary-2-binary. > I've seen the CSV interformat and I was thinking about using that. No bin-2-bin direct stuff, as that would complicate matters beyond control: given the 'cross-platform' tack, it would mean that a developer would have to code - and maintain - software which includes a table of file layout definitions, one for each supported platform (and probably the crm release version too). Compare this to databases: right now I'm in a project where I've found that Oracle cannot copy database files as-is across patch versions (that's the ultra-minor version number), let alone moving the binary database files as-is on to different unix architectures (HPUX vs. Linux, of course with differtent CPUs too). And that makes sense! The point? When Oracle DBAs are used to export-dumping and importing databases running in the many-multi-Gigabyte range to provide an migration/upgrade path for the data stored therein, I'd like to do _exactly_ the same. That means: use the CSV format (probably augmented) as an intermediate. (Or XML when I feel like getting fancy and really 21st century ;-) ) I've done direct bin-2-bin conversions in the past, but they're a true support nightmare. It's doable, but you can have someone spend a serious chunk of his/her life on that alone. And when that person quits supporting it, you're SOL as a tool provider, really. (Imagine your customers use a platform which you didn't support just yet. Maybe a new CPU type even. Can your _design_ of the bin2bin handle that? Or do you need to spend a significant amount of devel effort just to add the generation of these new-CPU-type files to your ware?) The easy way out is to provide all your customers with a single, portable format: they've got the software built on their own machines and who better than the machine itself can convert to/from that portable format? Thus, the conversion effort is off-loaded to the compiler vendor, who has to cope with it anyway. (sscanf/printf/etc.) XML is a good example as a solution invented for solving precisely this very issue. (cross-platform, cross-version, cross-X-whatever data transfer) We might even consider using XML as a replacement for the CSV format, though XML tends to be rather, er, obese, when it comes to data file sizes. XML is hierarchical, so we can easily store our header info and crm classifier data in there, while nicely separated/organized. > for spam filtering, it's easier (and usually better) to start from scratch, > but in other applications hashes DB might be precious stuff, so as people > extends crm114 use to other tasks, such tool might become highly desirable. > Yes, indeed. Verily. <off-topic> I have looked around at software supporting Bayesian/Markovian/etc. statistics and selected crm114 because it looked like it had the right amount of 'vim' (.i.e. lively dev community) while offering a feature set which might cover my needs - or get very close indeed. I intend to use crm114 for spam filtering (when combined with xmail) and for a second purpose: I'm not going to disclose what it is exactly, but think of it as a sort of fuzzy decision-making / monitoring process, which is a bit of a cross-breed between a constraint-driven scheduler and a _learning_ 'fuzzy' discriminator, which has to wade through a slew of 'crap' to arrive at a 'proper' rule or decision. Here I'm more interested in decision _vectors_ (rather small ones) than _scalars_, but I'll tackle that hurdle when I've got crm114 to a state where I can really dive into the classifiers themselves, because I believe right now it only supports single output bits(scalar) (pR?) but I'm not entirely sure there (lacking sufficient algorithm understanding). Anyway, I guess the 'vector solution' would be to use multiple crm (file) instances in parallel: one pR for each decision item in the output vector. Of course, that's a crude way, so the 'clean' approach I originally aiming for was convert crm114 into a library which could be called/used from within my own special purpose software. Alas, that's not a Q4 2007 target anyway. ;-) The problem for me is that I need to understand/learn the algorithm internals for this advanced statistics stuff as that is new to me and I want to understand what it's actually doing, i.e. how this stuff arrives at a decision, as I need to understand the implicit restrictions on the classifiers (and learning methods). Let's just say I don't want to join the mass who can't handle the meaning and implications of 'statistical significance', such as by just grabbing a likely classifier and 'slapping it on'. I fear that would cause some serious burn in the long term. You may have seen from my work so far, that I'm a bit paranoid at times ^H^H^H^H^H^H^H acutely aware of failure conditions, and it would be utterly stupid to fall into that beartrap at a systems level by grabbing this tool and applying it to a problem without really understanding where and what the limitations of the various parts are. I've met too many design decisions _not_ too worry. I've got the idea, I have a 'feeling' that this is the right direction, but it's really still just guesswork regarding feasibility so far. I arrived at crm114 while I had been looking for a decision filter which could easily handle _huge_ inputs for tiny outputs (spam: input = whole emails, output vector size = 1), produce consistent and significant decisions (spam: > 99% filter success rate in a very short learning period) while including a good 'learning' mode: somehow I don't think Bayesian is the bee's knees when it comes to my second goal. And it has been shown it's certainly not the end of it for spam either. And besides, crm114 isn't written in Perl (or some other interpreted language). Which in my world is a big plus. ;-) I don't mind too much if crm114 doesn't work out for goal #2 - though it would be a serious setback - as there's still the spam filter feature which is useful to me. So I don't mind spending some time on this baby to push it to a level where I can sit back, have a beer and say "yeah! Looks good, feels good. Let's do it!" </off-topic> Best regards, Ger |