md5

[Alain Descamps]

Hello,

Md5 is an algoritm, that calculates an encodes string of 32 characters out of a given string of theoretically *any* length. The nice thing is, that this md5 key is almost unique, and the second nice thing is, that you can use it as a subscript in globals.
We built a small add-on, zo that this M code could work :
s x="some string, that can be very,very,very large"
d &zmd5(x,.y)
w y
914c6e2ff8ab6ff576f7086121efa50e

My proposal is :
- If anyone wants to be this in GT.M, we could send the source code. (It is not even that hard to make it yourself..). You could do us a faver by putting it into the next release, it spares some time.
- Or is there already a md5 functionality, we missed ?

Greetings,

Alain

[Steven Estes]

Hi Alain,

There is not currently any md5 functionality built into GT.M But it is not clear exactly what you want to add to GT.M. A function or something else?

From time to time we do add new $Zxxx() functions, $Zxxx intrisic variables, or new Zxxx commands where the "Z" prefix means they are specific to the GTM implementation. We do this only after careful consideration when either there is no other way to do it or the performance gains are obvious or the usage is so widespread as to be beneficial to everyone. Other extensions to the language are done only in conjunction with changes to the M standard. Our own banking application ships with dozens of external calls to C code that don't warrant inclusion into the M language.

Your md5 code sounds like it would be a handy thing and if there was agreement from a sizeable section of the M community that they would like to see a function of this sort in GT.M I'm sure it could be considered.

By the way, in your example, it looks like you could change this to an external function and call it with $&zmd5(x) and return the value y rather than call it as a routine.

Hope this helps.

Steve

[LD Landis]

Hi,
  If there is consideration of implementing md5 in GT.M, please
  please consider building a protocol to the md5 engine so that
  one can send a significant dataset to the engine rather than
  just the contents of a single string.  An example of what I have
  in mind is (pseudo M):

    set f="filename.ext"
    do &md5init() ; initialize the md5 engine
    open f use f for  read buf quit:buf=""  do &md5collect(buf)
    close f
    set md5=$&md5sum()

  This way a large volume of data could be sent in for an
  md5sum.  As stated, the above is pseudo code... hopefully
  the desire is communicated.

Cheers,
  --ldl

[Alain Descamps]

Well,

I would be already happy, if one could do such a thing like :

s x="some string, that can be very,very,very large"
d &zmd5(x,.y)

And yes, if it is in the function form :

s y=$&zmd5(x)

it would be even better, but I am modest, so this is for me a luxury problem.

ldl, your proposal looks a very nice idea,
but looks to me as a good answer on a somewhat different question .

You could always do a zsy command and fetch the result, and this could be satisfying in some cases, where overhead could not harm. But this is just part 1 of the answer on your question..

My original idea was :
i cannot put very large strings into a global subscript,
or, if it is possible one day, i propably will be punished by a time-consuming $data or $get function.
Hey, we said, let's convert this string by a common algoritm, and you'll always get a short string.

Maybe GT.M could do both ?

Greetings,

Alain

[Steven Estes]

What is "very,very,very" long ?? The next production release (V4.4-002) due around the end of the month, while it has some interesting new features, does not have any relief in this area. But there is a field test release also coming out soon based on it that increases the maximum value length for local variables from 32K to 1MB (with a commiserate increase in storage usage) if that helps.

Steve

[Alain Descamps]

We have developed a software for libraries, called 'Brocade', and the only user interface we need is a web browser. We developed it with Cache, but used only the M part of Cache, and actually, our development system works with GT.M. The migration is not finished yet, there are some minor things around, but the engine works fine. It took us about a month, so this seems very short.
One of the things we store in M globals, are the description of books and articles. We often use xml as import/export medium. Some of the softwar is written in Python, where you can put a whole file in one string, if you want. For xml files, this might be interesting to do a read#x Megabyte !.

This limited record and variable length does not compete with other languages.

This is just a suggestion..

Alain

[Steven Estes]

We have had some discussions about this. For globals, allowing a record to span multiple GDS blocks is a major change to the database engine but for local variables, with the new code, raising the value size limit is currently as easy as modifying a compile time variable (#define) and rebuilding GT.M. We left it at 1M just as a sanity check and to "shake it out" as it were. It wouldn't take much work to turn the limit into a soft limit that is (re)settable by a VIEW command or something similar..

By the way, with the new call-in capabiility of V4.4 that allows C code to call M code, you can write some python C classes that allow you to call your M code directly from python if that is your wish. If I find the time to do this myself I'll post the routines as I'm a python aficionado as well.

Steve

[Alain Descamps]

If this new call-in capability works in both ways, that would mean i could have something like a
def setGlobalReference(globalref, value)

a
def getGlobalreference(value, globalref)

and a

def dollarOrder(globalref)

Or ?

Sounds great . I can't wait..

Alain

[Steven Estes]

Not quite at that level of granularity. The call-in functionality allows another language (usually C) to call an M routine . So if you had small M routines that would do the $ORDER or set global values for you, then yes, you could call them in the fashion you describe.

Steve

[Alain Descamps]

So this not the replacement for DAL ?

Alain

[Steven Estes]

Correct. This is not a direct call-in to the lower level GTM functions but a call interface to M routines without having to leave the process. But because of this new support, GTM is now packaged as a shared library so implementation of a DAL interface would be a much easier thing to implement if someone would like to pick that up. Especially since a DAL interface to GTM is defined and exists for VMS.

Steve

[Alain Descamps]

The essential functionality you'll need, is a bidirectional interface to M : python should not only feed M with information, but also get info back.

Maybe by starting an M entry $$Funct^routine(..),
so that the C program could get the return value ?

Or how do you see this ?

One can always put a result in some temp file, and the C program couls read it, but that is not the way we had in mind : let's call this an interface of level 1, level 3 would be better ..

You made me too curious..

Alain

[Steven Estes]

The callin interface is conceptually similiar to external call interface where M calls C except it is going in the opposite direction. The callin interface has the ability to pass in arguments which can be input, output or both, and get a return value and status code. Arguments can be passed by value or reference.

So there is plenty of functionality to allow python to return information without resorting to a "level 1" interface.. :-)

Steve