Running GM in batch or interactive mode

Kenneth Xu
2012-11-03
2013-03-27
1 2 > >> (Page 1 of 2)
  • Kenneth Xu
    Kenneth Xu
    2012-11-03

    Hi All,

    I searched the mailing list and forum but pardon me if the same topic did show up before and I missed it.

    When processing large amount of images, it can be expensive (30-50% overhead in my test) to start a new process every time to process an image. Yes, I'm aware of "Mogrify". But that's too limited for my use case.

    Thus, I'm wondering if we can make GM to run in interactive or batch mode? So that I can run large amount of commands in such a way like:

    cat my-gm-command-file | gm

    I looked at the code and did a quite prototype and working pretty well. So is there any interested to make this a feature?

    My motivation of this is to integrate GM with Java. I know there are two Java integrations available, JMagick and im4j. Both a good products but I cannot use either because (1) JMagick is JNI based, a failure in GM can crash our Java server which needs 24/7 uptime; (2) im4j starts a new process for each GM command. Using batch mode, I can start a number of GM processes from Java to form a GM process pool. That way I can enjoy both performance and reliability.

    Your feedback and comments will be very much appreciated.

    -Ken

    P.S.
    Test output in interactive mode:

    $ GM/utilities/gm
    GM> help
    GraphicsMagick 1.3.17 2012-10-13 Q8 http://www.GraphicsMagick.org/
    Copyright (C) 2002-2012 GraphicsMagick Group.
    Additional copyrights and licenses apply to this software.
    See http://www.GraphicsMagick.org/www/Copyright.html for details.
    Usage: /home/BNWEB/kxu/GM/utilities/.libs/lt-gm help command [options ...]
    Where commands include:
        animate - animate a sequence of images
      benchmark - benchmark one of the other commands
        compare - compare two images
      composite - composite images together
        conjure - execute a Magick Scripting Language (MSL) XML script
        convert - convert an image or sequence of images
        display - display an image on a workstation running X
           help - obtain usage message for named command
       identify - describe an image or image sequence
         import - capture an application or X server screen
        mogrify - transform an image or sequence of images
        montage - create a composite image (in a grid) from separate images
           time - time one of the other commands
        version - obtain release version
    OK
    GM> convert jpg/144107710.jpg -resize 60x60^ test.jpg
    OK
    GM> convert badname
    /home//ken/GM/utilities/.libs/lt-gm convert: Request did not return an image.
    NG
    GM>
    
     
  • It seems that you have implemented a batch mode which is very similar to what I had already been planning to implement (I had been planning 'gm batch').  It was already on my "TODO" list.  As you say, it is a good fit for how GM is put together since GM uses a single C function to access all command-line functionality.  The main issues are with how errors should be handled, and if there should be a syntax to allow use of an image produced in one statement to be used later without use of a disk file (there is already a syntax which should allow this).

    Are you willing bring your work to beyond prototype level and contribute it to the GraphicsMagick project?

     
  • Kenneth Xu
    Kenneth Xu
    2012-11-04

    Hey Bob,
    Thank you for your comment. I'm glad to know that was on your TODO list.

    > The main issues are with how errors should be handled,

    I have to admit my plan was very much limited to my use case but it may fit in general use:
    1. In interactive mode, human will be reading the same error message produced in single command line mode.
    2. When integrating with Java, I'll have java process to read stdout and stderror of GM until a status line is received. In the example I posted earlier, I have used OK and NG. We can make it configurable.

    > and if there should be a syntax to allow use of an image produced in one statement to be used later without use of a disk file (there is already a syntax which should allow this).

    I have never thought about but I like the idea. I assume the new syntax has to be compatible with existing single command line syntax. Then this feature can be an enhancement on top of basic batch mode.

    > Are you willing bring your work to beyond prototype level and contribute it to the GraphicsMagick project?

    Absolutely! I'll get back to you with a patch soon. Should I open a new issue ticket or there is already one?

    Thanks,
    Ken

     
  • See coders/mpr.c for an existing coder ('mpr') which supports caching decoded images and referring to them later. An operation can use the MPR coder to "write" and a subsequent operation can refer to the image which was "written".  I already take advantage of this to eliminate some repeated read operations in 'mogrify'.  There should be a way to cause a cached image to be removed from the cache so a long script can continue without wasting memory.  This facility will work between 'shell' commands as long as InitializeMagick()/DestroyMagick() are executed only when entering/exiting the shell mode.

    For errors, there should be a way to specify best-effort (blow past any errors), or quit on first error. This could be done via 'batch' sub-command options.

    While a default GM> prompt seems attractive, I still prefer 'gm batch' so that it is a GM sub-command because it does not change existing behavior, and because it allows adding other similar modes in the future.  For example, this simple batch scripting does not support the features of a proper language such as looping, conditions, and variables.  Something else on my dream TODO list is to add a Lua language (http://en.wikipedia.org/wiki/Lua_%28programming_language%29) scripting extension which is assured to be included in every build of GraphicsMagick so that it can always be depended on and even used to implement (re-implement?) GM high level commands.

    Even the simple shell extension should consider use with scripts in disk files with GM specified as the interpreter via POSIX bang syntax in the first line:

    #!/usr/bin/gm batch

    In this case, there would be a file argument provided by the operating system which is the script to open and execute.

    There is a 'patches' section in the GraphicsMagick tracker on SourceForge where you can upload patches and they will be tracked just like a bug report.

    Bob

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-19

    Hi Bob,

    Sorry to respond late, but I didn't forget this. Nov to Jan is the critical season in retail so I was buried in other things. But now I'm back on this and here is what I'm thinking.

    What you have described are all great things and I wish I would have enough time (and skill) implement them all. That being said, would you consider a phased approach? I can implement the first version of the batch by leaving out 'mpr'. We can discuss about the options and command formats to make sure whatever we choose will allow further development of advanced features.

    MayI propose below syntax and options for batch mode phase one?

    gm batch 
    -prompt off|<text>
    Specify text as command prompt or "off" to disable command prompt.
    Default to "GM> " when standard input is a TTY, otherwise off.
    -stoponerror on|off
    When set to on, batch execution exits premature whenever it encounters
    error executing a command. Default is off, which means batch execution
    continues regardless of the result of each command in batch.
    -escape unix|windows
    Unix style: using double quotes or backslash or a combination of them.
    Windows stype: use double quites only.
    Default is platform dependent.
    -feedback on|off
    Output a feedback to indicate the status of command execution. Default is off.
    -ok <text>
    When feedback is on, output given text when execution of command is sucessful. Default is "OK".
    -ng <text>
    When feedback is on, output given text when error executing the command. Default is "NG".

    when batch-file is not given, it reads from standard input.

    Please comment.

    Thanks,
    Ken

     
  • Ignoring the "dream" of a high-level scripting language (save for later), what you propose seems pretty good to me.  I am not sure why you would not want to support single-quotes on Windows as well since you are implementing the shell and don't need to depend on the Windows command shell for that.  It seems that behavior should be the same on Windows and Unix.

    Support for MPR automatically exists except for being easily able to deallocate a MPR image somewhere mid-stream in the script.  A new command option or even a new command could be added in order to manage MPR images (e.g. create from file, list, delete them).  Your scripting implementation would not need to know anything about it.

    Note that GM's command line parsing has special support for @file syntax in that the content of 'file' is injected into the command line and expanded into an argument list.  This helps avoid command-line length limitations and quoting issues.  I don't remember exactly how this works any more.

    Things to think about while you are doing the implementation is signal-handling (in case someone hits CTRL-C), progress monitoring, and trace logging.  All of this might just work based on the existing implementation, or perhaps it will need some tweaks to work better.

    Bob

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-19

    BTW, I noticed that we have TRUE, True and MagickTrue. Same set for false. I was trying to figure out a convention of when to use which but I guess better ask here.

     
  • Use MagickBool as the type and MagickFalse and MagickTrue as the values.  There is also MagickPassFail with values MagickPass and MagickFail and in the past boolean values were used for the same things (confusing).  I have been converting code gradually as I am working on the source modules.

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-20

    > It seems that behavior should be the same on Windows and Unix.
    That is actually easier for me as I otherwise I will need to provide two different implementations.

    But my only concern is that windows file path separator is '\', which is an escape character in Unix.

    For example, if you need to pass some text that contains both single and double quote: , you would use "She said: \"Let's go.\"" in Unix. But  becomes  according to the Unix shell escape rule.

     
  • The backslash escaping is a bit of a quandary.  Windows quoting/escaping syntax is weaker than POSIX quoting/escaping and Windows also uses syntax which causes harm/inconvenience for POSIX.  Windows (and X11 GUI desktops) also encourage spaces in paths which requires some form of escaping (double quote is all that works). Do what you think is best.

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-21

    Quick question. Major methods are defined mostly in the order of the name but not consistently. Is this the conversion that I need to follow or this just happen to be that way.

    ~/GM/magick$ grep '^%.*%$' command.c | grep '  ' | sed 's///g'
    NormalizeSamplingFactor
    AnimateUsage
    BenchmarkUsage
    BenchmarkImageCommand
    CompareImageCommand
    CompareUsage
    CompositeImageCommand
    CompositeUsage
    ConvertImageCommand
    ConvertUsage
    ConjureUsage
    DisplayUsage
    GMUsage
    HelpCommand
    IdentifyImageCommand
    IdentifyUsage
    LiberateArgumentList
    MagickCommand
    MogrifyImageCommand
    MogrifyUsage
    MontageImageCommand
    MontageUsage
    ImportImageCommand
    ImportUsage
    TimeUsage
    TimeImageCommand
    VersionCommand
    RegisterCommand
    GMCommand

     
  • It may be that some static functions were listed early in order to avoid needing to add explicit prototypes.  In general try to follow alphabetic order unless it is more convenient to do otherwise.  The source file is pretty large.

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-26

    OK, I think I have all the functionality implemented. Now I'll clean up the code a bit. A few questions.
    1. I'll follow the existing style as much as I can but other than that, please let me know if there is any specific things I need to pay attention to.
    2. Are you open to move some functions out to another file, e.g. command line parser maybe a good candidate.
    3. I don't have Windows development environment setup so I didn't test there. Is that OK with you? is there somebody in team can help to test?

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-26

    two difference styles of block indent. seen for "if", "for", "while", "switch" and etc. which one to follow?

      for (p=color_list; p != (const ColorInfo *) NULL; )
      {
        color_info=p;
        p=p->next;
        DestroyColorInfoEntry(color_info);
      }

      for (p=color_list; p != (const ColorInfo *) NULL; )
        {
          color_info=p;
          p=p->next;
          DestroyColorInfoEntry(color_info);
        }

     
  • The main reason to put functions into another file would be if they are reusable by some other code.

    There is currently a problem with GraphicsMagick in that even with the "MagickExport" declarations (which do work for Windows), the 'extern' symbols are all visible to users of the shared library on Unix and so they become part of the actual API/ABI because someone might use them.  I do plan to fix this someday (via a linker specification) but it requires a major release in order to do so and requires a lot of testing.  Due to this shortcoming, it is best if all the code remain together (as static functions) unless it is generally reusable or actually reused by other modules.

    I am thinking that sometime in the future the GM "command" support code will be moved to its own shared library (perhaps along with the X11 code) so that normal C API users are not exposed to the larger code and the dependencies the command-line code introduces.

    I have a Windows development environment and can take care of any Windows porting issues.  Key things to keep in mind for Windows is that 'int' and 'long' are 32-bit whereas pointers and size_t may be 32 or 64-bit.  Otherwise I doubt that there is much you might notice.

    Bob

     
  • As far as code formatting goes, the original format was a unique "Cristy" format which I was never able to quite match with any indent tool or editor due to being somewhat inconsistent.  The new code I write is formatted using standard "GNU" style conventions which are the default for the GNU EMACS editor, and GNU indent.  The GNU convention indents the brace a bit after 'for' whereas original "Cristy" style put the brace directly under the "for".  If achieving this is a problem, I can reformat your code a bit before its first commit.

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-27

    OK, I'll try to stay with GNU convention and keep all functions static in one filed. I didn't pay attention and thought the "command" support is already separated from API.

    My concern on the Windows part is mainly on line ending and the common functions that I used. For example I need a way to detect if stdin is a TTY. In Windows, it's _isatty in io.h. Thanks for offering help to test.

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-27

    I'd like to get some get feedback on the help text that I have below. Let me know if there is any change necessary.

    ken@Ubuntu:~/GM$ utilities/gm help
    GraphicsMagick 1.3.17 2012-10-13 Q8 http://www.GraphicsMagick.org/
    Copyright (C) 2002-2012 GraphicsMagick Group.
    Additional copyrights and licenses apply to this software.
    See http://www.GraphicsMagick.org/www/Copyright.html for details.
    Usage: utilities/gm help command [options ...]
    Where commands include: 
          batch - run GraphicsMagick in interactive or batch mode
      benchmark - benchmark one of the other commands
        compare - compare two images
      composite - composite images together
        conjure - execute a Magick Scripting Language (MSL) XML script
        convert - convert an image or sequence of images
           help - obtain usage message for named command
       identify - describe an image or image sequence
        mogrify - transform an image or sequence of images
        montage - create a composite image (in a grid) from separate images
           time - time one of the other commands
        version - obtain release version
    [code]
    
     
  • Kenneth Xu
    Kenneth Xu
    2013-01-27

    Spam detector doesn't let me put them all in one post. Sorry to split them into multiple.

    ken@Ubuntu:~/GM$ utilities/gm batch -help
    GraphicsMagick 1.3.17 2012-10-13 Q8 http://www.GraphicsMagick.org/
    Copyright (C) 2002-2012 GraphicsMagick Group.
    Additional copyrights and licenses apply to this software.
    See http://www.GraphicsMagick.org/www/Copyright.html for details.
    Usage: utilities/gm batch [options ...] [file]
    Where options include:
      -escape unix|windows force use Unix or Windows escape format for command line
                           argument parsing, default is platform dependent
      -feedback on|off     print text (see -ok and -ng options) feedback after each
                           command to indicate the result, default is off
      -help                print program options
      -ng text             when feedback is on, output the designated text if the
                           command returns error, default is \'NG\'
      -ok text             when feedback is on, output the designated text if the
                           command executed successfully, default is \'OK\'
      -prompt text         use the given text as command prompt. use text \'off\' or
                           empty string to turn off prompt. default is \'GM> \' when
                           standard input is a TTY, otherwise prompt is off
      -stop-on-error on|off
                           when turned on, batch execution quits prematurely when
                           any command returns error
    Unix escape allows the use backslash(\\), single quote(\') and double quote(\") in
    the command line. Windows escape only uses double quote(\").  For example,
        Orignal             Unix escape              Windows escape
        [a\\b\\c\\d]           [a\\\\b\\\\c\\\\d]             [a\\b\\c\\d]
        [Text with space]   [Text\\ with\\ space]      [\"Text with space\"]
        [Text with (\")]     [\'Text with (\")\']        [\"Text with (\"\")\"]
        [Mix: \"It\'s a (\\)\"] [\"Mix: \\\"It\'s a (\\\\)\\\"\"] [\"Mix: \"\"It\'s a (\\)\"\"\"]
    [code]
    
     
  • Kenneth Xu
    Kenneth Xu
    2013-01-27

    Entering batch mode

    ken@Ubuntu:~/GM$ utilities/gm batch
    GraphicsMagick 1.3.17 2012-10-13 Q8 http://www.GraphicsMagick.org/
    Copyright (C) 2002-2012 GraphicsMagick Group.
    Additional copyrights and licenses apply to this software.
    See http://www.GraphicsMagick.org/www/Copyright.html for details.
    GM> help
    Usage: help command [options ...]
    Where commands include: 
      benchmark - benchmark one of the other commands
        compare - compare two images
      composite - composite images together
        conjure - execute a Magick Scripting Language (MSL) XML script
        convert - convert an image or sequence of images
           help - obtain usage message for named command
       identify - describe an image or image sequence
        mogrify - transform an image or sequence of images
        montage - create a composite image (in a grid) from separate images
            set - change batch mode option
           time - time one of the other commands
        version - obtain release version
    GM> help set
    Usage: set [options ...]
    Where options include:
      -escape unix|windows force use Unix or Windows escape format for command line
                           argument parsing, default is platform dependent
      -feedback on|off     print text (see -ok and -ng options) feedback after each
                           command to indicate the result, default is off
      -help                print program options
      -ng text             when feedback is on, output the designated text if the
                           command returns error, default is \'NG\'
      -ok text             when feedback is on, output the designated text if the
                           command executed successfully, default is \'OK\'
      -prompt text         use the given text as command prompt. use text \'off\' or
                           empty string to turn off prompt. default is \'GM> \' when
                           standard input is a TTY, otherwise prompt is off
      -stop-on-error on|off
                           when turned on, batch execution quits prematurely when
                           any command returns error
    Unix escape allows the use backslash(\\), single quote(\') and double quote(\") in
    the command line. Windows escape only uses double quote(\").  For example,
        Orignal             Unix escape              Windows escape
        [a\\b\\c\\d]           [a\\\\b\\\\c\\\\d]             [a\\b\\c\\d]
        [Text with space]   [Text\\ with\\ space]      [\"Text with space\"]
        [Text with (\")]     [\'Text with (\")\']        [\"Text with (\"\")\"]
        [Mix: \"It\'s a (\\)\"] [\"Mix: \\\"It\'s a (\\\\)\\\"\"] [\"Mix: \"\"It\'s a (\\)\"\"\"]
    GM>
    
     
  • Kenneth Xu
    Kenneth Xu
    2013-01-27

    Sorry, not sure where come those backslashes. Here is the correct option for both

    Where options include:
      -escape unix|windows force use Unix or Windows escape format for command line
                           argument parsing, default is platform dependent
      -feedback on|off     print text (see -ok and -ng options) feedback after each
                           command to indicate the result, default is off
      -help                print program options
      -ng text             when feedback is on, output the designated text if the
                           command returns error, default is 'NG'
      -ok text             when feedback is on, output the designated text if the
                           command executed successfully, default is 'OK'
      -prompt text         use the given text as command prompt. use text 'off' or
                           empty string to turn off prompt. default is 'GM> ' when
                           standard input is a TTY, otherwise prompt is off
      -stop-on-error on|off
                           when turned on, batch execution quits prematurely when
                           any command returns error
    Unix escape allows the use backslash(\), single quote(') and double quote(") in
    the command line. Windows escape only uses double quote(").  For example,
        Orignal             Unix escape              Windows escape
        [a\b\c\d]           [a\\b\\c\\d]             [a\b\c\d]
        [Text with space]   [Text\ with\ space]      ["Text with space"]
        [Text with (")]     ['Text with (")']        ["Text with ("")"]
        [Mix: "It's a (\)"] ["Mix: \"It's a (\\)\""] ["Mix: ""It's a (\)"""]
    [code]
    
     
  • For batch help text I like

    batch - issue multiple commands in interactive or batch mode

    I like the batch suboptions as well.  The only thing debatable is the -ng option (stands for "not good"?).  People might tend to think of PASS/FAIL or OK/NOK.

    Bob

     
  • Kenneth Xu
    Kenneth Xu
    2013-01-27

    batch - issue multiple commands in interactive or batch mode

    changed.

    Yes, -ng stands for "not good". I'm open to the option name and default value. pass/fail?

     
  • I prefer "-pass" and "-fail", and default setting both on.

     
  • There is probably no perfect interface but the default text output should be really easy to parse.  PASS/FAIL are easier to parse than OK/NOK because the NOK includes 'OK'.

    The next people people will want from the command line is command line editing support and history via GNU readline or BSD Libedit.  The snowball is rolling.  :-)

    I am not suggesting to link more libraries into GraphicsMagick though.

     
1 2 > >> (Page 1 of 2)