HPCC Systems 5.0 Release

HPCC Systems 5.0 is now available!

• HPCC Systems downloads page: http://hpccsystems.com/download/free-co ... r-platform.
• HPCC Systems 5.0 Red Book pages: https://wiki.hpccsystems.com/display/hp ... elease+5.0
• Release notes and known limitations: http://hpccsystems.com/download/free-co ... imitations
• List of issues fixed in 5.0: https://track.hpccsystems.com/issues/?filter=11328
• HPCC Systems Community Forums: http://hpccsystems.com/bb/

HPCC Systems 5.0 is a major release including the following significant new features and improvements:

• First internationalised release.
We have provided the mechanism for translating ECL Watch into your native language and have also included translations for some languages including including Chinese, Spanish, Hungarian, Serbian and Brazilian Portugese.

• ECL Watch redesign.
You will already have seen evidence of the facelift in the technical preview released previously. The new ECL Watch uses modern techniques such as in place viewing so you can view related pages more easily. So for example, you can now view Workunit outputs, graphs, timers etc without having to navigate away from the Workunit Details page. The graph view has also been improved to handle larger graphs more efficiently and provide greater control over the details level. Spraying is now a much more efficient process. Where previously each spray had to be actioned separately, you can now spray multiple files in a single action saving time and effort. If you aren’t sure what type of spray to use or need more information to select the correct spray options, use the new hex preview mode which shows the contents of a file on the landing zone in hex for the first 32k. Read more at http://hpccsystems.com/bb/viewtopic.php?f=17&t=1383

• Visualizations of results for a number of chart/graph types including bar, scatter, pie, histogram etc. The ECL Playground includes two samples demonstrating the visualization of results as a bar chart or choropleth. You simply run the examples and click the chart button on completion to see the visualisations. Another way of viewing visualizations in ECL Watch is to include additional resources (such as an index web page) in your ECL code via the enhanced manifest mechanism, which can be viewed from within the workunit or a published query. Helper functions are also provided to facilitate calling Roxie from inside the web pages.

• Embedded language features including the ability to pass records and datasets into embedded functions, and to return them as results from embedded functions. You can also pass records and datasets into embedded functions, and return them as results from embedded functions. Moreover, only those records which are actually required will be evaluated. We’ve also extended the list of embedded languages to include some interesting new ones including MySQL.

• Extended Roxie capabilities including the ability to read and write persist files and launch Thor graphs. To try this out, deploy a cluster that contains both a Roxie and Thor and submit a job. The parts that require Thor will be executed on Thor.

• Roxie query managements GUI features.
We have extended ECL Watch to include a number of features for the management of Roxie queries which were previously available only from the command line. There are also more options available when publishing a query so, for example, you can now specify which cluster you want to get the files from and you can also get files from a remote DALI. You can also set a priority level for your published query and there is a comments field for recording information about a query you may want to refer back to in the future. In this release, you also get more information displayed about your deployed queries within ECL Watch. The queries page now shows a list of files used by the query and also indicates the reason why a query has been suspended. You can also clearly see which files and super files a query uses.

• Errors and warnings.
The ECLCC compiler has always carried out static analysis on your code to spot potentially incorrect ECL code and would generate a warning if any was detected. Sometimes this has resulted in a large numbers of warnings which may be ignored and new/significant warnings might be missed as a result. In 5.0 we have added a new feature which enables you to customize the warning settings to indicate which warnings should be shown as warnings, which should be shown as fatal errors and which should be suppressed altogether.

• Code Generator improvements focusing on generating smaller C++ code which reduces the compile time for large queries.

• Thor performance improvements which try to minimise spilling and increase the amount of row data that is retained in memory so that query throughput times are improved.

Also take note of the 3rd party plugins available for use with HPCC 5.0:

• HPCC Juju Charm.
If you want to get going quickly using HPCC and you are using either Ubuntu 12.04 (Precise) or 13.10 (Saucy), then try using our new HPCC Juju charm. The charm sets up your VMs and installs the HPCC software really easily and quickly. You get a single node system containing both a thor and a roxie cluster, which you can then customize to suit your requirements. So for example, using the HPPC Charm you can add as many nodes as you need or using the HPCC Configuration Manager you can completely reconfigure the clusters according to your requirements. What’s more, you can set up an HPCC Platform on either a local machine (LXC) or on the Amazon Web Services Cloud.
More information and download: http://manage.jujucharms.com/~xwang2713/precise/hpcc

• Nagios integration into HPCC.
This technical preview harnesses Nagios’s monitoring and notification systems to help identify and resolve infrastructure problems before the affect critical processes. Nagios hardware notifications can help keep your system highly available and alerts can assist in pre-emptive maintenance for processes which are down or behaving outside expected parameters to ensure system stability, reliability, and uptime. Scripts and tools are now provided to extract HPCC Platform system metrics and easily integrate that data into Nagios.
Download: http://hpccsystems.com/download/free-co ... dition/all
More information: http://hpccsystems.com/content/hpcc-mon ... al-preview

• WsSQL Web Service (Coming to the HPCC Systems website end July 2014).
This provides a mechanism for accessing HPCC data and published queries via SQL queries, supporting both ad-hoc and prepared queries. It provides an easy way to create read only connections from any SQL connection type to HPCC. There are a number of interesting benefits to be gained from using this service enabling you to harness the full power of HPCC ‘under the covers’. Your SQL requests generate ECL code which is submitted, compiled and executed on your target cluster. This means you can use the data processing features of HPCC without needing to learn and write ECL code. Moreover, you also get to take advantage of the Automatic Index fetching capabilities which produce faster data fetches. This service also supports simple SQL SELECT or CALL syntax so you can access HPCC data files as DB tables and published queries as DB Stored Procedures.