metaf2xml.pl is a very basic and simple script, unfortunately too basic
for this task. It merely demonstrates how the perl modules for parsing
and writing XML could be used (with a few features added).
First of all the script cannot handle the input format provided by your
example. To quote the manual for this program:
----
The input is expected to consist of complete messages, as specified by
WMO Manual No. 306, without additions or contractions for distribution
(e.g. providing section 0 of SYNOPs only once for several messages, or a
trailing "=" (equal sign)).
----
So this script could only process the input from your example if it was
changed to:
i.e. add header for each message, one line per message, no trailing =.
Neither does the script expect headers like 'SAUS70 KWBC 191518' or
'SMUS71 KWBC 201200' (which describe the type of message, who created
it, and the region it relates to). These headers, the newlines within a
message, and the trailing '=' were added for distributing the messages,
and are not part of a METAR or SYNOP message as such. Depending on the
source of messages, there may be even additional control characters.
However, the script does understand the keywords 'METAR', 'TAF', and
'SYNOP', if they are alone on a line, and switches to the corresponding
message type as default message type. So, to make your example work, the
input needs to be changed to:
Since version 1.36, there is a script to convert messages of various
input sources to the correct format: metafsrc2raw.pl. It can only handle
one input type at a time, so in your example it must be run twice, once
with the option -F metaf_XXX, and once with -F synop_XXX, where XXX
depends on the data source and can be 'nws' or 'cod' or 'fsu'. It's all
described (sufficiently, I hope :-) in the manual for metafsrc2raw.pl as
well. The script also prepends the WMO message type (SA for METAR and so
on), so this needs to be removed, too, before feeding it to
metaf2xml.pl. Also, in your input the starting line seems be missing,
which looks like e.g.:
####018000228####
for messages from NOAA/NWS. This is needed by metafsrc2raw.pl to
recognise the start of a distributed message.
I haven't sufficient knowledge how it can be done in operating systems
that use \ as directory delimiters, but with sh/ksh/bash it could be
done as follows (e.g. for data from NOAA/NWS and the input in the
files METARFILE and SYNOPFILE):
I assume you are still using it like that, and I also assume that 'code' contains the input string, possibly with multiple messages.
Now what I didn't notice the first time is that 'code' is prefixed by 'SYNOP ' on the command line. That's fine, as long as there is a SYNOP message following it in 'code'. However, in your example, you send 2 SYNOP messages:
AAXX 19124 72659 ...
AAXX 19124 72462 ...
Together with the 'SYNOP ' from the command line that becomes:
SYNOP AAXX 19124 72659 ...
AAXX 19124 72462 ...
That can also be seen in the output: the attribute 's' of the metar/taf/synop node in the XML contains the input string:
<synop s="SYNOP AAXX 19124 72659 ...
The message type for the first message (and the first message, only) is explicitly given with the keyword 'SYNOP '. However, as the default message type for metaf2xml.pl is 'METAR', the second message is interpreted as a METAR:
<metar s="AAXX 19124 72462 ...
You have several choices to avoid that:
- First, if you know that only SYNOP messages are in your input, the option '-T SYNOP' for metaf2xml.pl can be used to change the default message type. Then the 'SYNOP ' on the command line is not needed.
- The same result could be achieved by giving 'SYNOP' _and a newline_ as argument on the command line.
- The third possibility would be to prefix _each_ line with 'SYNOP ', again without the 'SYNOP ' argument on the command line.
Your second example in the previous post comes close to the correct solution - if you would have omitted the 'SYNOP' on the command line. The output shows that the 2 keywords are concatenated and constitute the first message:
<synop s="SYNOP SYNOP">
This, however, is not a valid message, so an error is produced and the default message type remains METAR. The following lines of the input are interpreted as METAR:
Without the 'SYNOP ' on the command line, this example should work.
The background to all of this is that it is mandatory to pass the correct message type to the parser module. metaf2xml.pl allows to give it as keyword for a message, simply at the beginning of a message. If the message does not start with a keyword, metaf2xml.pl uses the default message type, which is tracked internally. Initially, it is 'METAR'. It can be changed from the command line (with the option -T). It can also be changed by a special line anywhere in the input, containing only the new default message type: METAR, TAF, or SYNOP.
Hi martin
I want use metaf2xml in windows, I install activeperl but I can not work with it, I want decode METAR and TAF bulletins and save them into Database(SQL Server). please help me.
Best Regards
Hamid
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you mean you cannot work with ActivePerl or you cannot work with metaf2xml? Have you installed metaf2xml already? If not, at which step in the installation instructions did you get stuck?
Kind regards,
Thomas
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi
I am using the metaf2xml tool in my c# code
c:\\perl\\bin\\perl.exe", "metaf2xml.pl -x " + parserOutputFilename + " \"SYNOP " + code + "\"");
it works fine except when the code have more than one stations
here is a synop and metar exemple code that I send to the metaf2xml.pl tool, with these exemples, only the first station gets to be parsed
--------------------------------------------------
SAUS70 KWBC 191518
METAR
KVBT 191515Z AUTO 09017G23KT 10SM BKN025 OVC080 11/04 A3030 RMK
AO1=
KVIS 191515Z 16003KT 5SM CLR 11/09 A3003 RMK AO1=
KVTP 191518Z AUTO 00000KT 10SM CLR 01/M07 A3037 RMK AO2=
KVVV 191516Z AUTO 00000KT 10SM CLR M04/M09 A3049 RMK AO2=
KWVL 191515Z AUTO 27005G15KT 10SM OVC120 08/M10 A2990 RMK AO1=
KYKN 191515Z AUTO 10011KT 10SM FEW085 02/M02 A3044 RMK AO1=
KEVM 191515Z AUTO 00000KT 10SM CLR M07/M15 A3048 RMK AO1=
--------------------------------------
SMUS71 KWBC 201200
AAXX 20124
72659 35966 60000 10033 21011 39686 40170 55012 91153 333 10061
20011 555 92012=
72548 32966 41208 11006 21044 39970 40296 51004 91154 333 10094
21022 555 92012=
72462 35966 00904 10017 21039 37723 40160 56001 91152 333 10183
21011 555 92012=
thnaks
Martin
Hi Martin,
metaf2xml.pl is a very basic and simple script, unfortunately too basic
for this task. It merely demonstrates how the perl modules for parsing
and writing XML could be used (with a few features added).
First of all the script cannot handle the input format provided by your
example. To quote the manual for this program:
----
The input is expected to consist of complete messages, as specified by
WMO Manual No. 306, without additions or contractions for distribution
(e.g. providing section 0 of SYNOPs only once for several messages, or a
trailing "=" (equal sign)).
----
So this script could only process the input from your example if it was
changed to:
KVBT 191515Z AUTO 09017G23KT 10SM BKN025 OVC080 11/04 A3030 RMK AO1
KVIS 191515Z 16003KT 5SM CLR 11/09 A3003 RMK AO1
...
i.e. basically one line per message, no trailing =, and:
AAXX 20124 72659 35966 60000 10033 21011 39686 40170 55012 91153 333 10061 20011 555 92012
AAXX 20124 72548 32966 41208 11006 21044 39970 40296 51004 91154 333 10094 21022 555 92012
...
i.e. add header for each message, one line per message, no trailing =.
Neither does the script expect headers like 'SAUS70 KWBC 191518' or
'SMUS71 KWBC 201200' (which describe the type of message, who created
it, and the region it relates to). These headers, the newlines within a
message, and the trailing '=' were added for distributing the messages,
and are not part of a METAR or SYNOP message as such. Depending on the
source of messages, there may be even additional control characters.
However, the script does understand the keywords 'METAR', 'TAF', and
'SYNOP', if they are alone on a line, and switches to the corresponding
message type as default message type. So, to make your example work, the
input needs to be changed to:
METAR
KVBT 191515Z AUTO 09017G23KT 10SM BKN025 OVC080 11/04 A3030 RMK AO1
KVIS 191515Z 16003KT 5SM CLR 11/09 A3003 RMK AO1
KVTP 191518Z AUTO 00000KT 10SM CLR 01/M07 A3037 RMK AO2
KVVV 191516Z AUTO 00000KT 10SM CLR M04/M09 A3049 RMK AO2
KWVL 191515Z AUTO 27005G15KT 10SM OVC120 08/M10 A2990 RMK AO1
KYKN 191515Z AUTO 10011KT 10SM FEW085 02/M02 A3044 RMK AO1
KEVM 191515Z AUTO 00000KT 10SM CLR M07/M15 A3048 RMK AO1
SYNOP
AAXX 20124 72659 35966 60000 10033 21011 39686 40170 55012 91153 333 10061 20011 555 92012
AAXX 20124 72548 32966 41208 11006 21044 39970 40296 51004 91154 333 10094 21022 555 92012
AAXX 20124 72462 35966 00904 10017 21039 37723 40160 56001 91152 333 10183 21011 555 92012
Since version 1.36, there is a script to convert messages of various
input sources to the correct format: metafsrc2raw.pl. It can only handle
one input type at a time, so in your example it must be run twice, once
with the option -F metaf_XXX, and once with -F synop_XXX, where XXX
depends on the data source and can be 'nws' or 'cod' or 'fsu'. It's all
described (sufficiently, I hope :-) in the manual for metafsrc2raw.pl as
well. The script also prepends the WMO message type (SA for METAR and so
on), so this needs to be removed, too, before feeding it to
metaf2xml.pl. Also, in your input the starting line seems be missing,
which looks like e.g.:
####018000228####
for messages from NOAA/NWS. This is needed by metafsrc2raw.pl to
recognise the start of a distributed message.
I haven't sufficient knowledge how it can be done in operating systems
that use \ as directory delimiters, but with sh/ksh/bash it could be
done as follows (e.g. for data from NOAA/NWS and the input in the
files METARFILE and SYNOPFILE):
(echo METAR; metafsrc2raw.pl -F metaf_nws METARFILE | sed 's/...//';
echo SYNOP; metafsrc2raw.pl -F synop_nws SYNOPFILE | sed 's/...//';
) | metaf2xml.pl -x parserOutputFilename ...
If you need more info, just ask.
HTH,
Thomas
Thanks Thomas
I will give it a try...
Hi Thomas
It works fine for the metar
except that for the synop, it only recognize the first station
here is the code I send to the parser
AAXX 19124 72659 35966 00504 11067 21094 39833 40331 53005 91153 333 10050 21067 555 91912
AAXX 19124 72462 35966 02317 10167 21150 37696 40123 55006 92352 333 10194 21010 555 92100
Here is the result
<?xml version="1.0" encoding="UTF-8"?>
<reports query_start="Mon Jun 15 19:30:20 2009">
<synop s="SYNOP AAXX 19124 72659 35966 00504 11067 21094 39833 40331 53005 91153 333 10050 21067 555 91912">
<obsStationType s="AAXX">
<stationType v="AAXX"/>
</obsStationType>
<obsTime s="1912">
<day v="19"/>
<hour v="12"/>
<minute v="00"/>
</obsTime>
<reportModifier s="AUTO">
<modifierType v="AUTO"/>
</reportModifier>
<windIndicator s="4">
<windUnit v="KT"/>
</windIndicator>
<obsStationId s="72659">
<id v="72659"/>
<region v="IV"/>
</obsStationId>
<precipInd s="3">
<precipIndVal v="3"/>
</precipInd>
<wxInd s="5">
<wxIndVal v="5"/>
</wxInd>
<baseLowestCloud s="9">
<from v="8500" u="FT" q="isEqualGreater"/>
</baseLowestCloud>
<visPrev s="66">
<distance v="10" rp="1" u="SM"/>
</visPrev>
<totalCloudCover s="0">
<oktas v="0"/>
</totalCloudCover>
<sfcWind s="0504">
<wind>
<dir v="50" rp="4" rn="5"/>
<speed v="4" u="KT"/>
</wind>
</sfcWind>
<temperature s="11067 21094">
<air>
<temp v="-6.7" u="C"/>
</air>
<dewpoint>
<temp v="-9.4" u="C"/>
</dewpoint>
<relHumid1 v="81.05"/>
<relHumid2 v="81.34"/>
<relHumid3 v="81.21"/>
<relHumid4 v="81.08"/>
</temperature>
<stationPressure s="39833">
<hPa v="983.3"/>
</stationPressure>
<SLPhPa s="40331">
<hPa v="1033.1"/>
</SLPhPa>
<pressureChange s="53005">
<period v="3h"/>
<pressureChangeVal v="0.5" u="hPa"/>
<pressureTendency v="3"/>
</pressureChange>
<exactObsTime s="91153">
<hour v="11"/>
<minute v="53"/>
</exactObsTime>
<section3 s="333">
<tempMaxPeriod s="10050">
<period v="1d"/>
<temp v="5.0" u="C"/>
</tempMaxPeriod>
<tempMinPeriod s="21067">
<period v="12h"/>
<temp v="-6.7" u="C"/>
</tempMinPeriod>
</section3>
<section5 s="555">
<obsTime s="91912">
<day v="19"/>
<hour v="12"/>
<minute v="00"/>
</obsTime>
</section5>
</synop>
<metar s="AAXX 19124 72462 35966 02317 10167 21150 37696 40123 55006 92352 333 10194 21010 555 92100">
<ERROR errorType="obsTime" s="AAXX <@> 19124 ..."/>
<obsStationId s="AAXX">
<id v="AAXX"/>
</obsStationId>
</metar>
</reports>
I also tried with SYNOP TAG in the following code with no success
SYNOP
AAXX 19124 72659 35966 00504 11067 21094 39833 40331 53005 91153 333 10050 21067 555 91912
AAXX 19124 72462 35966 02317 10167 21150 37696 40123 55006 92352 333 10194 21010 555 92100
result:
<?xml version="1.0" encoding="UTF-8"?>
<reports query_start="Mon Jun 15 19:34:41 2009">
<synop s="SYNOP SYNOP">
<ERROR errorType="obsStationType" s="<@> SYNOP "/>
</synop>
<metar s="AAXX 19124 72659 35966 00504 11067 21094 39833 40331 53005 91153 333 10050 21067 555 91912">
<ERROR errorType="obsTime" s="AAXX <@> 19124 ..."/>
<obsStationId s="AAXX">
<id v="AAXX"/>
</obsStationId>
</metar>
<metar s="AAXX 19124 72462 35966 02317 10167 21150 37696 40123 55006 92352 333 10194 21010 555 92100">
<ERROR errorType="obsTime" s="AAXX <@> 19124 ..."/>
<obsStationId s="AAXX">
<id v="AAXX"/>
</obsStationId>
</metar>
</reports>
thanks
Martin
Hi Martin,
I think I know what the problem is. In your first post, you included the command line:
> c:\\perl\\bin\\perl.exe", "metaf2xml.pl -x " + parserOutputFilename + " \"SYNOP " + code + "\"");
I assume you are still using it like that, and I also assume that 'code' contains the input string, possibly with multiple messages.
Now what I didn't notice the first time is that 'code' is prefixed by 'SYNOP ' on the command line. That's fine, as long as there is a SYNOP message following it in 'code'. However, in your example, you send 2 SYNOP messages:
AAXX 19124 72659 ...
AAXX 19124 72462 ...
Together with the 'SYNOP ' from the command line that becomes:
SYNOP AAXX 19124 72659 ...
AAXX 19124 72462 ...
That can also be seen in the output: the attribute 's' of the metar/taf/synop node in the XML contains the input string:
<synop s="SYNOP AAXX 19124 72659 ...
The message type for the first message (and the first message, only) is explicitly given with the keyword 'SYNOP '. However, as the default message type for metaf2xml.pl is 'METAR', the second message is interpreted as a METAR:
<metar s="AAXX 19124 72462 ...
You have several choices to avoid that:
- First, if you know that only SYNOP messages are in your input, the option '-T SYNOP' for metaf2xml.pl can be used to change the default message type. Then the 'SYNOP ' on the command line is not needed.
- The same result could be achieved by giving 'SYNOP' _and a newline_ as argument on the command line.
- The third possibility would be to prefix _each_ line with 'SYNOP ', again without the 'SYNOP ' argument on the command line.
Your second example in the previous post comes close to the correct solution - if you would have omitted the 'SYNOP' on the command line. The output shows that the 2 keywords are concatenated and constitute the first message:
<synop s="SYNOP SYNOP">
This, however, is not a valid message, so an error is produced and the default message type remains METAR. The following lines of the input are interpreted as METAR:
<metar s="AAXX 19124 72659 ...
...
<metar s="AAXX 19124 72462 ...
Without the 'SYNOP ' on the command line, this example should work.
The background to all of this is that it is mandatory to pass the correct message type to the parser module. metaf2xml.pl allows to give it as keyword for a message, simply at the beginning of a message. If the message does not start with a keyword, metaf2xml.pl uses the default message type, which is tracked internally. Initially, it is 'METAR'. It can be changed from the command line (with the option -T). It can also be changed by a special line anywhere in the input, containing only the new default message type: METAR, TAF, or SYNOP.
A few examples (MT = message type, NL = newline):
option·-T··input···········default·MT··interpreted·as
(none)·····...·············METAR·······METAR
-----------------------------------------------------
-T·SYNOP···...·············->SYNOP·····SYNOP
-----------------------------------------------------
(none)·····SYNOP·AAXX·...··METAR·······SYNOP
-----------------------------------------------------
(none)·····SYNOP(NL)·······->SYNOP
···········AAXX·...(NL)····SYNOP·······SYNOP
···········METAR·...(NL)···SYNOP·······METAR
···········METAR(NL)·······->METAR
···········...(NL)·········METAR·······METAR
···········SYNOP·...·······METAR·······SYNOP
-----------------------------------------------------
-T·SYNOP···AAXX·...(NL)····->SYNOP·····SYNOP
···········AAXX·...(NL)····SYNOP·······SYNOP
···········METAR·...·······SYNOP·······METAR
-----------------------------------------------------
HTH,
Thomas
thanks thomas, It works fine now!
It was indeed the extra SYNOP tag on the command line that was the problem
Hi martin
I want use metaf2xml in windows, I install activeperl but I can not work with it, I want decode METAR and TAF bulletins and save them into Database(SQL Server). please help me.
Best Regards
Hamid
Hi Hamid,
> I install activeperl but I can not work with it
Do you mean you cannot work with ActivePerl or you cannot work with metaf2xml? Have you installed metaf2xml already? If not, at which step in the installation instructions did you get stuck?
Kind regards,
Thomas