phpweather-devel

Keeping Mears in Database

[Klaus D. <kla...@gm...>]

Hi all,

thanks for puting phpweather to work! Perhaps someone can give me a hint as I am not a great programmer. When running phpweather it 'overrides' teh metar per station, but I like to keep a history to see how weather changes, how good or bad last summer was. I digged around a bit in the coding. In 'data_retrieval.php' teh metar data are either inserted or updated (depending whether $icao already exist). The first try was simply to replace the method 'update' with 'insert' what failed as no unique index exist. I then modiefied the table settings to have 'timestamp' as part of the primary key. But even then an error for duplicate record occurs.

Is there a quick workaround to keep the old metars? Or do I have to adjust db/pw_db_mysql.php and subsequently date_retrieval.php to have a fourth column with the actual system time as part of the key?

Cheers,
Klaus

Re: Keeping Mears in Database

[Ondrej J. <ne...@po...>]

> Is there a quick workaround to keep the old metars? Or do I have to
> adjust db/pw_db_mysql.php and subsequently date_retrieval.php to have
> a fourth column with the actual system time as part of the key?

Several times ago I also wanted to have this feature. But there was no
such time for its implementation. Martin, I think this is written on the
long-term TODO list, isn't it? ;-)

--
  _/|   Ondrej Jombik - ne...@ph... - http://www.nepto.sk - OJ812-RIPE
 <_  \  Platon SDG - open source software development - http://platon.sk
   `\|  This message was written with Pine, the fastest IMAP mail client
    '`

Re: Keeping Mears in Database

[Martin G. <gim...@gi...>]

Ondrej Jombik <ne...@po...> writes:

>> Is there a quick workaround to keep the old metars? Or do I have to
>> adjust db/pw_db_mysql.php and subsequently date_retrieval.php to have
>> a fourth column with the actual system time as part of the key?
>
> Several times ago I also wanted to have this feature. But there was
> no such time for its implementation. Martin, I think this is written
> on the long-term TODO list, isn't it? ;-)

Yes, I guess it is... :-) Together with TAF reports, a working
www.phpweather.net/ and so on... If anybody wants to work with any of
this, then don't hesitate.

But as Klaus notes, then it shouldn't be that difficult for the SQL
based backends: just fetch the newest METAR for a given station per
default and then always insert new METARs instead of replacing the one
ones.

-- 
Martin Geisler                       My GnuPG Key: 0xF7F6B57B

See http://gimpster.com/ and http://phpweather.net/ for:
PHP Weather:   Shows the current weather on your webpage and
PHP Shell:     A telnet-connection (almost :-) in a PHP page.
Join Freenet:  http://gimpster.com/downloads/freenet/

Re: Keeping Mears in Database

[Klaus D. <kla...@gm...>]

On Sun, 06 Apr 2003 17:58:09 +0200
Martin Geisler <gim...@gi...> wrote:


> 
> But as Klaus notes, then it shouldn't be that difficult for the SQL
> based backends: just fetch the newest METAR for a given station per
> default and then always insert new METARs instead of replacing the one
> ones.

Martin,

as I mentioned before, inserting gets duplicate keys, I don't know
exactly why and how. Anyways, what I did so far:

Not to interfere with the flow logic of the program I look at what would
happen if I try to save teh metars in an own history table.
I created a new table (metars_history) with the colums req_time, icao,
metar, metar_time (time of request, icao, metar, time metar was
created). req_time and icao are set to UNIQUE.

I put the following statement into defaults-dist.php:
$this->properties['db_metars_hist']     = 'metars_history';

I inserted a new function to pw_db_mysql.php: 

function insert_metar_history($icao, $metar, $timestamp) {
$this->query(sprintf('INSERT INTO %s SET icao = "%s", ' .
                         'metar = "%s", metar_time = FROM_UNIXTIME(%d),
req_time = NOW()',
                         $this->properties['db_metars_hist'], $icao,
                         addslashes($metar), intval($timestamp)));
}


I added two lines in data_retrieval.php:

    /* We then cache the METAR in our database */
    if ($new_station) {
      $this->debug('get_metar_from_web(): Inserting new METAR for
<code>' .
                   $this->get_location() . '</code>');
      $this->db->insert_metar($icao, $metar, $timestamp);
--->     $this->db->insert_metar_history($icao, $metar, $timestamp);
    } else {
      $this->debug('get_metar_from_web(): Updating METAR for <code>' .
                   $this->get_location() . '</code>');
      $this->db->update_metar($icao, $metar, $timestamp);
-->      $this->db->insert_metar_history($icao, $metar, $timestamp);
    }

Unfortunately, the insert doesn' happen. Is it easy with this concept to get the history table populated?

Thanks and best regards,
Klaus

Re: Keeping Mears in Database

[Klaus D. <kla...@gm...>]

On Sun, 6 Apr 2003 23:38:21 +0200
Klaus Dahlke <kla...@gm...> wrote:

> On Sun, 06 Apr 2003 17:58:09 +0200
> Martin Geisler <gim...@gi...> wrote:
> 
> 
> > 
> > But as Klaus notes, then it shouldn't be that difficult for the SQL
> > based backends: just fetch the newest METAR for a given station per
> > default and then always insert new METARs instead of replacing the one
> > ones.
> 
> Martin,
> 
> as I mentioned before, inserting gets duplicate keys, I don't know
> exactly why and how. Anyways, what I did so far:
> 
> Not to interfere with the flow logic of the program I look at what would
> happen if I try to save teh metars in an own history table.
> I created a new table (metars_history) with the colums req_time, icao,
> metar, metar_time (time of request, icao, metar, time metar was
> created). req_time and icao are set to UNIQUE.
> 
> I put the following statement into defaults-dist.php:
> $this->properties['db_metars_hist']     = 'metars_history';
> 
> I inserted a new function to pw_db_mysql.php: 
> 
> function insert_metar_history($icao, $metar, $timestamp) {
> $this->query(sprintf('INSERT INTO %s SET icao = "%s", ' .
>                          'metar = "%s", metar_time = FROM_UNIXTIME(%d),
> req_time = NOW()',
>                          $this->properties['db_metars_hist'], $icao,
>                          addslashes($metar), intval($timestamp)));
> }
> 
> 
> I added two lines in data_retrieval.php:
> 
>     /* We then cache the METAR in our database */
>     if ($new_station) {
>       $this->debug('get_metar_from_web(): Inserting new METAR for
> <code>' .
>                    $this->get_location() . '</code>');
>       $this->db->insert_metar($icao, $metar, $timestamp);
> --->     $this->db->insert_metar_history($icao, $metar, $timestamp);
>     } else {
>       $this->debug('get_metar_from_web(): Updating METAR for <code>' .
>                    $this->get_location() . '</code>');
>       $this->db->update_metar($icao, $metar, $timestamp);
> -->      $this->db->insert_metar_history($icao, $metar, $timestamp);
>     }
> 
> Unfortunately, the insert doesn' happen. Is it easy with this concept to get the history table populated?
> 


Hi all,

I must have made something wrong on Sunday night. I just run phpweather and yes, the metar is with the above setup in the 'metar_history' table.

So, everything seems to work now.

Cheers,
Klaus

Extracting METAR Data from Personal Weather Station

[Jake O. <ja...@or...>]

OK, I know I discussed this before, but the output wasn't as good as I
would've liked, mostly because the data sucked.=20

Anyway, we'll try this again. I can get access to a "metar.txt" file =
from a
local weather station that contains the following:

"METAR SNRV 072215Z AUTO 18308KT 13/01 A3016 RMK A02 SLP212 NOSPECI =
P0000
T01310015"

And I currently have a site setup at
http://www.sunrayinc.com/sunriver/weather/index.php that loads weather,
parses it through a "print_table()" function (basically, a hacked =
version of
the original that came in the pw_text.php file that didn't work when it =
was
released, but I managed to make it work how I wanted -- I'll send code =
if
anybody wants it). I'd like to be able to somehow get this data into
something useful (automatically, if possible) as my default metar data =
so I
can use this data instead of the one I'm currently using (KRDM, which is =
40
miles from here, but still the closest), as well as pass variables via =
the
URL (so I can use my
http://www.sunrayinc.com/sunriver/weather/windimage.php?icao=3Dsnrv =
files that
I created to get the images I want and lay it out like I want).

Any ideas? I'm not a wonderful PHP programmer by any means, and it was
probably a fluke I got the print_table to work for me to begin with :).

-Jake

| Jake Ortman - Jack of All Trades Geek
| ja...@or... http://utterlyboring.com
|   *Instant Messaging Usernames:*
| AIM: jakeortman Y!: jake-ortman
| ICQ: 7739361 MSN: jak...@ho...

Re: Extracting METAR Data from Personal Weather Station

[Martin G. <gim...@gi...>]

"Jake Ortman" <ja...@or...> writes:

> OK, I know I discussed this before, but the output wasn't as good as I
> would've liked, mostly because the data sucked. 
>
> Anyway, we'll try this again. I can get access to a "metar.txt" file
> from a local weather station that contains the following:
>
> "METAR SNRV 072215Z AUTO 18308KT 13/01 A3016 RMK A02 SLP212 NOSPECI P0000
> T01310015"

Great, that should be enough.

> Any ideas? I'm not a wonderful PHP programmer by any means, and it
> was probably a fluke I got the print_table to work for me to begin
> with :).

Hehe, I don't think this should be that difficult :-) Since your file
contains just the one METAR for your local airport, then I think the
easiest way would be to change get_metar_file() in data_retrieval.php
to this

  function get_metar_file($icao) {
    if ($icao == 'SNRV') {
      return file('full/pathname/to/your/metar.txt');
    } else {
      $host = 'weather.noaa.gov';
      $location = "/pub/data/observations/metar/stations/$icao.TXT";
      return @file('http://' . $host . $location);
    }
  }

That should implement an override so that the METAR for SNRV will be
fetched from the local file somewhere on your filesystem, and the rest
will still be fetched from the NWS. The rest of the logic with caching
should still work, as far as I can see...

-- 
Martin Geisler                       My GnuPG Key: 0xF7F6B57B

See http://gimpster.com/ and http://phpweather.net/ for:
PHP Weather:   Shows the current weather on your webpage and
PHP Shell:     A telnet-connection (almost :-) in a PHP page.
Join Freenet:  http://gimpster.com/downloads/freenet/

Re: Keeping Mears in Database

[Ondrej J. <ne...@po...>]

On Sun, 6 Apr 2003, Martin Geisler wrote:

> >> Is there a quick workaround to keep the old metars? Or do I have to
> >> adjust db/pw_db_mysql.php and subsequently date_retrieval.php to
> >> have a fourth column with the actual system time as part of the
> >> key?
> > Several times ago I also wanted to have this feature. But there was
> > no such time for its implementation. Martin, I think this is written
> > on the long-term TODO list, isn't it? ;-)
> But as Klaus notes, then it shouldn't be that difficult for the SQL
> based backends: just fetch the newest METAR for a given station per
> default and then always insert new METARs instead of replacing the one
> ones.

But looking at the database schema of "metars" table [1], "icao" is
a primary key, thus inserting additional metar for particular airport
may cause problems.

So this won't be possible to implementent without database schema
change. We can remove primary key completely, or extend it with
"timestamp" field. In the second case new primary key will contain
"icao" and "timestamp" columns.

[1] Database schema of "metars" table

mysql/phpweather> \d metars
+--------+-----------+-----------+-----+--------------------------------------+
| table  |   name    |   type    | len |                flags                 |
+--------+-----------+-----------+-----+--------------------------------------+
| metars | icao      | string    | 4   | not_null primary_key unique_key      |
| metars | metar     | string    | 255 | not_null                             |
| metars | timestamp | timestamp | 14  | not_null unsigned zerofill timestamp |
+--------+-----------+-----------+-----+--------------------------------------+
mysql/phpweather>

--
  _/|   Ondrej Jombik - ne...@ph... - http://www.nepto.sk - OJ812-RIPE
 <_  \  Platon SDG - open source software development - http://platon.sk
   `\|  This message was written with Pine, the fastest IMAP mail client
    '`

Re: Keeping Mears in Database

[Klaus D. <kla...@gm...>]

On Sun, 6 Apr 2003 23:23:09 +0200 (CEST)
Ondrej Jombik <ne...@po...> wrote:

> On Sun, 6 Apr 2003, Martin Geisler wrote:
> 

> 
> So this won't be possible to implementent without database schema
> change. We can remove primary key completely, or extend it with
> "timestamp" field. In the second case new primary key will contain
> "icao" and "timestamp" columns.
> 
> [1] Database schema of "metars" table
> 
> mysql/phpweather> \d metars
> +--------+-----------+-----------+-----+--------------------------------------+
> | table  |   name    |   type    | len |                flags                 |
> +--------+-----------+-----------+-----+--------------------------------------+
> | metars | icao      | string    | 4   | not_null primary_key unique_key      |
> | metars | metar     | string    | 255 | not_null                             |
> | metars | timestamp | timestamp | 14  | not_null unsigned zerofill timestamp |
> +--------+-----------+-----------+-----+--------------------------------------+
> mysql/phpweather>
> 
> --

Ondrej,

I tried in the first place to extend the key and change 'update-metar' to 'insert_metar', but as the timestamp is the time of the metar, the system get still confused and reports 'duplicate entry'. On the weekend, I will check something if I add an auto-increment column to the table (either metars or using an additional history table).

Cheers,
Klaus

Cheers, 

Re: Keeping Mears in Database

[Martin G. <gim...@gi...>]

Klaus Dahlke <kla...@gm...> writes:

> I tried in the first place to extend the key and change
> 'update-metar' to 'insert_metar', but as the timestamp is the time
> of the metar, the system get still confused and reports 'duplicate
> entry'.

If you relax your demands on the keys in the database, and then make
update_metar call insert_metar, then it's easy to put the same row
into the database several times.

With no indexes at all, or just a normal index on the icao and perhaps
also on timestamp I get lots of METARs into the database. The problem
is that I get three new rows each time someone submits a station(!)

I think it's a problem with the logic in data_retrieval.php. The
get_metar_from_web() function retrieves a METAR from NWS and tries to
adjust the timestamp in a clever way if it think's it's too old. I've
not looked into it, but I think it must be there the problem lies.

If we're going to do this right, then there should be some support for
deleting old contents, support for selecting old reports and the logic
should be updated. It sure would be cool to have a fourth drop-down
box where people could select a different data to see data from, or
even to generate graphs that show the tendency over the last
day/week/month...

Klaus, if you want to work on this, then let me know and I can add you
to the project --- this project really needs someone who can put some
energy into it. Just create an account at SourceForge and mail me the
account name and I'll setup the CVS write permissions for you.

-- 
Martin Geisler                       My GnuPG Key: 0xF7F6B57B

See http://gimpster.com/ and http://phpweather.net/ for:
PHP Weather:   Shows the current weather on your webpage and
PHP Shell:     A telnet-connection (almost :-) in a PHP page.
Join Freenet:  http://gimpster.com/downloads/freenet/