finance-quote-devel — Developer discussions - NOT for reporting issues

Re: [Finance-quote-devel] Header too long error from YohooWeb

[Bruce S. <bs...@as...>]

> I started getting a new error message I've never seen before from
> all of you queries for stock prices from YahooWeb.  The message is
This list is not for issue and bug reporting.

There is already an issue opened on GitHub.

[Finance-quote-devel] Header too long error from YohooWeb

[Mike A. <mt...@um...>]

I started getting a new error message I've never seen before from all of 
you queries for stock prices from YahooWeb.  The message is

500 Header line too long (limit is 8192)

This seems to be generated in Net/HTTP/Methods.pm around line 266.  When 
I look at the URL being fetched (which is 
"https://finance.yahoo.com/quote/GE/history?p=GE") with Safari or curl 
it looks fine.  Has anyone else seen this?  Does anyone know what is 
causing it?  This started happening today and nothing has knowingly 
changed on my machine since this last worked.

I'm using Finance Quote 1.59 installed from Macports.

Mike

[Finance-quote-devel] Finance::Quote v1.59 Tentative Release Date

[Bruce S. <bs...@as...>]

Happy Holidays everyone.

It's short notice, but I intend to push Finance::Quote v1.59 to CPAN 
tomorrow, December 31, 2023.

The changes from v1.58 to v1.59 are:

* Fixed XETRA, Tradegate and SInvestor after webpage was restructured - 
Issue #350

* Fidelity.pm temporarily disabled - Issue #346

* Finanzpartner.pm - Fix scraper, did not work if quote was higher than
   the previous day's quote.

* GoogleWeb.pm - Updated to retrieve mutual fund and BATS prices - Issue
   #355

* BSEIndia.pm - Updated to use standardised data file at URL
   https://www.bseindia.com/download/BhavCopy/Equity
   BSE_EQ_BHAVCOPY_{DDMMYYYY}.ZIP
   Eliminated need to use temp folders by storing standardized file
   data from URL into a variable.
   Updated names of source fields to conform to those in the
   standardized data file

* IndiaMutual.pm - Eliminated need to use temp folders by storing
   nav file data from URL into a variable.

* TMX.pm - Correct a self-reference in documentation - PR #345

* Stooq.pm - Added new currencies and a fix for commodities' prices

* YahooWeb.pm - Skip rows in the price table where the prices ar "-".
   This seems to happen sometimes with TIAA (and perhaps other)
   securities including TILIX and QCILIX

* TSP.pm - Was not returning hash when the HTTP GET failed completely
   or the content did not contain the expected CSV file. - Issue #338

* BSEIndia.pm - Removed print when symbol not found - Issue #335

Bruce S

Re: [Finance-quote-devel] [GNC] Finance::Quote - AlphaVantage Trade Currencies

[john <jr...@ce...>]

> On Jul 16, 2023, at 15:52, Bruce Schuck <bs...@as...> wrote:
> 
> Hello again,
> 
> At some point I am going to take a break from F::Q, I thought this weekend I was going to escape.
> 
> ### For the question, skip to the next ###
> 
> Anyway, some time ago the issue was raised about prices from AlphaVantage for LSE traded stocks (see https://github.com/finance-quote/finance-quote/issues/281). While there is an issue, the user's perceived solution was not the answer. Apparently my attempt to explain that the root cause was that the data returned from the AV "GLOBAL_QUOTE" API did *NOT* contain the currency used for the pricing returned.
> 
> What the AV module does is assign a currency based on the suffix of the lookup symbol. So stocks like GBP.L and GBPG.L get mapped to GBP. The problem is that GBP.L (Global Petroleum Limited) is traded in GBX (aka GBp) and GBPG.L (Goldman Sachs Access UK Gilts 1-10) is traded in GBP. But the AV module as written cannot differentiate since the currency used is not part of the data. Stocks from South Africa have a similar issue.
> 
> Needless to say, since I was not ready to dig into the root cause and not wanting to remove the fix that was meant to convert GBp pricing to GBP pricing based solely on the symbol's suffix, the user decided to go on a rant and throw a tantrum in the comments and had to be blocked from the Finance::Quote repository.
> 
> ### Question here ###
> 
> The currency for the trades can be gotten from AV, but an additional call using the "SYMBOL_SEARCH" API is necessary. Obviously this would impact the throttling limitation of AlphaVantage. For non-US stocks, this means that including an additional currency lookup an average of 2.5 securities can be fetched per minute. I am curious how many of you all use AV for non-US traded securities? Has the pricing discrepancy affect you, and would you want it corrected even though that means increasing retrieval time due to the extra API usage?

It's not all non-US securities. In AlphaVantage it appears to be only LSE stocks; JSE stocks aren't supported and Tel Aviv stocks support only ILS. The Yahoo JSON module adjusts prices for Zac (South Africa) and ILA (Israel) as well as GBX, but Yahoo JSON apparently includes a currency code to make that easy. That seems to be missing from YahooWeb, perhaps an oversight on Vincent's part. The currency is available on https://finance.yahoo.com/quote/GBP.L prefixed by "Currency In" in div id="quote-header-info <>".

So for AlphaVantage it seems only .L and .IL suffix securities would need to be queried, and maybe only .L unless there are known instances where .IL-suffixed symbols are priced in GBP.

Rather than spending an extra query maybe AlphaVantage.pm could accept a special extra suffix indicating that the user wants quotes to be multiplied by 100, perhaps X, so a request for GBP.L priced in GBP would be GBP.L.X.

Alternatively we could create a GBX, ZAX, and ILA pseudo-currencies in GnuCash and users would create parent accounts in those currencies for stocks quoted that way. That would take a little finessing on users' part because using it would depend both on the currency the stock is quoted in and whether F::Q converts it to the "real" currency.

Regards,
John Ralls

[Finance-quote-devel] Finance::Quote - AlphaVantage Trade Currencies

[Bruce S. <bs...@as...>]

Hello again,

At some point I am going to take a break from F::Q, I thought this 
weekend I was going to escape.

### For the question, skip to the next ###

Anyway, some time ago the issue was raised about prices from 
AlphaVantage for LSE traded stocks (see 
https://github.com/finance-quote/finance-quote/issues/281). While there 
is an issue, the user's perceived solution was not the answer. 
Apparently my attempt to explain that the root cause was that the data 
returned from the AV "GLOBAL_QUOTE" API did *NOT* contain the currency 
used for the pricing returned.

What the AV module does is assign a currency based on the suffix of the 
lookup symbol. So stocks like GBP.L and GBPG.L get mapped to GBP. The 
problem is that GBP.L (Global Petroleum Limited) is traded in GBX (aka 
GBp) and GBPG.L (Goldman Sachs Access UK Gilts 1-10) is traded in GBP. 
But the AV module as written cannot differentiate since the currency 
used is not part of the data. Stocks from South Africa have a similar issue.

Needless to say, since I was not ready to dig into the root cause and 
not wanting to remove the fix that was meant to convert GBp pricing to 
GBP pricing based solely on the symbol's suffix, the user decided to go 
on a rant and throw a tantrum in the comments and had to be blocked from 
the Finance::Quote repository.

### Question here ###

The currency for the trades can be gotten from AV, but an additional 
call using the "SYMBOL_SEARCH" API is necessary. Obviously this would 
impact the throttling limitation of AlphaVantage. For non-US stocks, 
this means that including an additional currency lookup an average of 
2.5 securities can be fetched per minute. I am curious how many of you 
all use AV for non-US traded securities? Has the pricing discrepancy 
affect you, and would you want it corrected even though that means 
increasing retrieval time due to the extra API usage?

Bruce S.

[Finance-quote-devel] Taiwan stocks quotes

[Paul C. <pau...@gm...>]

Maybe it's time to say goodbye to yahoo finance API for global market
quotes.
I wonder if anyone has already been developing modules from Taiwan domestic
source like TWSE APIs <https://openapi.twse.com.tw/>, etc.

If yes, please raise your hand. Thanks and have a good weekend!

warmly yours,
Paul Chang (張仲銘)

Re: [Finance-quote-devel] [GNC] Finance::Quote 1.56_02 Available for Evaluation

[Bruce S. <bs...@as...>]

On 6/18/23 10:34, john wrote:

>> I'm at a loss. Although at the current moment, YahooWeb isn't
>> required as long other viable quote sources exist. I like the
>> Date::Business module, I just wish its install honored the
>> PERL_USE_MM_DEFAULTS environment variable. It would be trivial to
>> create different holiday subroutines for various
>> exchanges/countries and use the one for the exchange indicated in
>> the data returned. I have started the process CPAN/Pause requires
>> for adopting the Date::Business module namespace if the original
>> author/authors are unresponsive.

> OK, that seems to be the best alternative.

I have somewhat good news. I did some testing with Date::Manip without 
creating a config file. The Date_PrevWorkDay method does skip weekends 
properly as long the $offset is set to 1 when called on Sat/Sun.

I can replace Date::Business with Date::Manip in YahooWeb. At the very 
least it will skip back over weekends for all quotes, and not create 
issues during Date::Business install requiring for user input. We lose 
holiday calculations, but for an "emergency" use module, better than 
nothing.

Again, not optimal, but better than the original version where the trade 
date would be set to "today" regardless of what day F::Q was executed.

Happy Fathers' Day to all the dads.

Bruce S.

Re: [Finance-quote-devel] [GNC] Finance::Quote 1.56_02 Available for Evaluation

[john <jr...@ce...>]

> On Jun 17, 2023, at 15:03, Bruce Schuck <bs...@as...> wrote:
> 
> On June 16, 2023 at 15:11 EDT, John Ralls wrote:
> 
>> On 06/15/2023 12:23 -0700, Bruce S wrote:
> 
>>> Non-production release 1.56_02 is available for evaluation/testing.
> 
>>> It can be installed via cpan/cpanm as 'BPSCHUCK/Finance-Quote-1.56_02.tar.gz'.
> 
>>> The YahooWeb module currently requires the Date::Business module. Installing it cannot be done unattended, as the Makefile.PL used requires a yes/no question be answered. Not updated since January 2000, attempts to reach the author have been unsuccessful so far. Maybe you could use Date::Manip::Date_PrevWorkDay instead?
> 
> As I mentioned, I was looking at that along with Date::Business. I also looked at Date::Holidays today.
> 
> What I found about Date::Manip is that for business mode calculations the module requires a configuration file. See Date::Manip::Config where is notes:
> 
> "NOTE: if you use business mode calculations, you must have a config file since this is the only place where you can define holidays."
> 
> From the documentation (a rabbit hole of many man pages/PODs) that includes have date_PrevWorkDay skip over weekends (haven't tested if this just means you want to define your own custom work week).
> 
> I going to assume that requiring GnuCash users to have a config file for Date::Manip and keep it updated could be problematic for some.
> 
> What I found about Date::Holidays so far is that users would need to install all the Date::Holidays::<country> modules that Finance::Quote may use. F::Q would then be at the mercy of all the different authors with respect to those modules being updated. Also, the current US or USFederal modules do not account for Juneteenth or Good Friday. Juneteenth being a newly created holiday here in the US, and Good Friday is observed by the US stock market each year.
> 
>> But either way it's pretty US-centric because of the holidays enumeration. What about the Monday-Friday trading days assumption?
>> Is that valid for all of the stock markets that Yahoo can get quotes
>> for?
> I'm at a loss. Although at the current moment, YahooWeb isn't required as long other viable quote sources exist. I like the Date::Business module, I just wish its install honored the PERL_USE_MM_DEFAULTS environment variable. It would be trivial to create different holiday subroutines for various exchanges/countries and use the one for the exchange indicated in the data returned. I have started the process CPAN/Pause requires for adopting the Date::Business module namespace if the original author/authors are unresponsive.

OK, that seems to be the best alternative.

Regards,
John Ralls

Re: [Finance-quote-devel] [GNC] Finance::Quote 1.56_02 Available for Evaluation

[Bruce S. <bs...@as...>]

On June 16, 2023 at 15:11 EDT, John Ralls wrote:

> On 06/15/2023 12:23 -0700, Bruce S wrote:

>> Non-production release 1.56_02 is available for 
>> evaluation/testing.

>> It can be installed via cpan/cpanm as 
>> 'BPSCHUCK/Finance-Quote-1.56_02.tar.gz'.

>> The YahooWeb module currently requires the Date::Business module. 
>> Installing it cannot be done unattended, as the Makefile.PL used 
>> requires a yes/no question be answered. Not updated since January 
>> 2000, attempts to reach the author have been unsuccessful so far. 
>> Maybe you could use Date::Manip::Date_PrevWorkDay instead?

As I mentioned, I was looking at that along with Date::Business. I also 
looked at Date::Holidays today.

What I found about Date::Manip is that for business mode calculations 
the module requires a configuration file. See Date::Manip::Config where 
is notes:

"NOTE: if you use business mode calculations, you must have a config 
file since this is the only place where you can define holidays."

 From the documentation (a rabbit hole of many man pages/PODs) that 
includes have date_PrevWorkDay skip over weekends (haven't tested if 
this just means you want to define your own custom work week).

I going to assume that requiring GnuCash users to have a config file for 
Date::Manip and keep it updated could be problematic for some.

What I found about Date::Holidays so far is that users would need to 
install all the Date::Holidays::<country> modules that Finance::Quote 
may use. F::Q would then be at the mercy of all the different authors 
with respect to those modules being updated. Also, the current US or 
USFederal modules do not account for Juneteenth or Good Friday. 
Juneteenth being a newly created holiday here in the US, and Good Friday 
is observed by the US stock market each year.

> But either way it's pretty US-centric because of the holidays 
> enumeration. What about the Monday-Friday trading days assumption?
> Is that valid for all of the stock markets that Yahoo can get quotes
> for?
I'm at a loss. Although at the current moment, YahooWeb isn't required 
as long other viable quote sources exist. I like the Date::Business 
module, I just wish its install honored the PERL_USE_MM_DEFAULTS 
environment variable. It would be trivial to create different holiday 
subroutines for various exchanges/countries and use the one for the 
exchange indicated in the data returned. I have started the process 
CPAN/Pause requires for adopting the Date::Business module namespace if 
the original author/authors are unresponsive.

Bruce S.

[Finance-quote-devel] Google Finance Web Scrape

[Bruce S. <bs...@as...>]

Also posted to 
https://github.com/finance-quote/finance-quote/discussions/274

I have begun looking at screen scraping 
https://www.google.com/finance/quote/<symbol>. That page returns a list 
of URLs with the symbol for different exchanges. Assuming we want NYSE, 
NASDAQ, or NYSEAMERICAN (What Google uses for stocks on the American 
exchange), the module would then retrieve 
https://www.google.com/finance/quote/<symbol>:<exchange>. For the basic 
price data use HTML::TreeBuilder to get the <div tag with the 
data-last-price attribute. In that tag we have additional attributes 
data-exchange, data-currency-code, and 
data-last-normal-market-timestamp. The timestamp is in seconds since 
epoch. I already have a scratch script getting the link we want from the 
first link. The rest will be fairly trivial.

Bruce S

Re: [Finance-quote-devel] Unable to install F::Q on macOS 11.4

[Peter S. <psi...@ce...>]

Thanks for confirming.

> On Jul 1, 2021, at 3:41 PM, Bruce Schuck <bs...@as...> wrote:
> 
> On 7/1/21 1:51 PM, Peter Silsbee wrote:
> 
>> After taking MacOS 11.4 I cannot update F::Q: I’ve tried gnucash’s gnc-fq-update, cpanm, perlbrew. I get the following error:
> 
>> Is this a known issue? Since it failed with a brand-new perl installation under perlbrew, I doubt it is my system, but all
>> theories are welcome.
> Sorry, known bug. Will try to push out v1.51 this weekend, or at least make a tarball rc version available for downloading.
> 
> - Bruce S.

Re: [Finance-quote-devel] Unable to install F::Q on macOS 11.4

[Bruce S. <bs...@as...>]

On 7/1/21 1:51 PM, Peter Silsbee wrote:

> After taking MacOS 11.4 I cannot update F::Q: I’ve tried gnucash’s 
> gnc-fq-update, cpanm, perlbrew. I get the following error:

> Is this a known issue? Since it failed with a brand-new perl 
> installation under perlbrew, I doubt it is my system, but all
> theories are welcome.
Sorry, known bug. Will try to push out v1.51 this weekend, or at least 
make a tarball rc version available for downloading.

- Bruce S.

[Finance-quote-devel] Unable to install F::Q on macOS 11.4

[Peter S. <psi...@ce...>]

Hi devs,

After taking MacOS 11.4 I cannot update F::Q: I’ve tried gnucash’s gnc-fq-update, cpanm, perlbrew. I get the following error:

Is this a known issue? Since it failed with a brand-new perl installation under perlbrew, I doubt it is my system, but all theories are welcome.

[2014 Intel iMac, if it matters]

Oh BTW I did try a force install with cpanm but the module didn’t appear to work [invoked via gnc-fq-dump] so I expect the test failure is real. I should be able to reproduce some of that failure if it is of interest.

Thanks
Peter

========

t/fq-object-methods.t ...... 1/23 Undefined fetch-method usa passed to Finance::Quote::fetch at t/fq-object-methods.t line 14.
IEXCloud API_KEY not defined.  See documentation. at /Users/pnp/.cpan/build/Finance-Quote-1.50-7/blib/lib/Finance/Quote/IEXCloud.pm line 60.
# Looks like your test exited with 2 just after 5.
t/fq-object-methods.t ...... Dubious, test returned 2 (wstat 512, 0x200)
Failed 18/23 subtests 

Test Summary Report
-------------------
t/fq-object-methods.t    (Wstat: 512 Tests: 5 Failed: 0)
  Non-zero exit status: 2
  Parse errors: Bad plan.  You planned 23 tests but ran 5.
Files=56, Tests=71,  8 wallclock secs ( 0.12 usr  0.08 sys +  6.25 cusr  1.24 csys =  7.69 CPU)
Result: FAIL
Failed 1/56 test programs. 0/71 subtests failed.
make: *** [test_dynamic] Error 255
  BPSCHUCK/Finance-Quote-1.50.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
  reports BPSCHUCK/Finance-Quote-1.50.tar.gz

[Finance-quote-devel] Preparing for F::Q Release 1.50

[Bruce S. <bs...@as...>]

Hello everyone,

Thanks to Erik, I have the access to upload new releases to PAUSE/CPAN. 
While preparing for this, I did find that some changes should be made to 
the dist.ini file that Dzil uses to build the release.

Modify author list - update with output of git shortlog -s -n -e
for authors with >=2 commits.

Add files to [PruneFiles]
.cvsignore
.gitignore
.travis.yml
new-API-notes.txt
Modules-README.yml

These files are not necessary for the module to install or function and 
are more for developers for information.

Modify [EmailNotify]
from = Bruce Schuck <bs...@as...>
recipient = fin...@li...
recipient = gnu...@gn...
bcc = ec...@ec...
bcc = bs...@as...

Since Erik has indicated he has disconnected himself from F::Q emails, 
it makes sense that since I will be uploading releases to CPAN that his 
email should be swapped out for mine.

Please respond with any questions, comments, objections, etc.

- Bruce S.

Re: [Finance-quote-devel] Where's Erik?

[Erik C. <ec...@ec...>]

Bruce Schuck <bs...@as...> writes:

> It's been a while since we've heard from Erik (ecocode). Has anybody
> maybe chatted with him offline of the F::Q lists?

Hi Bruce and all,

Due to some personal health issues I had disconnected my mails related
to finance-quote.  I am still here but cannot help much for now.  I
removed you and Vincent from my filter but won't check any mails on the
ml.  Sorry for that...

best
--
erik colson

Re: [Finance-quote-devel] Where's Erik?

[Bruce S. <bs...@as...>]

On 6/16/21 8:04 AM, Erik Colson wrote:

> Bruce Schuck <bs...@as...> writes:

>> It's been a while since we've heard from Erik (ecocode). Has
>> anybody maybe chatted with him offline of the F::Q lists?

> Due to some personal health issues I had disconnected my mails
> related to finance-quote.  I am still here but cannot help much for
> now.  I removed you and Vincent from my filter but won't check any
> mails on the ml.  Sorry for that...

Understood Erik. I believe I speak for all when I wish you the best and 
hope that your situation will improve.

- Bruce S.

[Finance-quote-devel] Where's Erik?

[Bruce S. <bs...@as...>]

It's been a while since we've heard from Erik (ecocode). Has anybody
maybe chatted with him offline of the F::Q lists?

- Bruce S.

Re: [Finance-quote-devel] catching up and slowly handing over..

[Vincent L. <vin...@gm...>]

Hi,

Yes, please add me as an owner.  We’ll work towards a 1.50 release and then let you know.

Best,

Vince

> On Oct 10, 2020, at 9:30 AM, Erik Colson <ec...@ec...> wrote:
> 
> Hello !
> 
> I have been away for a long time and I don't see much amelioration
> coming in the near future.
> 
> We should probably plan a new release, but as I said I currently don't
> have much time to check all PRs and mails and stuff.  Therefor I think
> the ownership of the repo should probably be widened to other
> contributors.
> 
> So, I propose to :
> - first of all: add Vincent and/or Bruce (both are welcome :)) as
>  co-owners of the F::Q organisation on Github.
> - let Vincent and Bruce define a master branch commit which will be the
>  next release
> - I will do the next release and try to document the procedure so
>  I will be able to transfer in the future..
> 
> BTW, I especially wish to thank Vincent and Bruce for being present and
> keeping the project alive!
> --
> erik colson
> 
> 
> _______________________________________________
> Finance-quote-devel mailing list
> Fin...@li...
> https://lists.sourceforge.net/lists/listinfo/finance-quote-devel

[Finance-quote-devel] catching up and slowly handing over..

[Erik C. <ec...@ec...>]

Hello !

I have been away for a long time and I don't see much amelioration
coming in the near future.

We should probably plan a new release, but as I said I currently don't
have much time to check all PRs and mails and stuff.  Therefor I think
the ownership of the repo should probably be widened to other
contributors.

So, I propose to :
- first of all: add Vincent and/or Bruce (both are welcome :)) as
  co-owners of the F::Q organisation on Github.
- let Vincent and Bruce define a master branch commit which will be the
  next release
- I will do the next release and try to document the procedure so
  I will be able to transfer in the future..

BTW, I especially wish to thank Vincent and Bruce for being present and
keeping the project alive!
--
erik colson

Re: [Finance-quote-devel] CryptoCurrencies

[Erik C. <ec...@ec...>]

Hi Bruce,

I'd go for the second option.  Reason for that is, if not already the
case, someday stocks and funds will be tradeable in cryptocurrencies.
Going for option 1 will add complexity to F::Q.
However, I have no clue if that would require some change in GnuCash.
Feel free to ask the GnuCash team their opinion.

best
--
erik colson

[Finance-quote-devel] CryptoCurrencies

[Bruce S. <bs...@as...>]

Having done a little trading and purchasing using CryptoCurrencies over
the past year or so, I have been thinking about Crypto, GnuCash, and
Finance::Quote.

I think that before we go head first into creating or modifying F::Q
modules, we need to get a consensus on what path to take. By that I mean
we treat CrytoCurrencies in one of two ways.

1: Treat CryptoCurrencies like we do Stocks and Funds. In this path we
would likely have a module perhaps named Crypto.pm or CryptoCurrency.pm
using one of the available APIs that supply exchange data. Some of the
existing sources like AlphaVantage.co have APIs for crypto, but there
exists more to choose from, some do not require API keys.

2: Treat CryptoCurrencies as just another currency like USD, EU, etc.
And modify the existing currency related modules to incorporate the
various CryptoCurrencies. As of the date I write this email, May 23,
2020, I have not yet checked if there are conflicts with common
identifiers/abbreviations between currencies and crypto.

I may be wrong, but path 2 may require some input and coordination from
the team that maintains GnuCash?

- Bruce S.

[Finance-quote-devel] Pushed branch new-new to explore Erik's idea for FQ API improvements

[Vincent L. <vin...@gm...>]

Hi,

In June, Erik sent a message about his idea for cleaning up the FQ API to have consistent ways to set/get instance variables and expand new to allow for named parameters and module specific data.

One benefit is a sustainable way to enable users to register API keys for specific modules.

I just pushed the branch "new-new" to explore these ideas.

See the file https://raw.githubusercontent.com/finance-quote/finance-quote/new-new/new-API-notes.txt for details about the set/get methods.


I used the new API to thread an API key to the IEXCloud module from the test suite as an example:
https://raw.githubusercontent.com/finance-quote/finance-quote/new-new/t/iexcloud.t


If there is agreement to proceed with the API update there are a few things to do and I can help:

- updating the POD documentation for Quote.pm
- enabling AlphaVantage to use the module specific data as another source for its API key
- renew discussion with gnucash developers on enabling users to specify API keys for more modules



Best,

Vince

[Finance-quote-devel] FYI - Updated .travis.yml

[Bruce S. <bs...@as...>]

Hello,

I noticed the last few Travis-CI builds failed. The download of perl
versions 5.20 and older get 404 errors from the Google APIs site. So I
changed the Perl versions to 5.28, 5.26, 5.24, and 5.22. Looked into how
to compile against older versions, we may want to try using the "matrix"
configuration. Seems weird since the Ubuntu version is 16.04/xenial.

---

language: perl
matrix:
  include:
  - perl: "5.26"
  - perl: "5.24"
  - perl: "5.22"
    dist: trusty
  - perl: "5.20"
  - perl: "5.18"

---

- Bruce S.

Re: [Finance-quote-devel] Vanguard (US) investments daily closing price module

[Derek R. <der...@ym...>]

 Sorry - my attachment didn't come through. I've pasted the module below.


#!/usr/bin/perl -w
#    This module was written in November 2019 based on the 
#    Finance::Quote::Fool.pm module and prior versions of Fool.pm
#    that carried the following copyrights:
#
#    Copyright (C) 1998, Dj Padzensky <dj...@pa...>
#    Copyright (C) 1998, 1999 Linas Vepstas <li...@li...>
#    Copyright (C) 2000, Yannick LE NY <y-...@if...>
#    Copyright (C) 2000, Paul Fenwick <pj...@cp...>
#    Copyright (C) 2000, Brent Neal <br...@us...>
#    Copyright (C) 2001, Tobias Vancura <tva...@al...>
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    (at your option) any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#    
#    You should have received a copy of the GNU General Public License
#    along with this program; if not, write to the Free Software
#    Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
#    02111-1307, USA

package Finance::Quote::VanguardSite;

require 5.005;

our $VERSION = '1.49'; # VERSION

use strict;
use HTTP::Request::Common;
use HTML::TableExtract;
use HTML::TreeBuilder;
use Text::Template;
use Encode qw(decode);
use Time::Piece;

sub methods { 
  return ( vanguardsite   => \&vanguardsite );
}

my @labels = qw/date isodate nav/;
sub labels {
  return ( iexcloud => \@labels, );
}

sub vanguardsite {
    my $quoter = shift;
    my @investments = @_;
        my (%info, $symbol, $url, $reply, $code, $desc, $body);
    my $ua = $quoter->user_agent();
        my $quantity = @investments;

    foreach my $symbol (@investments) {
  my $t = localtime;
 my $t2 = $t - 7*86400;
  my $URL = Text::Template->new(TYPE => 'STRING', SOURCE => 'https://personal.vanguard.com/us/funds/tools/pricehistorysearch?radio=1&results=get&FundIntExt=INT&FundId={$symbol}&fundName={$symbol}&radiobutton2=1&beginDate='.$t2->mon.'%2F'.$t2->mday.'%2F'.$t2->year.'&endDate='.$t->mon.'%2F'.$t->mday.'%2F'.$t->year.'&year=#res');
 # Vanguard price history doesn't like invalid dates, so we pass a date range from a week ago through today - this range will always return SOME valid data, unless markets close for several days.
         # Get the web page
        $url   = $URL->fill_in(HASH => {symbol => $symbol});
        $reply = $ua->request( GET $url);
        $code  = $reply->code;
        $desc  = HTTP::Status::status_message($code);
        $body  = decode('UTF-8', $reply->content);
         if ($code != 200) {
            $info{ $symbol, 'success' } = 0;
            $info{ $symbol, 'errormsg' } = $desc;
            next;
        }
  # Extract first table with Date and either Price or NAV as headers.  my $te = HTML::TableExtract->new( headers => [qw(Date Price|NAV)] );
        $te->parse($body);
 my $ts = $te->first_table_found();
  #The last row in this table contains the date and price we want.
 my $mostrecent = ($ts->rows)[-1];
         eval {
 $info{$symbol, 'symbol'}             = $symbol;
 $info{$symbol, 'method'}             = 'vanguardsite';
 $info{$symbol, 'nav'}                = @$mostrecent[1] =~ s/[\$,]//g ? @$mostrecent[1] : die('failed to parse last price');
 $info{$symbol, 'currency'}           = 'USD';
 $info{$symbol, 'success'}            = 1;
 $quoter->store_date( \%info, $symbol, { usdate => @$mostrecent[0] } );
 }
 or do {
 $info{$symbol, 'errormsg'} = $@;
 $info{$symbol, 'success'}  = 0;
        }
  }

    return wantarray() ? %info : \%info;
}

1;

=head1 NAME

Finance::Quote::VanguardSite - Obtain day-end price (NAV) quotes for Vanguard investments from Vanguard.com.

=head1 SYNOPSIS

    use Finance::Quote::VanguardSite;

    $q = Finance::Quote::VanguardSite->new;

    %stockinfo = $q->fetch("vanguardsite","1679");

=head1 DESCRIPTION

This module obtains day-end price (NAV) quotes from Vanguard.com for any investment that has a Vanguard fund ID. Note that it
will NOT return intraday prices for ETF's - look those up on another provider using their ticker if you want them.

The symbol passed to the vanguardsite method must be the fund ID, not the ticker symbol.

This module is loaded by default on a Finance::Quote object.  It's also
possible to load it explicity by placing "VanguardSite" in the argument list to
Finance::Quote->new().

Information returned by this module is governed by Vanguard's terms and conditions.

=head2 Labels Returned

The following labels may be returned by Finance::Quote::VanguardSite:
symbol (an internal Vanguard fund ID), nav, currency, method.

=head1 SEE ALSO

Funds, trusts and ETFs: Vanguard Institutional, https://institutional.vanguard.com/web/c1/fund-list/?filters=&sortBy=alphabetical&viewType=quarterEndReturnsNAV
529 portfolios: Vanguard 529 plans, Vanguard 529 Plan portfolios: Your investment options | Vanguard

| 
| 
|  | 
Vanguard 529 Plan portfolios: Your investment options | Vanguard

Explore Vanguard's 529 plan investment options, including money market, bond, balanced, and stock portfolios.
 |

 |

 |


Vanguard fund ID's are the four-digit numbers embedded in the links to each investment found at those sites. 


Finance::Quote.

=cut

    On Wednesday, November 20, 2019, 02:44:09 PM EST, Derek Robinson via Finance-quote-devel <fin...@li...> wrote:  
 
 (posting here at Frank Ellenberger's suggestion; this originated in the GnuCash user mailing list)

Vanguard investors may be frustrated that many Vanguard products (like their 529 portfolios, any 401(k) / 403(b) products with the word "Trust" in the title) do not have prices (really NAVs, or Net Asset Values) that are easily obtainable from one of the regular price sources. These products do not have ticker symbols and cannot be accessed with AlphaVantage (as far as I know). Vanguard manages over $5T USD in assets, so price discovery for its investments should be helpful to a broad audience.

The attached module extracts prices from the Vanguard price history page and (in its original intent, anyway) adds them to the GnuCash price database. It does not attempt to extract anything other than a NAV and date although it could be extended to find other attributes on a different part of the Vanguard site.


Please let me know if I can clarify anything in the module.


Regards,Derek_______________________________________________
Finance-quote-devel mailing list
Fin...@li...
https://lists.sourceforge.net/lists/listinfo/finance-quote-devel
  

[Finance-quote-devel] Vanguard (US) investments daily closing price module

[Derek R. <der...@ym...>]

(posting here at Frank Ellenberger's suggestion; this originated in the GnuCash user mailing list)

Vanguard investors may be frustrated that many Vanguard products (like their 529 portfolios, any 401(k) / 403(b) products with the word "Trust" in the title) do not have prices (really NAVs, or Net Asset Values) that are easily obtainable from one of the regular price sources. These products do not have ticker symbols and cannot be accessed with AlphaVantage (as far as I know). Vanguard manages over $5T USD in assets, so price discovery for its investments should be helpful to a broad audience.

The attached module extracts prices from the Vanguard price history page and (in its original intent, anyway) adds them to the GnuCash price database. It does not attempt to extract anything other than a NAV and date although it could be extended to find other attributes on a different part of the Vanguard site.


Please let me know if I can clarify anything in the module.


Regards,Derek