I have been investigating possibly building the ability into OFFL to use other sources for stats, scores and player information. There are several RSS feeds available that can provide this information although most at a price. I do plan to implement RSS subscriptions for NFL player news. This can provide player information real time as the news is released.
Stats.com has a streaming XML feed that can handle scoring in realtime or allow for solicited querys. Stats.com provides their info to FOX, CBS and NFL.com.
So currently OFFL gets stats like this
XML(stats.com) ---> NFL.com---(XML/HTML)<----OFFL
Any interest in this, or am I just beating a drum to make noise?
Billy
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The PHP feed is actually a serialized array transported via XML. You would still have to parse the array and re-structure it. XML RSS feeds are industry standard and are actually the original source of the stats used at myffl.net. They screen scrape NFL.com as we do, then convert to XML.
Pros for XML
-XML RSS is standardized which will enable better error handling and less maintenance
-XML interfaces can provide much more information than stats. News/Blogs/IR most for free from these web sites (Yahoo, SI, ESPN, FOX and NFL. Source Forge news can also be subscribed to....OFFL project news and release information, bugs, security alerts etc.
-RSS is what PODCasts use allowing video and audio information.
Cons:
-The current OFFL OO environment was not built around CSS standards, so this could require a great deal of code to change.
-Too many RSS feeds could be resource intensive on OFFL servers(If running Windows)
-True XML sources for Stats are not free.(MyFFL converts HTML->XML) but could serve as a decent testing ground.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here's what I like about moving to XML through myFFL.net:
- instead of all the machines that run OFFL continuously hitting the NFL's servers during game week, we instead take lean, mean XML from myFFL.net which is updated more often than most OFFL servers scrape,
- if OFFL servers started using myFFL.net XML feeds instead of having to parse the stats themselves, it could remove a computational burden from the OFFL server (XML is easier for a cpu to parse versus taking entire webpages from all games and breaking it down),
- if OFFL gets a wider user base than it already has, and all the OFFL servers are nailing the NFL's website every Sunday afternoon, nfl.com may start tweaking their code every week, possibly breaking our current stat scraping method and making a mess for us to try and troubleshoot,
- by piggybacking XML from myFFL.net, we leverage the knowledge from that project rather than try to duplicate the same capability within OFFL - right now, OFFL's stat scraping doesn't parse all the categories that myFFL.net provides.
It would be nice to build OFFL with some redundancy by making each one of the stat scraping methods a user definable option. Hence, if nfl.com changed their code before a given Sunday, you could toggle to scrape stats through myFFL.net instead. This might be pie in the sky thinking, but in a way, I think we're running OFFL on borrowed time...
I'm not sure if any of you were following this, but MLB.com was considering charging for their stats. It went to court and they lost. The courts felt MLB was trying to gouge for something that people have really already paid for... Where am I going? At one time, I was worried the NFL would start charging for stats, but I think the precendent has been set in court and those days are behind us.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I completely agree on the 'hitting the nfl.com servers' point, Brad ... and the MLB thing is interesting. I'll have to look that up.
The myFFL feed is a serialized php array. Just fetch it and unserialize(). The xml feed is separate and different and, probably, will be discountinued in the near future in favor of a true RSS format.
I'm curious, though, what you mean by "in a way, I think we're running OFFL on borrowed time...". I'm really glad to see you're still at it and you've got a good community here which seems like it will live on whether you personally are actively involved or not. So, hopefully, you are not referring to the project itself.
If, instead, you are talking about Open-Source Fantasy Football in general the, yeah, its kind of an interesting question. I got into it when all the big league managers started charging buttloads to play. Now things seem to have swung back and almost every major site has a free option. So perhaps the need for open-source FF is dwindling.
Or, if you mean you're worried about the NFL shutting us all down ... well that's a game of whack-a-mole I don't think they can win. Even if they went to a (hard to parse) all-flash interface to display stats, there are ways to get at the text behind the flash and there will always be people hacking their way in.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yeah, when I said something to the effect of OFFL working on
"borrowed time", I was referring to the NFL getting cute and going to a Flash-like interface like you mentioned.
The reason I'm most interested in myFFL.net for the stats feed is that I think we are burdening ourselves by trying to develop a great web-based league manager AND a package to scrape stats from the NFL. Things have worked well so far, but if there's another project (myffl.net) actively working on the stat scraping ability, why should we attempt to recreate this effort our own way inside OFFL?
I guess this becomes an even more pressing issue, because I'm looking at trying to squash these stat-scraping bugs in the next several days and I'm not sure I'll have time to pull it off.
If anyone has been able to patch OFFL to fix the "player update" feature, I'd love to hear about it. I'll patch the code on the CVS and roll out a patch pronto.
Brad
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
All,
I have been investigating possibly building the ability into OFFL to use other sources for stats, scores and player information. There are several RSS feeds available that can provide this information although most at a price. I do plan to implement RSS subscriptions for NFL player news. This can provide player information real time as the news is released.
Stats.com has a streaming XML feed that can handle scoring in realtime or allow for solicited querys. Stats.com provides their info to FOX, CBS and NFL.com.
So currently OFFL gets stats like this
XML(stats.com) ---> NFL.com---(XML/HTML)<----OFFL
Any interest in this, or am I just beating a drum to make noise?
Billy
Billy,
We'd love to have this functionality. I'd gladly add your modified code to the project if you decide to start working on this...
Also, for a free XML stats feed that is updated every three minutes, check out this site:
http://sourceforge.net/forum/forum.php?forum_id=406854
Or head to MyFFL.net...
Brad
Actually myffl has a php feed now which might be easier to work with than the xml feed.
The PHP feed is actually a serialized array transported via XML. You would still have to parse the array and re-structure it. XML RSS feeds are industry standard and are actually the original source of the stats used at myffl.net. They screen scrape NFL.com as we do, then convert to XML.
Pros for XML
-XML RSS is standardized which will enable better error handling and less maintenance
-XML interfaces can provide much more information than stats. News/Blogs/IR most for free from these web sites (Yahoo, SI, ESPN, FOX and NFL. Source Forge news can also be subscribed to....OFFL project news and release information, bugs, security alerts etc.
-RSS is what PODCasts use allowing video and audio information.
Cons:
-The current OFFL OO environment was not built around CSS standards, so this could require a great deal of code to change.
-Too many RSS feeds could be resource intensive on OFFL servers(If running Windows)
-True XML sources for Stats are not free.(MyFFL converts HTML->XML) but could serve as a decent testing ground.
Thanks for the breakdown, Billy.
Here's what I like about moving to XML through myFFL.net:
- instead of all the machines that run OFFL continuously hitting the NFL's servers during game week, we instead take lean, mean XML from myFFL.net which is updated more often than most OFFL servers scrape,
- if OFFL servers started using myFFL.net XML feeds instead of having to parse the stats themselves, it could remove a computational burden from the OFFL server (XML is easier for a cpu to parse versus taking entire webpages from all games and breaking it down),
- if OFFL gets a wider user base than it already has, and all the OFFL servers are nailing the NFL's website every Sunday afternoon, nfl.com may start tweaking their code every week, possibly breaking our current stat scraping method and making a mess for us to try and troubleshoot,
- by piggybacking XML from myFFL.net, we leverage the knowledge from that project rather than try to duplicate the same capability within OFFL - right now, OFFL's stat scraping doesn't parse all the categories that myFFL.net provides.
It would be nice to build OFFL with some redundancy by making each one of the stat scraping methods a user definable option. Hence, if nfl.com changed their code before a given Sunday, you could toggle to scrape stats through myFFL.net instead. This might be pie in the sky thinking, but in a way, I think we're running OFFL on borrowed time...
I'm not sure if any of you were following this, but MLB.com was considering charging for their stats. It went to court and they lost. The courts felt MLB was trying to gouge for something that people have really already paid for... Where am I going? At one time, I was worried the NFL would start charging for stats, but I think the precendent has been set in court and those days are behind us.
I completely agree on the 'hitting the nfl.com servers' point, Brad ... and the MLB thing is interesting. I'll have to look that up.
The myFFL feed is a serialized php array. Just fetch it and unserialize(). The xml feed is separate and different and, probably, will be discountinued in the near future in favor of a true RSS format.
This year, I have implemented a scores/schedule feed which is available as html, serialized php, var_export php and RSS. You can see the different formats for the preason week 2 using the following URLs respectively: http://myffl.net/feed/scores/?2006pre/2/html/verbose http://myffl.net/feed/scores/?2006pre/2/php/verbose http://myffl.net/feed/scores/?2006pre/2/dump/verbose http://myffl.net/feed/scores/?2006pre/2/rss/verbose (Other applications will be using the scores feed to correct their schedule in the last weeks of the season when the Sunday nights games change.)
The plan is to use provide similar formats for the game feed (which is currently only available as php and the non-standard xml) sometime soon.
I'm curious, though, what you mean by "in a way, I think we're running OFFL on borrowed time...". I'm really glad to see you're still at it and you've got a good community here which seems like it will live on whether you personally are actively involved or not. So, hopefully, you are not referring to the project itself.
If, instead, you are talking about Open-Source Fantasy Football in general the, yeah, its kind of an interesting question. I got into it when all the big league managers started charging buttloads to play. Now things seem to have swung back and almost every major site has a free option. So perhaps the need for open-source FF is dwindling.
Or, if you mean you're worried about the NFL shutting us all down ... well that's a game of whack-a-mole I don't think they can win. Even if they went to a (hard to parse) all-flash interface to display stats, there are ways to get at the text behind the flash and there will always be people hacking their way in.
Dan,
Yeah, when I said something to the effect of OFFL working on
"borrowed time", I was referring to the NFL getting cute and going to a Flash-like interface like you mentioned.
The reason I'm most interested in myFFL.net for the stats feed is that I think we are burdening ourselves by trying to develop a great web-based league manager AND a package to scrape stats from the NFL. Things have worked well so far, but if there's another project (myffl.net) actively working on the stat scraping ability, why should we attempt to recreate this effort our own way inside OFFL?
I guess this becomes an even more pressing issue, because I'm looking at trying to squash these stat-scraping bugs in the next several days and I'm not sure I'll have time to pull it off.
If anyone has been able to patch OFFL to fix the "player update" feature, I'd love to hear about it. I'll patch the code on the CVS and roll out a patch pronto.
Brad