Carl - looking at Dave's Zap2it file there are no programmes with the category of 'Series' so programmes like The Late Late Show with James Corden won't get flagged as generic.
Why is that when I see 'Series' category in other Zap2it files and you also recommended it as a reliable way of identifying a series. Is it dependent on a Zap2it setting?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
After checking "Set empty previously shown tags to default date" and re-downloading the guide, I see only 1 Simpson episode scheduled to record in the next 10 days... Thanks !
I found another issue, I have not gone through the logs yet fully today...
The show Fast N' Loud, it's set to only record NEW episodes in WMC.
On Discovery channel they replay the same episode later for the west coast feeds (I am on the EAST coast).
For tonight, in Guide, the 9pm one is scheduled to record(as it should) but, the replay for the West coast at 1am, with the same desc, is also scheduled to record.
In WMC, the 2nd show is scheduled for 5/5 (1am) but, it still shows Orignal Air date as 5/4..but, is still scheduled to record...same Episode number, of course.
Last edit: Dave Evans 2020-05-04
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
None of the programmes in your xml file have a category of 'Series' which is what's needed to make scheduled recording and updating of generic programe descriptions work properly.
I have seen this category in several other Zap2it files so I assume it's some sort of option in Zap2it to retrieve that information. I'm not familiar with Zap2it and have asked Carl if he knows as it is present in his files. He hasn't replied yet.
👍
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
-u <username>
-p <password>
-d <# of days> (default = $days)
-n <# of no-cache days> (from end) (default = $ncdays)
-N <# of no-cache days> (from start) (default = $ncsdays)
-B <no-cache day="">
-s <start day="" offset=""> (default = $start)
-o <output xml="" filename=""> (default = "$outFile")
-c <cachedirectory> (default = "$cacheDir")
-l <lang> (default = "$lang")
-i <icondirectory> (default = don't download channel icons)
-m <#> = offset program times by # minutes (better to use TZ env var)
-b = retain website channel order
-x = output XTVD xml file format (default = XMLTV)
-w = wait on exit (require keypress before exiting)
-q = quiet (no status output)
-r <# of connection retries before failure> (default = $retries, max 20)
-e = hex encode entities (html special characters like accents)
-E "amp apos quot lt gt" = selectively encode standard XML entities
-F = output channel names first (rather than "number name")
-O = use old tv_grab_na style channel ids (C###nnnn.zap2it.com)
-A "new live" = append " *" to program titles that are "new" and/or "live"
-M = copy movie_year to empty movie sub-title tags
-U = UTF-8 encoding (default = "ISO-8859-1")
-L = output "<live>" tag (not part of xmltv.dtd)
-T = don't cache files containing programs with "$sTBA" titles
-P http://proxyhost:port = to use an http proxy
-C <configuration file=""> (default = "$confFile")
-S <#seconds> = sleep between requests to prevent flooding of server
-D = include details = 1 extra http request per program!
-I = include icons (image URLs) - 1 extra http request per program!
-J <xmltv> = include xmltv file in output
-Y <lineupid> (if not using username/password)
-Z <zipcode> (if not using username/password)
-z = use tvguide.com instead of zap2it.com
-a = output all channels (not just favorites)
-j = add "series" category to all non-movie programs``</zipcode></lineupid></xmltv></configuration></live></icondirectory></lang></cachedirectory></output></start></no-cache></password></username>
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The j flag marks anything not a movie as a series. I think most that tested it didn't find it useful but others can comment and it would depend on what you are wanting to accomplish.
Hmmm, maybe my terminology isn't correct. Maybe adding the series category to everything not a movie is somehow different than marking everything as a series? The full implications of these tags isn't always clear to me.
Last edit: s armstrong 2020-05-06
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Steve,
This is my fault. I didn't realized that I use the "-j" option (using someone else setup file"), so "Series" category is always generated for my xmltv files. I just generated two xmltv files, one with "-j" and the second without, and they are same except for the "<category lang="en">Series</category>" line.
Ihavereadthezap2xmlperlprogram, andI'm pretty sure that the best way of figuring out if a show is a series or not is to use the "<episode-num system="common">" line. That line will only be generated is series > 0 and eposide > 0 (ie a series).Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Carl - the suggestion to use the 'common' episode-num tag won't work. That's only present for EP entries so you might as well use the ID beginning with EP in any case.
It also means that shows like The Late Late Show with James Corden that have a generic description and the SH prefix (and don't have the 'common' attribute) wouldn't be considered a series and therefore wouldn't have the isGeneric flag set.
I think the only way to set the isGeneric flag and not rely on -j on the Zap2xml command line is to set it for all shows prefixed with SH. That must be pretty much what the -j is doing anyway..
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Carl - looking at one of the Zap2it xml files I have which has cast and crew and categories for many programmes (which I assume means the Zap2itxml command line included -D) the category Series is not on every non-movie programme but only only the ones with EP prefix and SH ones that we would want to mark with the isGeneric flag eg James Corden.
So it looks to me like the only way to correctly set the isGeneric flag for a programme is to have -D on the Zap2xml command line and incur the extra HTTP request for each programme. Alternatively as I said above we could mark every SH programme with isGeneric.
What do you think?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Gary is the expert on the isGeneric flag, he the one that gave us this information. Here what he said:
The isGeneric flag is used for those, well, generic programs that don't have episode specific information. It will tell WMC that though this program may not be properly identified, it will be treated as unique and will always be considered "new". The isGeneric flag also allows you to record the nightly news every day as part of a series recording. When to use isGeneric="true"? EPG123 uses it when the program ID starts with "SH", is a series, is not a miniseries, and is not paid programming OR starts with "SH" and is a TBA sports event (this one isn't easy to detect in the xmltv file).
To me, that means that ALL "SH" programs should have isGeneric set to true, but there are exceptions, SH000000010000 (Paid Programming), do NOT. I'm not sure how to detect **a series, is not a miniseries, or is a TBA sports event (this one isn't easy to detect in the xmltv file).
In my xmltv files, "SH" programs are either News programs, Late night shows, Generic series placeholders or Paid Programming. All but the Paid Programming should have the flag set.
I have been getting 14 days worth of data from zap2it. The data for the next 7 days seems fine, its the data that more then 7 days out that can get the Generic placeholders. Because zap2xml caches the pages from zap2it, the reality is that it uses the pages that were previous cached to generate the xml, reload 13 days worth of data from cache, only downloading day 14. So what happens is after a few weeks of download data, all the cache data is 14 days old. The data from days 1 to 13 never get reload unless you use the N option. To make sure that I get the most current data for the next 24 hours, I now use the "-N 1" to make it reload that data.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ah, interesting about the -N 1 command, I will udpate my script to apply this, and I will do a command to delete all the cache before downloading new data.
I only set my download to give 10 days. I head issues going out past 7-10 days with generic data. I DO want it to update data if there is a change on the new import.
I don't want to cause a lot of traffic on Zap2it but, I want good data.
Thanks for info guys... If you need anything else from me, please let me know.
I do know the Discovery channel issue is still shown for next week... Like I said above. When they have shows for the east and west coast, it will record both, even the next day.
It records the first episode fine In WMC, the 2nd showing is scheduled for 5/5 (1am) but, it still shows Orignal Air date as 5/4..but, is still scheduled to record...same Episode number, of course.
I will update with the -N 1 and re-download to see if that helps with that
Edit: after doing above, there has been no change to the Fast N' Loud show, it will record it at 9pm on Monday, and record the same episode at 1am on Tuesday(when it does the 2nd showing).
Last edit: Dave Evans 2020-05-07
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Carl - Yes i was following Gary's lead on this as he does know his stuff when it comes to WMC.
I have Zap2xml files where isGeneric can be set using his rules because the categories are present. There is a category of Series where it is needed (not every SH) and Miniseries appears as a category so can be detected. Paid Programming is easy to detect and I'm ignoring the SH for a TBA sports event as I've never seen it so can't test any code for it anyway.
However I assume these files were created with the -D Zap2xml command line parameter which means an extra web access, more strain on the Zap22it servers and longer download times.
So if the isGeneric flag is to work it either needs everybody to use -D or EPGC sets isGeneric true for every SH programme except Paid Programming. That means it will be set for miniseries.
-D or flagging SH programmes incorrectly? Which do you want.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, the problem is only with Miniseries shows which only get flag with the -D option. I search the only xmltv file I have generated with -D and there were 17 shows marked as Miniseries, but ALL of them are 'EP' shows, not 'SH', so they would never be flaged. The question I have, how often does a 'SH' Miniseries happen in the xml files that you have.
I don't understand why Miniseries are an exceptions. In my mind, they are the same as any other series.
If I was given the choice of -D or flagging SH programmes incorrectly, I would go with incorrectly, but it would be nice to know how often it happens. If it only happens once in a blue moon, then I'm willing to go with it. The other factor is that setting the flag to true will only have an effect if that program has been scheduled to be recorded. Yes, it will look like a new program in the guide, but so what.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Well, it has happened that a miniseries had an 'SH', or I wouldn't have added the exception I don't think. More to the point, anything that is a miniseries should never be generic... the whole thing about being a miniseries is that all episodes are already defined and completed before the first episode is even shown.
But agree, it should be a rare event and the only ones that may care are the ones recording it. Worst case is they will record the same episode multiple times.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think what I will do is, if EPGC is processing a Zap2xml file, check the file for category tags and if any are present look for Series, Miniseries etc before setting the isGeneric flag.
If they are not present then all SH programmes except Paid Programming will be flagged with isGeneric set.
Then it's up to the user as to how they want to download the Zap2it file.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have Zap2xml files that have categories present but none of the programmes have a category of Series. I have other Zap2xml files that have categories present and do have programmes with the Series category. This is for the same programme. I tend to check The Late Late Show with James Corden.
So maybe to get categories in the Zap2xml file and have programmes marked with the Series category you need both -D and -j flags in the Zap2xml command line.
👍
1
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Added the -j to my command line, and all it did was add repeats that was scheduled to record (10 of them over a week) and they were clearly repeats, I used EPG123's clear guide data feature, wiped out the guide info, Re-downloaded with out the -j and all the repeats that it added were not scheduled any more.
So in my case the -j caused more problems....
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Steve,
Zap2xml files without -D will generate at least 5 categories, Family, Movie, News, Sports and Talk, but only because those categories are hard coded into zap2xml program. It looks to me as if the categories information is on the main web page, but encoded as a number. My guess is that those categories are converted from a number into text when you use -D. With -D, I found one xml file with 101 categories. The category "Series" is NOT generated in either case. The Zap2xml perl program will only add that category for the -j option, and then it will generate it for EVERY program except for movies ("MV").
Your software should detect series for all programs except for movies, and not depend on it showing up in the xml file. I'm pretty sure that we could upgrade the zap2xml perl program to generate the "Miniseries" category (and any others you might need) if we see a need for it down the road.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
On the Import tab entry for your Zap2it file you need to have the option 'Set empty previously shown tags to default date' ticked.
The original air date is then set to 1/1/1970 for the SH version you list above and that will stop it being recorded.
Carl - looking at Dave's Zap2it file there are no programmes with the category of 'Series' so programmes like The Late Late Show with James Corden won't get flagged as generic.
Why is that when I see 'Series' category in other Zap2it files and you also recommended it as a reliable way of identifying a series. Is it dependent on a Zap2it setting?
After checking "Set empty previously shown tags to default date" and re-downloading the guide, I see only 1 Simpson episode scheduled to record in the next 10 days... Thanks !
I found another issue, I have not gone through the logs yet fully today...
The show Fast N' Loud, it's set to only record NEW episodes in WMC.
On Discovery channel they replay the same episode later for the west coast feeds (I am on the EAST coast).
For tonight, in Guide, the 9pm one is scheduled to record(as it should) but, the replay for the West coast at 1am, with the same desc, is also scheduled to record.
In WMC, the 2nd show is scheduled for 5/5 (1am) but, it still shows Orignal Air date as 5/4..but, is still scheduled to record...same Episode number, of course.
Last edit: Dave Evans 2020-05-04
None of the programmes in your xml file have a category of 'Series' which is what's needed to make scheduled recording and updating of generic programe descriptions work properly.
I have seen this category in several other Zap2it files so I assume it's some sort of option in Zap2it to retrieve that information. I'm not familiar with Zap2it and have asked Carl if he knows as it is present in his files. He hasn't replied yet.
Here is an updated XMLTV if it will help. I see the same issue for next week's episode, I am sure it's affecting other shows...
Thanks again for helping getting to the bottom of this...
Dave - my post above yours is meant for you. Do you know where there is a list of the Zap2xml command line parameters?
Steve,
-u <username>
-p <password>
-d <# of days> (default = $days)
-n <# of no-cache days> (from end) (default = $ncdays)
-N <# of no-cache days> (from start) (default = $ncsdays)
-B <no-cache day="">
-s <start day="" offset=""> (default = $start)
-o <output xml="" filename=""> (default = "$outFile")
-c <cachedirectory> (default = "$cacheDir")
-l <lang> (default = "$lang")
-i <icondirectory> (default = don't download channel icons)
-m <#> = offset program times by # minutes (better to use TZ env var)
-b = retain website channel order
-x = output XTVD xml file format (default = XMLTV)
-w = wait on exit (require keypress before exiting)
-q = quiet (no status output)
-r <# of connection retries before failure> (default = $retries, max 20)
-e = hex encode entities (html special characters like accents)
-E "amp apos quot lt gt" = selectively encode standard XML entities
-F = output channel names first (rather than "number name")
-O = use old tv_grab_na style channel ids (C###nnnn.zap2it.com)
-A "new live" = append " *" to program titles that are "new" and/or "live"
-M = copy movie_year to empty movie sub-title tags
-U = UTF-8 encoding (default = "ISO-8859-1")
-L = output "<live>" tag (not part of xmltv.dtd)
-T = don't cache files containing programs with "$sTBA" titles
-P http://proxyhost:port = to use an http proxy
-C <configuration file=""> (default = "$confFile")
-S <#seconds> = sleep between requests to prevent flooding of server
-D = include details = 1 extra http request per program!
-I = include icons (image URLs) - 1 extra http request per program!
-J <xmltv> = include xmltv file in output
-Y <lineupid> (if not using username/password)
-Z <zipcode> (if not using username/password)
-z = use tvguide.com instead of zap2it.com
-a = output all channels (not just favorites)
-j = add "series" category to all non-movie programs``</zipcode></lineupid></xmltv></configuration></live></icondirectory></lang></cachedirectory></output></start></no-cache></password></username>
OK thanks.
To get the isGeneric flag set EPGC needs to see the Series category so you would set the -j flag. Is that correct?
The j flag marks anything not a movie as a series. I think most that tested it didn't find it useful but others can comment and it would depend on what you are wanting to accomplish.
Hmmm, maybe my terminology isn't correct. Maybe adding the series category to everything not a movie is somehow different than marking everything as a series? The full implications of these tags isn't always clear to me.
Last edit: s armstrong 2020-05-06
Steve,
This is my fault. I didn't realized that I use the "-j" option (using someone else setup file"), so "Series" category is always generated for my xmltv files. I just generated two xmltv files, one with "-j" and the second without, and they are same except for the "<category lang="en">Series</category>" line.
Thanks Carl, I'll check it out in the various xml files peple have given me.
Carl - the suggestion to use the 'common' episode-num tag won't work. That's only present for EP entries so you might as well use the ID beginning with EP in any case.
It also means that shows like The Late Late Show with James Corden that have a generic description and the SH prefix (and don't have the 'common' attribute) wouldn't be considered a series and therefore wouldn't have the isGeneric flag set.
I think the only way to set the isGeneric flag and not rely on -j on the Zap2xml command line is to set it for all shows prefixed with SH. That must be pretty much what the -j is doing anyway..
Carl - looking at one of the Zap2it xml files I have which has cast and crew and categories for many programmes (which I assume means the Zap2itxml command line included -D) the category Series is not on every non-movie programme but only only the ones with EP prefix and SH ones that we would want to mark with the isGeneric flag eg James Corden.
So it looks to me like the only way to correctly set the isGeneric flag for a programme is to have -D on the Zap2xml command line and incur the extra HTTP request for each programme. Alternatively as I said above we could mark every SH programme with isGeneric.
What do you think?
Steve,
Gary is the expert on the isGeneric flag, he the one that gave us this information. Here what he said:
The isGeneric flag is used for those, well, generic programs that don't have episode specific information. It will tell WMC that though this program may not be properly identified, it will be treated as unique and will always be considered "new". The isGeneric flag also allows you to record the nightly news every day as part of a series recording. When to use isGeneric="true"? EPG123 uses it when the program ID starts with "SH", is a series, is not a miniseries, and is not paid programming OR starts with "SH" and is a TBA sports event (this one isn't easy to detect in the xmltv file).
To me, that means that ALL "SH" programs should have isGeneric set to true, but there are exceptions, SH000000010000 (Paid Programming), do NOT. I'm not sure how to detect **a series, is not a miniseries, or is a TBA sports event (this one isn't easy to detect in the xmltv file).
In my xmltv files, "SH" programs are either News programs, Late night shows, Generic series placeholders or Paid Programming. All but the Paid Programming should have the flag set.
I have been getting 14 days worth of data from zap2it. The data for the next 7 days seems fine, its the data that more then 7 days out that can get the Generic placeholders. Because zap2xml caches the pages from zap2it, the reality is that it uses the pages that were previous cached to generate the xml, reload 13 days worth of data from cache, only downloading day 14. So what happens is after a few weeks of download data, all the cache data is 14 days old. The data from days 1 to 13 never get reload unless you use the N option. To make sure that I get the most current data for the next 24 hours, I now use the "-N 1" to make it reload that data.
Carl
Ah, interesting about the -N 1 command, I will udpate my script to apply this, and I will do a command to delete all the cache before downloading new data.
I only set my download to give 10 days. I head issues going out past 7-10 days with generic data. I DO want it to update data if there is a change on the new import.
I don't want to cause a lot of traffic on Zap2it but, I want good data.
Thanks for info guys... If you need anything else from me, please let me know.
I do know the Discovery channel issue is still shown for next week... Like I said above. When they have shows for the east and west coast, it will record both, even the next day.
It records the first episode fine In WMC, the 2nd showing is scheduled for 5/5 (1am) but, it still shows Orignal Air date as 5/4..but, is still scheduled to record...same Episode number, of course.
I will update with the -N 1 and re-download to see if that helps with that
Edit: after doing above, there has been no change to the Fast N' Loud show, it will record it at 9pm on Monday, and record the same episode at 1am on Tuesday(when it does the 2nd showing).
Last edit: Dave Evans 2020-05-07
Carl - Yes i was following Gary's lead on this as he does know his stuff when it comes to WMC.
I have Zap2xml files where isGeneric can be set using his rules because the categories are present. There is a category of Series where it is needed (not every SH) and Miniseries appears as a category so can be detected. Paid Programming is easy to detect and I'm ignoring the SH for a TBA sports event as I've never seen it so can't test any code for it anyway.
However I assume these files were created with the -D Zap2xml command line parameter which means an extra web access, more strain on the Zap22it servers and longer download times.
So if the isGeneric flag is to work it either needs everybody to use -D or EPGC sets isGeneric true for every SH programme except Paid Programming. That means it will be set for miniseries.
-D or flagging SH programmes incorrectly? Which do you want.
Steve,
OK, the problem is only with Miniseries shows which only get flag with the -D option. I search the only xmltv file I have generated with -D and there were 17 shows marked as Miniseries, but ALL of them are 'EP' shows, not 'SH', so they would never be flaged. The question I have, how often does a 'SH' Miniseries happen in the xml files that you have.
I don't understand why Miniseries are an exceptions. In my mind, they are the same as any other series.
If I was given the choice of -D or flagging SH programmes incorrectly, I would go with incorrectly, but it would be nice to know how often it happens. If it only happens once in a blue moon, then I'm willing to go with it. The other factor is that setting the flag to true will only have an effect if that program has been scheduled to be recorded. Yes, it will look like a new program in the guide, but so what.
Carl
Well, it has happened that a miniseries had an 'SH', or I wouldn't have added the exception I don't think. More to the point, anything that is a miniseries should never be generic... the whole thing about being a miniseries is that all episodes are already defined and completed before the first episode is even shown.
But agree, it should be a rare event and the only ones that may care are the ones recording it. Worst case is they will record the same episode multiple times.
I think what I will do is, if EPGC is processing a Zap2xml file, check the file for category tags and if any are present look for Series, Miniseries etc before setting the isGeneric flag.
If they are not present then all SH programmes except Paid Programming will be flagged with isGeneric set.
Then it's up to the user as to how they want to download the Zap2it file.
I had to modify this process again.
I have Zap2xml files that have categories present but none of the programmes have a category of Series. I have other Zap2xml files that have categories present and do have programmes with the Series category. This is for the same programme. I tend to check The Late Late Show with James Corden.
So maybe to get categories in the Zap2xml file and have programmes marked with the Series category you need both -D and -j flags in the Zap2xml command line.
Added the -j to my command line, and all it did was add repeats that was scheduled to record (10 of them over a week) and they were clearly repeats, I used EPG123's clear guide data feature, wiped out the guide info, Re-downloaded with out the -j and all the repeats that it added were not scheduled any more.
So in my case the -j caused more problems....
Steve,
Zap2xml files without -D will generate at least 5 categories, Family, Movie, News, Sports and Talk, but only because those categories are hard coded into zap2xml program. It looks to me as if the categories information is on the main web page, but encoded as a number. My guess is that those categories are converted from a number into text when you use -D. With -D, I found one xml file with 101 categories. The category "Series" is NOT generated in either case. The Zap2xml perl program will only add that category for the -j option, and then it will generate it for EVERY program except for movies ("MV").
Your software should detect series for all programs except for movies, and not depend on it showing up in the xml file. I'm pretty sure that we could upgrade the zap2xml perl program to generate the "Miniseries" category (and any others you might need) if we see a need for it down the road.
Carl