I'm using zap2it to build a simple OTA EPG data for Windows Media Center.
I'm not using the -D option.
I'm not using Lookups.
I'm trying to use WMC to find the repeats (I thought you said that WMC does a better job of repeat dectection, plus I find that Lookups take a long time to run).
The system is mostly working, but I find that none of the programs recorded using EPGC data have an original record date in WMC. I'm using WMC to find repeats, and I find that it rarely finds repeats. The problem is mostly with recording programs on my PBS channels. I have three in my area, and I find that some programs (like NOVA) can be brocasted many times over a week. In my xmltv file, I have five showing of "Mysteries of Sleep", two new (different channels) and three repeats. WMC is schedule to records all five. Looking at the TVGuide.MXF file, I see the five programs (see A.txt), with the two new programs having originalAirdate="0001-01-01T00:00:00", and the three repeats having originalAirdate="2020-02-27T00:00:00". It seems to me that programs that are new shoud have the originalAirdate set to the time of the showing, not "0001-01-01". How is WMC able to identifiy new programs vs programs where we don't know the originalAirDate?
It seems to me that it would be better if the "Series and repeats" first option should be "Use WMC repeat checking" instead of "Not used".
Is there a way to generate the MXF without having it loaded into WMC (so I can test some changes and then manual load it into WMC)?
You often say in your release notices of new Fixpacks that "See the release notes for details.", but I have looked at the release notes for Fixpack 21 and I have a hard time seeing what has been changed (for 19, 20 and 21). Could you either add a Fixpack log at the end with details of each fix, or label your notes with the Fixpack to make it easier to see what changed?
Lookups may take a long time the first time you run with it enabled but it caches the datalocally so after that only data not held locally is downloaded so it should be a lot quicker. Saying that I wouldn't bother with it for Zap2it data as it seems pretty comprehensive.
It's correct you should run with Series and repeats set to not used. If this is set to anything else EPGC messes around with the orIginal air date and it has proved less successful than using the repeat checking in WMC (which also uses the original air date). I have considered actually removing the option altogether.
The original air date of 0001-01 etc basically means not set ie new programme. EPGC merely transfers any previously-shown tag in the Zap2it file to the original air date field. It has also been observed that any original air date within 7 days of the next showing means the next showing will not be considered a repeat by WMC and will be recorded so it would make no difference if EPGC set the original air date to the date of broadcast in your PBS example. They would still be recorded and there is no way to stop it (AFAIK).
WMC does seem to hold data for a long time, possibly never deleting it. As long as you don't reset the database maybe it can detect duplicate recordingsoutside the 7 day window even though the original air date is not set. I don't know.
The command line to manually import an MXF file into WMC is
LoadMXF -v -i "mxf file name including full path"
LoadMXF is in the Windows/ehome directory. Run it from there.
I'll endeavour to make the fixpack notes more comprehensive. TBH it's only lately since you NA guys have popped up that there has been a rash of fixes and changes. EPGC has been around a long time and for most users it just worked with the occasional one asking for a new feature.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, IhackedupmylastloadedTVGuide.mxf, changedtheoriginalAirdateofthetwonew"Mysteries of Sleep"programsfrom"0001-01-01"to"2020-02-27"andreloadeditintoWMC(andrescheduledtheindexing). NowwhenIlookattheshowsscheduledforNOVA, thetwonewshowingof"Mysteries of Sleep"showaoriginalairdate, andnow, thethreererunsalsoshowaoriginalairdate(theyneverdidbefore). NootherprogramthatEPGChasschedulehasevershowanoriginalairdate. AllotherscheduledNOVAshows(notMysteriesofSleep)stillshownooriginalairdates(sonoteffected). Istillhavefivecopiesof"Mysteries of Sleep"scheduled, butfromwhatIunderstand, onceWMCschedulesaprogram, itstaysscheduleunlesstheprogramisdeletedorthedatabasegetreset(whichIdon't want to do). So from this test, it looks like setting the original air date to the time of the program is scheduled for NEW programs should work and is the right thing to do.IlookedattheMicrosoftdocumentationforMXF, butitnotclearwhenoriginalAirdateshouldbeset, butatleastintheexample, thethreeprogramsintheexampleallhadoriginalAirdateset. Itnotcleariftheprogramsareneworrepeats, buttheylooknewtome(theyallhaddifferentdescriptions, sonorepeatofpreviousprograms).
Inthepast, ItriedLookups, for6000showsovertwoweeks, ittookoveranhourthefirsttimeIranit, andthenIranitasecondtimewiththesameinputdata(soitshouldhavebuffermostofthelookups), itstilltookover30minutestorun. Anditdidn't seem to fix the original air date problem. From what I can tell. The Zap2it file without the -D option provides me 99% of what I what (or willing to live with), it just missing the original air date. One option that you could add that would cut down on the lookup time is to only process programs that could be schedule. That should cut out 90% of the lookups. For now, I can live with the recording of repeats from 6 months ago, but I want to stop the scheduling of repeats that I just recorded for the next week.IhavethoughtabouthackingmyownprogramtofixtheoriginalairdateandkeeptrackofthedateforprogramsIschedule, butIwouldneedtomodifyyourmxffileafteryougenerateit, butbeforeitgetsloadintoWMC, butIhaven't been able to figure out how to stop EPGC from automatically loading it after it generate it. Maybe you could provide an option on the Output tab to not load it after generating it.Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The reindex job doesn't affect the schedules, Did you mean the update PVR schedule task?
If the previously-shown tag is in the Zap2it file is present you will see it in the original air date field in WMC as long as the Series and repeats is set to Not used. Other settings for that option do mess around with the original air date.
In the MXF document which, being kind here, is pretty vague on most counts the original air date is set on the 3 program but in no case is it set to the schedule date. Also In the description of the field it does say 'Use this value to determine whether this program is a repeat'.
It's probable that your first lookup run timed out after the default of 60 mins (nowhere near long enough for 6000 programs) so not all the programs collected were processed. That's why the next run also took a while. Once all the programmes in the collection have been processed it is a lot quicker. Sying that I would still not bother with it. The original air date on which you are focussing is likely to be more accurate from Zap2it than that entered on TMDB or TVDB.And if a new episode of a program is broadcast and then repeated a day or so later it probably won't even have made it to TVDB before you trty and look it up. What's the problem with using the -D option? The data from Zap2it looks pretty good to me with that option.
I could certainly provide an option to just create the MXF file and not import it. But if you manually imported a modified MXF file after an EPG run it would overwrite fields like the original air date anyway.
i'll do some more testing on this topic on my system and let you know the results. One thing I was thinking of trying was only creating one MXF program record for each unique program. At the moment EPGC creates an MXF program record for each program in th Zap2it file but the MXF format allows for a single program to be linked to any number of schedule records. it would be interesting to see the effect the original air date would have in this scenario.
I do know multiple recordings have been an issue in WMC for a long time. There have been many discussions on the Green Button site about it and someone even produced a utility to delete redundant recordings..
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
OK, I didn't reliese that it didn't finish processing and timeout, so not a good test of the overhead.
I'm trying to not use the -D option since that will cause another hit on the Zapit website for every show in the xmltv file. I'm trying to keep my impact on their website as small as posible. It looks to me that most of the time, zap2it sets the original air date for most shows that I care about (repeasts). If we can fix the repeat scheduling issure, it looks like I can live without the originalAirdate being set.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've done some further testing on this topic. I realised that the EPGC Series and repeats option set to 'Use programme title to generate links' does pretty much what you talk about.
It generate one program entry in the MXF file for each program with the same title and, if they are present, the same season and episode. If season and episode is not avaialbe then same description. It also sets the original air date on every program entry.
This seems to work as expected when you record a series 'New' and the programs differ by season/episode or description,
Where it doesn't work is when you try and record a series where all the descriptions are the same. So if I try to record The News at 6 which has the same description every day I would have to specify new and rerun otherwise it only records the first day. It then also unfortunately records News at 6 on every channel it is on (we have 4 TVNZ One channels here all showing the same programs with regionalized advertising) so I end up with multiple copies of the recording for each day.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Imighthavefoundasimpliersolution. Ihavebeenplayingaroundwiththeperlprogramxmltv-to-mxf. Ionlyhaveafewhourswithit, butIfoundthatitseemstodotherightthingasfarasrepeatsgoes. WhenIresetmydatabaseandloadit's mxf, I now only have one copy of "Mysteries of Sleep" being scheduled. When I look at the mxf file, it looks similar to yours, here is the output of the "Mysteries of Sleep" shows,<Programid="3856"uid="!Program!EP00003163_0953"title="NOVA"episodeTitle="Mysteries of Sleep"description="Scientists study why animals and humans need to sleep, what happens to the brain during sleep, and the role sleep plays in memory, trauma, and emotion regulation."episodeNumber="3"season="sn507"series="si439"isSeries="true"/><Programid="3862"uid="!Program!EP00003163_0953"title="NOVA"episodeTitle="Mysteries of Sleep"description="Scientists study why animals and humans need to sleep, what happens to the brain during sleep, and the role sleep plays in memory, trauma, and emotion regulation."episodeNumber="3"originalAirdate="2020-02-27T00:00:00.000Z"season="sn507"series="si439"isSeries="true"/><Programid="5056"uid="!Program!EP00003163_0953"title="NOVA"episodeTitle="Mysteries of Sleep"description="Scientists study why animals and humans need to sleep, what happens to the brain during sleep, and the role sleep plays in memory, trauma, and emotion regulation."episodeNumber="3"season="sn507"series="si439"isSeries="true"/><Programid="5060"uid="!Program!EP00003163_0953"title="NOVA"episodeTitle="Mysteries of Sleep"description="Scientists study why animals and humans need to sleep, what happens to the brain during sleep, and the role sleep plays in memory, trauma, and emotion regulation."episodeNumber="3"originalAirdate="2020-02-27T00:00:00.000Z"season="sn507"series="si439"isSeries="true"/><Programid="5063"uid="!Program!EP00003163_0953"title="NOVA"episodeTitle="Mysteries of Sleep"description="Scientists study why animals and humans need to sleep, what happens to the brain during sleep, and the role sleep plays in memory, trauma, and emotion regulation."episodeNumber="3"originalAirdate="2020-02-27T00:00:00.000Z"season="sn507"series="si439"isSeries="true"/><ScheduleEntryprogram="3856"duration="3600"/>
....
<ScheduleEntryprogram="5063"duration="3600"isRepeat="true"/>Twothingsstandouttome, first, fornewshows, hedropstheoriginalAirdatefield, andforrepeats, heusestheisRepeatfieldontheScheduleEntry. NotsurewherehefoundtheisRepeatdocumentation, itnotintheMicrosoftdocumentation. I'm guessing that isRepeat is the big change here, but I could be wrong.WhenIchecktheNOVAseriesinWMC, Ionlyhaveoneshowingof"Mysteries of Sleep"schedule, theotheroneatthesametimebutdifferentchannelisNOTscheduled, andtheotherfourversionsallshowthattheprogramisbeingrecordedelsewhere. TheotherthingInoticedisthatrepeatprogramsnowshowupwith"(Repeat)"intheGuide, whichisnice.
Ionlyhadacoupleofnewshowsrecordedtonight, butIfoundthatbothofthemdidNOTdisplaytheoriginalairdateinWMC, sothathasn't been fixed, but its only a minor problem in comparison to the repeat scheduling problem.Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm looking into this as well and also have some recent experience with xmltv-to-mxf. If you could post the related zap2xml lines from which the mxf above was generated, I'll see if I can offer any additional insight.
Last edit: Panda Moanium 2020-02-26
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Looking at the part of the MXF file you posted it looks wrong to me because the uid field is meant to be unique for each program but is the same for each of those programs..
The isRepeat flag won't do anything. The isRepeat flag in the definition of the ScheduleEntry class in the WMC code is read only. It can't be set from the MXF file. In fact the reading of this property is where the check for 7 days is made which I mentioned before.
Do you get the original air date in the Zap2it file without the -d option? Thinking about it some more Lookups definitely won't give you the result you want. Updating of the TVDB database isn't immediate enough so you would probably not find new episodes on there until after they have been broadcast..
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
"Do you get the original air date in the Zap2it file without the -d option?" That's also what I wanted to see... although it's actually the -D (upper case) option.
P.S. - 100% agree about TVDB not being a viable method. I think there may be a lot of different volunteers helping to maintain various shows on there, which means sometimes on vacation, sick, etc. I say this because at various times over the years I've seen outdated info for shows last for weeks or even months! And in any case, for the many years when Microsoft used to source their data from Zap2it I never had a problem with repeats being accidentally recorded. So, there must surely be a way to get this sorted out. Whether it can be done without having to resort to the -D option is (I think) a big part of the question.
Last edit: Panda Moanium 2020-02-27
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I didn't noticed that the uid value was the same for all of the "Mysteries of Sleep" shows, but I just check a newer verion and it the same way. He using the dd_progid value from the xmltv file (with underscore convert to '.') for his uid, so maybe its not wrong.
I just check, and zap2it used the same dd_progid for "Mysteries of Sleep" across two different days (pulls), so maybe zap2it is doing that so that shows can have the same uid over time.
The isRepeat flag might be for the Guide only, but about 90% of the shows in my OTA pulls are repeats.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Here the xmltv.xml file generated by zap2it and the TVGuide.mxf output file that the java xmltv-to-mxf program generated.
I had 24 hours of WMC usage of the xmltv-to-mxf output and found as far as the scheduling of new programs and the not scheduling of repeats, it seems to do it right thing 100% of the time. The xmltv-to-mxf still has problems, for one, it seems to drop almost all News programs as not being in a series (60 Minutes, 48 Hours, etc). It also doesn't generate the categories tags, but I think they are only being used by the Guide and I didn't noticed them missing until pointed out by someone else. I also miss the sesion and episode information that EPGC puts into the show descption. From what I can tell, the xmltv-to-mxf version also doesn't schedule repeats for new shows outside the 7 day window as well. Since they are in the mxf output file, it looks to me that WMC is correctly handling them.
zap2it will give the originalAirdate correctly for previously-shows that happen within the timeframe of the zap2it pull. In my case, I have been only pulling 7 days at a time. Repeats of new programs that are outside that window shows the previously-shown tag, but without a date. I have been giving the "Mysteries of Sleep" show as an example since the first show and repeats all occur within the 7 day window, and I would have expected that EPGC or WMC would have handle it correctly.
It looks to me that the originalAirdate as being the solution to repeat scheduling is not the main fix now that the xmltv-to-mxf version shows that a mxf can be generated from the zap2it simple download and not have all repeats being scheduled. All the programs that were recorded using Microsoft data have oiginalAirdates, but none of the new programs record last night or tonight (or with EPGC data) have that set either. The one repeat program that I ask to have record tonight does show the original air date (1/1/1970). It looking like the originalAirdate is only a minor factor, one I could live without.
Based on the above, it sounds like the issue will be that if a show was previously shown further back than the current window of Zap2it data then it will NOT be recognized as a repeat and therefore be recorded even if the series settings are for new episodes only. The question remains if the -D option is used if that will provide the previously shown date even if it's further back? And, if so, is that the only way to avoid recording older repeats? It definitely would be preferable to avoid having to use -D if at all possible since that does apparently create a substantial extra load against the Zap2it servers... which could potentially lead to them taking measures to kill the ability to d/l data from them.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The problem is that even when the previously has a date (within window), the EPGC version still schedule it.
I download using the -D option, and the previously shown date (and originalAirdate) is given for all programs (plus crew, credits, and extra categories ), include ones that outside the current window. I'm 98% sure that I loaded that version and I still had the repeat problem. I have also used the Lookup tag, and I'm sure that I still had the repeat problem, which is one of the reasons that I stop using it. I agree that the prefer way would be to not using the -D do to the substantial extra load. The fact that the java version works without the -D indicates to me that it can be done, we just have to figure out how.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The Zap2it file does contain previously-shown dates way back. I've seen some as old as 1995. There are also a lot that are blank. But that shouldn't matter. WMC keeps the original program record for every recording so should check these old programs when scheduling new recordings. After all a previously-shown date set for a program doesn't mean you have a recording of it.
The date 1/1/1970 you saw in a recording is similar to 1/1/0001. It's a default date not a real one.
Do you set up a new WMC database each time you try something different? If you don't the results can't be relied on.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
But if you setup a series record for the first time and set it to only record new shows (because, for example, you watched the prior seasons in some other way outside of WMC) then there will be no record of prior recordings and it would apparently therefore create recordings you don't need/want. Of course deleting them afterwards is possible but not ideal for various reasons... needlessly wasting electricity to wakeup the PC to record something you don't want, shows such as Carl's example which have multiple repeats (indeed NUMEROUS repeats for those of us with access to several PBS stations) could result in dozens of unwanted recordings thus becoming a problem if you don't have a lot of spare disk capacity, the unnecessary extra wear on drives recording airings you don't want, etc.
I still think there must be a way to avoid recording previously aired showings since I genuinely don't remember WMC mistakenly recording them even when I setup a new series record and there were subsequent repeats aired for which there was no prior recording of in its database. If it was previously able to do this properly there must be a way, right?
BTW, Carl, you said "I download using the -D option". Can I just confirm that was the case for what you supplied in your zip file? I ask because I do recall seeing an entry with the previously shown tag but no date after it. And for such a situation, I don't think even xmltv-to-mxf will work properly... it's my understanding that it ignores the previously shown tag if it's not accompanied by a date = per the current version release notes: "Repeats only marked if previously-shown has a date"
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
1) The Series.zip file included the xmltv file that was build without the -D option.
2) I rebuild my WMC database, loading the xmltv-to-mxf version, but before I did, I strip out the isRepeat flag from the ScheduleEntry. I can't see any difference in either the Guide or what was schedule, so I think that proves Steve correct in that the isRepeat flag has no effect.
3) I download the xmltv file with both with and without the -D option (one after the other to make them the same as possible, non D version second since it so fast). I rebuild WMC and loaded the -D version. I found that this version did NOT schedule the repeat shows. It seems to do the right thing as best as I could tell. I did find a show (SEAL Team) in the TVGuide.mxf output file that had both the isSeries and isMovie flags set to true, but the xmlt file only had the isSeries set.
Here is the entry for "Ask This Old House", a series that I record that PBS currently is only showing repeats.
</programme><programmestart="20200229043000 -0500"stop="20200229050000 -0500"channel="I62.2.35795.zap2it.com"><titlelang="en">Ask This Old House</title><desclang="en">Answers to home repair questions.</desc><date>20200222</date><categorylang="en">Home improvement</category><categorylang="en">House/garden</category><categorylang="en">How-to</category><categorylang="en">Series</category><lengthunits="minutes">30</length><iconsrc="https://zap2it.tmsimg.com/assets/p186265_b_v9_ad.jpg"/><url>https://tvlistings.zap2it.com//overview.html?programSeriesId=SH00529109&tmsId=EP005291090468</url><episode-numsystem="common">S18E16</episode-num><episode-numsystem="dd_progid">EP00529109.0468</episode-num><episode-numsystem="xmltv_ns">17.15.</episode-num><previously-shownstart="20200222000000"/><subtitlestype="teletext"/><rating><value>TV-G</value></rating></programme>
4) I reloaded the xmltv file without the -D and now WMC reschedule ALL of the repeats. The only difference in the xmltv file between the two for this show, other then a few extra category entries is the D version has a <date> (20200222). Both versions have the previously-shown tag, but the non-D version doesn't have a date value. I did check the D version of the xmltv file, and while most of the previously-shown tags have what looks to be the correct dates, there are a number with no dates value (I'm guessing that zapit knows they are not new, but doesn't know the date).</date>
5) I hand edit the non D version, changing the uid of the Program id of one show (Ask This Old House) that had gothen reschuled five times. I took the uids from the xmltv-to-mxf output. There was two different episodes, each with a different uid. After a reset, I then load this version, and ALL five episodes are not longer schuduled. I did this again on another show with repeats, but this time, I made the uid the same, but used the EPGC value. It didn't remove all the shows, but now there is only one scheduled for each episodes. So, it sure looks like WMC uses the program uid to help it identify programs that are the same with in a mxf. I did a third show, like Ask Thid Old House, I used the same uids that xmltv-to-mxf used, but this time, it still scheduled shows, but only one per episode. I'm not so sure why it didn't remove all episodes for this show, but at least it only schedule one.
Summary, it sure looks like WMC uses the program uid to help it identify episodes that are the same. It also looks like WMC stores the program uid to help it identify shows that have been recorded in the past. It possible that the format that xmltv-to-mxf uses is the same as Microsoft was using. Since zapit keeps track and has an identifier for each show, I think it make sense to use that value.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you still have them, could you post the 2 different xmltv files you created back-to-back with and without the -D option? I'd like to do some comparisons. Thanks.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yours also show the same weirdness I've seen regarding various sports broadcasts. In the Zap2it guide date most, but not all, sports events include a "Live" indicator. I'm guessing the ones which don't are perhaps on a tape delay? Regardless of the reason, they are NOT repeats. And, indeed, Zap2it doesn't indicate they are repeats - in fact, within the details/episode guide data the do show the Live/New tags (see below for examples). However, the problem is that apparently if there is no "Live" or "New" indicator in the main Zap2it grid data then zap2xml includes the previously-shown tag. And, as you can imagine, this can cause recording problems. I've run across this several times already during just the few weeks since having switched from the WMC Rovi guide. In other words, it's not an isolated or rare situation. So far I've only noticed it for certain sports broadcasts; however, I don't have enough time with it yet to know if this anomaly is happening in other situations.
If you'd like to see a couple of examples:
CBS (main broadcast channel) Sat. 2/29 12:30-1:30 eastern time "Rodeo" = no "LIVE" indicator (like you'll see for most other sports that same day from them) despite the fact it is NOT a repeat. Indeed, if you click on the title to see the details and look on the Episode Guide tab, they indicate "LIVE" there for this airing. However, for some unknown reason, they don't provide the "LIVE" indicator within the main guide listing.
CBS (main broadcast channel) Sun. 3/1 12:30-1:30 eastern time "PBR Bull Riding" = same exact thing. Similar to above, if you click on the title to see the details and look on the Episode Guide tab, they indicate "NEW" there for this airing. However, for some unknown reason, they don't provide the "NEW" indicator within the main guide listing.
I'm unsure how to deal with this situation since the issue is related to the incoming xml file and, I'm not seeing a way to handle this in the xmf conversion. For example, the current xmltv-to-mxf version would NOT treat the above as a repeat IF the xml file was created WITHOUT -D (since there'd be no date after the previously-shown tag.) However, I believe it would treat it as a repeat IF the xml file was created WITH -D (since there would be a date after the previously-shown tag.) Yet not using the -D option isn't viable because then other types of shows, which actually are repeats, wouldn't have a date after the previously-shown tag. And you can't even code the xmf converter to ignore the previously-shown tag if the date following it is the same date as the airing since, as Carl pointed outed previously, there are various PBS shows which air more than once per day. Anyone have any thoughts on how to resolve this quandary?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The Microsfot MXF guide states that the UID must be unique and when it is processing an MXF file if the UID does not exist in the WMC database it is added otherwise it is updated. This is for all data types that have a UID.
I have just confirmed that this is true for programs.I created a new WMC database with channels but no guide entries. I then edited an MXF file so that there were 4 programs all with the same UID. I set the original air date on each one and incremented it by 1 day so each program had a different original air date.
After running LoadMXF I used a program I wrote some time ago which looks at the contents of the WMC database and there was only 1 record for the program with the original air date set to the original air date of the last program entry in the MXF file.
I confirmed this result with several more runs with different settings for the 4 programs.All that ever happens is the program record in the WMC database is updated with the contents of the last program entry in the MXF file.One proviso to this is that if a field is not present in the program entry in the MXF file (eg original air date) the field in the database is unchanged.
So the script that converts xml to mxf is doing nothing by creating multiple MXF program records with the same UID. All that is happening is that the WMC database ends up with a single program record for each unique UID that contains the values from the last program entry in the MXF file for the matching UID.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
IthinkthistestjustshowsthattheWMCdatabaseonlyupdatestheoriginalairdateifgiven. Itheory, youshouldNEVERhavethesameprogramintheMXFwithdiffentoriginalairdates. Iftheyareallthesameprogram, theyshouldallhavethesameoriginalairdate.
TheMicrosoftdocssaysforprogramuid:
"A unique ID that will remain consistent between multiple versions of this document."Tome, "this document"meansthemxf, sothataprogramuidforthesameepisodeshouldbesameacrossdifferentmxfloads(ieeachdayload). Iguessyoucouldmakemulitpleuidsforthesameepisodethatoccurinthesamemxf, buthowdoyoumakesuretheystaythesameacrossdiffenentmxfloads.
Carl
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have Zap2it files I'm sure were created with -D that have previously shown tags with and without a date in the same file. Maybe without a date can indicate it's a repeat but they don't know when it was originally shown.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm using zap2it to build a simple OTA EPG data for Windows Media Center.
I'm not using the -D option.
I'm not using Lookups.
I'm trying to use WMC to find the repeats (I thought you said that WMC does a better job of repeat dectection, plus I find that Lookups take a long time to run).
The system is mostly working, but I find that none of the programs recorded using EPGC data have an original record date in WMC. I'm using WMC to find repeats, and I find that it rarely finds repeats. The problem is mostly with recording programs on my PBS channels. I have three in my area, and I find that some programs (like NOVA) can be brocasted many times over a week. In my xmltv file, I have five showing of "Mysteries of Sleep", two new (different channels) and three repeats. WMC is schedule to records all five. Looking at the TVGuide.MXF file, I see the five programs (see A.txt), with the two new programs having originalAirdate="0001-01-01T00:00:00", and the three repeats having originalAirdate="2020-02-27T00:00:00". It seems to me that programs that are new shoud have the originalAirdate set to the time of the showing, not "0001-01-01". How is WMC able to identifiy new programs vs programs where we don't know the originalAirDate?
It seems to me that it would be better if the "Series and repeats" first option should be "Use WMC repeat checking" instead of "Not used".
Is there a way to generate the MXF without having it loaded into WMC (so I can test some changes and then manual load it into WMC)?
You often say in your release notices of new Fixpacks that "See the release notes for details.", but I have looked at the release notes for Fixpack 21 and I have a hard time seeing what has been changed (for 19, 20 and 21). Could you either add a Fixpack log at the end with details of each fix, or label your notes with the Fixpack to make it easier to see what changed?
Carl
Lookups may take a long time the first time you run with it enabled but it caches the datalocally so after that only data not held locally is downloaded so it should be a lot quicker. Saying that I wouldn't bother with it for Zap2it data as it seems pretty comprehensive.
It's correct you should run with Series and repeats set to not used. If this is set to anything else EPGC messes around with the orIginal air date and it has proved less successful than using the repeat checking in WMC (which also uses the original air date). I have considered actually removing the option altogether.
The original air date of 0001-01 etc basically means not set ie new programme. EPGC merely transfers any previously-shown tag in the Zap2it file to the original air date field. It has also been observed that any original air date within 7 days of the next showing means the next showing will not be considered a repeat by WMC and will be recorded so it would make no difference if EPGC set the original air date to the date of broadcast in your PBS example. They would still be recorded and there is no way to stop it (AFAIK).
WMC does seem to hold data for a long time, possibly never deleting it. As long as you don't reset the database maybe it can detect duplicate recordingsoutside the 7 day window even though the original air date is not set. I don't know.
The command line to manually import an MXF file into WMC is
LoadMXF -v -i "mxf file name including full path"
LoadMXF is in the Windows/ehome directory. Run it from there.
I'll endeavour to make the fixpack notes more comprehensive. TBH it's only lately since you NA guys have popped up that there has been a rash of fixes and changes. EPGC has been around a long time and for most users it just worked with the occasional one asking for a new feature.
Steve,
Hi Carl,
The reindex job doesn't affect the schedules, Did you mean the update PVR schedule task?
If the previously-shown tag is in the Zap2it file is present you will see it in the original air date field in WMC as long as the Series and repeats is set to Not used. Other settings for that option do mess around with the original air date.
In the MXF document which, being kind here, is pretty vague on most counts the original air date is set on the 3 program but in no case is it set to the schedule date. Also In the description of the field it does say 'Use this value to determine whether this program is a repeat'.
It's probable that your first lookup run timed out after the default of 60 mins (nowhere near long enough for 6000 programs) so not all the programs collected were processed. That's why the next run also took a while. Once all the programmes in the collection have been processed it is a lot quicker. Sying that I would still not bother with it. The original air date on which you are focussing is likely to be more accurate from Zap2it than that entered on TMDB or TVDB.And if a new episode of a program is broadcast and then repeated a day or so later it probably won't even have made it to TVDB before you trty and look it up. What's the problem with using the -D option? The data from Zap2it looks pretty good to me with that option.
I could certainly provide an option to just create the MXF file and not import it. But if you manually imported a modified MXF file after an EPG run it would overwrite fields like the original air date anyway.
i'll do some more testing on this topic on my system and let you know the results. One thing I was thinking of trying was only creating one MXF program record for each unique program. At the moment EPGC creates an MXF program record for each program in th Zap2it file but the MXF format allows for a single program to be linked to any number of schedule records. it would be interesting to see the effect the original air date would have in this scenario.
I do know multiple recordings have been an issue in WMC for a long time. There have been many discussions on the Green Button site about it and someone even produced a utility to delete redundant recordings..
Steve,
OK, I didn't reliese that it didn't finish processing and timeout, so not a good test of the overhead.
I'm trying to not use the -D option since that will cause another hit on the Zapit website for every show in the xmltv file. I'm trying to keep my impact on their website as small as posible. It looks to me that most of the time, zap2it sets the original air date for most shows that I care about (repeasts). If we can fix the repeat scheduling issure, it looks like I can live without the originalAirdate being set.
Carl
I've done some further testing on this topic. I realised that the EPGC Series and repeats option set to 'Use programme title to generate links' does pretty much what you talk about.
It generate one program entry in the MXF file for each program with the same title and, if they are present, the same season and episode. If season and episode is not avaialbe then same description. It also sets the original air date on every program entry.
This seems to work as expected when you record a series 'New' and the programs differ by season/episode or description,
Where it doesn't work is when you try and record a series where all the descriptions are the same. So if I try to record The News at 6 which has the same description every day I would have to specify new and rerun otherwise it only records the first day. It then also unfortunately records News at 6 on every channel it is on (we have 4 TVNZ One channels here all showing the same programs with regionalized advertising) so I end up with multiple copies of the recording for each day.
Steve,
Carl,
I'm looking into this as well and also have some recent experience with xmltv-to-mxf. If you could post the related zap2xml lines from which the mxf above was generated, I'll see if I can offer any additional insight.
Last edit: Panda Moanium 2020-02-26
Looking at the part of the MXF file you posted it looks wrong to me because the uid field is meant to be unique for each program but is the same for each of those programs..
The isRepeat flag won't do anything. The isRepeat flag in the definition of the ScheduleEntry class in the WMC code is read only. It can't be set from the MXF file. In fact the reading of this property is where the check for 7 days is made which I mentioned before.
Do you get the original air date in the Zap2it file without the -d option? Thinking about it some more Lookups definitely won't give you the result you want. Updating of the TVDB database isn't immediate enough so you would probably not find new episodes on there until after they have been broadcast..
"Do you get the original air date in the Zap2it file without the -d option?" That's also what I wanted to see... although it's actually the -D (upper case) option.
P.S. - 100% agree about TVDB not being a viable method. I think there may be a lot of different volunteers helping to maintain various shows on there, which means sometimes on vacation, sick, etc. I say this because at various times over the years I've seen outdated info for shows last for weeks or even months! And in any case, for the many years when Microsoft used to source their data from Zap2it I never had a problem with repeats being accidentally recorded. So, there must surely be a way to get this sorted out. Whether it can be done without having to resort to the -D option is (I think) a big part of the question.
Last edit: Panda Moanium 2020-02-27
Steve,
I didn't noticed that the uid value was the same for all of the "Mysteries of Sleep" shows, but I just check a newer verion and it the same way. He using the dd_progid value from the xmltv file (with underscore convert to '.') for his uid, so maybe its not wrong.
I just check, and zap2it used the same dd_progid for "Mysteries of Sleep" across two different days (pulls), so maybe zap2it is doing that so that shows can have the same uid over time.
The isRepeat flag might be for the Guide only, but about 90% of the shows in my OTA pulls are repeats.
Carl
Here the xmltv.xml file generated by zap2it and the TVGuide.mxf output file that the java xmltv-to-mxf program generated.
I had 24 hours of WMC usage of the xmltv-to-mxf output and found as far as the scheduling of new programs and the not scheduling of repeats, it seems to do it right thing 100% of the time. The xmltv-to-mxf still has problems, for one, it seems to drop almost all News programs as not being in a series (60 Minutes, 48 Hours, etc). It also doesn't generate the categories tags, but I think they are only being used by the Guide and I didn't noticed them missing until pointed out by someone else. I also miss the sesion and episode information that EPGC puts into the show descption. From what I can tell, the xmltv-to-mxf version also doesn't schedule repeats for new shows outside the 7 day window as well. Since they are in the mxf output file, it looks to me that WMC is correctly handling them.
zap2it will give the originalAirdate correctly for previously-shows that happen within the timeframe of the zap2it pull. In my case, I have been only pulling 7 days at a time. Repeats of new programs that are outside that window shows the previously-shown tag, but without a date. I have been giving the "Mysteries of Sleep" show as an example since the first show and repeats all occur within the 7 day window, and I would have expected that EPGC or WMC would have handle it correctly.
It looks to me that the originalAirdate as being the solution to repeat scheduling is not the main fix now that the xmltv-to-mxf version shows that a mxf can be generated from the zap2it simple download and not have all repeats being scheduled. All the programs that were recorded using Microsoft data have oiginalAirdates, but none of the new programs record last night or tonight (or with EPGC data) have that set either. The one repeat program that I ask to have record tonight does show the original air date (1/1/1970). It looking like the originalAirdate is only a minor factor, one I could live without.
Carl
Based on the above, it sounds like the issue will be that if a show was previously shown further back than the current window of Zap2it data then it will NOT be recognized as a repeat and therefore be recorded even if the series settings are for new episodes only. The question remains if the -D option is used if that will provide the previously shown date even if it's further back? And, if so, is that the only way to avoid recording older repeats? It definitely would be preferable to avoid having to use -D if at all possible since that does apparently create a substantial extra load against the Zap2it servers... which could potentially lead to them taking measures to kill the ability to d/l data from them.
The problem is that even when the previously has a date (within window), the EPGC version still schedule it.
I download using the -D option, and the previously shown date (and originalAirdate) is given for all programs (plus crew, credits, and extra categories ), include ones that outside the current window. I'm 98% sure that I loaded that version and I still had the repeat problem. I have also used the Lookup tag, and I'm sure that I still had the repeat problem, which is one of the reasons that I stop using it. I agree that the prefer way would be to not using the -D do to the substantial extra load. The fact that the java version works without the -D indicates to me that it can be done, we just have to figure out how.
Carl
The Zap2it file does contain previously-shown dates way back. I've seen some as old as 1995. There are also a lot that are blank. But that shouldn't matter. WMC keeps the original program record for every recording so should check these old programs when scheduling new recordings. After all a previously-shown date set for a program doesn't mean you have a recording of it.
The date 1/1/1970 you saw in a recording is similar to 1/1/0001. It's a default date not a real one.
Do you set up a new WMC database each time you try something different? If you don't the results can't be relied on.
Another option is not to worry about duplicates being recorded and use a utility to delete the duplicates.
Available via The Green Button. Just Google WMC utility to delete duplicate recordings.
But if you setup a series record for the first time and set it to only record new shows (because, for example, you watched the prior seasons in some other way outside of WMC) then there will be no record of prior recordings and it would apparently therefore create recordings you don't need/want. Of course deleting them afterwards is possible but not ideal for various reasons... needlessly wasting electricity to wakeup the PC to record something you don't want, shows such as Carl's example which have multiple repeats (indeed NUMEROUS repeats for those of us with access to several PBS stations) could result in dozens of unwanted recordings thus becoming a problem if you don't have a lot of spare disk capacity, the unnecessary extra wear on drives recording airings you don't want, etc.
I still think there must be a way to avoid recording previously aired showings since I genuinely don't remember WMC mistakenly recording them even when I setup a new series record and there were subsequent repeats aired for which there was no prior recording of in its database. If it was previously able to do this properly there must be a way, right?
BTW, Carl, you said "I download using the -D option". Can I just confirm that was the case for what you supplied in your zip file? I ask because I do recall seeing an entry with the previously shown tag but no date after it. And for such a situation, I don't think even xmltv-to-mxf will work properly... it's my understanding that it ignores the previously shown tag if it's not accompanied by a date = per the current version release notes: "Repeats only marked if previously-shown has a date"
1) The Series.zip file included the xmltv file that was build without the -D option.
2) I rebuild my WMC database, loading the xmltv-to-mxf version, but before I did, I strip out the isRepeat flag from the ScheduleEntry. I can't see any difference in either the Guide or what was schedule, so I think that proves Steve correct in that the isRepeat flag has no effect.
3) I download the xmltv file with both with and without the -D option (one after the other to make them the same as possible, non D version second since it so fast). I rebuild WMC and loaded the -D version. I found that this version did NOT schedule the repeat shows. It seems to do the right thing as best as I could tell. I did find a show (SEAL Team) in the TVGuide.mxf output file that had both the isSeries and isMovie flags set to true, but the xmlt file only had the isSeries set.
Here is the entry for "Ask This Old House", a series that I record that PBS currently is only showing repeats.
4) I reloaded the xmltv file without the -D and now WMC reschedule ALL of the repeats. The only difference in the xmltv file between the two for this show, other then a few extra category entries is the D version has a <date> (20200222). Both versions have the previously-shown tag, but the non-D version doesn't have a date value. I did check the D version of the xmltv file, and while most of the previously-shown tags have what looks to be the correct dates, there are a number with no dates value (I'm guessing that zapit knows they are not new, but doesn't know the date).</date>
5) I hand edit the non D version, changing the uid of the Program id of one show (Ask This Old House) that had gothen reschuled five times. I took the uids from the xmltv-to-mxf output. There was two different episodes, each with a different uid. After a reset, I then load this version, and ALL five episodes are not longer schuduled. I did this again on another show with repeats, but this time, I made the uid the same, but used the EPGC value. It didn't remove all the shows, but now there is only one scheduled for each episodes. So, it sure looks like WMC uses the program uid to help it identify programs that are the same with in a mxf. I did a third show, like Ask Thid Old House, I used the same uids that xmltv-to-mxf used, but this time, it still scheduled shows, but only one per episode. I'm not so sure why it didn't remove all episodes for this show, but at least it only schedule one.
Summary, it sure looks like WMC uses the program uid to help it identify episodes that are the same. It also looks like WMC stores the program uid to help it identify shows that have been recorded in the past. It possible that the format that xmltv-to-mxf uses is the same as Microsoft was using. Since zapit keeps track and has an identifier for each show, I think it make sense to use that value.
Carl
Carl,
If you still have them, could you post the 2 different xmltv files you created back-to-back with and without the -D option? I'd like to do some comparisons. Thanks.
Here they are.
Yours also show the same weirdness I've seen regarding various sports broadcasts. In the Zap2it guide date most, but not all, sports events include a "Live" indicator. I'm guessing the ones which don't are perhaps on a tape delay? Regardless of the reason, they are NOT repeats. And, indeed, Zap2it doesn't indicate they are repeats - in fact, within the details/episode guide data the do show the Live/New tags (see below for examples). However, the problem is that apparently if there is no "Live" or "New" indicator in the main Zap2it grid data then zap2xml includes the previously-shown tag. And, as you can imagine, this can cause recording problems. I've run across this several times already during just the few weeks since having switched from the WMC Rovi guide. In other words, it's not an isolated or rare situation. So far I've only noticed it for certain sports broadcasts; however, I don't have enough time with it yet to know if this anomaly is happening in other situations.
If you'd like to see a couple of examples:
CBS (main broadcast channel) Sat. 2/29 12:30-1:30 eastern time "Rodeo" = no "LIVE" indicator (like you'll see for most other sports that same day from them) despite the fact it is NOT a repeat. Indeed, if you click on the title to see the details and look on the Episode Guide tab, they indicate "LIVE" there for this airing. However, for some unknown reason, they don't provide the "LIVE" indicator within the main guide listing.
CBS (main broadcast channel) Sun. 3/1 12:30-1:30 eastern time "PBR Bull Riding" = same exact thing. Similar to above, if you click on the title to see the details and look on the Episode Guide tab, they indicate "NEW" there for this airing. However, for some unknown reason, they don't provide the "NEW" indicator within the main guide listing.
I'm unsure how to deal with this situation since the issue is related to the incoming xml file and, I'm not seeing a way to handle this in the xmf conversion. For example, the current xmltv-to-mxf version would NOT treat the above as a repeat IF the xml file was created WITHOUT -D (since there'd be no date after the previously-shown tag.) However, I believe it would treat it as a repeat IF the xml file was created WITH -D (since there would be a date after the previously-shown tag.) Yet not using the -D option isn't viable because then other types of shows, which actually are repeats, wouldn't have a date after the previously-shown tag. And you can't even code the xmf converter to ignore the previously-shown tag if the date following it is the same date as the airing since, as Carl pointed outed previously, there are various PBS shows which air more than once per day. Anyone have any thoughts on how to resolve this quandary?
The Microsfot MXF guide states that the UID must be unique and when it is processing an MXF file if the UID does not exist in the WMC database it is added otherwise it is updated. This is for all data types that have a UID.
I have just confirmed that this is true for programs.I created a new WMC database with channels but no guide entries. I then edited an MXF file so that there were 4 programs all with the same UID. I set the original air date on each one and incremented it by 1 day so each program had a different original air date.
After running LoadMXF I used a program I wrote some time ago which looks at the contents of the WMC database and there was only 1 record for the program with the original air date set to the original air date of the last program entry in the MXF file.
I confirmed this result with several more runs with different settings for the 4 programs.All that ever happens is the program record in the WMC database is updated with the contents of the last program entry in the MXF file.One proviso to this is that if a field is not present in the program entry in the MXF file (eg original air date) the field in the database is unchanged.
So the script that converts xml to mxf is doing nothing by creating multiple MXF program records with the same UID. All that is happening is that the WMC database ends up with a single program record for each unique UID that contains the values from the last program entry in the MXF file for the matching UID.
Steve,
Also with every Zap2it file I have from different users every program either has a new tag or a previously shown tag (with or without a date).
I have Zap2it files I'm sure were created with -D that have previously shown tags with and without a date in the same file. Maybe without a date can indicate it's a repeat but they don't know when it was originally shown.