As others have mentioned with the increasing occasions the Microsoft EPG is offline I am now using EPG Collector more and more often. I currently have EPG Collector 4.3 FP7 installed in Windows 7 64bit.
EPG Collector does run successfully but many channels have big gaps in the data and running it multiple times over consecutive days does not always help. A real TV with its own tuner and built-in ability to view the same EPG data from DVB-T and DVB-T2 does not have the same gaps. I therefore feel confident this issue is something in my PC setup.
One possibility is that I am using a SiliconDust HDHomeRun HDHR4-2DT network tuner. I have already seen that similar software like DVBGuide 0.96beta and EPGScan 1.0.3.3 are completely unable to use the HDHomeRun to scan for EPG data at all. EPG Collector can but perhaps based on this problem only partially. I am currently running HDHomeRun software/firmware version 20161117/12302928 which appears to be the latest for my tuner and Windows 7.
Note: Whilst I currently have Windows Media Center 7 disabled from using its own builtin 'inband' DVB scanner which would also scan for DVB EPG data my recollection from the past is that it could get a full set of EPG data this way with no gaps and this of course is via the exact same tuner. For the benefit of others the Microsoft inband scanner cannot cope with the Huffman compressed data used on Freeview HD channels and all Freesat channels - unlike EPG Collector which can.
I have tried multiple frequencies in EPG Collector - usually 506000 but also 490000 both show 100% signal but both resulted in the same gaps, and I have made sure it is using 8MHz bands which is what the UK uses compared to Austrlia which I believe uses 7MHz bands. I am on the Crystal Palace transmitter for London if that helps.
I don't believe this is a setting issue as stated above it might be down to compatibility with the HDHomeRun. What sort of test - preferably for a single channel, could I run to give you some evidence to check? Perhaps capturing the raw transport stream?
On a related topic having just read some other posts. If like me you want to get EPG data for Freeview HD channels which are broadcast over DVB-T2 and not DVB-T then you do need to run EPG Collector also on a DVB-T2 tuner. The channel scan in EPG Collector needs to see these channels to include them in the list fo sync data for which it can only do if it has a DVB-T2 tuner. So even if possibly the guide data itself is included in DVB-T transmissions you do still need to use a DVB-T2 tuner for EPG Collector to find and include the HD channels in the list.
Last edit: John Lockwood 2017-02-17
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Which Mux do you collect from and what time do you schedule it? I collect from just BBCA and did get gaps from midnight on several channels at 02:30 but not seen any gaps since I moved the time to 05:30. The same EPG data is broadcast on all Mux but native channel data is broacast at a higher rate. Have you tried collecting from several or all Mux which some software does. I also collect Freesat data as well but tend to use mainly the Freeview EPG data.
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As far as I can see rather than specifying a Mux you specify a frequency which typically therefore corresponds to a Mux. As mentioned I normally use the frequency of 506000 but also tried some others including 490000. You can see a list of the frequencies for each Mux on my Crystal Palace transmitter here https://ukfree.tv/transmitters/tv/Crystal_Palace
The C23 Mux aka BBCA = 490.0MHz = 490000
The C25 Mux aka SDN = 506.0MHz = 506000
I could add multiple frequencies i.e. muxes but all the instructions I have seen suggest that normally you only need to add one.
I had tried multiple runs during a single day and not seen the gap I was particular worried about when testing filled. Fortunately I was around to watch that program live that fell in that gap.
I will however as an experiment try adding multiple frequencies to cover additional muxes, especially the muxes that include the problem channels.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ok, added multiple frequencies/muxes now got about five or six listed. Still plenty of gaps, other than adding every single mux not more that I could do with this approach. As all the instructions for all three of these tools i.e. DVBGuide, EPGScan and EPG Collector suggest just using a single frequency is sufficient this does not look like the solution.
What I might do next is reduce it down to a single frequency again but increase the timeout value for EIT processing from the default of just five minutes. It does seem on every run to timeout and it is not clear why. It could be simply too much data to process in just that time, or it could be getting stuck due to compatibility issue as previously alluded to.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As you say only one frequency is needed but sometimes adding multiple can fill in the blanks. Some software needs the time out extending as the EPG now takes a bit longer to broadcast than say 10 years ago as there are so many channels but not 5 minutes!
Do the blanks in the EPG match what is reported in the log? Are the blanks always on the same channels?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Channel 11 i.e. Pick is one that has been proving particularly bad but ITV1 London, Channel5 and lots of others also have problems.
When I get home I will add every single mux and see how well it does. Having doubled the timeout from the default 5 minutes to 10 minutes with no better results I am going to change that back since it seems to always take the maximum amount of time before giving up and this timeout applies to each frequency added so with six plus frequencies it would take well over an hour!
I would have to more carefully look at the log but no I don't think the gaps match completely, that is there are more actual gaps than the log indicates. The last run only logged seven gaps but there was way more than that. Just looked at gaps in the log specfically, according to the log Pick, ITV1, Channel5 had no gaps but actually they were some of the worst affected channels!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If your collection finishes with a timeout you will have gaps.
EPGC decides it has finished collecting by not receiving any new data in the number of attempts defined by the 'Number of retries' field on the Advanced tab. If this is the case it finishes with a zero completion code.
If the 'Data Collection' time expires before the above condition has occurred you get a timeout completion code.
It is possible with the EIT protocol to determine that all data has been collected. However I have seen some examples where the protocol was corrupt so the collections timed out so that is why EPGC uses the 'has any new data arrived' method instead.
Apart from the Data Collection timeout another couple of parameters you can try changing are the Buffer Size and Buffer Refills parameters also on the Advanced tab. Try changing the Buffer Size parameter first to say 250MB and leave the Buffer Refills alone.
You only need a single frequency. Don't bother adding multiple frequencies to the collection.
I'm not sure your comments about the DVB-T2 tuner are correct. All my testing with UK Freeview data in the past suggest that the HD channels EPG data can simply be collected along wih the non-HD channels. The HD channels are visible to EPGC on any frequency not just the T2 frequencies and the EPG data itself is the same on all frequencies or at least it used to be.
If you want me to check out the T2 issues give me a transport stream dump of a non-T2 frequency and I'll take a look. 30 secs should be enough.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have re-run EPG Collector with just a single frequency, increased the buffer size and confirmed it resulted in lots of gaps in the EPG data. Here are the log results http://pastebin.com/CFc2GDgG
Let me know what you want me to try next.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes I had already doubled the timeout from 300 to 600 i.e. 10 minutes and that had not made a notieable difference, I have since even before seeing your last post quadrupled it to 1200 seconds i.e. 20 minutes. This does appear to have improved matters but I still found a few gaps during the week ahead.
This suggests it is not a processing problem but perhaps it is taking that long to listen multiple times to find more and more EPG data, perhaps the signal quality is not as good as the 100% figure implies. (The EPG data is retransmitted in a loop so listening longer gives you more chances to pickup the data.)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have not been looking at this for a few days as Microsoft have (temporarily) revived their EPG. However I will do as requested when I get a chance. More recent runs suggest that it might be skipping the same programs on multiple days - reducing the chance it is reception problems. I have therefore been wondering if something in the data itself is upsetting EPG Collector although my own brief skim of the log did not show anything obvious.
Note: It does seem to particularly affect some channels but if one scrolls through the resulting seven days one can see gaps in most channels especially towards the end of the seven day period. Would the resulting MXF file be of any use?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
As others have mentioned with the increasing occasions the Microsoft EPG is offline I am now using EPG Collector more and more often. I currently have EPG Collector 4.3 FP7 installed in Windows 7 64bit.
EPG Collector does run successfully but many channels have big gaps in the data and running it multiple times over consecutive days does not always help. A real TV with its own tuner and built-in ability to view the same EPG data from DVB-T and DVB-T2 does not have the same gaps. I therefore feel confident this issue is something in my PC setup.
One possibility is that I am using a SiliconDust HDHomeRun HDHR4-2DT network tuner. I have already seen that similar software like DVBGuide 0.96beta and EPGScan 1.0.3.3 are completely unable to use the HDHomeRun to scan for EPG data at all. EPG Collector can but perhaps based on this problem only partially. I am currently running HDHomeRun software/firmware version 20161117/12302928 which appears to be the latest for my tuner and Windows 7.
Note: Whilst I currently have Windows Media Center 7 disabled from using its own builtin 'inband' DVB scanner which would also scan for DVB EPG data my recollection from the past is that it could get a full set of EPG data this way with no gaps and this of course is via the exact same tuner. For the benefit of others the Microsoft inband scanner cannot cope with the Huffman compressed data used on Freeview HD channels and all Freesat channels - unlike EPG Collector which can.
I have tried multiple frequencies in EPG Collector - usually 506000 but also 490000 both show 100% signal but both resulted in the same gaps, and I have made sure it is using 8MHz bands which is what the UK uses compared to Austrlia which I believe uses 7MHz bands. I am on the Crystal Palace transmitter for London if that helps.
I don't believe this is a setting issue as stated above it might be down to compatibility with the HDHomeRun. What sort of test - preferably for a single channel, could I run to give you some evidence to check? Perhaps capturing the raw transport stream?
On a related topic having just read some other posts. If like me you want to get EPG data for Freeview HD channels which are broadcast over DVB-T2 and not DVB-T then you do need to run EPG Collector also on a DVB-T2 tuner. The channel scan in EPG Collector needs to see these channels to include them in the list fo sync data for which it can only do if it has a DVB-T2 tuner. So even if possibly the guide data itself is included in DVB-T transmissions you do still need to use a DVB-T2 tuner for EPG Collector to find and include the HD channels in the list.
Last edit: John Lockwood 2017-02-17
Which Mux do you collect from and what time do you schedule it? I collect from just BBCA and did get gaps from midnight on several channels at 02:30 but not seen any gaps since I moved the time to 05:30. The same EPG data is broadcast on all Mux but native channel data is broacast at a higher rate. Have you tried collecting from several or all Mux which some software does. I also collect Freesat data as well but tend to use mainly the Freeview EPG data.
Chris
As far as I can see rather than specifying a Mux you specify a frequency which typically therefore corresponds to a Mux. As mentioned I normally use the frequency of 506000 but also tried some others including 490000. You can see a list of the frequencies for each Mux on my Crystal Palace transmitter here https://ukfree.tv/transmitters/tv/Crystal_Palace
The C23 Mux aka BBCA = 490.0MHz = 490000
The C25 Mux aka SDN = 506.0MHz = 506000
I could add multiple frequencies i.e. muxes but all the instructions I have seen suggest that normally you only need to add one.
I had tried multiple runs during a single day and not seen the gap I was particular worried about when testing filled. Fortunately I was around to watch that program live that fell in that gap.
I will however as an experiment try adding multiple frequencies to cover additional muxes, especially the muxes that include the problem channels.
Ok, added multiple frequencies/muxes now got about five or six listed. Still plenty of gaps, other than adding every single mux not more that I could do with this approach. As all the instructions for all three of these tools i.e. DVBGuide, EPGScan and EPG Collector suggest just using a single frequency is sufficient this does not look like the solution.
What I might do next is reduce it down to a single frequency again but increase the timeout value for EIT processing from the default of just five minutes. It does seem on every run to timeout and it is not clear why. It could be simply too much data to process in just that time, or it could be getting stuck due to compatibility issue as previously alluded to.
As you say only one frequency is needed but sometimes adding multiple can fill in the blanks. Some software needs the time out extending as the EPG now takes a bit longer to broadcast than say 10 years ago as there are so many channels but not 5 minutes!
Do the blanks in the EPG match what is reported in the log? Are the blanks always on the same channels?
Channel 11 i.e. Pick is one that has been proving particularly bad but ITV1 London, Channel5 and lots of others also have problems.
When I get home I will add every single mux and see how well it does. Having doubled the timeout from the default 5 minutes to 10 minutes with no better results I am going to change that back since it seems to always take the maximum amount of time before giving up and this timeout applies to each frequency added so with six plus frequencies it would take well over an hour!
I would have to more carefully look at the log but no I don't think the gaps match completely, that is there are more actual gaps than the log indicates. The last run only logged seven gaps but there was way more than that. Just looked at gaps in the log specfically, according to the log Pick, ITV1, Channel5 had no gaps but actually they were some of the worst affected channels!
If your collection finishes with a timeout you will have gaps.
EPGC decides it has finished collecting by not receiving any new data in the number of attempts defined by the 'Number of retries' field on the Advanced tab. If this is the case it finishes with a zero completion code.
If the 'Data Collection' time expires before the above condition has occurred you get a timeout completion code.
It is possible with the EIT protocol to determine that all data has been collected. However I have seen some examples where the protocol was corrupt so the collections timed out so that is why EPGC uses the 'has any new data arrived' method instead.
Apart from the Data Collection timeout another couple of parameters you can try changing are the Buffer Size and Buffer Refills parameters also on the Advanced tab. Try changing the Buffer Size parameter first to say 250MB and leave the Buffer Refills alone.
You only need a single frequency. Don't bother adding multiple frequencies to the collection.
I'm not sure your comments about the DVB-T2 tuner are correct. All my testing with UK Freeview data in the past suggest that the HD channels EPG data can simply be collected along wih the non-HD channels. The HD channels are visible to EPGC on any frequency not just the T2 frequencies and the EPG data itself is the same on all frequencies or at least it used to be.
If you want me to check out the T2 issues give me a transport stream dump of a non-T2 frequency and I'll take a look. 30 secs should be enough.
I have re-run EPG Collector with just a single frequency, increased the buffer size and confirmed it resulted in lots of gaps in the EPG data. Here are the log results http://pastebin.com/CFc2GDgG
Let me know what you want me to try next.
Increase the data collection timeout to 600 secs.
Yes I had already doubled the timeout from 300 to 600 i.e. 10 minutes and that had not made a notieable difference, I have since even before seeing your last post quadrupled it to 1200 seconds i.e. 20 minutes. This does appear to have improved matters but I still found a few gaps during the week ahead.
This suggests it is not a processing problem but perhaps it is taking that long to listen multiple times to find more and more EPG data, perhaps the signal quality is not as good as the 100% figure implies. (The EPG data is retransmitted in a loop so listening longer gives you more chances to pickup the data.)
Can you post a log of a 20 minute run with buffer size set to say 500mb.
There are some entries in the log that may give me a clue.
I have not been looking at this for a few days as Microsoft have (temporarily) revived their EPG. However I will do as requested when I get a chance. More recent runs suggest that it might be skipping the same programs on multiple days - reducing the chance it is reception problems. I have therefore been wondering if something in the data itself is upsetting EPG Collector although my own brief skim of the log did not show anything obvious.
Note: It does seem to particularly affect some channels but if one scrolls through the resulting seven days one can see gaps in most channels especially towards the end of the seven day period. Would the resulting MXF file be of any use?
No, I need to see the log.