I recently noticed ?? appearing several times in the EPG so looked at the TV EPG and found it was supposed to be a UK Pound sign (£). I searched the xml file and found over a 100 occurrences of ?? but only some seemed to be the Pound sign. Some of the other examples found were:-
I assumed the other problem character was (’) but in the file there are also :-
Victim’s, World’s, son’s, Mother’s, Path??e’s
This is using my Freeview ini file but using the Freesat ini the characters are removed so:
Pathé becomes Path and Fiancé become fianc and café becomes caf
The matter gets more confused after I built a Freeview test server to investigate the problem and using the same ini file Pathé now becomes PathÃ,e and £4000 becomes £4000 in the xml file.
I now remember noticing a problem with Pathé last year but the programme disappeared from the EPG before I could investigate
All three ini files have the same default settings using v4.3 update 2
Its not a major problem as it as it hasn’t stopped me recording any programmes! Most appear in the descriptions few of which I ever see as I mostly skim through the evening programme titles.
UK English is the install language but the language in the Advanced settings is Undefined as I know setting it can upset the decoding of the HD EPG data. However having seen that you suggested someone change the setting I changed it to UK but no change. However I then changed the character set to UTF8 caused an Error 5 crash and lots of odd things in the log!
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've processed both a UK Freeview and FreeSat ts file and pound signs and apostrophes both appear to be OK. There are also no ?? combinations present.
Can you use EPG Centre to do a transport stream dump of your Freeview frequency? The ones I have are over 2 years old so there may be some differences. Upload it somewhere and post the link.
Can you also attach your ini files to a post and give me an example of a programme that is in the ts dump where the title or description is wrong.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Steve. Although I have spent some time investigating the problem I will spend a little more time as I dont get consistant results . I did a clean install of EPG Collector on a test PC and created a new ini and in EPG sometimes pound sign is there and sometimes missing.
On the test PC with the clean install the pound sign seems to now display ok but in the title Pathé appears in the xml file and EPG as PathÂ. It dosnt help that in the current episode Pathe is used in the description. However when the same ini was used on my main server PC it gave different results.
I will get back to you when I have investigated more.
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You shouldn't need to select/change anything to do with languages etc on the Advanced dialog/tab for Freeview or FreeSat. The undefined or default selections should work with these data types.
Also for EIT collections there is are comments in the log showing which character sets were used for decoding the text. These character sets are specified as part of the EIT protocol. ISO-8859-1 is the default.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It looks like some of the problem is probably caused by a problem with the Windows installation on the server. When EPG Collector runs on the server with the Freeview ini file then lots of ?? appear in the xml file where various punctuation should be. If the same ini file is used on the test PC then the UK Pound sign appears correctly in the xml file and EPG and no ?? appear. However one odd thing is that on one channel which is a region option, pound signs are missing but its not important as I don’t need that EPG data.
However the original Freesat ini file still does not display the pound sign and some other punctuation in the xml file and EPG when the ini file moved to the test server. I also created a new ini file and this behaves the same. Below is link to dump of Freesat EPG you asked for earlier. Do you see any pound signs which is the easiest to identify of the various missing punctuation.
After more investigation the inconsistency I was seeing with the Freeview setup is becoming clearer!
If I run EPG Collector from within EPG Centre the xml file has no ?? and the pound sign is OK.
If I run EPG Collector from Task Scheduler as User the xml file is full of ?? and no pound signs! This is the way it normally runs.
If I run EPG Collector from the command prompt using exactly the same command no ?? in the file.
However if I do the same on my test PC the xml contain no ?? and the pound sign displays ok each time even run with Admin rights.
This is ok as the disk in the test PC is going to replace the current server disk. Both are copies of the original install of Windows 8 last year so Windows on the new disk is already Activated.
I have made no progress with the Freesat problem!
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There seems to be a fault in the decoding of the FreeSat text. This is Huffman compressed but allows for special escape sequences for character like e with an accent and it's the escape sequences that I don't think are being processed correctly.
I'm still working on a fix.
Is it possible the faults you are seeing with the Freeview data is only in the HD EPG entries? They use Huffman compression as opposed to the remainder of the channels that use uncompressed text.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Thanks Steve. Yesterday I started using the hard disk from my test PC in my main server and using the Freeview ini files from the original disk EPG Collector now correctly decodes the pound sign and other punctuation. I don’t know what the cause of my Freeview EPG decoding problem was but it has to be caused by the particular install of Windows. It was definitely not related to just HD channels. All I know is that when SFC was run on the original disk that it failed!
I would probably not have noticed the Freesat decode problem had I not been searching the Freesat xml file to find the pound sign and other punctuation which were a problem in the Freeview xml file.
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I agree the Freeview problems seemed to be specific to that rig. If you have time could you get me a ts dump of your Freeview frequency? The ts files I have for UK Freeview are quite old.
I'll issue a fixpack for the FreeSat problem at the end of the month.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've fixed the decoding problem which actually affected a large amount of text. All the titles and descriptions containing special Gaellic or Welsh characters were wrong, pound signs were missing, any accented character was missing etc etc.
Incidentally there are several different spellings of 'fiancee' in the data I saw, some with an accented e and some without. It's not a decoding issue, it's what the broadcaster sent.
Thanks Steve I will install the new version and report back in a day or two.
I did notice an inconsistency between channels when I was trying to find the problem with Freeview channels. A lot of it was I think on minor USA owned channels which seemed to sometimes use different punctuation and spelling! As I said the Freesat problem was not obvious unless you were looking for it when I noticed the missing pound signs.
Thanks Steve. I can confirm that the Freesat EPG xml file now contains the pound sign which was the most noticeable problem. I will let you know if I notice any other problems.
Chris
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I recently noticed ?? appearing several times in the EPG so looked at the TV EPG and found it was supposed to be a UK Pound sign (£). I searched the xml file and found over a 100 occurrences of ?? but only some seemed to be the Pound sign. Some of the other examples found were:-
Path??e, what??s, stonemason??s, she??d, Ireland??s, Jos??e, fianc??e, caf??e, canaop??es, br??ul??ee, clients??, ??real crimes??, we??ll.
I assumed the other problem character was (’) but in the file there are also :-
Victim’s, World’s, son’s, Mother’s, Path??e’s
This is using my Freeview ini file but using the Freesat ini the characters are removed so:
Pathé becomes Path and Fiancé become fianc and café becomes caf
The matter gets more confused after I built a Freeview test server to investigate the problem and using the same ini file Pathé now becomes PathÃ,e and £4000 becomes £4000 in the xml file.
I now remember noticing a problem with Pathé last year but the programme disappeared from the EPG before I could investigate
All three ini files have the same default settings using v4.3 update 2
Its not a major problem as it as it hasn’t stopped me recording any programmes! Most appear in the descriptions few of which I ever see as I mostly skim through the evening programme titles.
UK English is the install language but the language in the Advanced settings is Undefined as I know setting it can upset the decoding of the HD EPG data. However having seen that you suggested someone change the setting I changed it to UK but no change. However I then changed the character set to UTF8 caused an Error 5 crash and lots of odd things in the log!
Chris
I've processed both a UK Freeview and FreeSat ts file and pound signs and apostrophes both appear to be OK. There are also no ?? combinations present.
Can you use EPG Centre to do a transport stream dump of your Freeview frequency? The ones I have are over 2 years old so there may be some differences. Upload it somewhere and post the link.
Can you also attach your ini files to a post and give me an example of a programme that is in the ts dump where the title or description is wrong.
Thanks Steve. Although I have spent some time investigating the problem I will spend a little more time as I dont get consistant results . I did a clean install of EPG Collector on a test PC and created a new ini and in EPG sometimes pound sign is there and sometimes missing.
On the test PC with the clean install the pound sign seems to now display ok but in the title Pathé appears in the xml file and EPG as PathÂ. It dosnt help that in the current episode Pathe is used in the description. However when the same ini was used on my main server PC it gave different results.
I will get back to you when I have investigated more.
Chris
You shouldn't need to select/change anything to do with languages etc on the Advanced dialog/tab for Freeview or FreeSat. The undefined or default selections should work with these data types.
Also for EIT collections there is are comments in the log showing which character sets were used for decoding the text. These character sets are specified as part of the EIT protocol. ISO-8859-1 is the default.
It looks like some of the problem is probably caused by a problem with the Windows installation on the server. When EPG Collector runs on the server with the Freeview ini file then lots of ?? appear in the xml file where various punctuation should be. If the same ini file is used on the test PC then the UK Pound sign appears correctly in the xml file and EPG and no ?? appear. However one odd thing is that on one channel which is a region option, pound signs are missing but its not important as I don’t need that EPG data.
However the original Freesat ini file still does not display the pound sign and some other punctuation in the xml file and EPG when the ini file moved to the test server. I also created a new ini file and this behaves the same. Below is link to dump of Freesat EPG you asked for earlier. Do you see any pound signs which is the easiest to identify of the various missing punctuation.
https://drive.google.com/file/d/0By6PKMAbJoLba0o5M2VNdDcwTTA/view?usp=sharing
Chris
After more investigation the inconsistency I was seeing with the Freeview setup is becoming clearer!
If I run EPG Collector from within EPG Centre the xml file has no ?? and the pound sign is OK.
If I run EPG Collector from Task Scheduler as User the xml file is full of ?? and no pound signs! This is the way it normally runs.
If I run EPG Collector from the command prompt using exactly the same command no ?? in the file.
However if I do the same on my test PC the xml contain no ?? and the pound sign displays ok each time even run with Admin rights.
This is ok as the disk in the test PC is going to replace the current server disk. Both are copies of the original install of Windows 8 last year so Windows on the new disk is already Activated.
I have made no progress with the Freesat problem!
Chris
There seems to be a fault in the decoding of the FreeSat text. This is Huffman compressed but allows for special escape sequences for character like e with an accent and it's the escape sequences that I don't think are being processed correctly.
I'm still working on a fix.
Is it possible the faults you are seeing with the Freeview data is only in the HD EPG entries? They use Huffman compression as opposed to the remainder of the channels that use uncompressed text.
Thanks Steve. Yesterday I started using the hard disk from my test PC in my main server and using the Freeview ini files from the original disk EPG Collector now correctly decodes the pound sign and other punctuation. I don’t know what the cause of my Freeview EPG decoding problem was but it has to be caused by the particular install of Windows. It was definitely not related to just HD channels. All I know is that when SFC was run on the original disk that it failed!
I would probably not have noticed the Freesat decode problem had I not been searching the Freesat xml file to find the pound sign and other punctuation which were a problem in the Freeview xml file.
Chris
I agree the Freeview problems seemed to be specific to that rig. If you have time could you get me a ts dump of your Freeview frequency? The ts files I have for UK Freeview are quite old.
I'll issue a fixpack for the FreeSat problem at the end of the month.
Could you give the attached a try with FreeSat?
I've fixed the decoding problem which actually affected a large amount of text. All the titles and descriptions containing special Gaellic or Welsh characters were wrong, pound signs were missing, any accented character was missing etc etc.
Incidentally there are several different spellings of 'fiancee' in the data I saw, some with an accented e and some without. It's not a decoding issue, it's what the broadcaster sent.
Thanks Steve I will install the new version and report back in a day or two.
I did notice an inconsistency between channels when I was trying to find the problem with Freeview channels. A lot of it was I think on minor USA owned channels which seemed to sometimes use different punctuation and spelling! As I said the Freesat problem was not obvious unless you were looking for it when I noticed the missing pound signs.
Enclosed are two Freeview 60 second dumps from mux with different owners in case there is any difference.
https://drive.google.com/file/d/0By6PKMAbJoLbblhXekktXzdiNGs/view?usp=sharing
https://drive.google.com/file/d/0By6PKMAbJoLbMzBLNGtobHlBaXc/view?usp=sharing
Chris
Thanks Steve. I can confirm that the Freesat EPG xml file now contains the pound sign which was the most noticeable problem. I will let you know if I notice any other problems.
Chris