PyBuckets ReadMe!
=================
What the heck is PyBuckets?
Cough, well, the Enigma TV receiver boxes organise their TV channels into groups (Entertainment, Movies, Kids, etc.)
which for some reason are called bouquets. Since I'm not French and we're not discussing flowers, I'd rather organise
my channels into buckets! You may have seen e2m3u2bouquet / Autobouquets maker as a plugin utility which lets you do
exactly the same thing. I wrote PyBuckets (also in Python, hence the Py) because I had problems using e2m3u2bouquet
with IPTV service providers. I found compatibility issues between Python 2 and 3, and e2m3u2bouquet is really old and
not supported anymore, and really doesn't do everything I want it to do. If you do use an IPTV provider, it may come
with 20,000 channels from 200 countries and in many different languages. It may also come with VOD / On-Demand
channels like movies and TV shows, and these can really clutter up the channel list. Furthermore, very often it will
also come with hundreds of channels you never ever want to watch and don't want to see in the channel list. For
example, religion, sport, adult, etc. I wanted to build a processor for IPTV channels which works with regular cable/
satellite/terrestrial/IPTV channels equally, and provides lots of configurability to organise all of the channels.
But why?
The real driving factor to develop PyBuckets, though, comes from the Episode Programme Guide (EPG) not working for
IPTV channels in general. For me this was always a deal-breaker, because it meant one could not use the Enigma box
for scheduled recordings. Even for general TV surfing, I feel naked if I open the EPG Grid view for all channels and
everything is empty. If I can't see what's on, it becomes impractical to watch live TV and impossible to use recorded
TV as well, which rather renders the entire platform useless. The crux of the problem is that the channel names rarely
match exactly the names of the channels in the EPG, so they are not assigned. This is the main problem I am trying to
solve with PyBuckets - channel mapping. I wanted the Channel-EPG mapping to be 99% automatic, as it is not practical
to expect anyone to manually map thousands of channels. So with some cleaning and synonym-substitution (with smoke and
mirrors), this is now possible. I'm not aware of any other utility to do this.
No, but why?
Because I fear big companies like Sky have too much of a monopoly. Their receivers are closed and proprietary. As
soon as you stop paying for Sky+, you can no longer record [even free-to-air] channels. You cannot even play back
anything you have recorded. They took this further with Sky Q, so that all of the hardware is rental only. You
would own nothing. And further again with Sky Glass - getting rid of the receiver box altogether so that the TV
is streamed over the internet. It sounds like IPTV, but it's so much worse. Sky Glass is rented. Stop paying and
you give the TV back as well. Now I'm not a luddite, I promise, but if you look back it all seems rather clever.
Video tape became extinct, replaced with digital media over which Digital Rights management (DRM) gives them more and
more control. You bought the disc, but they can revoke your license to play it at any time. Physical media is also
starting to go extinct in favour of streaming. Now TV has gone the same way, so the provider controls everything,
whilst forcing you to pay for both a TV License AND watch commercial advertising. But what annoys me most of all is
you can't get the channels you want without buying bundles/packages of channels. Then you can't even get the shows
you want all from one provider. American TV is controlled by networks, so you have to buy Sky, Netflix, Disney and
Apple TV (and probably lots more) and you still cannot watch everything you want. Some shows are just never exported
to other countries. I'm not a pirate, and I'm willing to pay, but due to legalities, licenses, monopolies, it becomes
very difficult to get you want and virtually impossible to get it all on one single usable platform. The great
crumble has started, but if WW3 happens, all of the "Web 3.0" interdependencies of services will see to it that
when part(s) of the Internet go down, companies fail, cities fall you will have no TV / Movies to watch! I think
the now elderly generation (also tech-averse) are being excluded from new TV, my generation have it but are being
milked for it and are increasingly dissatisfied, and the younger generation are just completely ignorant and are
happy to take what they get (YouTube on their phone) because they don't know much different. All quite sad.
Everything is being taken away from us, we get less and less, but pay more and more. Rant over!
Donations
This software is free-of-charge, and I don't make money from this or any similar software. But I do have a mountain
of debt to service and like most people I am struggling. I would really welcome your donations, which would help me
do more of this kind of thing, and break out of the rat race. Please send any amount of Bitcoin (BTC) to:
3DqvFKkRjN1UVnNBLo4jJjKDz8ogwThZNw
FEATURES
========
* Makes Buckets automatically based on Country + Category
* User-definable Virtual Buckets!
* Maps M3U Channels to EPG Channels automatically, by name (doesnt trust TVGID)
* Supports multiple EPGs (and multiple providers), optionally by country
* Consolildates and combines EPGs
* Multi-country, multi-lingual logic
* Manual mapping of M3U Channel to EPG Channel possible
* Fuzzy matching possible
* Full Substitutions (for mapping) are user configurable
* Partial Substitutions (for mapping) are also user configurable
* Country Whitelisting (e.g. choose to take UK, US, etc.)
* Term Blacklisting at Bucket & Channel level (e.g. exclude VOD, Arab)
* Preserve your user-defined Buckets (favourites).
* Updates the Stream URLs in [Preserve] Buckets if URLs change.
* Installs bouquet*.tv files and EPG files to Enigma
* Runs nightly
* Exports M3U + EPG as CSV files as well to aid with mapping.
* Supports TVGLogo (downloaded separately)
* Supports Picons (downloaded separately)
* Supports multiple channels (different quality levels)
* Highly configurable. Options include:
- Stream Type
- Service Type
- Buffer Mode
REQUIREMENTS
============
* Any Enigma2 box, and a TV :-)
* Python3. (Required Libs: diffLib, xml.etree.ElementTree, urllib, os, json, re, datetime)
* CPU good enough to run PyBuckets overnight (it's quite heavy, and make take hours)
* An EPG Source (EPGDirect recommended)
* An IPTV Provider (optional - it should work with Cable/Sat/Terrestrial channels only as well)
* Internet access to download M3U and EPG files
* Some basic Linux skills to install files to your box
RECOMMENDED SETUP
=================
PyBuckets has been tested on:
* VU 4K Duo SE+ box
* OpenATV / Enigma2
* EPGDirect as a provider
* An "always-on" VPN, for your protection, ideally installed on the router.
INSTRUCTIONS
============
Overview
--------
1. Upload all files to /etc/enigma2/PyBuckets
2. Execute /etc/enigma2/PyBuckets/install.sh
3. Edit the config.json to your needs (see configuration below).
CONFIGURATION
=============
Configuration files:
* config.json
A sample configuration file has been provided, set up for EPG Direct separate EPGs by country, plus a shared EPG from
your IPTV provider. You just need to edit it, which mostly means deleting stuff you don't want.
Minimum steps:
* Edit download-m3u.sh, enter the URLs, Usernames, Passwords for any IPTV providers you use
* Edit download-epg.sh, enter the URLs, Usernames, Passwords for any EPG providers you use
* Edit config.json:
- ensure m3ufile filename matches the one being downloaded [by download-m3u.sh]
- ensure epg_xml block filenames match the one(s) being downloaded [by download-epg.sh]
- specify the Bucket to EPG level mapping
- check/edit the partial replacements (delete any you really dont want to do)
- check/edit the full replacements (delete any you really dont want to do)
- check/edit the blacklist terms (delete any you really dont want to exclude)
- check/edit the country whitelist
- check/edit the stream url prefix whitelist
FUTURE IDEAS
============
* Diacritics need removing. Not sure how without a big mapping.
* All blacklisted Buckets/Channels can be dumped in a 'blacklisted' bucket, rather than discarded.
* All non-whitelisted countries could also be dumped somewhere accessible, rather than discarded.
* All VOD could also be dumped somewhere accessible, rather than discarded.
* Move installation into Python? (Crontab, etc.)
* Move download tasks into Python?
* Check for incomplete downloads. Retry. In case of failure, keep last.
* Mix Cable/Sat/Terrestrial channels into same Buckets with source suffix?
* Plugin for Enigma2 menus needed
* Test with other IPTV providers
LEGAL
=====
* PyBuckets ("It") is an Open-Source utility for data processing and mapping of lists Channels.
* Any and all data it processes is provided by the end-user.
* It does not provide or facilitate access to "illegal streams".
* It does not contain or host any media (pictures/video/sound/text, etc.)
* It is for individual personal "fun" use only, and it is expressly forbidden for any company or organisation to use this software.
* It may not be (re)sold, and is provided free-of-charge under the usual open-source license.
* It may not be included in the distribution of pre-build "illegal TV boxes".
* It may not be used for any nefarious or shady purpose!
* Its use is entirely at the end-user's own choice and risk, and comes with "NO WARRANTY".
* No guarantees can be made either regarding compatibility with particular services.
* The developer cannot be held responsible for writing some Python code and making it freely available!
* The developer does not advocate or condone the use of unlicensed copyrighted works.
* By obtaining/using PyBuckets, you agree to the above terms and conditions.
* Ñañañañañaña!
MY NOTES (Various Sources)
==========================
Service Reference Code
Example:
#SERVICE 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12
#SERVICE 1: 0: 19: C472: 837: 2: 11A0000: 0: 0: 0: : CNN HD
#SERVICE 1: 0: 1: 0: 0: 0: 0: 0: 0: 0: http%3A//myiptv.provider/$USERNAME/$PASSWORD/481718: CHANNELNAME
StreamType : Special : ServiceType : SID : TID : NID : Namespace : 0 : 0 : BufferType : URL : Name
#1 Stream Type / Source Type:
- 1 = DVB-S/T/C/IP
- 5001,5002,5003 = ServiceApp plugin and additional players such as ffmpeg + exteplayer3
- 8192 = external HDMI input
- 8193 = Youtube /8xxx services
- 4097 = Gstreamer multimedia player
#2 Special:
- 0 = Default
#3 Service Type / Type of Service:
- 1 = TV ; subtypes:
- 16h = MPEG (SD)
- 19h = MPEG / H264 (HD,FHD)
- 1Fh = H265 (UHD,4K)
- 2 = RADIO
#4 Service ID / SID
#5 Transponder ID / TID
#6 Network ID / NID
#7 Namespace:
- DVB-S: PP PP 00 00 (Satellite longitude)
- 01 3B 00 00 => 013B hex = 315 dec => 31.5 °E
- 0E 08 00 00 => xx08 hex = 008 dec => 0.8 °W
- DVB-T: EE EE MM MM
- EE EE = Destination code
- MM MM = Frequency
- DVB-C:
- FF FF = Destination code
- MM MM = Frequency
#8 Unused
#9 Unused
#10 Buffer Type:
- 0 = stream-buffer is not used
- 1 = stream-buffer in RAM memory
- 3 = stream-buffer in the file on the internal disk
#11 URL: Stream URL of IPTV source etc.
#12 Name: Display Name of the channel.
++++++++++++++++++++++++++++++
+++ SERVICE REFERENCE CODE +++
++++++++++++++++++++++++++++++
1 2 3 4 5 6 7 8 9 10
---------------------------------------
#SERVICE 1:0:1:445C:453:1:C00000:0:0:0:
---------------------------------------
order → explanation
---------------------------------------
first
→ service type (stream source) - to select the appropriate player for this service:
(1 = DVB-S/T/C/IP - DVB-Mediasink decoder; 4097 = Gstreamer multimedia player; 5001,5002,5003 = ServiceApp plugin and additional players such as ffmpeg + exteplayer3; 8192 = external HDMI input; 8193 = Youtube /8xxx services do not work in every Enigma!!!!/)
second
→ special purpose, normally there is always 0 ... but when there is an information line (so-called "marker") in the channel list, then the second position is 64 and the third position is the consecutive number of the information line, for example 1:64:consecutive_number:0:0:0:0:0:0:0: (the remaining portion of ServRefCode is filled with zeros in this case)
third
→ type of service (or type of service with its more detailed specification)
1=TV ;
2=RADIO ... in case of TV it is possible to specify a specific stream format: 16h = MPEG (SD); 19h = MPEG / H264 (HD,FHD); 1Fh = H265 (UHD,4K); etc.
- some examples:
1:0:1:... DVB-stream:0:TV
1:0:2:... DVB-stream:0:RADIO
4097:0:1:... internet-stream:0:TV
4097:0:2:... internet-stream:0:RADIO
fourth
→ Service ID (445C => "SAT.1") - the identification code of a specific channel (service)
fifth
→ Transponder ID (453) - transponder number on a particular satellite
sixth
→ Network ID (1) - I did not find out what it means
seventh
→ Namespace (00 C0 00 00) - satellite position in case of DVB-S or specific channel frequency in case of DVB-T / C
- together as a whole, this 4-byte data is called as the so-called "namespace" (in translation it could mean something like "the name of space in free space - in the orbit of the Earth"?!)
- if it is DVB-S, then the first 4 characters of the hexadecimal value represent the position of the satellite in space and the remaining 4 characters are ignored (always zeros = 0000)
- some examples for DVB-S "namespaces", to determine the angle of the world side:
....from the point of view of developers, this is a non-traditional determination of decimal places - in contrast to the use of the so-called "floating point" or data / number type "float"
....there is always only one decimal place in the final decimal value (after conversion from hexadecimal)
0E 08 00 00 => xx08 hex = 008 dec => 0.8 °W
-- NOTE:
-- if the first byte is equal to "0E", it means the opposite side of the world to determine the angle - it is therefore the angle from the West, not the angle from the East, which represents the designation "°W"
-- of course, this first byte is then not used to determine the value of the angle
01 3B 00 00 => 013B hex = 315 dec => 31.5 °E
00 82 00 00 => 0082 hex = 130 dec => 13.0 °E
00 A0 00 00 => 00A0 hex = 160 dec => 16.0 °E
00 C0 00 00 => 00C0 hex = 192 dec => 19.2 °E
00 EB 00 00 => 00EB hex = 235 dec => 23.5 °E
- in the case of DVB-T / C, then the first 4 characters of the hexadecimal value represent either the EEEE destination code of the DVB-T technology or FFFF as the destination code of the DVB-C technology ... and the remaining 4 characters may (but need not) contain the frequency of the specific DVB-T / C channel
- factual note to the hexadecimal number system:
the zeros at the beginning (on the left) of the hexadecimal numeric value are irrelevant! they are usually used only to accurately specify the maximum value (bit length) for a particular variable
eight to ten (10th)
→ 3 zeros at the end of the service reference code are currently unused by Enigma, perhaps used earlier or later. even now, as far as I know, they are using for experimental purposes (for testing) only
- the last value of the three digits (10th position in total) when using service type 4097, has the following usage in some Enigmas:
0 = stream-buffer is not used
1 = stream-buffer in RAM memory
3 = stream-buffer in the file on the internal disk
---------------------------------------
#NAME UK| Documentaries
#SERVICE 4097:0:1:a00e8:0:0:0:0:0:0:http%3A//myiptv.provider/$USERNAME/$PASSWORD/158460:
#DESCRIPTION UK: DISCOVERY CHANNEL FHD
#SERVICE 4097:0:1:a00e8:0:0:0:0:0:0:http%3A//myiptv.provider/$USERNAME/$PASSWORD/158460:
#SERVICE 1:2:3:4:5:6:7:8:9:10:http%3A//myiptv.provider/$USERNAME/$PASSWORD/158460:
4097:0:1:0:0:0:0:0:0:0:URL:NAME (no buffering)
4097:0:1:0:0:0:0:0:0:1:URL:NAME (buffering enabled)
4097:0:1:0:0:0:0:0:0:3:URL:NAME (progressive download and buffering enabled)
4097:2:1:4:5:6:7:8:9:3:URL:NAME (progressive download and buffering enabled)
10th
---------------------------------------
Another example of a service reference code from a particular Skylink provider ... on a specific satellite "Astra 23.5E" ... for a concrete channel Markiza:
#SERVICE 1:0:19:13B4:C95:3:EB0000:0:0:0:
- if we neglect the first three data, the other data means ...
- 13B4 hex / 5044 dec means ServiceID - the unique number of a particular TV channel on a transponder or on a satellite (on other transponders or satellites, the same Markiza can be assigned a totally different number)
- C95 hex / 3221 dec means TransponderID
- value 3 represents NetworkID (I do not know what does it mean)
- and finally the value of EB0000 hex represents the position of the satellite in the universe (EB hex => 235 dec => 23.5 E position)
--------------------
Example for DVB-IP (IPTV, MPEG-TS):
```
#NAME IPTV (CZ/SK/PL/DE/HU)
#SERVICE 1:0:19:13B4:C95:3:EB0000:0:0:0:http%3a//192.168.1.1%3a4444/udp/232.48.42.17%3a4000/:Markíza HD
#DESCRIPTION Markíza HD
#SERVICE 1:0:19:1329:CA2:3:EB0000:0:0:0:http%3a//192.168.1.1%3a4444/udp/232.48.42.121%3a4000/:Jednotka HD
#DESCRIPTION Jednotka HD
#SERVICE 1:0:19:132A:CA2:3:EB0000:0:0:0:http%3a//192.168.1.1%3a4444/udp/232.48.42.122%3a4000/:Dvojka HD
#DESCRIPTION Dvojka HD
```
From: https://forums.openpli.org/topic/40312-iptv-channels-service-ref/
```
4097:0:1:SID:TID:NID:Namespace:0:0:0:
```
```
4097: indicating it is an IPTV stream (or stream)
Type: 1 for TV and 2 for Radio
SID: station identifier: a separate number for each station (1 to .... a 1 to 4 digit hexadecimal number)
TID: 0 for NL and 1 for FR (each country can have its own sequence)
NID: 32 for BE, 31 for NL, 49 for Germany, 44 for UK etc..... (international telephone country code)
namespace: 0 for the moment
```
```
4097:0:1:0000:44:44:0:0:0:0:
```
From: https://forums.openpli.org/topic/28766-faqs-about-epgimport-previously-called-xmltv-import/
A working example:
In the bouquet file you have this as channel reference (here the Nostalgie radiostation in Belgium)
```
#SERVICE 4097:0:1:1000:1000:0:0:0:0:0:http%3a//stream10.fluoline.net%3a80/:Nostalgie
#DESCRIPTION Nostalgie
```
You have to make a /etc/epgimport/channels.xml file containing:
```
<?xml version="1.0" encoding="latin-1"?>
<channels>
<channel id="PrivaSpice.ero">1:0:1:1000:1000:0:0:0:0:0:http%3a//example.com</channel> <!-- Private Spice -->
</channels>
```
The part http%3a//example.com is really needed, but can be the same for all channels.
Also the 4097 has to be replaced by a 1 : Not needed anymore
I called this file test.channels.xml and placed it in etc/epgimport
Now you have to reference this channels file in the /etc/epgimport/sources.xml file:
```
<?xml version="1.0" encoding="latin-1"?>
<sources>
<source type="gen_xmltv" channels="test.channels.xml">
<description>test XMLTV</description>
<url>http://url1/rytecxmltvero.gz</url>
<url>http://url2/rytecxmltvero.gz</url>
<url>http://url3/rytecxmltvero.gz</url>
<url>http://url4/rytecxmltvero.gz</url>
<url>http://url5/rytecxmltvero.gz</url>
</source>
</sources>
```
You could edit the rytec.sources.xml by adding the above, but I do not recommend it as the file will be overwritten when you update your box. The advise is thus to make a separate something.sources.xml file. The XMLTV importer will read both this file and the rytec.sources.xml file.
PS: with something I mean really any name you can dream up.
