IPTV USA EPG working on E2

Dont know if many of you guys are on twitter but [MENTION=223292]tommycahir[/MENTION] has managed to get non UK IPTV EPG working with E2

Its all done manually at the min but he has put us on the right track now

Hi All

Recently I started messing around to see if I could get epg data working on US IPTV channels and turns out it is possible but you need to have a good source of data available, as a bonus the same process can be used for IPTV streams from any country once you have a good epg data source.

# I do not take responsibility for any lost data or damage to your system or device, This is done at your own risk! #

Points of Note
This solution means that you will manually have to update the source epg data file on a periodic basis unless you have access to a server hosted updated epg file.
This solution has been tested with epgimport only and is not tested in anyway with crossepg.
This has been tested on a base OpenATV install on a ZGEMMA H2S, I can not gaurentee it will work on all boxes or images especially if they have had modifications to how epg data is imported or stored etc.
Thanks to the guys over on Rytec support forum on openpli site who answered some queries on epgimport processing and handling. They were the key to this being possible

Prerequisites:
IPTV channels: These should be imported into your Enigma2 box and preferably(but not necessary) in a single bouquet
epgimport plugin: this is the main engine that does heavy lifting once you have configuration completed.
FTP client: a good commonly used one is filezilla but there are loads out there that will do the job.
- Filezilla basic user guide
- Youtube guide on using Filezilla
XML editor: I personally use PsPad but others recommend Notepad++
Enigma2 Box IP Address: To find your IP Address go to Menu-> Setup-> Network-> Device Setup-> Lan Setup (WLan option if that is used)-> Adapter settings
- Note that if you have DHCP set to Yes then your IP Address may change periodically, if you want a static IP address then you need to change the settings here.
Enigma2 Box userID and password: This varies from box to box and image to image so google is your friend here.
E-Channelizer: You will need some form of editor to update the Stream reference and E-Channelizer is very simple and intuitive to use, it can be downloaded from http://echannelizer.com/. This can also be used to load your IPTV to the Enigma2 box.

Optionally:
Grep: This is needed if you want to use a data source which is not already preformatted for epgimport, this is built into linux boxes and is available on windows box as part of GNUWin32 gnuwin32 site
zap2xml: Again this is needed if you want to extract epg data if a ready to use one is not available, download and install from http://zap2xml.awardspace.info/
7Zip: This can optionally be used to compress your source epg xml file to save space on your box as you will be uploading the source data to flash boxes with low flash memory could encounter problems, its available at www.7-zip.org

Getting US EPG data from zap2it.com
Download and install http://zap2xml.awardspace.info/, your AV may give a false positive if so you usually can choose to ignore the quarantine or you can decide to use the perl scripts instead.
Register your free zap2it.com TV Listings account
Input zip/postal code (10111 directtv sat seems to have the most channels)
Click Set Preferences:
Manage Favorite Channels
- Select your favorite channels from the "Available Channels" and put them in the "My Favorite Channels" list
Additional Settings
- Checkmark [✔] "Show six hour grid"
- Checkmark [✔] "Show only my favorite channels in the grid"
Click "Save"
Run zap2xml with the userEmail and password parameters of your account, this needs to be done from the command line in the same directory as the zap2xml.exe you downloaded earlier
- Example: zap2xml -u zap2xmluser@email.xx -p password
This will create a new file xmltv.xml which contains all the needed info based on your account settings from zap2it.com
There will also be a folder called "cache" which contains the zipped data files which were downloaded, if you rerun the command the cache files will be used to rebuild the xmltv.xml file.

Sanitizing and compressing the xml epg data file for use in epgimport
An enigma2 receiver only uses 4 tags for epg. The timing, title, subtitle and description tags.
Extra tags like category, episode and previously shown are not used.
Also a tag like: <previously-shown /> could create problems.
Although this format is perfectly OK in xml. The epgimporter plugin does not recognize it.
Therefore the plugin team recommends to pass the xmltv.xml file through grep (in linux a standard installed utility - for windows it is in the GNUwin32 package, see above)
- For windows GNUwin32 package users you need to run this from the installed bin file location "C:\Program Files (x86)\GnuWin32\bin" and provide full paths to the files in quotes below (below command assumes all of these are in the same folder).
grep -v -f "GREPpattern.txt" "xmltv.xml" > "OUTPUTXMLTV.XML"
I have attached their GREPpattern below.
GREPpattern.zip
(224 Bytes) Downloaded 1 time

As this file can be quiet large (I've seen <250MB size files) so should really be compressed.
You should follow the below guide to create a .gz file if you are stuck for space.
- https://linhost.info/2012/08/gzip-files-in-windows/
NOTE: if you do this step you will need to modify the file name in some other configuration files later.

Modifying your IPTV Type and Stream Reference
You will need to connect to your Enigma2 box and have the ability to edit the IPTV streams.
There are guides on the E-Channelizer website in their tutorials section on how to do this and other operations on your bouquets etc

I have found that this works best if you set your stream type to DVB
- Double click on the IPTV channel and in the resulting popup dialog set Stream Type to be DVB
- You can select Multiple IPTV Streams, right click, choose Stream Type and then choose DVB to edit more than one at once.
Once that is complete you need to update the stream reference for each individual IPTV Stream you want to assign an epg to.
This Stream reference needs to be a unique Hexadecimal Number
- It is very important that this is unique as if there is duplication of there Stream References then the same epg will be assigned to both IPTV streams.
In E-Channelizer Right-click on the stream, and select Edit. Alternatively, double-click it to edit.
In the Edit Stream dialog, enter a custom reference for example if the current stream reference is 1:0:1:0:0:0:0:0:0:0 make it 1:0:1:1:0:0:0:0:0:0 (similarly if it is showing 4097:0:1:0:0:0:0:0:0:0 make it 4097:0:1:1:0:0:0:0:0:0 )
Then the next IPTV stream reference as should be changed from 1:0:1:0:0:0:0:0:0:0 to 1:0:1:2:0:0:0:0:0:0
- I have found that if you use the last piece of the stream reference to be the unique identifier then this will not work.
- Remember this unique identifier has to be a hexadecimal number or you will get
Click OK to save changes and close.

Constructing the USIPTV.channels.xml file
In file this you define the link between the stream ref you have given the channel on your Enigma2 receiver and the value in the downloaded epg.xml file eg channel id="I16485.labs.zap2it.com"
If you have defined an IPTV stream/channel in E-Channelizer to have the Stream Reference 1:0:1:1:0:0:0:0:0:0
The definition in the channels file looks like this:
<channel id="I15952.labs.zap2it.com">1:0:1:1:0:0:0:0:0:0:http%3a//example.m3u8</channel> <!-- EPL 1 SD (NBCSN) -->
- If you have 4097 in the stream reference then you need to change this to 1in the channels file.
- <channel id="I15952.labs.zap2it.com"> comes from the epg data you downloaded earlier and refers to the IPTV stream
- http%3a//example.m3u8 could be anything but is needed to denote that this is a IPTV channel. You can have there the actual URL, but it is not needed a dummy (but syntactically correct) URL is enough. my recommendation is to keep the dummy id for all entries
- <!-- EPL 1 SD (NBCSN) --> This is just a comment to allow you understand what channel this row entry is for
Repeat this step for all the channels you want to assign an epg to, making sure you have one row per channel.
I have attached my USIPTV.channels.xml file with 2 channels mapped as an example.
USIPTV.channels.xml
(372 Bytes) Downloaded 3 times

Note: If you map the channel ID and stream reference here incorrectly you will end up with the wrong epg data assigned to the IPTV stream.

Constructing the USIPTV.sources.xml file
This file ties all the others together and is the main reference file for the epgimport plugin
All the other files can be called pretty much anything eg epgdataforus.xml.gz and magicmapping.channels.xml but they need to be correctly referenced in this file otherwise the data will not be imported and the epg assigned to the IPTV stream.
There are 3 main lines of code in this file
CODE: SELECT ALL
<source type="gen_xmltv" channels="/etc/epgimport/USIPTV.channels.xml">
<description>Test US IPTV</description>
<url>/etc/epgimport/OUTPUTXMLTV.XML</url>
</source>

channels="/etc/epgimport/USIPTV.channels.xml" should be set to the location where you will be storing the custom channels.xml created earlier I personally like /etc/epgimport/ as it is a persistent storage location but it could as easily be on /media/hdd if you like.
<description>Test US IPTV</description> This is the description you will see in the epgimport plugin sources section so again naming here is your preference.
<url>/etc/epgimport/OUTPUTXMLTV.XML</url> this is the sanitized epg data that you downloaded earlier, here I have stored it in /etc/epgimport/ but you could have on /media/hdd/ if you want or on a server out in the scary www eg <url>http://epgalfasite.dyndns.org/xmltv.xml</url>. Remember if you compressed it earlier you need to reflect that in the name also.
This file should then be saved as with whatever name you want .sources.xml, I named mine USIPTV.sources.xml as it specifically dealt with US IPTV.
You could name this as custom.sources.xml and add more section of the code above if you had custom mappings created for other regions and clean epg data.
I have attached my USIPTV.sources.xml below for reference
USIPTV.sources.xml
(239 Bytes) Downloaded 2 times

Transferring the files and running the import
Open an FTP connection to your enigma 2 box
Transfer the files you have created above to the box, the custom sources.xml MUST go in /etc/epgimport
The channels.xml file MUST go where you have specified in the opening <source tag of sources.xml
The epg data MUST be where you have specified in the <url> tag of sources.xml

Open the epgimport plugin on your enigma 2 box.
Click the blue button on your remote to open the sources menu, you should now see a new entry matching you would put as description in the sources.xml file.
Click the green button to save the changes.
Once you return to the main epgimport menu, click the yellow button to run a manual scan.
In the popup asking to do a NIT scan select Yes.
You should now see a count of the events being imported.

If you make a mess of your epg while doing this you can easily clear it by setting "Clearing Current EPG before Import" to Yes then running a manual scan with the IPTV source deselected to return to normal service.

Change Log
05/October - Initial version created