Converts xmltv files from any source to MediaStar compatible EPG dat files. I've tested it with xmltv files from 6 different sources (including IceTv).

To use just download, unzip, copy your xmltv file to the Data directory and name the xmltv file xmltv.xml. Or you can call it from the command line or a batch file and use the path/name of your xmltv file as a parameter. i.e. xml2ms "C:\Program Files\somegrabber\listings.xml"


Requirements: Windows - and an xmltv file

Download: http://pvr-tools.com/freeware/xml2ms.zip

Very nice Dave... Works well.

I was hoping someone would say that... because I don't have a mediastar so I was programming 'blind' with no way of testing it myself.

I still have to tweak the category flag (it currently only sets the flag for 'Movie') as at the moment I have no idea what number the ms firmware uses for other categories. Also the ratings may be incorrect... I had to estimate what they should be.

This is my parental code in the EPG_uploader TAP, converts the harvested strings to the EIT defined ratings.

I have only created a MS dat file with it... havnt tested on the MS yet, but the dat file looked allright, as do the ratings. Have you thought about adding some configeration to the program such as channel mapping to LCN and ICE ID. This will help me with adding sat channels for my 820 (when I finally get it to work).

The Ice ID number is not needed for the MS epg - at least not in the current firmware. I only inserted the channel names as the iceid# so people could spot any errors in the dat file.

Currently it has channel-to-lcn mapping for all Oz broadcasters (and all known xml files with their different channel naming conventions).

You can add extra channel mapping for sat/paytv channels by editing the xml2ms.awk file in a text editor (channel mapping is about 1/2 way down). Just bare in mind that the order is important (ABC2 has to come before ABC, and 31 has to come before 3 etc), and every line, except the last line, needs an 'else' at the end. So to add Cartoon Network, assuming that's what the channel's <display-name> is in the xml file, you'd add


replacing '0123' with the Cartoon Network's LCN.

Good work Dave!

In Perth (or maybe its just my mediastar) i needed to change the LCN for SBS 1 from 1283 to 1282, other than that it works... (this will also need to be fixed once mediastar bug fix this bad LCN...)

But, a few feature requests

1) we have a variable for input filename, i would like a variable for *output* directory that way i can just plug in the Mediastar and tell it what directory i want the EPG file to end up in.

2) The jrobbo files are not in GMT. Could we have a variable so we can fix the offset to GMT...

I tried looking how to hardcode these manually and couldnt find it easily... and as for the GMT on second thought that is a hard problem... Are there any converters already out there that will manipulate the XML file times?

I get this when using the new dugg importer:



Looks like the ICE ID and LCN are missing and the Event ID of 1234 is the same for all programs.

BTW, I did manage to get my sat channels incorporated by editing the awk file though.

Ah, I meant to ask you about this sometime Tony - I was curious if the lack of a (P)reschool rating was a limitation of EPG_Uploader, the Toppy, or the EIT standards. I hadn't noticed the lack of a 'C' rating. I was going to look up all the relevant specs, but seeing as you posted that snippet here...

I had posted this in another thread re the mediastar ratings available, and was wondering what the "P" rating was....

Updated to Version 1.1

- Added TDT channel mapping
- Fixed ratings
- Changed optional input file parameter
- Added optional output path parameter
- Added optional UTC offset parameter

See the included readme.

Can someone test numbers from 2 up to about 20 for 'flag1' to see what number relates to what category?

Wow, that was quick, but Dave, there are a few quirks that crept in... run in a fresh directory and you will see... Ie needs input file name or it errors... and wants to create an iceepg directory in the target (but the target should be what i want)

Will test those flags later, you meant the first of thost numbered tags?

The Event ID and ICE ID currently aren't used by the MS firmware... so I just added 1234 as the Event ID to all programs. I only put the channel name in as the ICE ID so users could easily spot mistakes in the dat file. I could have just left them both out.

But missing LCNs are a problem. Can you zip and email me your awk file and xml file to 'support AT toppytools.com'

Both Fixed. (version 1.1.2)


Yep

Will version 1.1.3 have a working UTC converter? the one you gave doesnt appear to work with jrobbo DW files. What does it work with?

I also tested out the flags and didnt notice a thing with 0-23 in both the 8th and 9th fields. it might be a live thing, but there is nothing to the EPG. Not implemented yet?

It works with files in this format:
<programme start="20060803001000 +1000" stop="20060803003500 +1000" channel="2">

But apparently not in this format:
<programme channel="2" start="20060803001000 +1000" stop="20060803003500 +1000">

I just included someone else's texta.exe. Looks like I'll have to code it myself.



Maybe it'll be in a later firmware release.

I had noticed you had included the texta, and that the comment from dialupxmltv is that it works with oztivo data. (oztivo blocks my IP for no good reason). Some may like to know that the D1xmltv grabber also has the correct format. {edit: ... has the correct format for textA}

I just checked xmltv files from 10 different grabbers/sources and only jrobbo's xmltv files are in the wrong format.

Jrobbo's format is not strictly "wrong" as XML does not place significance on the order of attributes. A proper XML parser like Expat can handle jrobbo's files fine.

Yeah, I know. I meant wrong as far as TEXTA was concerned.

All fixed. It includes a new version of texta.

Havent uploaded it yet (no notebook till tonight) but looks ok on the GMT-ization (FYI Perth is -480 ). Thanks ! The jrobbo files are the most detailed free souce i have found yet (why is it so hard to find sources, is it a legal thing?) Have i said too much already?

With this version you left out Chan 31 in the mapping (which used to be in the wrong place ), so 31 data is getting into the sbs guide still.

It also still insists on writing to an "iceepg" directory and fails without it, so you have to say h:\PVR (with an existing \iceepg) rather than h:\PVR\ICEEPG

Either way, i say again, good work! I uploaded some D1 info last night and have now started programming by the EPG. Yay! no more manual entry. Might check out a trial of the iceTV guides to see how they go later.

I removed it last night because ch31 is analogue. But if it's in your xml file you'll have problems... so I've put it back now (in the correct order ).


I was unaware of the PVR directory on the MS. I assumed the ICEEPG directory would be in the root, like h:\ICEEPG. I coded it so people wouldn't have to type the whole path (and possibly get it wrong).


Once you go EPG you'll never go back!

Updated to Version 1.1.5

Version 1.1.5
- Changed IceEPG directory to PVR\ICEEPG
- Added title truncating to 39 characters
- Added channel 31 deletion

Version 1.1.4
- Fixed 'Channel 31 appearing as SBS' bug

Version 1.1.3
- Fixed UTC changing in jrobbo xmltv files

Version 1.1.2
- Bug fix - I'd included an old file in 1.1.1

Version 1.1.1
- Fixed 2 bugs introduced in 1.1
- Added TDT channel mapping

Version 1.1
- Added TDT channel mapping
- Fixed ratings
- Changed optional input file parameter
- Added optional output path parameter
- Added optional UTC offset parameter

Version 1.0
- First release

One small bug in the awk script:

These two lines should be transposed, since "SC10" (Canberra's Channel 10) will trigger for the first line and so not enter the second line, giving an incorrect lcn of 0001,0010.



FWIW, here's the entire section from my version of your script; I've set it up so ABC 0022 shows up too, as well as Prime (PrimS):


Nick

Updated to Version 1.1.6

Version 1.1.6
- Fixed some channel mapping errors

If only Microsoft fixed errors at this pace...

I've got a beta v1.2 if anyone wants to test it. The only difference between v1.2 and v1.1.x is the UTC conversion is built in to xml2ms.exe instead of in the separate texta.exe.

Download: http://pvr-tools.com/freeware/xml2ms.exe - you need to already have the other files from a recent version xml2ms.zip

I only added UTC conversion internally in xml2ms.exe because:
a.) I like a challenge.
b.) I'd already spent a couple of hours on it before the new version of texta was released.
c.) Because the author of texta didn't show me his source code.
d.) There is no reason d.

Thanks once again Dave®.

For reason A above... despite what you may be planning to do with this converter, i made a very crude 1 click uploader (in batch file!) using wget (or here) and unzip and, of course, xml2ms.
Put wget and its files in directory \xml2ms\wget and unzip.exe itself in \xml2ms

Call this xml2ms1click.bat or something more exciting and put it in the xml2ms directory
This batch file should get only a new file, unzip it (and overwite the old unzipped file) and then de-convert from Perth time to GMT... (it may not need the "del xmtlv.xml" to start with..., i havnt been able to test yet whether the switches in wget will overwrite or create a new file, need to wait for an update ) It has zero error handling, it works or gives obscure messages, use at your own risk, and only in fine weather.

Edit:The del line is not needed and the wget commands should *not* have -r -nd, just -N

Nothing wrong with batch files. I often resort to batch commands when the language I currently use can't do what I need.

A few of my programs started life as simple batch files (Send2-Toppy, USB Monitor, Up2Date etc). Though Up2Date grew to 1500 lines!

Nice find on Info-Zip. I'd been looking for command line zip and unzip exes that can handle long file names and output directory paths.

BTW what happens with your batch file when the zip file is 0kb or non-existant? Will it download from jrobbo's mirror? Will it download one of jrobbo's backups? Will it still run xml2ms? Will the world end?

Basic and batch are all i have coded in. I have some moderately elaborate (and probably inefficient) programs, but nothing on this scale (the batch language in XP is surprisingly powerful compared to the days when i read the dos 5.0 manual from front to back, when programs actually came with manuals...) I had ideas how to code this, but doing it would have taken me a lot longer than a few days to code it! and it might have been hard in batch...

What happens when something unexpected happens with the batch... it explodes... mercilessly... if it has been run before it will probably just process the old file, or xml2ms will complain of not having a file... If one step fails, then the next command just errors worse... and so on. But i dont think it will escalate to end the world... I guess i could watch for errorlevels if the programs used them and i could be bothered, but i figure if it doesnt work, then failing with an error message wont really help I guess you are promting me to do more with it

Infozip and wget were the first things i got from a google search...

I wouldn't use -r and -nd together here, as you are depending on Robbo not changing his directory structure. If he decides to archive old files in sub-directories, you'll end up pulling every one of them down to your PC, even if they do get saved on top of each other. It would be wasted bandwidth.

Simply use -N and you'll be fine.

BTW, great minds think alike . I've also been using wget, and I use 7zip to uncompress them.

Never hacked with wget before (only knew it existed because of a few linux excursions...). Was trying to find a way to get wget to just overwrite a file rather than rename it (the manual seemed to imply -r would do that... but then you get the directories...). You are right that -N is sufficient. The del is not needed either now that i have tested it.

Now all i want to do is abort the batch if the file is indeed old, because at the moment it just plods along. wget doesnt seem to output an errorlevel whether it downloaded a new file or not. Someone has already tried to do this here, but i cant get it to work based on the for /f example and dont know why (it doesnt seem to get the date into the variable)

Wouldnt you know it, answered my own question. Kept tinkering with that example and this works (Windows XP .bat file, probably wont work with other windows/9x/NT? etc...)

This works on comparing filedates. Better than the other 'accepted' kludge of desetting the archive attribute and then testing for the presence of +a if a new file was created. Apparently the other accepted ways of processing wget output is to capture and grep? the output text (arrgh!)

Updated to Version 1.2.1

Version 1.2.1
- Added removal of CDATA tags

Since jrobbo's dwxmltv data now contains CDATA tags in the title and description it now checks if CDATA tags exist and if so removes them.

jrobbo also went and changed some of the channel IDs... what has he been smoking this week?

here is the one i use in my .bat file
thats for sydney of course, i cant get the unzip call to work in the bat file...and mine had trouble coping with finding the file in the xmltv directory



wget checks the file (well my version does) and only downloads it if it a new file
i had to refer to the abosolute path of 7zip

I think i might need to add a new line to my batch file to get the latest xml2ms version every day too

Thanks Dave for all your effort!

i second that, Dave you are a champion!!!

nothing like a program that does exactly what you want, yet has 0 GUI!

ingenious!

Thanks heaps to everyone who has been working on this - for someone who doesn't understand the how this all works, it is really helpful.
I have tried to get the EPG working on my MS using Dave's xml2ms (V 1.2.1) against www.jrobbo.com/dwxmltv/xmltvSydney.zip
Everything seemed to work fine, but I am only getting EPG info for channel 2. Not sure if there is something I have missed??? [actually, to test, I just deleted the dat file from the box, and am still getting epg data for channel 2...same as I was with the dat file: for today (Sat) until 4:30am Tue morning. The channel 2 I use is ABC TV Sydney. I remember someone saying they had extra broadcast epg from ABC in another post. Looks like this is what I have.]

Is there anything I need to do to map my channels to line up with the dat file?
(I haven't updated the f/w yet - would this make a difference?)
Below is the begining of a few lines from the dat file - not sure if something has gone slightly awry - most columns are delimited by a TAB, but some are delimited by a comma.

2aagaibabeaaa0 10 0001,0010 1234 115 The UpLate Game Show 0 0
2aagaibabeaaa0 7 0007 1234 30 20 Things To Do Before You're 30 0
2aagaibabeaaa0 9 0009 1234 90 The Afl Footy Show (Cont.) 0 0
2aagaibabeaaa0 ABC2 0021 1234 60 Rugby League (Qld) 2006 (Cont.) 0 0
2aagaibabeaaa0 SBSD 0033,1281 1234 30 Arabic News 0 0 0


The other question I have is:
Once this is working, I am not 100% sure on how to make the most use of it. I bring up the epg screen, select the program I want to record and the details should be automatically filled in in the scheduler? This seems to work so far with the ABC EPG guide. If I want to record the same show each day/week, I imagine I just set that accordingly as a daily or weekly recording. My question is, as I continue to upload new epg data to the box, if my weekly program is moved one week to a different time slot, does this info get passed to the scheduler or do I still need to adjust the schedules?

Thanks again to everyone involved in these forums!!!

BW

You need to turn on the 'Use Ice' option in the MediaStar's menu.



Yes


Yes


The timers are set by channel, start time and duration only. So if the broadcasters change their scheduling you have to adjust your timers - or delete them and create new ones.

Dave,

thanks heaps for this tool. It works really smoothly.

I need to off-set the times but haven't worked out how to modify the script (my skills in this area are low: the computer has buttons and I push them until something works... well, not quite). The help file gives the command line, but not how or where to stick it. Could you enlighten me?

Cheers
CA

P.S. I've only been able to bring up OzTiVo and JRobbo's EPG guides with google. You mentioned that you've tested the converter on a range of EPG's. What are some of the others?

Edit: Frustration central. I just tired getting hold of an IceTV guide since they have a 14 day free trial. What a disaster. I signed up OK, but then spent an hour trying to work out how to download the xml file. Still haven't done it. I figured if you paid for the service you might just get a gui that made things simple -- like JRobbo's. Was I wrong. I much prefer the hack already!

Part of the problem is that IceTV doesn't have support for the MediaStar (that I could see anyway). Maybe this will come, but for now... big thumbs down from me.

Changing the time offset is done via a command-line switch. It's all in the included readme.txt

The following would change the time offset by -30 minutes, using "C:\Program Files\somegrabber\listings.xml" as the source file and create the dat file in "H:\PVR\ICEEPG"

xml2ms -offset=-30 -output="H:\" -input="C:\Program Files\somegrabber\listings.xml"

All 3 parameters are optional. If you need/want to use one of the parameters create a text file in the same directory as xml2ms.exe, change the text file's extension to bat, copy and paste the above commands in the bat file, then edit the red parts of the commands to suit your setup and save the file. Now when you want to run xml2ms you run the bat file instead of xml2ms.exe.



Some of the others are IceTV, D1 (Development One), frreepgstream, and all the tvguide websites (using various grabbers - a few of which the authors don't want publicly known). In fact all the better ones are kept 'under the radar' and aren't publicised. There's at least one grabber available for almost every tvguide website.


Getting the IceGuide in xmltv format is 'relatively easy'. Download webget.exe here: http://www.icetv.com.au/support/howto/webget.exe

Then create a text file in the same directory, then copy and paste the following:
webget.exe "http://iceguide.icetv.com.au/cgi-bin/epg/iceguide.cgi?op=xmlguide" –u username -p password -o "C:\xml2ms\xmltv.xml"

Replace username and password with your IceGuide username and password. Edit "C:\xml2ms\xmltv.xml" to where you want to download the xml file to, and the name you want to save the xml file as. Then save the text file and rename the extension to bat. You then run this bat file when you want to download the IceGuide.

Make sure you've set your area and time zone on the icetv website - click on "My Account".


If Ice don't hurry up and release a simple GUI application for the MediaStar... I will.

edit, oops didnt read the next page, deleted unnecessary reply...

Also i have re-tested some field lengths.
The Title must be 30 or less (i know i said otherwise, but that doesnt seem to hold now... )
quick fix, line 193 " title = substr( getfield( "title", rec ), 1, 39 )" change the 39 to 30

The subtitle is good for at least 400 characters, havent tested more yet (suspect that it is between 500-550 as..)

Desc tag is limited to somewhere below 550 (504 is ok, still testing). No problem other than the desc is not shown if it is over, i guess it could be added with similar lines to the title field limiter? havnt tried hacking it yet.

Also, i dont know how hard it would be to add a sanity check to the "length". Yesterday the perth guide was stuffed by seinfeld being 770 minutes long. Even though the next program's start time had passed, the MS doesnt roll over. I guess this is a firmware bug anyway... (and reflecting the 'free' quality of the guide )
"<programme channel="10" start="20060812131500 +0800" stop="20060813014500 +0800">" so it wasnt your script Dave

Version 1.2.2

- Changed maximum title length to 30
- Added subtitle length limit
- Added description length limit

Thanks Dave - I had looked through the menu's thinking there must be something, but couldn't find it in any obvious place. I went back and found it in: Menu\ User Preferences\ Display Set Up

Something else strange - I have tried adding schedules from the EPG info - I can only seem to get it to import the data (time, name etc) to the shedule for Channel 9 - but it picks programs from channel 7 at a similar timeslot to record on channel 9...ie I pick CSI Miami Ch 9 9:30pm Sun, and the data that is imported to the scheduler is record Channel 9 at 11pm Filename JAG (which is the next program on Channel 7 after 9:30pm

I haven't switched the box off since setting this up, but will look again once I have done.

Trying to record from the EPG for other channels takes me to the main shedule screen, listing current schedules, and adding a new shedule at that point contains the default data.

Sounds like this problem:


Surlyrich - did you ever get this to work?

If anyone else has any other ideas - I'd certainly welcome them!

Thanks,

BW

Edit: Looks like this is happening for others too: see here

I'm glad you found that... it saves me repeating the answer here.

The 30 char title limit, coupled with the limited screen width of the EPG display, shows up a bit of a problem with some of the feeds. I use Robbos, and his feeds preface some titles with things like "Documentary Series:", "Sport: ", "Cooking Series:", etc. It means that you get a display on the TV EPG of

19:00 Cooking Series: Ming shows what he is coo

which doesn't show what the episode is about.

I mucked around with Dave's awk script and added some lines to remove some of these things. It searches for some particular entries, then removes the first part of the entry up to the ':' and the blank space after it.

Here is the section that changes the titles:


You can see what I've done. I searched the xml listing and these were the only ones I wanted to remove; you don't want to remove all entries with colons because some of them are descriptive, like "The West Wing: President Barrett etc". You may like to add some more entries, just follow the format above and don't forget the 'else' at the end of the line.

I should mention a big kudos to Dave for the very flexible way he has designed his tool. By putting the grunt of the editing into an awk script, he has made it possible for us to edit it to suit our own preferences. Very good software design Dave, as I'm sure you're aware .

Nick

Cool padeen. I had tackled this at the moment by putting the full length title into the subtitle box...
I like your way better I have also played with putting the desc in the subt box so you dont have to hit info to find out if there is any desc or not...

MS said to me the other day they were working on the layout and perhaps improving the space usage on 4:3 screens. Is it just as limited on widescreens?

Not sure if anyone else has trouble with wget - it seems to have trouble getting through the firewall....
I found another utility that is almost as good - doesn't appear to be able to check if the file is newer than the destination file and downloads regardless: iopus
command for the batch file:
.\iopus\Downloader.exe -download http://www.jrobbo.com/dwxmltv/xmltvSydney.zip "C:\...\xml2ms\Data\xmltvSydney.zip"