Mediastar Recording File Format Hacking, Getting uploaded recordings (mpeg2) to play Options

[temporary1]

I have been looking into the Mediastar recording file format to see how we might be able to upload video. Downloading is straight forward (projectX handles the files fine). Uploading of MPEG TS files dont work, so here is a breakdown of what the Mediastar appears to have, which i guess we will have to emulate.
My goal here is to eventually have a converter like topfield owners can use that would add the appropriate headers (projectX can do this). Coding that is probably beyond me, as is working it all out it seems... Anyone willing to help out, or with good ideas please post below (IMG:style_emoticons/default/smile.gif)
I am going to see what else I can add later. Apologies in advance for poor use of technical words, I am merely self taught at computer jargon (IMG:style_emoticons/default/smile.gif)

The basics:
The Mediastar makes recordings with at least 2 files, an information file and the recording datastream files split at 2GB intervals. The name is derived from: filename_#00n_YYYYMMDDHHMM.AVF/.AVR
Both files must be present. If only the .AVR is present it appears in the file menu but will not play (you cant even delete the file from the PVR menu).

.AVR file
MPEG2 TS container
Has header of 32KB starting with

CODE

     A  R  A  V                   C:\PVR\AV\{filename}.avf
0x “41 52 41 56 10 00 00 86 80 00”C:\filename etc…

Null “00” to TS sync start at 0x8000=32Kb

.AVF file
“Info container”, why when there is 32Kb in the recording? (but I digress...)

15KB, plain text or big endian words and dwords. Mostly padded with “00”

hex offset (hexcomp2)
0x0 =“ANRF” common
0x04 =11 00 00 04 common
0x08 =3f 04 00 0a common
0x0c =?4 bytes variable
0x10 =channel (ie 00 1D=29=SBS Perth) (?haven’t found LCN) “07 00”common
0x14 =00 48 01 00 common
0x18 =Service PID (2 bytes) (unsure ?Number of audio streams 2 bytes?)
0x1c =Audio MP2 PID (2 bytes) (unknown variable 2 bytes) (no PID for ac3 audio found)
0x20 =?video type (0x 11 11 common) Video PID (2 bytes)
0x24 =?teletext language? (00 00 common) Teletext PID (2 bytes)
0x28 =?2 bytes variable, then 00 common
0x2B =Plaintext Channel name (“ABC2”) null space to 0x43?
0x3c =?4 bytes variable usually null…
0x44 =01 11 00 00 common
0x48 =00 01 common, 2 bytes variable
0x4c =null common to 0x5c

0x5C =02 CA common
0x58 =Program starttime from EPG, 2 byte year (0x07d6=2006) 1 byte each M D hrs mins (nb not the recording start time)
0x64 =Duration from EPG 1 byte hr, 1 byte minute
0x66 =Plaintext Title (to 0xa5) 63 characters
0xA6 =Plaintext Subtitle (to 0x125)127 characters
0x126 =Plaintext Description (to 0x325) 511 characters

0x326 =Plaintext Filename to 1st AVR (to 0x3A4) 126 characters (no reference of 2nd found)
0x3a4 =File size of 1st AVR in bytes (big endian) less 32kB ie 0x3a4=00 04 f8 00=13 MB
0x3a8 =File size of 2nd AVR... (guess there is space for more)
0x6C4 =Bitrate in bytes/second ie 0x6c4=00 08 EA DB=584KB/s (educated guess)
0x6C8 =?dword minutes (total recording length)
0x6CC =??dword not seconds, frames what else could it be? 00 01 00 00 or 00 02 00 00
0x6D4 =Total recording file size of AVR “0x3a4+0x3a8…”

Multipart recording has… need to analyze more..
0x6E0 =?0000 or 01 01
0x6E8 =?nulls or 10 00 00 01
0x6F4 =?nulls or value (00 0f 6d 58)

0x1F48 =”1BEE” common
0x1F4A =Local start time of recording “YYMDmh” ie 0x1F4a=07 D6=2006, 0x1F4C=08, 4D=0x14=20, 4E=0C=12, 4F=0D=13 (ie 200608201213)

0x1F50 =1 byte, number of files? or number of EPG program information blocks?

0x1F51 =Information of 1st file, also repeat of 0x5C again to 0x325, ie
0x1F50 =01 02 00 CA common
0x1F54 =EPG info starttime
0x1F5A =Duration from EPG, 1 byte hr, 1 byte min
0x1F5C =Plaintext title
0x1F9C =Subtitle
0x201C =Description

0x221C =info of 2nd file, starts “02 CA” as 0x5c
0x24E6 =info of 3rd file… 714 bytes each file… Space for 10 files?

Null “00” padding to 0x3b38=15KB

To Do
Parts marked "variable" I havnt worked out yet. Not sure what is left to find so i dont know what they are... The aspect colour etc for all the files i have compared is the same afaik. There is probably a managable amount of unknowns now to start hacking anyway.

Still have not found how seconds are encoded, minutes appear to be 0x6c8, but seconds are not there as seconds.

If bitrate is important, how am i going to obtain that from an mpeg to feed into the avf?

Have not yet looked for bookmarks, but they should be in the .avf somewhere (but is unimportant for uploading)

Finally (IMG:style_emoticons/default/smile.gif)
All of this is from just doing file compares of a few 10 second recordings and a lot of guess work. I have no idea how much is essential or what effect hacking the information does yet. If anyone else wants to join in this challenge that is all I know at the moment... As editing of this post will/has expired, check down the thread for updates...

Why? As someone else said, "because i like a challenge"

This post has been edited by temporary1: Aug 21 2006, 02:58 PM

[jokiin]

Don't want to put you off the project at all but I don't think you would ever be able to get this chipset to decode DivX or any similar Mpeg4, AVI etc formats

[temporary1]

Don't want to put you off the project at all but I don't think you would ever be able to get this chipset to decode DivX or any similar Mpeg4, AVI etc formats

Oh yes, i know that (IMG:style_emoticons/default/smile.gif) ... I want it to play *transcoded* MPEG 2 transport streams... where else am i going to get those but from my collection of divx and DVDs that i own (really!)

[jokiin]

Perhaps your thread title through me off, any form of MPEG2 file would be infinately easier than any MPEG4 format

[temporary1]

Perhaps your thread title through me off, any form of MPEG2 file would be infinately easier than any MPEG4 format

My bad... topic title changed if anyone is wondering what we were on about...

[Dave®]

Somewhere in there, there should be the number of 'part' files. And info for whether there are subtitles (and what type). And the type of audio streams.

I'd imagine the date/time would be in a fomat like the UNIX or DOS formats where it's the number of seconds since some arbitrary start date (01-01-1980 for DOS). Or if it's like the toppy it could be from 01-01-2003.

BTW it's interesting that the recordings can contain longer titles and descriptions etc than the epg allows.

[dischucker]

Has anyone tried deleting the .AVF file and seeing whether the recording will play or appear in the PVR list?

It might be possible to simply upload the AVR file alone and not worry about hacking the AVF file format at all?

Just a thought...

[temporary1]

Somewhere in there, there should be the number of 'part' files. And info for whether there are subtitles (and what type). And the type of audio streams.

I'd imagine the date/time would be in a fomat like the UNIX or DOS formats where it's the number of seconds since some arbitrary start date (01-01-1980 for DOS). Or if it's like the toppy it could be from 01-01-2003.

BTW it's interesting that the recordings can contain longer titles and descriptions etc than the epg allows.

Thanks dave
1) Found the PID's , but dont understand the values next to them yet.
2) The date/time is there (at 0x1f4a)... just not the length of recording... what other values would be logical to search for? If the bitrate and filesize is there, is that reasonable?edit: think i found the recording length in minutes, but cant work out seconds
3) field length of the titles etc is a guess based on the (apparent?) null space between the obvious information

Edited the first post

have deleted an avf file associated with the an avr and it doesnt work anymore... havnt tried editing out the reference to the avf in the avr file yet. Someone tried uploading TS files before, and it didnt work

[Surlyrich]

You obviously dont have enough recordings on your HD. I am trying to find the time to get the bastards off my HD, rather than put them on, and my server which is serving 1.5tb is almost full (IMG:style_emoticons/default/smile.gif) On a side note, has anyone tried a larger HD in the unit? I was thinking about that 400gb Samsung I saw for $250, or maybe even one of the 750gb jobies. Anyone know what the limitation on the HD size is for the firmware?

[temporary1]

You obviously dont have enough recordings on your HD. I am trying to find the time to get the bastards off my HD, rather than put them on, and my server which is serving 1.5tb is almost full (IMG:style_emoticons/default/smile.gif)

Exactly. I dont have a media server, i decided to go off the shelf with the mediastar. That leaves me without a good way to play "backed up" video. Even if it is offline, requiring transcoding, this is still better than sitting at my desk with a 14" screen. Yes i could purchase a divx player but i already blew the budget on the Mediastar (IMG:style_emoticons/default/smile.gif)

Plus: Bump for editing the 1st post again... has almost everything worked out now (except recording length seconds, help? or ill just have to blindly test...). Will get to hacking/testing later tonight or in the week

[CameronD]

I am finding a few differences so far, but I only have one radio and one HDTV recording to work with just now.

The repeated blocks are not for files, they are for the EPG info over the duration of the recording.
So for example my one hour recording started a bit early and overran the 5 minutes of news at the end. That gave me 4 EPG entries (number 4 at 0x1f50).

[Dave®]

All of this is from just doing file compares of a few 10 second recordings and a lot of guess work.

Are you able to put some of these avr and matching avf files online so I can download them and have a look at them?

I have no idea how much is essential or what effect hacking the information does yet. If anyone else wants to join in this challenge that is all I know at the moment...

At a guess I'd say that a lot of the info in the avf file is superfluous for an uploaded file. e.g. if you upload a home movie you're not going to have, or need, the description, subtitle and epg info etc.

Why? As someone else said, "because i like a challenge"

I wonder who said that? (IMG:style_emoticons/default/wink.gif)

Of course you know as soon as you, or someone else, creates an application to add medistar headers and create a matching avf file... programs like ProjectX will be updated to include this anyway.

[temporary1]

I am finding a few differences so far...
The repeated blocks are not for files, they are for the EPG info over the duration of the recording.

I would be interested what differences you have found. I havnt done any more compares yet, and i hope nothing is machine/location specific.

And yes i am convinced that the repeated blocks are EPG info. But i havnt actually tested it yet. I have made a few recording long enough to be split and will analyse them tonight.

Are you able to put some of these avr and matching avf files online so I can download them and have a look at them?
At a guess I'd say that a lot of the info in the avf file is superfluous for an uploaded file. e.g. if you upload a home movie you're not going to have, or need, the description, subtitle and epg info etc.

Of course you know as soon as you, or someone else, creates an application to add medistar headers and create a matching avf file... programs like ProjectX will be updated to include this anyway.

Sorry Dave® i am mostly on dialup or through work so dont have bandwith for that (or a site to host, or the files on me at the moment...). Minimum file size is about 10MB, unless i cut the ends off them (and then they wont be good for stream analysis...). And i have had to compare half a dozen to make sence of them. Can anyone else help Dave out ?

I would guess too that most of the info isnt needed, or wont be 100% necessary. What i dont know yet is how to automatically get the PIDs, times, bitrates, etc into the file. I have a couple of hours tonight to upload test files to the MS, so i will see if (as close as i can get to) 100% compliance works, and then what is actually necessary.

I fully hope someone will update something like ProjectX (I am not in this for fame, glory and riches (IMG:style_emoticons/default/laugh.gif) ), but with only a few users in Australia i wouldnt have thought it likely that it would be done without some people starting the ball rolling... I wonder if the european equivelent of the mediastar uses the same formats... i tried a little searching before but found no hacking out there...

[temporary1]

(IMG:style_emoticons/default/cool.gif) Success (IMG:style_emoticons/default/cool.gif)

Ok i have mananged to get a DVD derived recording up and going (or at least 100MB/2mins of it)

I can send my sample avr header and avf files if you PM me your email (32k + 15k), but these are easy to make yourself, instructions below. I dont have a one click solution just yet (IMG:style_emoticons/default/smile.gif)

Here is how... (sorry for the length)

Uploading files:
Must have 256kbs 48khz mp2 audio (128kbs does not work…)

The filesize at 0x3a4 has no effect (on single file recordings at least)

The time does come from 0x6d4 / 0x6c4 (ie filesize/bitrate). Actual numbers are unimportant, but must be about or longer than the recording. A small ratio will not play the whole file, a long ratio will terminate at the end of the data, but before the progress bar thinks it will.

Method for .AVR
*** Haven’t tried over 2gb yet (only up to 140mb/2 min 30), very small files also don’t play (less than 5MB).
Obtain mpeg file… ie VOB from DVD
Demux in projectX
If audio is not 256kbs 48khz mp2 then make it so…
Ie D:\VIDEO_TS\BeSweet>besweet -core( -input ..\VTS_01_1.ac3 -output ..\VTS_01_1.mp2 ) -mp2enc( -b 256 )
Re mux the video and audio (imagoMPEGmuxer, only mux the mp2 file)
Transcode the mpg to TS using projectX (you should see that “mpeg audio 0xc0” “mpeg video 0xe0”) Also write down the number of bytes written so you dont have to look later.

Now you need to put the 32k header on the file. I have a generic file that has a generic filename in it at the moment. Create it from any AVR file by saving the initial 32kb and changing the filename in the header.
I then copy the header to the .ts with
D:\video>copy /b Test000001_#001_200601011200.avr.header + Test000001_#001_200601011200.ts Test000001_#001_200601011200.avr

That was the easy part…

Now create an AVF file…
Best to start with any from a previous recording and edit it with these changes... Not sure if every change is required yet.

CODE

0x18    =0x102
0x1c    =0x00 c0 07 00
0x20    =0x11 11 00 E0
0x24    =0x00 00 00 00
0x2B    =text ie “UPLOAD”
0x58    =Program starttime ie 0x07d601011200
0x64    =Duration from EPG 1 byte hr, 1 byte minute ie “0x0100”=1hr
0x66    =Plaintext Title (to 0xa5) 63 characters
0xA6    =Plaintext Subtitle (to 0x125)127 characters
0x126    =Plaintext Description (to 0x325) 511 characters

0x326    =Plaintext Filename to 1st AVR (to 0x3A4) 126 characters (C:\PVR\AV\Test000001_#001_200601011200)
0x3a4    =File size of 1st AVR in bytes (big endian) less 32kB ie 0x3a4=00 04 f8 00=13 MB (not important with 1 file at least), Remember that number from projectX i told you to write down?
0x6C4    =Bitrate in byte/second ie 0x6c4=00 08 EA DB=584KB/s
0x6D4    =Total recording file size of AVR “0x3a4+0x3a8…”

Note the bitrate at total recording size must give a ratio of approximately and slightly larger than the recording. This is how time is calculated. Can be larger, but progress bar will report longer than the file actually is.

CODE

Probably optional, just fill in with generic info
0x1F4A    =local time of recording, as 0x58
0x1F50    =1 byte, number of EPG info blocks
0x1F51    =Information of 1st file, also repeat of 0x5B again to 0x325, ie
0x1F50    =01 02 00 CA common
0x1F54    =time again
0x1F5C    =Plaintext title
0x1F9C    =Subtitle
0x201C    =Description

No idea how multipart recordings behave yet. Don’t know if 2gb split is essential etc etc…

It looks like coding this together should be easy (i have a very rough batch for my generic filenames), it is just handeling all the format conversions that is hard. I also think that i may have gone a few steps too far in all the transcoding demuxing remuxing etc, but i havnt seen where or how i can shorten the path yet...

Wont get time to play more till the weekend now (IMG:style_emoticons/default/sad.gif)

[temporary1]

OK, ill put this in a new post as those of you interested may have already read my last post.

Some of you are much more experienced at transcoding and scripting than i so here is a shopping list of programs i would like to compile and automate... I dont think this is too ambitious (but correct me if i am wrong). I dont plan on making this 100% automatic, but this is a recipe of what i do manually...

These are mostly windows programs, so if multiplatform alternatives are around please suggest.

• First stop is something to check what you are feeding the converter. Mediainfo (windows, has source) works on command line and outputs a log of what the audio and video streams are. I presume parsing that to make some further decisions is relatively easy... If it is an m2v/256Kbps 48khz mp2 mpeg then go straight to transcoding to TS.
• Audio: It appears that divx and dvd files are usually .ac3. Besweet (windows only) appears to take almost anything and gives a good (enough) output.
  besweet -core( -input {filename}.ac3 -output {filename}.mp2 ) -mp2enc( -b 256 )
• Video: Things could get complicated here... there are a few programs that do avi-m2v, but i havnt seen any that output TS. If the source is Mpeg2 then projectX (windows/*nix) will demux on the command line.
  java -jar ProjectX.jar "{inputfile}" -demux -out {location} -name {filename}
  I havnt done a lot of mp4-mp2 conversions, what command line is out there ?(ffmpeg?) I took the *long* road and used avi2dvd then worked on the VOB files...
• Remux: Now you have m2v and mp2 (256kbps 48khz) files, you need to stick them back together... ImagoMPEGmuxer(windows only) has a command line interface.
  ImagoMPEG-Muxer {input.m2v} {audio.mp2} -o -mpg {outputpath}
  There are others, if anyone knows one that outputs mpeg TS please post.
• Convert to TS. ProjectX again
  java -jar ProjectX.jar "inputfile" -tots -out {location} -name {filename}
• AVR File header. All i need here is something to write out the initial 10 bytes of code, plaintext filename plus file number and date (C:\PVR\AV\Filename_#00n_YYYYMMDDHHMM.AVF), and then pad with 0x00 to 32768 bytes. Then copy that header and the .ts together "copy /b header + tsfile Filename_#00n_YYYYMMDDHHMM.AVR"
• AVF file. Probably only needs filename, filesize, date and a time calculation. I have no idea how to automate this, would like ideas. Since projectX always appears to output files with the same PIDs for audio and video we dont need to change these from (0xc0 and 0xE0), but can get them. Playtime and filesize are also output in the projectX logfile. All that appears to be really needed over a generic .avf file is the filename and recording length:
  0x326 =Plaintext Filename to 1st AVR (to 0x3A4) 126 characters (C:\PVR\AV\filename_#00n_YYYYMMDDHHMM.AVR)
  0x3a4 =File size from projectX output
  
  Time
  0x6C4 =Bitrate in byte/second ie 0x6c4=00 08 EA DB=584KB/s
  0x6D4 =Total recording file size of AVR “0x3a4+0x3a8…”
  where these numbers give a ratio equal to the time. I havent actually tested this but think it would work, if the dword at 0x6c4 was decimal 1000 (0x000003e8), then 0x6d4 could be actual number of thousandths of seconds.
  
  for completeness you could add relevant (i just use dummy placeholders at the moment)
  0x58 =Program starttime (todays date?)
  0x64 =Duration
  0x66 =Plaintext Title (to 0xa5) 63 characters
  0xA6 =Plaintext Subtitle (to 0x125)127 characters
  0x126 =Plaintext Description (to 0x325) 511 characters
  
  0x1F4A =local time of recording, as 0x58
  0x1F50 =1 byte, number of EPG info blocks
  0x1F51 =Information of 1st file, also repeat of 0x5B again to 0x325 etc

Doing this a couple of times is ok for the few manual steps. I wont be rushing to code it on automatic, but may do it for the hell of it if i get time (it would be windows batch, so you perl nuts out there might like to do it better (IMG:style_emoticons/default/smile.gif) ).

Larger files:
Still dont know how files larger than 2gb should be handled.
In the AVF: only 1 avf file is required
File sizes are at 0x3a4 0x3a8 0x3ac 0x3b0 (at least) each max file size is =0x7f990000=2GB
0x6cd=appears to refer to the number of AVR files
0x6cb=is the number of total minutes, but appears to be unutilised.

AVR:I dont know how the joins are managed. Need to look at the beginning and end of multipart AVR files to see what headers/footers would be needed. I hope that we can just tell projectX to split at 2GB when making the TS ("-split 2000" or 1999).

[CameronD]

Larger files:
...
AVR:I dont know how the joins are managed. ...

ProjectX has format-specific options for toppys and Humax, among others.

I would presume the MediaStar stuff could be built into projectX along the same lines; they might be just a bit more complex. The TS file splitting and AVR output could all be done at once.
There must be some block boundaries to the video and audio streams and I would think these might need specific chopping points - but perhaps not (it's something else to check).

[padeen]

Larger files:
Still dont know how files larger than 2gb should be handled.
In the AVF: only 1 avf file is required
File sizes are at 0x3a4 0x3a8 0x3ac 0x3b0 (at least) each max file size is =0x7f990000=2GB
0x6cd=appears to refer to the number of AVR files
0x6cb=is the number of total minutes, but appears to be unutilised.

AVR:I dont know how the joins are managed. Need to look at the beginning and end of multipart AVR files to see what headers/footers would be needed. I hope that we can just tell projectX to split at 2GB when making the TS ("-split 2000" or 1999).

I have a couple of TV programs that were split over two files. I had a look at them and I can't find anything in the way of a footer on file 1 or header on file 2 that indicates a lead-out -> lead-in. File 2 has the same 0x8000 sized file header as file 1 as far as I can tell (apart from the obvious, such as filename).

BTW, I had a bit of trouble finding a freeware hex editor that can handle 2GB or bigger files. I ended up with HexExtreme, but it's very basic, although it handles big files very well, with no lag. Have you found a good editor?

Kudos to you for starting this thread! I've long wanted to watch DivX etc on my TV instead of a small computer monitor.

Nick

[temporary1]

... I can't find anything in the way of a footer on file 1 or header on file 2 that indicates a lead-out -> lead-in.

File 2 has the same 0x8000 sized file header as file 1 as far as I can tell (apart from the obvious, such as filename).

BTW, I had a bit of trouble finding a freeware hex editor that can handle 2GB or bigger files. I ended up with HexExtreme, but it's very basic, although it handles big files very well, with no lag. Have you found a good editor?

Thanks for that.

1) i wonder how we can tell what the end is, one presumes it might be a P frame (just learning mpeg terminology) or it could just be that it is seamless data across the two files... Guess we will have to try unless someone knows of a program that can tell us what the first and last frames are...

2) So the header still has the *same* 10 initial bytes, followed by the path to the AVF for file #001 ???

3) I have been using hexcmp for loading and editing (notepad++ is ok for small files, but chokes on larger ones). It is trial shareware, so i have to get this finished in 8 days ! (you get 15)

I cant get to work on mine till sunday probably so if anyone out there is playing with uploading, would you let us know if you find what the filename length limit is and if the date on the filename is necessary File_#001_200601011200.avr File_#001.avr File.avr Thanks!

I have also had a look at the source to projectX. Those who understand java might get more out of it than I, but the code for the Topfield (TF5000) headers is there in the \thirdparty\ts.java file to give you an idea. Poking around in the other sources i can see that projectX will output additional files for somethings, so could realistically be built to output the AVF also. I also see source for decoding mpeg and ac3 audio and encoding mpeg audio. Does that mean that one could possibly do the audio transcoding here also? I dont see it being used like that in the current build. I would talk to the projectX forums but everything appears to be in german...

[CameronD]

Larger files:
A bit more information and observation.
The files are never larger than 2140766208 bytes (0x7f998000), as you noted before.
This is about 6MB below the max possible with 31bit addressing.
This number happens to be 32k plus 188 * 11386880.
188 being the length of a TS packet.
Now the last number is 0xadc000, so the MS might like to use packet counts that are multiples of 0x4000.
Whether it likes to or insists on are different matters. One possibility for a policy like this is that the file size will always be a multiple of the fat-32 cluster size. I think it's unlikely that it is looking for anything fancier, such as using GOP boundaries.

I have another largish recording (1 and a half files), but I edited out the first few minutes on the 920. It happily makes the first file shorter, without changing the second one, so the large-chunk files do not have to be the exact same size.
This file is also a multiple of 0x8000

File names:
The names inside the files cross-reference each other - the relevant AVR file is named inside the AVF file and vice versa. If either have a mismatch then nothing appears on the pvr list. In some cases of mismatch the AVR file is actually deleted automatically. I think it is if the AVF file named in the AVR file does not exist.

[Dave®]

I'm about 70% done writing a simple (command line/send to) application to add the avr header and create the avf file. Should be finished on the weekend.

The code for adding the avr header is done. I'm just fine tuning the avf file creation code trying to decide what's actually needed. And I'm still trying to work out how to automatically get the bitrate from the original file.