Incomplete 4DTV schedule load

[LKO]

I've just rebuilt my MythTV system, and switched over to SD (Thanks a ton for keeping the guide information available!). I was hoping that the problem in this thread had been solved during the switchover, but it cropped up again when I was setting up the video sources and all:
http://www.gossamer-threads.com/lists/m ... 084#240084

Any hope of a fix for the few 4DTV users out here? For now I can do what I did back then: hand enter the channels we use the most. Which brings up another question:

How can I manually download the guide XML (I guess that's the word for it) like I did in the thread above, from SD?

[3gem]

I second that. How do we download it manually?

[LKO]

It looks like the latest XMLTV will handle SchedulesDirect, so you should be able to use that like I did way back when I hand-entered channels from it. I haven't had a chance to try it yet; Real Life intrudes....

[LKO]

OK, this isn't elegant, and I'm not yet proficient enough yet to get this into a script, but I finally have a workaround for the failure to import 4DTV lineups into mythTV.

1) tv_grab_na_dd -configure
This will get XMLTV set up to download the channel information. Be sure to configure it for your 4DTV lineup.

2) tv_grab_na_dd --dd-data dd_data.xml --days 0 > xmltv.xml
This grabs the channel information. At this point, pick your favorite text editor, something that can do global find/replace, and open up xmltv.xml.
Here's a snippet of what we'll be modifying:
<channel id="I10021.labs.zap2it.com">
<display-name>G3701 AMC</display-name>
<display-name>G3701 AMC 4DTV:-</display-name>
<display-name>G3701</display-name>
<display-name>AMC</display-name>
<display-name>AMC</display-name>
<display-name>Satellite</display-name>
</channel>

Evidently, it's the channel ID that's messing up mythTV. We need to do two global find/replaces. The first will be to change all
channel id="I
to
channel id="
That gets rid of all the "I"'s in front of the channel id. The second global find/replace will be to change all
.labs.zap2it.com"
to
"
That gets rid of the ".labs.zap2it.com" stuff in the channel id. At this point all our channel id's should look like this:
<channel id="10021">
<display-name>G3701 AMC</display-name>
<display-name>G3701 AMC 4DTV:-</display-name>
<display-name>G3701</display-name>
<display-name>AMC</display-name>
<display-name>AMC</display-name>
<display-name>Satellite</display-name>
</channel>

One last change that isn't necessary, but it keeps mythfilldatabase from complaining later: delete everything after the last channel id block. That gets xmltv.xml down to just the channels.

3) mythfilldatabase --only-update-channels --do-channel-updates --do-not-filter-new-channels --file 1 xmltv.xml
This actually gets all the 4DTV channels into mythTV. Hooray! But we're not quite done yet. I noticed that even though the channel id's were in and good and it all worked, the display information was all weird. The next command seems to fix it:

4) mythfilldatabase --only-update-channels --do-channel-updates --do-not-filter-new-channels

Voila! It may not be pretty at all, but after all that (which doesn't really take long to do by hand) all your 4DTV channels should be in mythTV correctly, and better still, they are updated properly by mythfilldatabase! The only thing that doesn't happen automatically is the addition of new channels. Those will still be mysteriously ignored by mythfilldatabase when run automatically. But, all you have to do is hand-enter the hopefully few new channels, repeat the above process.

I don't pretend to know why this works, and I hope that this might provide a clue to the devs as to why 4DTV lineups just won't go into mythTV otherwise, and perhaps have an easy code fix.

[rmeden]

XMLTV requires the channel-id to be RFC2838 complaint so it can't begin with a number.

There is an "--old-chan-id" switch to give a different channel-id, but that's still not the TMS ID, Myth is looking for.

Here's your command with the substitution added. (BTW, tv_grab_na_dd has --list-channels which is similar to --days 0)

tv_grab_na_dd --dd-data dd_data.xml --days 0 | perl -pe "s/I(\d+).labs.zap2it.com/$1/;" >xmltv.xml

Robert

[LKO]

Thanks for the shortcut!
Ok, so evidently at some time Zap2it suddenly added the "I" and "labs.zap2it" stuff to the 4DTV data, which broke mythTV (because a couple years ago everything worked). Should SD modify the data, or mythfilldatabase get modified to handle the problem, or -- since there are so few 4DTV folk left -- should I just be happy with the workaround?

[rmeden]

Ok, so evidently at some time Zap2it suddenly added the "I" and "labs.zap2it" stuff to the 4DTV data, which broke mythTV (because a couple years ago everything worked). Should SD modify the data, or mythfilldatabase get modified to handle the problem, or -- since there are so few 4DTV folk left -- should I just be happy with the workaround?

Nope.. I'm the guilty party when I wrote tv_grab_na_dd for Zap2IT labs. (4+ years ago)

The XMLTV spec strongly encourages a RFC2838 style channel-id. I wanted to base it on the TMS channel-id, so added a "I" (not sure why that letter, but it couldn't start with a number) and the "labs.zap2it.com" domain. The old Zap2IT.com scraper used a channel-id based on the callsign and channel number, which --old-chan-id provides for compatability.

When the conversion to SchedulesDirect was done, there was no reason to change the channel id (and it would cause grief), so the labs.zap2it.com part was left in.

I don't think anyone suddenly changed the channel-id... tv_grab_na_dd has always done that. Maybe you used to use a native MythTV grabber (which didn't mangle the TMS-id), but now you're using XMLTV to avoid upgrading?

If we (SD) does self-host the data, I don't see us changing the channel-id.

Robert

[LKO]

Nope.. I'm the guilty party when I wrote tv_grab_na_dd for Zap2IT labs. (4+ years ago)

The XMLTV spec strongly encourages a RFC2838 style channel-id. I wanted to base it on the TMS channel-id, so added a "I" (not sure why that letter, but it couldn't start with a number) and the "labs.zap2it.com" domain. The old Zap2IT.com scraper used a channel-id based on the callsign and channel number, which --old-chan-id provides for compatability.

When the conversion to SchedulesDirect was done, there was no reason to change the channel id (and it would cause grief), so the labs.zap2it.com part was left in.

I don't think anyone suddenly changed the channel-id... tv_grab_na_dd has always done that. Maybe you used to use a native MythTV grabber (which didn't mangle the TMS-id), but now you're using XMLTV to avoid upgrading?

If we (SD) does self-host the data, I don't see us changing the channel-id.

Robert

Nah, it was actually when I switched to running SVN from v18 last year that I "discovered" the problem. v18 of mythTV evidently got the 4DTV channels in ok. It wasn't until I rebuilt the system and went to SVN that I discovered it wouldn't import the channels (well, technically it seems to import a random 9 or 10). I've been running SVN ever since. I only started fiddling with XMLTV to try to noodle out why mythTV didn't like my 4DTV channels. And the really nice thing about this workaround is that once I go through the XMLTV->mythTV process to import the channels, mythfilldatabase is happy and from then on works as it should. So, like I said, the only time I have to do the XMLTV thing again is if a bunch of channels get added. I think it even handles channels moving satellite-wise.

As you said, I'm probably ignorant of what changed and when, since I ran a long time on v18 of mythTV. I'm just happy now that I could get the channels in without painful grief, and maybe shed some light on what needs to be fixed to make "a more permanent solution"!