DataDirect replacement service - BETA 1

Discussion about Schedules Direct grabber code and data formats.

Re: DataDirect replacement service - BETA 1

Postby rmeden » Mon Oct 06, 2014 6:30 am

I'm pretty sure now the SD-DD service (Schedules Direct - Data Direct) effectively has only 13 days. Not the end of the world.

Robert
rmeden
SD Board Member
 
Posts: 1491
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: DataDirect replacement service - BETA 1

Postby bill6502 » Mon Oct 06, 2014 12:05 pm

aircarver wrote:...
Result was the grabber reported success, but the number of days of data incremented down each day.
...

That's one of the classic signatures of a stale file in /tmp. Most frequently caused by running
mythfilldatabase as a user that doesn't have write permission on that file. E.g running manually
as root and then when mythbackend (running as user mythtv) starts mythfilldatabase, it appears
to run OK, but the new data is never written to the /tmp file.

Type: ls -l /tmp/mythtv_ddp_data . Remove the file if it exists. If that doesn't help, you should
pastebin your mythfilldatabase log and put a link to the pastebin here. Edit, removed space
before mythtv_ddp_data.
Last edited by bill6502 on Tue Oct 07, 2014 4:55 pm, edited 1 time in total.
bill6502
 
Posts: 16
Joined: Fri Sep 19, 2014 4:52 pm

Re: DataDirect replacement service - BETA 1

Postby aircarver » Tue Oct 07, 2014 1:56 pm

bill6502 wrote:
aircarver wrote:...
Result was the grabber reported success, but the number of days of data incremented down each day.
...

That's one of the classic signatures of a stale file in /tmp. Most frequently caused by running
mythfilldatabase as a user that doesn't have write permission on that file. E.g running manually
as root and then when mythbackend (running as user mythtv) starts mythfilldatabase, it appears
to run OK, but the new data is never written to the /tmp file.

Type: ls -l /tmp/ mythtv_ddp_data . Remove the file if it exists. If that doesn't help, you should
pastebin your mythfilldatabase log and put a link to the pastebin here.


I got:
greg@greg-IBM-CORPORATION:~$ ls -l /tmp/ mythtv_ddp_data
ls: cannot access mythtv_ddp_data: No such file or directory
/tmp/:
total 2420
drwx------ 2 greg greg 4096 Sep 4 11:23 keyring-hHhu1H
-rw-rw-r-- 1 greg greg 2429216 Oct 5 14:40 mythtv_ddp_data
drwx------ 2 greg greg 4096 Sep 4 11:23 ssh-xJafAZVX1574
drwxr-xr-x 2 mythtv mythtv 36864 Oct 7 08:03 tvdb_api_110


I believe they are working now (at 13 days max) after I changed to the current IP from the early one that was listed previously.
aircarver
 
Posts: 3
Joined: Sun Oct 05, 2014 1:04 pm

Re: DataDirect replacement service - BETA 1

Postby bill6502 » Tue Oct 07, 2014 5:04 pm

aircarver wrote:...
I got:
...
greg@greg-IBM-CORPORATION:~$ ls -l /tmp/mythtv_ddp_data
-rw-rw-r-- 1 greg greg 2429216 Oct 5 14:40 mythtv_ddp_data
...


My mistake, there was a space before the mythtv_ddp_data file name. Since that file is
owned by greg, if some other user (e.g. mythtv), without write permission on the file, tries
to use it, then it will fail. But sounds like you're fine now. I'd remove the file.
bill6502
 
Posts: 16
Joined: Fri Sep 19, 2014 4:52 pm

Re: DataDirect replacement service - BETA 1

Postby johnsonx » Fri Oct 10, 2014 9:21 pm

hey all...

Tonight I tried testing the DataDirect replacement service with NextPVR v3.2.9 on behalf of those users who for whatever reason won't upgrade to NextPVR 3.3.x when it's released later this month, which I can already say works just spiffy with the new JSON service. Hopefully most users will upgrade, but there's always a certain percentage that lag behind quite awhile... there's even a handful of GB-PVR users still around.

First I couldn't get any response from the IP posted in this thread, but some minor sleuthing led me to 54.210.29.17. At first that didn't seem to work either, but I finally got a response after nearly 2 minutes and it did indeed populate my EPG. My lineup has only 7 channels, I don't know if a more normal sized lineup would take even longer...

The problem is that when I attempted to download a new or updated lineup in the NextPVR Settings->Channels screen, the program times-out before it gets a response (it seemed like it waited 90 seconds), so as it stands now there's no way to complete the initial setup of channels in NextPVR (or add new channels) using the DD replacement service. I got my test setup to work only by completing the initial setup using the TMS server, then switching to your DD server to actually populate the EPG.

May it be assumed that before the DD replacement service goes fully live, it will be much faster?
johnsonx
 
Posts: 4
Joined: Fri Oct 10, 2014 9:03 pm

Re: DataDirect replacement service - BETA 1

Postby rmeden » Sat Oct 11, 2014 11:04 am

wgmaker wrote:Our SD client based on this DataDirect replacement service is ready. Is it OK to publish it ? If not could you give a date?
regards Jan (WebGrab+Plus client)


Yes, ok to push it out!

BTW, static hostname users, it seems Amazon changes the load balancer address quite often. Please use this address now: 54.85.117.227 (parent post updated).

If it's determined we need to use a load balancer, will just run our own instead of using Amazon's.
rmeden
SD Board Member
 
Posts: 1491
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: DataDirect replacement service - BETA 1

Postby johnsonx » Sat Oct 11, 2014 1:37 pm

ok, using the new virtual IP 54.85.117.227 I'm now getting very quick response, about 4 seconds, when downloading the lineup(s) in NextPVR, so the problem I reported with it timing-out before the lineups could be downloaded is solved.

The listings download itself though still takes quite awhile, about 77 seconds:
Code: Select all
2014-10-11 13:27:45.032 [DEBUG][7]      Processing ScedulesDirect LISTINGS
2014-10-11 13:27:45.032 [DEBUG][7]      Requesting listings for the period: 2014-10-11T00:00:00Z to 2014-10-19T23:59:59Z
2014-10-11 13:29:02.653 [DEBUG][7]      Got SchedulesDirect response
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Vary=Accept-Encoding
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Connection=close
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Content-Encoding=gzip
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Content-Length=4268439
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Content-Type=text/xml;charset=utf-8
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Date=Sat, 11 Oct 2014 20:28:18 GMT
2014-10-11 13:29:02.653 [DEBUG][7]      Response header: Server=Apache/2.2.15 (Red Hat)
2014-10-11 13:29:02.653 [DEBUG][7]      Received compressed stream from SchedulesDirect

However, it is working.
johnsonx
 
Posts: 4
Joined: Fri Oct 10, 2014 9:03 pm

Re: DataDirect replacement service - BETA 1

Postby rmeden » Sat Oct 11, 2014 8:58 pm

How long does it take with TMS-DD?

Most of the folks I've heard from say SD-DD is faster.

Robert
rmeden
SD Board Member
 
Posts: 1491
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

Re: DataDirect replacement service - BETA 1

Postby johnsonx » Sun Oct 12, 2014 10:36 am

Downloading my EPG from TMS takes just over 6 seconds:
Code: Select all
2014-10-12 10:20:29.216   [DEBUG][8]   Processing ScedulesDirect LISTINGS
2014-10-12 10:20:29.216   [DEBUG][8]   Requesting listings for the period: 2014-10-12T00:00:00Z to 2014-10-20T23:59:59Z
2014-10-12 10:20:35.403   [DEBUG][8]   Got SchedulesDirect response
2014-10-12 10:20:35.403   [DEBUG][8]   Response header: Connection=close
2014-10-12 10:20:35.403   [DEBUG][8]   Response header: Content-Encoding=gzip
2014-10-12 10:20:35.403   [DEBUG][8]   Response header: Content-Type=text/xml;charset=utf-8
2014-10-12 10:20:35.403   [DEBUG][8]   Response header: Date=Sun, 12 Oct 2014 17:20:30 GMT
2014-10-12 10:20:35.403   [DEBUG][8]   Response header: Server=Apache-Coyote/1.1
2014-10-12 10:20:35.403   [DEBUG][8]   Response header: X-Powered-By=Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
2014-10-12 10:20:35.403   [DEBUG][8]   Received compressed stream from SchedulesDirect


Not knowing how it's all working behind the scenes, I could imagine that SD-DD has some initial semi-fixed processing delay that makes my small lineup take longer, but it would catch up on a bigger lineup and eventually surpass TMS?

In any case, the only reason I registered and made my initial post was because downloading the lineup(s) was taking so long that NextPVR.exe was timing out, so it wasn't possible to complete initial setup or add new channels with the SD-DD service; now that problem has gone away (that download now only takes about 3 seconds), so I really have no complaint at this point. SD-DD works fine as-is for existing NextPVR v3.2.9 and prior users, and most users will upgrade to NextPVR 3.3.x and use the new JSON service anyway. All good in my book.

By the way, here's the log of the lineup download from SD-DD, the part that used to take so long that it timed out:
Code: Select all
2014-10-12 10:32:51.450   [DEBUG][1]   Using NetworkCredential(******, *******)  ===egads! sub logged my SD username and password in clear text!===
2014-10-12 10:32:53.559   [DEBUG][1]   Got SchedulesDirect response
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Vary=Accept-Encoding
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Connection=close
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Content-Encoding=gzip
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Content-Length=49947
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Content-Type=text/xml; charset=utf-8
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Date=Sun, 12 Oct 2014 17:33:24 GMT
2014-10-12 10:32:53.559   [DEBUG][1]   Response header: Server=Apache/2.2.15 (Red Hat)
2014-10-12 10:32:53.559   [DEBUG][1]   Received compressed stream from SchedulesDirect
johnsonx
 
Posts: 4
Joined: Fri Oct 10, 2014 9:03 pm

Re: DataDirect replacement service - BETA 1

Postby rmeden » Sun Oct 12, 2014 10:29 pm

Not sure if it was the problem, but the very first download takes longer because we have to get your lineups from TMS.
rmeden
SD Board Member
 
Posts: 1491
Joined: Tue Aug 14, 2007 2:31 pm
Location: Cedar Hill, TX

PreviousNext

Return to Developers Corner

Who is online

Users browsing this forum: No registered users and 3 guests

cron