Why zip files?

Why zip files?

Postby cwchapma » Sat Mar 30, 2013 12:03 pm

I'm working through trying to get the data out of zip files. I'm getting there but it's not easy and somewhat inconsistent. Before I go requesting changes, it would probably be good if I understood the reasoning behind the zip files (as opposed to just giving the json directly). Is it a bandwidth, cost, caching issue? It seems like a lot of complication and I'm just not understanding the benefit. I would have thought you could force gzip compression and let the server take care of caching. I don't know for a fact that is possible - it's just how I would have expected it to work.

Hopefully someone can enlighten me :)

Thanks,
Clint
cwchapma
 
Posts: 42
Joined: Fri Mar 29, 2013 8:32 am

Re: Why zip files?

Postby Slugger » Mon Apr 01, 2013 11:40 am

Robert tried sending just the json, but building the json on demand proved to be too much for the server. I was around for that testing, it didn't go well. So instead, the json is prebuilt and stored in the db for each object and then when you need a certain set of objects you get them all zipped up in a single file. Joining all the objects into one single object proved to be equivalent to building the 'super' object at runtime, and didn't go any better. So the best/cheapest/effecient/CPU friendly/RAM friendly way to get a set of objects for a request is to zip up all the individual objects and send the zip file.

The other issue with presenting a single 'super' json object is that it's memory intensive for the client. With a larger lineup (DirecTV, Dish, larger cable cos), a single lineup could present a 200-300MB json structure. Obviously, that would be compressed to send it across the wire so maybe it's only a 10MB or so download, but you still need to store that string and/or its parsed native data structure in memory and process it, which in some envs (like Java), you would end up blowing a large chunk, if not most, of your available heap space just storing the json object. With the multiple object approach, you process tiny objects and they are quickly garbage collected when they're no longer needed.
Slugger
 
Posts: 77
Joined: Sun Sep 18, 2011 1:22 pm

Re: Why zip files?

Postby cwchapma » Mon Apr 01, 2013 11:56 am

Thanks slugger. That makes sense to me now. It helps a lot knowing why those decisions were made and keeps me from making a bunch of silly suggestions.

Clint
cwchapma
 
Posts: 42
Joined: Fri Mar 29, 2013 8:32 am


Return to Developer

Who is online

Users browsing this forum: No registered users and 2 guests