View the original community article here
Last tested: Dec 19, 2017
The Looker API can stream to a client, but the client cannot control what size each chunk is and if they are not using the Ruby SDK, they will have to write their own code to handle streaming. We have examples for streaming using the Ruby SDK.
"Streaming" can mean different things in different contexts. For video playback, streaming usually means the ability to view the video as it is being downloaded from a server, with no requirement to save to disk first. This also requires that the data be consumed linearly from start to finish, with no "seeking" to a different position in the data, forward or back.
Internal to the Looker instance, we refer to streaming as being able to pass data through the Looker instance with little or no buffering or post-processing - just relaying the data from the db to the API caller. (See "Streaming")
For the API caller, streaming usually means being able to receive large data in a series of small chunks, so that the client app can process any amount of data with very little memory use.
Streaming is primarily driven by how the client application retrieves the data from an HTTP response. Most of the time, a client app just deserializes the response.body json text into an in-memory object. This requires all the data to reside in memory in the client application. For small objects this is fine but for very large data, this will be a problem.
If instead the client app processes the http response body as a series of chunks of bytes, then the app can handle arbitrarily large data responses with very limited memory consumption - one chunk at a time.
Every Looker API is capable of streaming its HTTP response. The API client application decides whether to retrieve all the data from the HTTP response into memory, or to retrieve and process the data chunk by chunk.
In order to stream using the official Ruby SDK, you can use this guide.
We have a guide in https://github.com/looker-open-source/sdk-codegen on how to stream from generated SDKs or normal HTTP requests.
Sources
Streaming downloads Github how-to: https://github.com/looker/looker-sdk-ruby/blob/master/streaming.md
This content is subject to limited support.