Analysing gamer data is vital for any online game developer, especially for data-driven organisations such as Jelly Button, as Liad Livne, BI Manager, explains. “We analyse user behaviour and try to improve on KPIs, such as retention and average revenue per user, by assessing which offers and promotions to suggest to players. We use the same data to improve our games, focusing on technical issues and difficulty levels.” Adds Nadav, “To do that, we would execute an hourly and a daily ETL, extracting raw data from our business analytics service and loading it into Google BigQuery to access the information.” Unfortunately, running costs for the business analytics service part of the process became disproportionately high as games went viral, so the Jelly Button team began to talk with the cloud experts at DoIT International about building something new.

Using the Google Reference Architecture for Mobile Gaming Analytics as a baseline, Jelly Button began streaming data directly into Google BigQuery from its clients. “We considered several alternatives and this was the best one, both in terms of cost and in terms of technology,” says Nadav. Autoscaling the solution with Google Kubernetes Engine, the team set up Google Cloud Pub/Sub to relay events with low latency through to Google Cloud Dataflow, which filters, maps, and aggregates the raw data for Google BigQuery, making the data available for analysis in near real time. Because the data and analytics on each game is stored in a separate Google BigQuery project in a modular pipeline, managing the process is simple for administrators.

“Licenses for our previous analytics service were expiring,” says Nadav. “So we had to work to a tight deadline. DoIT International provided us with its best talent to launch this project in just under five weeks from start to production, with weekly meetings to review progress. It’s the latest in several successful initiatives with DoIT International, including cost optimisation and a cloud security review.”

Beyond speed and savings, the new pipeline offers greatly improved stability. “In terms of ETL data streaming, we now have the big advantage that we can keep all of our data within Jelly, instead of on a third party,” says Liad. “In the past we had problems with APIs, which meant that we couldn’t extract data and experienced days of shutdown without any communication.” Shutdowns would occur around once a month, according to the team, seriously affecting the company’s ability to respond to issues. “If we don’t have that data, we cannot react to gamer complaints, or bugs, or alerts. We’re really in the dark.”

“The new pipeline is much more stable because we have all the data within our own systems,” says Liad. “We tested it within the game by shutting down the process and waiting for 24 hours, then enabling it again. All the data was queuing within the accumulator and was recovered with no data loss whatsoever.”