We have just released a major update of MyBillingExplorer, - our software for collecting, analyzing and optimizing Google Cloud Platform costs used by hundreds of our customers.

It's a big release for us as we completely revamped our backend, allowing us to refresh the billing data every 15 minutes (instead of 24 hours in previous version) and basically rewriting the whole application from ground up providing a solid foundation for future features like cost optimization recommendations and more.

There are literally few dozens of bug fixes as well as new features, most notably:
 - Improved user experience throughout the app
 - Much faster user interface (±500% performance improvement over the last version)
 - Mobile friendly UI
 - The UI now automatically fetches new data without the need to refresh the page (thanks, Firebase!)
 - Pivot Builder now shows 6 months of billing history (instead of 3 months)
 - Billing prediction for current month
 - 100% SSL (with automatic redirect from HTTP)
 - Support for Google Commitment Discounts
 - Multiple Billing Accounts per customer's account
 - Integration with Geckoboard for those who want to publish billing data to a TV dashboard

New Dashboard View

New Dashboard View

Breakdown by Product/Project Widget

Breakdown by Product/Project Widget

We are continuing to work on new features, bringing more value to our customers. Some of the things we are working on are: customer classification and clustering, cost optimization recommendations, mobile app for iOS and Android and more.

I'd like to thank my team, who tirelessly worked on this release. Well done, guys!

Posted
AuthorVadim Solovey

Recently, I came across an interesting benchmark of BigData systems based on "A Comparison of Approaches to Large-Scale Data Analysis" by Pavlo et al. (SIGMOD 2009). Based on the benchmark methodology, the APMLab guys from Berkeley University developed an open-source software that allows anyone to run this benchmark using public cloud (AWS in this case).

This benchmark measures response time on a handful of relational queries: scans, aggregations, and joins across different data sizes. They have an impressive website with benchmark results, comparing Amazon RedshiftHiveSharkImpala and Stinger/Tez.

Since a lot of my time is devoted to working with Google BigQuery, I was intrigued how BigQuery would stand being compared to other solutions based on exactly the same dataset. Clearly, BigQuery has a very different nature since it's shared service and not dedicated, customer-deployed (and maintained) solution.

Posted
AuthorVadim Solovey
Categorieshomepage
11 CommentsPost a comment

Want to know how much data is in your BigQuery project? Thanks to bqdu.info it's easy and fast.

Inspired by Linux's 'du', I created a small service which does exactly what 'du' does for folders but only for Google BigQuery's projects, datasets and tables. BQdu will scan your project/s and display a treemap visualization of your project, datasets and tables.

Posted
AuthorVadim Solovey
Categoriescloud, homepage

Google is rolling out new version of UI for Google Drive (which I like a lot so far). The only 'issue' for me so far has been the inability to change density of the layout. The 'old' version of Drive had various options to control density - you could choose from Comfortable/Cozy/Compact layouts. The new Google Drive UI doesn't seem to have this kind of setting. 

Luckily, I was able to find that pressing 'qq' while in Google Drive UI changes the layout density very similarly to what we had in the previous version of UI. 

Posted
AuthorVadim Solovey
2 CommentsPost a comment

Ubuntu is a popular choice of Linux distribution among developers and DevOps people, especially due to long-term-support (LTS) and package availability.

At the present, Google Compute Engine doesn't have any of Ubuntu images, which gives us the joy of cooking the image by ourselves - thanks to Custom Kernels functionality of GCE. Please note this post represents just the general idea of how to make your own Ubuntu image on GCE, your actual deployment may vary.

Posted
AuthorDoron Offir
Categorieshomepage
9 CommentsPost a comment

Sometimes you get to the point that we need to reassign the instance on GCE with new external IP address. The cause can be varying starting for security and goes up other reasons. I've decided to summarize the process of re-assigning external IP address on GCE instance.

But first couple of side notes:

1) As GCE instance can have multiple IP addresses to get connections from using IP forwarding (see previous post of: How to assign multiple external ip addresses to instance in gce), at this point in time you can only assign a single external IP address for outgoing connections.

2) During the IP assignment your external connectivity to the server will be down

So how we do it?

Step 1:
Make sure all needed services and running process who needs external connectivity can stand a small downtime of you network interface. 

Step 2: 
Delete ephemeral/static external IP address from the instance by running the following command

$ gcutil --project=<project_name> deleteaccessconfig <your_instance> --access_config_name="External NAT" --network_interface_name=nic0

Step 3a (for ephemeral): 
Assign an ephemeral external IP address to your instance by running the following command:

$ gcutil --project=<project_name> addaccessconfig <your_instance> --access_config_name="External NAT" --access_config_nat_ip=ephemeral --access_config_type=ONE_TO_ONE_NAT --network_interface_name=nic0

Step 3b (for static): 
Assign a static external IP address to your instance (given that you have already reserved it, say 100.200.30.40), run the following command

$ gcutil --project=<project_name> addaccessconfig <your_instance> --access_config_name="External NAT" --access_config_nat_ip=100.200.30.40 --access_config_type=ONE_TO_ONE_NAT --network_interface_name=nic0 

The process won't take more than 10 seconds, and you'll be assigned with a brand new IP address 

/ Dave

Posted
AuthorVadim Solovey

Today, Google is making it easier for everyone to run Hadoop jobs directly against data in Google BigQuery and  with the Preview release of Google BigQuery connector and Google Cloud Datastore connector for Hadoop. 

The Google BigQuery and Google Cloud Datastore connectors implement Hadoop’s InputFormat and OutputFormat interfaces for accessing data. These two connectors complement the existing Google Cloud Storage connector for Hadoop, which implements the Hadoop Distributed File System interface for accessing data in Google Cloud Storage.

Posted
AuthorVadim Solovey
Categorieshomepage
Tagshadoop

Sometimes you need to assign more than a single external IP address to an instance running on Google Compute Engine. It might be when you are deploying a load balancer service and you'd like to configure multiple IP addresses for different services, so you'll have a better control over where and how your traffic goes within the farm.

Posted
AuthorVadim Solovey
Categorieshomepage