My Favorite BigQuery Moments
After 7.5 incredible years, I recently announced my departure from Google. It’s been a fun, rewarding, yet challenging ride… I’ve been blessed to have helped grow BigQuery into a leader in a dynamic rapidly expanding CDW market..
I feel like reminiscing about the good ol’ days and sharing my favorite BigQuery stories (being mindful of confidentiality). I hope you enjoy the banter. Consider this a virtual beer with me — a suitable replacement during the pandemic!
Favorite Customer Engagement: Yahoo
Around 2016 Yahoo had a very big big data problem. Gobs of it. They tried everything, but nothing worked. That is, until their friends recommended BigQuery.. And then that didn’t work either…
Speed, size and shape of the data was generating so many files, that the resulting metadata was dragging performance down. I knew how important this prospect was to our business, so I mediated between Nikhil Mishra, Yahoo’s chief architect, and BigQuery’s world-class engineers to resolve this problem. The team also recognized that this is bound to be a recurring pattern, so a solution would have positively impacted a broad set of customers.
We pulled out all the stops. I asked, begged, and even demanded relevant engineers to reroute their effort to figure this one out. We flew down to Sunnyvale to pair-troubleshoot with Nikhil. We shut down an entire data center to re-process poorly-materialized data. And I was beginning to get a reputation as the guy who short-circuits the process and randomizes engineers — I didn’t care…This was the most important thing for BigQuery at that moment.
The backend improvement was quarters away from being fully operationalized. However, we could trigger a custom optimization manually, which we scheduled for Yahoo. For the next 6 months Nikhil would periodically call me to see if we can run some more background processes to clean up his datasets. At some point Nikhil joked that all he needed to do to make BigQuery fly was to call Tino (which was very marginally true)!
Watch Nikhil talk about his journey here, or read about it.
What Yahoo taught us: Sometimes close collaboration with a key prospect is so important that you need to stop EVERYTHING to make that prospect succeed.
Honorable Mentions: Discord, Home Depot, Viant, Kabam, Twitter, NY Times, Snap, Plaid, HSBC, King, Disney, Demandbase, Spotify
Most Impactful Product Moment: Flat-Rate
In 2015 BigQuery was doing just swell — all the business and usage metrics were skyrocketing, the CDW market was beginning to take shape, and we were starting to get recognition for the quality of our product. However, we had a major problem. We couldn’t sell a darn thing to ANY serious Enterprise.
Why? In a world dominated by appliances, CAPEX, and subscriptions we were selling a pure consumption-based model. It was and still is the most efficient pricing model for data warehouses, but it is absolutely not predictable and totally unbounded. Companies would ask us to approximate spend, to play a flat contract, and to guarantee against runoffs. We had no good answers.
Then after lots and lots of brainstorming sessions someone (I think Jordan Tigani) had an idea — why not package storage and compute together for the price of storage alone? You pay for storage, and all your queries are free, bound by performance characteristics of associated compute. I wrote up a proposal and named it “Free Willy”. You see, just like in the 1993 cult classic, we were going to help the whales (Enterprises) escape the little data pond and live free in the data ocean (BigQuery). I am not sure I have a future in marketing.
A GTM tiger team quickly spun up, and we launched the program privately in a mere three months. Will Grannis gave us his session during the sales kickoff, we announced it, and we were off. We iterated more, and less than a year after ideation, BigQuery flat-rate was launched publicly. In the following years we up-leveled flat-rate with Reservations, Flex Slots, and a slew of other landings. Nowadays, virtually every BigQuery pricing discussion begins with flat-rate.
This was a crazy effort, but it would not have happened without unconditional support from some of the best leaders I’ve been lucky to work with — Greg Czajkowski (now SVP@SNOW), Jim Caputo (now VP@SoFi), Will Grannis (GCP OCTO), and Jordan Tigani (now CPO@SingleStore). These folks know how to positively elevate their organizations to execute on innovative yet outlandish ideas.
What flat-rate taught us: No matter how innovative you are, listen to the market. Your prospects will tell you what you need to do to convert them to paying customers.
Honorable Mentions: Reservations, Flex Slots, Standard SQL, DML, Federated Parquet, BQML, Materialized Views
Best Team Moment: The Hawaii Death-Trip
Success of flat-rate led to us winning an all-expenses-paid trip to Hawaii. During the trip Tess, Greg’s awesome longtime EA, planned a “fun team activity” to take a boat tour to see red hot lava flow into the ocean.
We were a little concerned that morning when ocean swells reached ten feet, expecting the tour to be canceled. We persevered and arrived at the designated meeting spot — a desolate gravel parking lot half a mile from shore. This made little sense: this was a boat tour after all! We should have read the warning signs….
Then the boat pulled up, on top of a makeshift platform, towed by a huge truck. The captain propped up a painters’ ladder against the side of the boat and yelled “Get In, we goin’!”. This was our last chance to escape, yet no one dared to confront the authority…
So naturally we all got in, and the boat was taken to the shore and launched off the truck. In a near instant the four 250 engines sprung to life and bolted us above the huge waves. Over the next hour we were horrified by the constant jolting of the craft going over ten footers, often getting flung into the air to momentarily plop back down on the hard cold steel benches.
While everyone aboard suffered quietly, Jordan’s wife Tegan announced every little bump and toss by producing what can only be described as heinous panic mixed with extreme exhilaration. That made us all laugh. More importantly, Tegan distracted us all from the reality of the situation.
We arrived at the lava, and I wished we didn’t… For what seemed like an eternity the captain navigated us tantalizingly close to the lava flows. We could feel the heat and smell the sulphur. Solid hot rocks floated by us. Ocean boiled. Underwear soiled. And the captain continuously navigated the ship to within mere feet of what seemed like horrid death by lava.
To this day every time I see Tess, she apologizes profusely for almost killing us all… as she should!
Naturally, in 2018 this particular touring company was sued for… you guessed it.. lava-related injuries sustained by passengers.
Lesson Learned: Maintain a healthy skepticism around any “fun team building activities” — you may not survive..
Encore…
Some other fun moments to mention:
Spotify. When Spotify announced that they picked Google Cloud as their infrastructure partner, Nic Harteau called out Google’s data prowess as the chief reason for the decision. This sent shockwaves through GCP leadership — Spotify caused Google to reluctantly realize that BigQuery is THE tip of the spear.
OCTO. I was blessed to have been just the second member of the Office of CTO, founded by the amazing Will Grannis and Brian Stevens. OCTO brought together a team of world-class experts and liberated them to innovate through customers. I departed because I just had a newborn and needed less travel, but not a day goes by that I don’t miss that team.. To this day I call many of them friends and some mentors…
BigQuery is….Serverless?. In 2016 I wrote a quick one-pager, proposing that BigQuery co-opt the relatively-nascent “serverless” buzzword. The idea was met with mixed signals, but was shared broadly across the organization. However, two months later Urs Holzle at the Google NEXT keynote announced BigQuery as the world’s only serverless data warehouse, and the rest is history.
Halloween-themed Parquet launch. After a while I established a strong cadence with the four horsewomen of landings — Kathy (PR), Stephanie (Analysts), Christine (Blog), and Anita (Marketing). They got really good at landing features, and we started to have some fun. My favorite outcome of our collaboration was the Halloween-themed launch of Parquet and ORC on BigQuery. What a howler!
