2021

family waving

Greetings earthling (Tera always makes fun of me for saying “greetings”). You’re here! You scanned our QR code… OR you actually typed out our link (how quaint)! Congratulations and welcome!

Travel travel, fo-favel

This year, we enjoyed getting back to some traveling, after being pretty restricted for most of 2020 (for obvious reasons).

Spring break in the George!

In March, we visited St George for the boys 3rd grade Spring Break, and had a great time at the Las Palmas resort. These little condos have all the essentials, and the property boasts multiple heated pools, water slides, hot tubs, and other outdoor activities.

We had a nice tour and a few hikes through Snow Canyon State Park, including some sand dune fun! Wes had a great time skateboarding at Ivins City skate park, which has some great street terrain.

Gamble town

In June, we headed to Las Vegas and spent a few weeks enjoying a house with a private pool, and all of our favorite Sin City vegan restaurants. We spent time at the Children’s museum to beat the heat, and jettisoned to the top of the Stratosphere to partake in their “top of the world” amusement park rides. Initially, the boys didn’t want to go on the vertical drop ride on the roof, but after we finished the ride, they wanted to go again!

We visited the Meow Wolf Omega Mart interactive art exhibit in Vegas. I loved it! Phenomenal blend of visual art, immersive surroundings, tactile variety and audio/visual experiences.

One of the funnest parts of Vegas was the escape room that we played through. It was a Pharoah’s tomb theme and we had a blast working together as a family, solving the puzzles, finding clues, and strategizing together.

Chowin down at True Food in Summerlin. Misters on the patio ftw!

Rollin’ on the river

Near the end of July, we spent a week at the Riverside Hotel in Boise, Idaho. It was the first time in Boise for most of the family, and we absolutely loved it! Boise was much nicer than I expected and the Riverside Hotel was a fantastic place to stay. The hotel is nestled immediately beside the Boise river, with walking and bicycle paths stretching for miles in both directions, alongside the river.

We toured the old Idaho prison site, we rafted down the river (Wes’ favorite part!), we bicycled almost every day, went skateboarding, hung out at the pool, and listened to live music nightly at the outdoor hotel bar, next to the pool. Surprisingly, Boise had quite a few vegan restaurant options to visit, and we had a good time exploring them.

Wave ridin’

In August, we visited Bear Lake with Papa and Nana, and had a great time on the lake. We stayed at the KOA in Garden City. We cruised the waters, played in the sand, the boys learned how to SUP (Stand Up Paddleboard), Archer (Papa and Nana’s Yorkshire terrier) explored the marshy weeds, we went tubing, and we shredded some waves with the wakeboard. Wes was stoked that he got his first ever air on a wakeboard.

Arizona dreamin’

In October, we drove down to Phoenix to visit Wes’ Mom, brothers, and niece Ava. This was our first time back to Phoenix in years and we enjoyed the visit, although the 11 hour drive from SLC is a bit much.

Separately, we also had a visit in SLC from our aunt’s Giogi and Amy, who live in Washington state. They bought a new RV! We were happy to see them and share a few meals together.

New job, who dis?

In August, Wes started a new job working at Lucidpress. In May, Lucidpress spun out from Lucid Software into its own independent entity. We’re a small company, with caring, motivated folks, who put teamwork over ego and are here to win. I am so grateful to get to work with wonderful people in a phenomenal environment. I have been so impressed with our people and the enormous opportunity in front of us. We are building something truly special here at our little company.

Farewell Pluralsight

The move to Lucidpress meant a departure from Pluralsight. I spent 6 and a half years building Pluralsight, and I was sad to leave so many wonderful coworkers and colleagues, with whom I have shared so many incredible experiences. I will always cherish the fond memories, the opportunities, and the many accomplishments that we made together there. Thank you to all of my coworkers, my team mates, my mentors… my friends. I miss working with so many of you!

Our keystone

Tera at Tacoterian in Las Vegas

Tera continues to hold everything together for our family. She is so organized, thoughtful, and hard working.

The family always has everything that we need thanks to her, including our beautiful, loving home. I have so much admiration for her attention to detail, and her dedication to ensuring that Ashton and Dylan have a wonderful childhood and home.

Our little sprouts

Dylan and Ashton continue to sprout like weeds and are now 56 inches and 57 inches, respectively. They’re almost as tall as Tera! These 10 year olds are just the greatest little dudes. We are so blessed to have the best kids in the world. They are kind, caring, funny, and great students. They continue to excel in their school work and are at the top of their class. Every one of their teachers tells us they’re such a pleasure to have as students.

They’re enjoying this Christmas season by making Gingerbread houses, opening 1 toy per day from their Lego Advent calendars (Star Wars and Harry Potter themed), and eating fresh snow when we get a good snowstorm.

The boys have gotten much better at skateboarding this year, thanks to Dad forcing them to go to the skateparks with him. Some of their favorite activities include reading, wrestling each other, playing Roblox, watching video game YouTubers, and creating things with Roblox Studio.

This year, we’ve been tearing through “The Land of Stories” book series, which Dad usually reads to the boys every night. We’re currently on the last book in the series and we will need to find something new soon! It is so engrained in our routine that the boys get upset if we miss one story time before bed at night.

Dylan continues to blossom as a writer. He is so creative and writes the most amazing stories and newspaper articles. We love reading his creations!

Ashton has jumped right into Roblox Studio and is teaching himself how to create games, game elements, shirts, and other Roblox items. It blew my mind to see him download Blender all on his own and start experimenting with it.

Thanks so much for reading about our family! Happy Holidays to you and we wish you the best this season and all of next year!

Scaling the modern data center

Written 2020-12-16.

I recently met with a group of passionate technologists and we had an intriguing conversation around the challenges that they currently face with operating and scaling their SaaS applications hosted within their own data centers. This company has provisions in many of their customer contracts that prohibit them from moving to the cloud.

They are having difficultly scaling and responding quickly enough to handle high burst workloads that inundate their backend infrastructure. The problems are exacerbated when their sales team inks a deal with a new large customer, resulting in significant increases to their user base and a corresponding increase in web traffic, on short notice. The load is generated from consumers using mobile and web applications.

As is the case with many complex issues, this is an interesting and multi-faceted problem that cannot be solved with any one course of action. Thankfully, constraints are the mother of creativity and innovation!

After thinking on this problem, I suggest putting significant effort into exploring the following strategies going forward:

  1. Performance Engineering
  2. Adding elasticity to the data centers
  3. Optimizing the data center supply chain
  4. Partnering with Legal to eliminate anti-cloud provisions on contract renewals
  5. Partnering with Sales to form clear agreements on capacity expansion

Let’s explore what I mean with each of these points.

Performance Engineering

When we started the discussion, it centered around the data centers specifically, and I did not immediately consider application performance. The team shared with me that they have been able to surgically rewrite key portions of their Ruby based application in Go, which has improved performance and allowed them to squeeze more out of their existing data center capacity.

This is brilliant! When you have a data center(s) with compute resource constraints, make sure that you are maximizing the resource efficiency of those limited assets. The team achieved significant performance improvements with this strategy. This result should not surprise you, as compiled languages generally exhibit notably increased performance characteristics compared to interpreted languages. In addition, the concurrency features built into the Go language may have helped them with achieving even greater efficiencies.

This strategy has already paid off for them, and there should be an emphasis on this continuing, but what else could we look at?

Are they using an APM (Application Performance Monitoring) product, which could help them shine a light onto the inner workings of each application? This could unveil bottlenecks and potential areas to target for further improvements.

Has a Content Delivery Network been implemented? This would ensure that their static assets are served at the edge, for optimal end user performance and would offload additional work from their backend servers.

Are they caching data sourced from expensive database queries by employing Redis, Memcached or a similar solution? Depending on data workloads, database types, query patterns, and other factors, database calls can be the most expensive transactions for SaaS applications. If you can eliminate some of the database queries via caching, you can often see significant performance improvements.

Caching is not the only lever for improving expensive data queries. There are many additional factors to consider in this space. Are there database indexes in place in all the appropriate places? Is the current database solution the most optimal one, based on the data shape, query patterns and load? Has a NoSQL solution been explored for query intensive use cases where low latency reads under heavy traffic is of utmost importance?

I would also look at the application architecture and the division of compute labor within the product. Is everything highly coupled together, requiring that every piece of the architecture be scaled, regardless of the discreet need? If the answer is yes, then a decomposition of the application might be beneficial, to allow for independent scaling of just the pieces that require additional computing power under increased load.

Additionally, load testing in a non-production environment can unveil weak points, assist with forecasting capacity capabilities, and help to inform future architectural needs.

Adding elasticity to the data centers

One of the biggest issues with running your own data centers is that your capacity is mostly static. It can take weeks, or even months to acquire new hardware, install it, wire it up, configure it, and finally put it into service.

The cloud revolutionized our industry by providing resource elasticity. If you have to run your own data centers, it is imperative that you add elasticity to your compute layer.

What do I mean by elasticity? Creating your own PaaS (Platform as a Service) based on Kubernetes is one compelling option. With unpredictable workloads and bursting traffic, you won’t have a good time waiting in the range of 5-10 minutes for VM’s to launch and enter service. Compare that to Kubernetes HPA (Horizontal Pod Autoscaling) capabilities, where you can horizontally scale out your applications in milliseconds.

Kubernetes isn’t the only solution, of course, but it’s difficult to ignore the vast mind-share, open source community support, CNCF projects, and numerous other resources that are dedicated to its ecosystem.

Additionally, are the data centers being managed and operated using a Software Defined Data Center (SDDC) model? While this term is heavily used in marketing materials and should set off your buzzword alarms, the underlying concept of managing all (or as much as you can) of your data center resources with software has validity. Leveraging this strategy can increase the time to provision or reallocate resources in your data center, leading to greater elasticity.

Optimizing the data center supply chain

Having more customers than you can handle is a great problem to have. In the context of data center capacity, some of the biggest issues revolve around the fact that it is often slow to add capacity.

This is a classic supply chain issue. Don’t settle for “this is how it’s always been done”. If this is your business’ bottleneck, then you should be fanatical about pushing the limits, exploring new innovations, talking with many suppliers, renegotiating contracts and anything else you can do to start chipping away at your expansion timelines.

If you don’t know how long it takes to increase capacity now, the first step is measuring the entire process, from the start of when new resources are needed, to the very end point at which you have these new resources in service. Once you have that baseline, get the team involved in brainstorming how to reduce those timelines, devise experiments to undertake, and get to work on reducing the cycle time.

Minting a new role within the organization for a dedicated project manager, tasked with data center provisioning optimization, is another option that could be considered.

Partnering with Legal to eliminate anti-cloud provisions on contract renewals

As mentioned in the introduction, this company has a “no cloud” provision written into many of their customer contracts. This was commonplace and expected 5-10 years ago, but in today’s business landscape, even the world’s largest banks and financial institutions are undergoing massive cloud transformations, shifting from their own home grown data centers to the public cloud.

It’s important to listen to your customers and give them what they want, but this can be a delicate balance in certain circumstances. On some things, you have to push back. I believe this is one of those areas where significant effort should be put in place to attempt to eliminate the anti-cloud provisions upon contract renewals. I expect this effort to be difficult, and that some customers would not agree.

How do you help make your customers more comfortable with a cloud solution? Highlight specific practices and safeguards that you will guarantee when moving them to the cloud. Discuss encryption at rest, AWS Well Architected Framework adherence, Cloud Governance processes, Security Incident Response processes, routine Pen testing, and other practices that help mature organizations ensure that their infrastructure and customer data is secured in the cloud.

The messy middle is when some of your customers agree to use cloud backed services and when others are adamant about avoiding it. In this case, I have often seen product capabilities and features diverge, where cloud customers get more features and different product capabilities, more quickly, compared to their on premise peer customers.

This is a delicate strategy and should be carefully considered with intention, before putting into practice. If this is the path taken, then the cloud hosted version of your services offering more features is one carrot that can be used to entice your legacy customers to shift over.

Partnering with Sales to form clear agreements on capacity expansion

Speaking with the team, it was clear that one of the pain points of the data center capacity expansion was missing Agreements (yes, capital A) between the Sales teams and the Engineering teams.

It is critical to get Data Center Engineering and Sales on the same page. There should be no ambiguity. If Sales is pursuing a deal to add 10 million new customers to the platform, then there should be clear expectations on how long it will take the Data Center team to expand capacity to be able to serve that new user base.

A crude example; if Data Center Engineering needs 3 weeks to add capacity to meet 10 million new customers, then Sales should set expectations with the customer that they will not be able to onboard until 4 weeks after the deal is signed. Leaving some room for error is always advisable, so that you can meet or exceed your customer expectations.

Conclusion

Operating a SaaS application out of your own data centers, pressured by tremendous growth, is a challenging endeavor. There are many nuanced considerations to be made in this problem space, and it is imperative that you get creative and take a look at all possible strategic approaches while weighing the pros and cons of each direction.

No one has all the answers, and the specific strategies employed will be dependent on the context of the business, its leaders, and their particular preferences. It’s important to brainstorm with your teams on this, hear out all the ideas, run experiments, iterate, and focus on continual improvement.