👩‍🎨 Being "rockstars": when software was a talents/creatives industry 🎭

🎵 The song for this post is Torn, by Natalie Imbruglia. 🎵

On vacation, visiting Guatemala for the first time in ~5 years 😍 Finishing this post like dis.

What was behind the flames of Where have all the hackers gone?? The biggest one I'm conscious of is Kellan's Software and its Discontents series. It's an excellent set of articles that try to provide answers to the question "why is everyone so miserable about software these days?"

I really loved it, but I felt it didn't articulate the overarching narrative I observed behind all the glum feelings: application software is no longer a game of "creatives" and "talents," it's now a "commodified sludge" version with a playbook on how to produce it. This playbook describes a game that's a) not fun or motivating for most of the people involved, b) based on risk management (fear) rather than outcome maximization (hope), and c)… still doesn't seem to work, actually? But everyone yells at you and calls you immature or Not Serious if you deviate?

The labor/hiring market was very slow to catch up to this, so for a while, remuneration and narrative were about creatives/hackers, but the work and outcomes were conservative and managerial. Software was Taylor Swift-priced but "uncredited studio musician"-outcomed.

For many people working in software, this gets incredibly demotivating, but also, of particular interest to me: when we overreact and play to a playbook too much, we prevent any Taylor Swifts from ascending, since we decide to work with mediocre tools and "lowest common denominator" teams. Getting back to "that magic feeling" comes from reaching for the ceiling and maximizing your people, rather than flattening them, as scary as that may be.

This post describes what I mean by a "creative"/"talents" industry, argues that moving away from this model made software more miserable for everybody, and that maybe we need to reconsider the playbook that represents that move, at least in part.

In 2009, I bought this book, because when you walked into a codebase, it had a style. You had something of a "code handwriting." Now we have black and gofmt.

"Do they have "the juice""?!

Something you see a lot in these industries: scouts and producers looking for talent that delivers, and using their "gut" to find it, because it's not obvious on inspection what's going to work. These industries are… colorful. An example is entertainment: if you were a music executive who wanted to cash in on grunge in the 90's, you could try to create a new grunge band with professional musicians, or you could have talent scouts finding teenagers in garages.

One of these is strictly the better option: there's no formula to create successful grunge; some bands have "the juice" that resonated with disaffected White teens who bought CDs en masse, and others can technically play the instruments and say the words, but it's missing that magic, and the teens reject it. So like shrimping boats, most successful grunge bands (like most other "youth culture whisperers" in entertainment) weren't farmed and harvested, they were wild-caught.

Here's a clip from one of the most famous Reply-All episodes, where a musician gets a meeting with a major record executive who heard his demo. I recommend giving it a listen but I'll put the transcript in the summary tab if you're just skimming. Consider the lifestyles and values of the various people involved, what "work" looks like to them, and how funny it is that these two men even met and could be collaborators:

The point is: software was more like that. Getting flown as an undergrad to SF in 2009 for an interview at Facebook, put in a hotel, served free food, all because you could code? You felt cool, especially if you'd spent the last two years in libraries hacking up side projects, but also: you were dazed and confused by whatever machine was spending that much money just to see if you had the magic they needed.

Here are the ingredients for a talents/creatives industry:

• It needs to be very valuable, like professional athletes, musicians, movie stars. Winning big needs to mean winning big.

• It must be hard or impossible to replicate success "synthetically." We even have a show meant to farm pop stars (American Idol) and while many contestants had some kind of career in music, it's really only produced one major pop star (Kelly Clarkson, the very first winner). An industry like this must have some degree of mystery of who's got "the juice."

• "The juice"… has actually got to matter. Here's where I think tech went wrong: there needs to be an actual force multiplier to having that ineffable talent. Kendrick Lamar being Kendrick Lamar needs to actually produce a Kendrick Lamar-sized result. I think tech really convinced ourselves that if we wrote Lisp like Paul Graham we could Smart Person our way into riches, but it turns out it's much easier to convince yourself you're a tech genius who innovates than actually be a tech genius who innovates. Very, very few companies actually made profits investing in tech with genius hackers: for most, the path of less resistance was to adopt the brand and sheen of "tech company," make Engineering a cost center, and play finance instead.

Remember when we wrote programming language books like this? For the young'uns, this is from _why's poignant guide to ruby. Also, remember when a weird Danish racecar enthusiast took a language from 1995 to write a framework and people… welcomed it?

Okay, but why bring this up?

Because while there's nothing wrong with the analysis in Software and its Discontents, I think much of its diagnoses are mitigated by teams that embrace the narrative (and build a reality of) "being in an industry of creatives." That said, he and I will likely never see eye-to-eye on this because he more-or-less invented Choose Boring Technology, and most of what I advocate is to break playbook, embrace narrative, and be interesting.¹

He brings up four major bullet points in his essays:

• an explosion in the complexity of software development
• tech talent becoming significantly more expensive
• success becoming more elusive than ever, with startups having “lost that magic feeling”
• conflicts over changing expectations of the work environment

Let's tackle these. If you get the "creative industry" mindset, you can operate out of the worst of them.

tech talent becoming significantly more expensive

First: hire fewer people. If you aren't feeling desperately, desperately constrained, don't hire. At around the 5-7 year mark engineers start smugly saying "it's best to write no code at all! Every line is expensive and must be maintained," and while that's absolutely true I want you to imagine a new hire as at least as operationally expensive to your Eng org as a 200k line service.

When WhatsApp was acquired by Facebook, they had 35 engineers. They were able to do this in part because they didn't "follow playbook," and while you probably can't go that extreme, per the creative industries point: do you think SNL would be improved if they had 400 more writers? If HBO added 15 showrunners to Game of Thrones?

Teams are an immutable data structure: when you change them, you get a new one. Adding too many people, too quickly, adds links to the game of Telephone that is every conversation your company already has. If people are introduced to a company without enough seasoned mentors, and without its culture and vision established, they add process and produce documents because it feels like productive work. Your customer doesn't care about any of those things.

The example I give is the ecosystem of a pond: imagine it's got turtles and fish and bacteria and algae. That's your company's culture. If you add 5% of its volume in water from a different pond (with different bacteria, fish, algae), the resulting pond over a few weeks is pretty much the first pond but with more water. But if you ever add 40% of its volume in water, the new pond's fauna may win out, and the resulting pond may be unrecognizable.

Hire carefully, but moreover: hire as slowly as you can manage. Bonus, this also works well with being extremely selective about who you hire.

(I recognize that one of the reasons we all hired so much was investor pressure. If you're leadership at a company, your daily dashboard includes cash-on-hand, and the investors don't want that cash back: they want "cash go down, stock price go up," and you needed to make that happen by the next board meeting. In the 0% era, one way you made that happen was hiring expensive engineers.)

Other "pretend we're a creative industry" things you can do is break playbook:

• Consider colocating servers instead of cloud. Squarespace did it, Twitter did it, Basecamp is doing it, Stack Exchange does it. It's not only possible but often way, way cheaper.

• Hell, even consider a "pets, not cattle" model while you get started. Don't get trapped into creating a shitty version of Heroku and your product.

• Consider stateful deploys, at least as another layer on top of stateless. If you've got a Docker-based flow that git clones from a fresh base image, consider hosting a persistent build server that git pulls. Fred Hebert talks about how companies he's worked for lost functionality and flexibility that they never recovered (ditto) in the name of removing stateful deploys and moving from BEAM to Go + Docker ("following playbook, Choosing Boring").

Each of these can be their own blog post, with lots of caveats and supporting evidence. But I'm feeling this get pretty long already, so I'll jet through the rest.

"an explosion in the complexity of software development"

I hate the word complexity. As an industry we don't share a definition, it usually means "shit that I don't like." Luckily Kellan offers some precision:

In both systems thinking and software the term "complex" is a technical one. It refers to the number of distinct parts in a system, and the connections between them. Complex systems are characterized by nonlinearity, randomness, emergence, and surprise. Complexity is why communication and coordination dominate all other costs when it comes to building software.

Did you notice that the examples from the previous point also removed links between things? People, an entire infra stack, and the idea that when you want a cake you should build and maintain a full-on bakery. All gone!

But it doesn't have to end there. Breaking playbook, be a little creative:

• Does every company need a full team of FTEs to support a React client, that's SSR-ed with extra infra for "hydration" + more engineering built into its build system and node_modules folder than the entire Flash Runtime had? Consider htmx, or server-side templates, or LiveView.

• Do you really need microservices? Kubernetes? Terraform? Pretend it's 2011 again and deploy JARs or binaries, keep and maintain a monolith. Dropbox, YouTube, Instagram, Facebook Blue App, Shopify… all (mostly) monoliths, all (mostly) pre-date the Misery Era.

• Can you use technology that has better scaling properties than Ruby, Python, or Node? I can write more about these, but talk to any eng who's worked on giant codebases in these stacks, bad things happen that don't happen in, say, the JVM. I said in another blog post: "you've created an environment that's optimized for your founding engineers to build something quickly in the first 7 months, but the price is a set of recurring obstacles which your engineers will pay down over the next 7 years."

Part of why I'm so damn bullish on the BEAM and LiveView is precisely because you remove so many links and connections. It has a learning curve but if you study a bit, it's incredibly powerful how much you have available to you.

"success becoming more elusive than ever, with startups having "lost that magic feeling""

All my suggestions above radically reduce costs of developing software, which hopefully leads to some increased version of success.

Additionally, the talent you'll acquire to paint outside the lines here will both be more inspired, and inspire others on the team. A favorite article of Ramp CEO Eric Glyman's is Amp it Up!, which contains a lot of ideas, but a core one: keep the energy high! Someone on a team of 50 really inspiring engineers who own the whole product and stack, WhatsApp-style, are going to perform better (and be a lot happier) than someone on a team of 50 React devs who owns 3 pages.

Tech won't be the reason your business wins, but it can help create that high-energy, high-motivation environment, which will strongly up your chances.

I'll end with a question I don't have the answer to: is the application software industry "irreducibly creative?" By that I mean: can it ever produce a set of "established solutions" that are cost-effective?

I'll say some industries cannot reliably be made into a cost-effective sludge factory: you must catch wild, and you can't produce "the goods" reliably and synthetically. I gave the example of grunge music, but a more favorite example is the content wars of HBO, Disney, and Netflix.

See, HBO and Disney have been kings of content for decades. They have pipelines for producing quality content people love. They know that you gotta pay a bunch of desperate weirdos in LA and take strange bets based on gutfeel. In this story about HBO, look how close they were to losing the biggest show in television for a decade, shepherded by a person they were firing:

Carolyn Strauss is among them. Despite coming up with hit after hit for HBO, in 2008 she was shown the door. In her telling, it was because she passed on one show, which was Mad Men. And as a result, months later, she was fired. And on the way out, she's like, 'There's one show I'm developing that I want to keep my hands on. It's the dragon show.' And they're like, 'You want that? Sure. I'm sure we'll make that.' And they did. And Carolyn Strauss is a top producer of HBO's most successful drama of all time, Game of Thrones.

These are very human processes. Netflix was a newcomer, and tech-first, so when they got into original programming, they tried to data their way out of it. "Data says new subscribers fall off after two seasons, shows will only have two seasons" (bonus, it's easier to exploit LA entertainment workers like this).

And they're losing the bet. Turns out people want Marvel movies, Andor, and The Rehearsal more than they want a $200m movie with hit stars nobody watched or a clumsy Will Smith racism allegory.

(Anecdotally, I have a friend who worked in Disney Animation during Frozen, and she said offhandedly that many of them were convinced it was going to be a dud. You can't predict success in this game)

So some industries don't "sludge." There comes a point where the humans gotta human. I wonder to what degree software is? Probably more than entertainment, since it was easier for entertainers to sludge into a streaming service than it was for a streaming service to sludge into entertainment. But I don't think it's as deterministic as our "playbook" is sold as.

1. ^ In his (and my) defense, the tenets of Choose Boring Technology (when you read the articles) have the disclaimers and asterisks that make it pretty unimpeachable. It should be understood as an argument against a certain type of "novelty-driven development." It never explicitly says "popularity is worth suspending your Engineering sensibilities," and in Kellan's Discontents post, he writes "As an engineering leader raising the quality of technical decision making is arguably your most important job after building the team itself." So I don't think he wants people saying "Choose Boring Technology!" to mean "we'll use Kubernetes and microservices because everyone else is using it" for a pre-launch B2B.

Similarly, please read my posts with the same understanding. I add a million similar disclaimers on the bottom of my "Against Boring" post for similar reasons. Choose Boring was a reaction to a real problem.

I keep saying the point of these posts is to emphasize that the tech has to matter. To use the framework of the last post, we should consider soil in addition to atmosphere, and based on his quote on technical decisions, I think he'd agree! Unfortunately, I don't think that's how most tech leaders use "Boring."