Coming up for Air

I didn’t notice the changes at first. I just thought I was busy.

But when I finally had some space to breathe, it hit me. I had lost sight of what was important.

Perspective can be really hard when you’re stuck in your house.

Now, in a time when so many have been impacted so deeply by loss of jobs, eviction, sickness, and even death - and as the coronavirus pandemic seems to have compounded inequality in a time when inequality was already on the rise - being stuck at home is probably not something to complain about.

And I before I go further, I must acknowledge that the world is hurting right now in some pretty profound ways, and if you’re lucky enough to still have a stable job and a healthy family, please take a little time to help those most impacted. Give some money, give some time - there’s countless highly impactful causes right now - such as food banks, which according to Feeding America have experienced a 55% increase in demand through the course of the pandemic.1

For me professionally, last year absolutely kicked my butt. My team was responsible for building and shipping a new production grade large scale distributed deep learning system for a new manufacturing customer. 2

We eventually were successful, and learned and grew a bunch along the way, but holy crap it was intense.

One of the more memorable moments of the project hit right after our phase one go-live. Late on a Friday afternoon, we realized we had a problem. 3

Our Edge Server was experiencing a memory leak (that had somehow escaped our testing), that caused the front end of our application to crash every 8-12 hours. With no fix in sight, 24/7 uptime requirements, and a project plan that was already slipping, we quickly made a rotation for the weekend, where one of us would manually restart the app every 6 hours.

Thanks to some real hustle from the team, we were able to ship a fix the following week, but the issues were far from over, and just reaching this go-live had already taken a toll. This made for 4 months of 60-80 hour weeks for me personally, and plenty of stress for the team as well.

Now, I don’t think there’s anything intrinsically wrong with putting in extra time toward a goal that you believe to be important - I’ve definitely found intense sprints both essential and rewarding throughout my career.

Where the pattern can become destructive is when we lose sight of why we’re doing things in the first place, when sprinting feels like the new normal, we spend too much time in reaction mode putting out fires with no end in sight, and not enough time making sure that we’re working as a team on the right thing.

And for me all of this was definitely compounded by being stuck at home during the pandemic. It was extra hard for me to get outside of my short term reactive way of thinking.

The analogy is so obvious is seems silly - but it’s really true - when your physical perspective is fixed inside the same four walls - this can really limit your mental perspective.

Travelling is a nice counter-example. More often than not, I return from travelling, especially vacation, with a renewed sense of energy and focus, and and whole set of new approaches for the problems that felt impossible or painfully slow moving when I left. I don’t know about you, but I didn’t do a lot of travelling in 2020.

So, how do we expand our mental perspective when we’re physically stuck on lock down?

Well, for me it really started with a purge - for the first time in a while I took the full week off between Christmas and New Years to rest and reflect. After catching up on some sleep, I poured over 2020 - I knew something was off, but what was it? General pandemic and quarantine malaise? Why did it feel like a layer of haze had descended on the year?

As nicely put by Ray Dalio: “Pain plus reflection equals progress”

This reflection period was really critical for me to see some of the negative patterns I had fallen into, and helped drive some changes I’m in the middle of making now.

There’s a type of creeping instability I feel in these periods when I’m not really sure what the core of me really wants or needs to do - when I’m just going through the motions. I feel it in conversations. I feel when I find myself trying to justify to others how I’ve been spending my time. Not knowing what you want, where you’ve been, or where you’re going sucks. And that’s where I was.

In taking a deep look at the year, it was also really interesting to see that I really had learned and grown is some significant ways - but hadn’t had the time to notice, appreciate, or think about these explicitly. I think this is really an important piece - recognizing how you’ve learned or how your thinking has changed will impact what you think is important and where you should focus next.

So. What is important then? Where is it worth spending time? What is the work that will connect our soul to our actions?

These answers are of course different for everyone, I think that’s part of the beauty here.

Here’s my list for 2021.


1. Empowering Non-Technical People with Deep Learning

This is probably favorite parts of my job right now, and I think a really critical area in deep learning. I sense the “AI glow” is starting to fade from deep learning, but that doesn’t mean there aren’t still plenty of interesting opportunities. One reason I like working in manufacturing is that the industry is generally underserved when it comes to machine learning powered solutions - deep learning can really make a big impact, and we’ve found that the size of that impact is often proportional to how much we can actually get data science “out of the way” of the domain experts. The more autonomy we can deliver through well-designed UIs on the front-end, and powerful automated tools on the back-end, the better.

I also really like this area because it intersects with another area I really love - education. I really don’t think anyone has nailed down the right abstraction layer between non data scientists and deep learning models - what are the right tools, education, and abstractions to empower the world with deep learning? How can we make deep learning just another (but more powerful) tool in the tool belt of quality, efficiency, or continuous improvement experts in manufacturing and other domains?


2. Teams, Really Collaborating, and Software Engineering

Something I thought a lot about in 2020 are teams. Microsoft Teams, that is. Just kidding, although I did spend way more time using it than I wanted to.

To date, a big portion of the work I’ve been most proud of has been individual contributor type stuff. This really changed for me over the last few years, and came even more into focus in 2020.

I’m still way at the bottom of the learning curve on this, here’s my thinking/learning today:

  1. No one succeeds alone. I’m sure this is obvious for a lot of people, but it hasn’t always been for me. Also, this doesn’t mean that everything important happens on a traditional team, or that individual effort and contribution have no place. It just means that whatever your idea of success is, especially if it’s big, you need to work with others to get there and to sustain it.
  2. A team is not just a group of individual contributors. For some types of problems, you can patch a bunch of individual contributors together through centralized planning - but this doesn’t work very well in software4 - so if you’re going to build something together, you have to really embrace open collaboration and team work the whole way through - this is of course a big part of Agile methodologies.
  3. Leadership and management is its own discipline, and definitely not one you learned in school.
  4. Teams don’t write novels. Individuals don’t write operating systems. It’s important to know when something should be tackled by an individual vs a team, and how these pieces fit together. Here’s what these different modes feel like to me today:
coming-up-for-air-figure-1-01.png

In each of these areas, I’m sure that someone much smarter than myself as already given these some deep thought, and I’m excited to learn more and expand my thinking and practice on teams and collaboration.

Finally, an interesting and incredibly comprehensive book I picked up last year, Software Engineering at Google, gives a really nice definition of software engineering that I think fits in nicely here:

“We believe it is important to differentiate between the related-but-distinct terms “programming” and “software engineering.” Much of that difference stems from the management of code over time, the impact of time on scale, and decision making in the face of those ideas. Programming is the immediate act of producing code. Software engineering is the set of policies, practices, and tools that are necessary to make that code useful for as long as it needs to be used and allowing collaboration across a team.”


3. Education

For the last three years I’ve taught a graduate-level computer vision course at the University of North Carolina Charlotte each fall.

In 2020 we did the course fully remote. And it sucked.

Ok, it did’t fully suck. And I need to say right away that we had some fantastic teaching assistants, great students, and the that UNCC administration was incredibly helpful and supportive.

But. It still kinda sucked.

Teaching in-person gives me energy, teaching remotely drains me. Students can feel this too. There’s something magical about being in the same room with other people that I took for granted pre-pandemic.

All of this got me thinking more about education and in general 2020, and especially how the internet has changed and will continue to change education.

On one hand the internet age has been incredible for education - the quality and quantity of online education resources has exploded in the last 10 years - I still remember having my mind blown by MIT OpenCourseWare as an undergraduate student at Georgia Tech, the Linear Algebra lectures were so much better than the ones I was in! 5 On the other hand, when we take the online thing further and run a whole class online, it can really cheapen the experience. Maybe this is just a technology gap - I hope that one day we look back at today’s video communication technology and laugh at how primitive it is - and I hope this happens soon.

In my course this year, we did our best to make lectures interactive - but gosh I struggled with the “bandwidth” of the communication. When you stand in front of a group of other humans, you have so many queues to draw on to make the experience better for everyone - and most of these queues just don’t make it across the internet.

I don’t have any answers right now, just lots of questions - but 2020 definitely got me thinking more about education and where it makes sense to invest time next. Here’s my big list today.

What is the future of education?

What is the future of technical education? This stuff changes so quickly, you can’t just learn it once in college.

Why is so much of what we learn in school not relevant on the job?

What is the future of the business of education?

Is the cost of universities sustainable?

Does it make sense to couple education with research as most technical universities do?

Where do universities fit in? What is the role of universities in the 21st century? Is it different than in the 20th century?

The quality and quantity of online education resources has exploded in the last 10 years - where is this going?

Will virtual instruction replace in-person one day?

Do students need interactive lectures, why not just watch videos? I think we know the answer is that videos/information delivery are just one piece, but what exactly is missing? Interaction with the material? Interaction with the instructor? Interaction with class mates? Some pressure to get students to put in work?

How can instructors like myself best serve our students?

How can we make education less artificial? Or does it need to be on some level?

A quick sidebar on this last question - for myself, over the years I’ve found a few patterns that work really well for myself for learning new things - one of the best patterns for me is something like:

  1. Go try something hard, fail (hopefully quickly)
  2. Buy every book on the subject that might help me not fail
  3. Read the parts of the books that will help me
  4. Reflect on what I’m doing wrong and make a new plan
  5. Try again
  6. Repeat as needed

Step 1 is really important for me - this ensures I have some “skin in the game” - failing at something first makes it real. As soon as I’m in a learning setting where things feel artificial, I’m just not motivated.6 This was a huge struggle for me in school - it mostly felt fake so I didn’t care about it.


4. Algorithms: Unsupervised and Semi-Supervised Learning

One of the biggest friction points in my work today is data labelling. Getting the right people to consistently create the right labels that both benefit the business application and are actually learnable by today’s models is rarely straightforward.

Clearly, other folks have these issues as well - just look at the myriad of labeling tools that have hit the market in the last 5 years to address this need.

I don’t think labeling is fully going away anytime soon, but I do think we can do a lot better. Part of this is certainly software & tools to maximize and streamline human efforts, but another piece has to be better algorithms that need less labeled data to perform well.

There are especially unique challenges in my current domain of manufacturing. In my previous role in autonomous driving, many of the ML challenges were kinda “global”. Everyone in the space needed to have a really good vehicle detection model, for example.

In manufacturing, every customer is a freaking snowflake. Even the same customer making the same product but a little later down the production line can present new challenges. The scale and variability of manufacturing is astounding - it demands tools that are quick and adaptable - we can’t spend a year making a perfect model for a certain type of failure mode.

Another unique challenge in manufacturing relative to some other domains (but not as bad as healthcare!) is the lack of quality open datasets. I know of exactly one today. We need more to push the industry forward, and provide the foundation for developing and proving out the next generation of unsupervised and semi-supervised algorithms.


5. Sales, Commercialization, Startups

In 2020, I participated in 100+ sales calls. These are sometimes really exciting, sometimes boring, sometimes frustrating. I’ve done my fair share of these before, but this year felt more visceral, more tangible, more real.

When I was building my first startup, Intuitive Pickups, one of my advisors recommended I read Steve Blank’s Four Steps to the Epiphany.

At the time I thought it was incredibly dry and mostly irrelevant.

In 2020, listening to potential customer after customer explain their problems, and doing our best to explain how our product would help, I found myself thinking through some of the ideas of Four Steps to the Epiphany again.

How do we know if we’re making the right thing? (product market fit)

How should engineering and sales work together? Should this change with scale?

How much of sales success or failure is about the product itself vs how it’s sold?

Are we talking to the right people in a given organization?

With these questions in mind I revisited Four Steps to the Epiphany, and holy crap was it a different experience. It’s like Steve Blank knew what was going to happen to us before I did. 7 Now don’t get my wrong, the book is still dry as hell - but happily, there’s a cheat sheet out now, that I wish I had the first time I read it!


6. Fundamentals of Computer Science

Relative to previous years, in 2020 I spent bunch more time worrying about and working on the full stack (much of my previous work was more focused on just the ML components). This gave me a much deeper appreciation for some of the more fundamental building blocks that I think can be easy to take for granted when you’re training models all day.

My biggest epiphany of the year went something like: “HOLY SHIT, DATABASES”.

The job of a modern, especially large scale and/or distributed database is pure insanity - and the fact that humans been able to engineer software that is so reliable and performant, even when the underlying hardware and networking components are intrinsically noisy and faulty, is incredible.

A part of the stack my team built and maintains involves shipping images and metadata around the internet. If an operator at a plant in Michigan realizes that one our models isn’t detecting a new type of manufacturing defect, she can correct our model right on the spot in our “In-Factory” app - this image and label then gets shipped to the right database to be pulled into our model retraining pipeline.

As we engineered this part of the system last year, I couldn’t stop thinking about how impressed I was that systems like Instagram and Netflix actually work, and are so freaking reliable. Anyone in the world with a phone can post an image on Instagram, and just assumes that magically all their friends will have the same image ready to view on their phones - almost instantaneously. Witchcraft.

Just thinking hard about the idea of concurrency alone last year was enough for me to have that moment on the learning curve where it’s like “holy crap, I don’t actually understand anything.” This is a fun place to be, and I’m excited to learn more about databases and other, more fundamental parts of computer science: computation, compilers, programming languages, and operating systems.


7. Open Source & Why it Matters

Maybe I’m just way behind on this, but a couple things happened in 2020 that got me thinking more deeply about open source software, why it’s important, where it’s going, and how it fits and doesn’t fit in with today’s commercial and legal systems.

For me, I think as a result of on when I came of age, it feels intuitively obvious to me that open source model had some significant advantages over a closed model. I still remember discovering Wikipedia in high school in 2004, and being just blown away - reading article after article, diving deeper and deeper into the web of human knowledge - it felt so incredible and alive in a way that closed system never could.

I think this and other experiences let me to just assume something along these lines to be true:

The most reliable and maintainable software in the world is created using open source models.

I never really challenged myself on that assumption until 2020.

Is this really true?

If so, why, and what should we do about it?

Should everything be built in the open? If not, when and why should we not build things in the open? How do we balance the very real commercial concerns of the business of software with these facts?

As part of digging into this a bit more I picked up The Cathedral and the Bazaar by Eric S. Raymond - this is a fantastic read, and I don’t think I’ve had the bandwidth to fully unpack it yet. One of the more popular ideas is what Raymond calls “Linus’s Law”, after Linux Torvalds:

“given enough eyeballs, all bugs are shallow”

I think this really captures a key part of why open source works so well.

What I also found really interesting in 2020 is that I believe there’s a type of “inverse corollary” to Linus’s Law.

I wasn’t explicitly aware of it until I watched a great introspective video by the incredible YouTuber CGP Grey. Grey talks about his process research and writing process, how it go wrong, and the anxiety knowing that what he produces will be judged by the internet.

This really resonates with me. I made a video about my process at Welch Labs back in 2017 where I share some of the same struggles.

Where I think this ties into Linus’s Law, is that since the approach to making videos taken by CGP Grey, and lots of other YouTubers including myself on Welch Labs, is to go out into the world, do a bunch of research, and create something as interesting and as accurate as you can - but, since the creation process is intrinsically an individual or small team effort - you can’t leverage the wisdom of the crowds - Linus’s law is working against you!

Here’s my Linus’s law inverse corollary:

”when you make something for enough eyeballs and care about being right, you will screw up eventually.

Now, there’s also so big advantages to crafting content and ideas this way - when something is written mostly by one person it gives the work it such a unique and beautiful personality and perspective that wikipedia articles will never have.

I wasn’t really aware of it at the time, but when I was actively working on Welch Labs content for YouTube, Linus’s Law really worked against me, especially when trying to start a new series - it’s easy to joke about the fear of being wrong on the internet - but it’s real. It’s really really real and if you don’t explicitly do something about it, it can cripple your creative process.

Like most topics here, I have no idea what to do about this, but I think it’s really interesting, and am curious and excited to learn more.


8. Social Justice

I don’t think you can watch what happened to George Floyd in the summer of 2020, and the subsequent global response, and not feel like you should do something about it. That something has to change. That we can’t continue as we have.

And I have to admit here that as painful as it was to watch this summer, did I really do anything? Did I really make any changes?

Aside from becoming more aware of these issues, and donating some small amounts to a couple non-profits in this area, I really haven’t done anything. And that can’t be how systems like this change. I’m part of this system, and I haven’t done anything.

This sucks, and is not something I’m proud of.

And this really hits home for me when I think about how the city I live in, Charlotte North Carolina, and how we were ranked dead last (50/50) for upward mobility amongst American Cities in 2017.

Each Fall I have around 60 students in my computer vision course at UNCC, and in three years, none of them have been a black or brown students from Charlotte. Not one.

Now, the course is very internationally diverse - but how is it that not one black or brown kid from Charlotte ends up in my course over three years? Maybe I’m being self-centered here, maybe this is a poor measure of the real issues out there, maybe students from Charlotte go elsewhere for school, and of course computer science is just one area - maybe I’m way oversimplifying here - but when I think about social mobility it’s hard to ignore the relatively high salaries that can follow a computer science degree - and this doesn’t feel like a coincidence to me.

So what exactly are the barriers between black and brown kids in Charlotte and my computer vision course? What can we do about them?

This is clearly another an area where I have more questions than answers - zooming out a bit I think that fastai founders Jeremey Howard and Rachel Thomas are doing some important work at the intersection of technology and ethics.


9. Information in Politics

Watching the political madness of 2020 and working in technology, I found myself wondering how the work my peers and I do impacts the political landscape.

The internet was supposed to make the world a more transparent place - but it seems to be doing exactly the opposite right now.

How do we know what’s really true?

How much freedom should people, organization, and elected officials have online?

To what extent and how should these systems be regulated?

Are there technical solutions here? Or are these the types of problems that are better solved through other means?


10. Climate Change

In the Fall of 2010 I headed off to graduate school to study environmental engineering at UC Berkeley, fired up to do something about climate change. Barack Obama was fresh in office, appointing Berkeley’s own Steven Chu as secretary of energy - and it seemed clear (to me at least) that climate change was widely recognized as a real and critical issue - and that the country was serious and staring to move forward.

In my first semester, searching for the right research fit, I spent some time learning about the the work happening at Berkeley - I was especially interested in the work of the Renewable & Appropriate Energy Laboratory (RAEL). I remember being surprised to learn how policy focused RAEL was. It seemed to me, at the time anyway, that idea that drove this focus was something like this:

The technology we need to curb climate change is already here - we know what to do - what we need now are the policies to make these things a reality.

This viewpoint well captured by the stabilization wedges paper, which I recall being pretty central in a couple of my courses. Not really seeing myself as the kind of person who could make in impact in politics/policy (I saw my strengths more in engineering), I ultimate decided to focus my research in an area outside of energy, and ended up working on a really interesting project using machine learning and sensor networks to study snowmelt in the Sierra Nevada mountains - a critical water supply for the state of California.

In 2020, listening to Greta Thunberg, I found myself reflecting on the 10 years since that fall, and how it really felt that we had taken one step forward and two steps backward on climate change, and wondering if my cursory take on what I could do to mitigate climate change back in 2010 was too swift.

Is addressing climate change mostly a public policy issue?

Are there new key technologies waiting to be discovered or invented?

Are there things aside from reducing my own carbon footprint I can do to make a difference?

~

Stephen Welch

February 2021

Charlotte, NC

  1. https://www.feedingamerica.org/take-action/coronavirus
  2. Super easy, right? :)
  3. This seems to be when all the really fun production bugs like to pop up.
  4. I think the reasons why are super interesting, and probably deserve at least a whole separate blog post.
  5. Shoutout to Gilbert Strang! Also, I emailed him once about some solutions to the exercises in one of his books, and we responded right away and was super nice.
  6. This reminds me of some really beautiful passages about education in Zen and Art of Motorcycle Maintenance.
  7. I think this is a great example of learning with “skin in the game” - there’s a time life, based on our experiences, when we’re ready to learn certain lessons, and this is really hard to recreate in formal education.