Brian Lee


Lessons from Software Engineering

I worked in the software industry for about 18 years, starting in 2006. Along the way, I learned a few hard lessons because I am an idealist and tend to be experimental. I used to think I was a perfectionist, but I am not, as you will see.

These hard-learned lessons seemed relevant only to software engineering. But now, I realize much of what I learned is about general problem-solving. It took me a long time to see that. Now, I apply these “engineering” ideas to my day-to-day life.

Lesson 1: Solve the Smallest Problem Possible

Finding a general solution to a general problem can be quite challenging. I believe that career advancements come with tackling more complex issues. For instance, a junior software engineer starts by making minor updates and fixing bugs, which have a low impact if done incorrectly. A senior software engineer, on the other hand, oversees or designs projects that have a long-term impact.

At every level, problems are complex relative to one’s skill and experience. Thus, everyone is always solving overwhelmingly intricate problems in their world. A complex problem is daunting because it involves many variables, including unforeseen requirements, events, and oversight. I used to try to solve the whole problem in one go, hoping for a perfect solution. However, this approach often led to overly complicated and still inadequate solutions because the complexity of the problem remained unchanged.

Somewhere along my career, I discovered that breaking down a larger problem into smaller, manageable parts works best. Start with the smallest problem possible. For example, if we’re designing a system that must serve 1,000 customers a second, we first need to figure out how to serve one customer efficiently. When focusing on the single-customer solution, we have a few choices. Should we design a system that can serve one or more customers, or should we assume there is only one customer? I vote for the latter.

Some caution is necessary here. When you build an elaborate single-customer solution, you may create a complex system with many assumptions. Therefore, the key is to build the simplest solution possible for a single customer, which can then reveal paths to the final 1,000-customer solution. Some may call this minimal solution a prototype or demo.

It is important that you spend as little time as possible at each step, solving the problem just well enough so the system does not embody poor assumptions. Gradually, you can remove design flaws while continuously using partial solutions to meet business needs over time. This approach allows for steady improvement and avoids the pitfalls of overly complex initial designs.

In life, I find that applying this principle gives me a sense of accomplishment while continuing to make progress toward solving complex problems. This book is a great example. I am not a writer, and I don’t know what systems are used to write a book. So, I started with what I knew best: opening up a text editor and just typing a simple HTML page with minimal formatting.

Every day, I make adjustments or research more suitable systems for continued writing. Over the past three months, I have gone through many iterations of tools and frameworks. It was easy to switch from one system to another because I didn’t invest time in developing an elaborate setup. Every solution was just “good enough” to help me manage writing and publishing the book on the website.

Every time I adopted a new solution, I took the best parts from the previous one, allowing me to compound my learnings over time. This gradual approach not only made the process more manageable but also ensured continuous improvement without overwhelming me with complexity.

Lesson 2: Find the Right Tools

I love finding tools with surgical precision. With a background in software, I get a kick out of discovering ways to automate or simplify my tasks. AI-assisted tools like ChatGPT have made great advances in recent years, and I’ve been learning about them while working on software projects and tinkering for amusement.

Here is how I use a customized ChatGPT, the Python3 Pandas Helper, to write a daily stock portfolio review script in Python. My daily portfolio review used to take about three hours in the morning. With this little tool, I finish a review in just one hour.

The script uses the numerical analysis framework called pandas. Pandas has a steep learning curve and continues to be puzzling at times, even when you understand the basic concepts. I found that ChatGPT does a great job generating scripts for my purposes, so I chose to customize ChatGPT to build the Python3 Pandas Helper. For example, this customized ChatGPT can handle prompts like:

Given the following table, reverse sort by column A and then B:
item, A, B
foo, 1, 2
bar, 3, 4

ChatGPT’s response includes Python code and a sample output:


import pandas as pd

# Create the DataFrame
data = {
    'item': ['foo', 'bar'],
    'A': [1, 3],
    'B': [2, 4]
df = pd.DataFrame(data)

# Reverse sort by columns A and then B
df_sorted = df.sort_values(by=['A', 'B'], ascending=[False, False])

# Display the sorted DataFrame

Sample Output:

   item  A  B
1  bar  3  4
0  foo  1  2

See the full output here.

It’s really fun to use technology this way because it makes me both productive and amused.

Here is another example. I use a customized ChatGPT to edit my writing. This ChatGPT is instructed to write at a 6th grade reading level while preserving my concise writing style. I think it’s been doing a splendid job. I can’t understate the improvements to my writing process and productivity. I am sure my writing style and taste will change as I read and write more. With more future edits for this book, I look forward to continuing to use the editing helper. I can’t share a link because this ChatGPT “violates” the usage policy due to the mention of “6th grade.” I’m appealing the decision and it’s in review. Here is a prompt you can try yourself.

This GPT is a helpful editor that preserves the original writing style as much as possible while correcting common writing issues such as grammar and readability. It aims to improve clarity with minimal changes to the original text, aiming for a 6th grade reading level. The target writing style should be less terse, more introspective and reflective, with a gentle motivational tone. When appropriate, it uses rhymes to enhance the text. It varies sentence lengths, uses synonyms where possible, and adds rhythm to the writing.

Copyright 2024, Brian Lee