Respect the Production

To me, the first rule as a software engineer is, respect the production.

I learnt this rule while having lunch a few years ago.

One time, I was having lunch with a colleague. Suddenly, my phone rang.

A PagerDuty came in!

I had to rush back to the office and roll back the last deployment.

It was not the first time.

During that period, my team had a daily deployment schedule. The deployment took an hour for the cluster.

We usually started deployment around 11 am, monitored the system, and then went to lunch.

Interestingly, after multiple consecutive smooth deployments, the deployment time would slowly move towards noon, and the monitoring duration became shorter.

Not long, production taught us lessons, as expected.

---

Respecting the production contains more detailed rules, e.g., never make a deployment when I don’t have X minutes to monitor afterwards.

However, this rule sets my standard.

In Chinese, it is known as “居敬”, means to stay in respect, be polite and be cautious.