Oops, no trace in logs

The Chinese colleagues in my organization usually go out every Monday to have lunch in a nearby Chinese restaurant. It’s a good time to share some common concerns about the economy, the stock market or the food security in China. Topics like work or technologies seldom emerge in the lunch discussion, unless somebody start the complaint, like what happened this Monday.

One of my colleagues complained that he spent hours trying to decipher the myth that some messages failed in processing with no reason in Production environment. Eventually, he found out the deploy team rolled out a new server last weekend, which was not configured correctly. The new server, participating in a cluster, grasped some of the messages and failed them. That explained why only some messages failed while other went through successfully.

But, why did it take him so long to figure it out? Two reasons, first of all, he wasn’t aware of the environment change (roll out of new server). Second, there weren’t enough logs to show which server processed the messages. He was browsing through the logs of all the known servers trying to find traces of the failure, with no luck. If there are some central logs or database records showing which server in the cluster processed the message, the error will be obvious.

Software Development Process Frameworks

As a developer and later a technical architect, I have been doing software development for almost nine years after I graduated from school. I witnessed the fast evolvement of new technologies, for example, from Java Applet in the early days to AJAX recently in front end field and from EJB to Spring in the business logics field. However, the process we do software development changes much slower. In one organization, the same process is usually used for all types of projects, from new and complicate multi-year projects to simple and repetitive bug fix releases. I feel the knowledge we have in software development processes are way behind the knowledge we have in technologies. To me, the software development process and methodologies are more fundamental and important than specific technologies. Because it affects the entire lifecycle of software development project, the entire software development team and every aspects of the software product.

Run Development as a Start-Up

Most of the days, we wake up in the morning, brush our teeth, wash face, have a brief breakfast and rush out to work. But, there are some special mornings. When we open our eyes, we start to see the world differently. This morning is one of such for me.

Maybe I have stayed in the development department in a big organization for too long. I was so used to the layback, slow-moving, stable, orderly working style. So, when I had the opportunity to go through a demo session of a competitor’s application yesterday, I was completely shocked, not only by the application, but also by the attitude they treat their work.

We developed our application in a typical “elegant” way of big organization. And they developed their application like a “startup” company.The differences are in sharp contrast. I listed some here:

• We wait for the requirements being given to us by other departments like Sales or Marketing vs. They go out to talk to frontline Sale people and to customers directly, for example in the trade show, to figure out the requirements firsthand.
• We are proud of the technologies we used and focus on latest and advanced technologies vs. They are proud of the product itself and focus on how it can satisfy customer needs.
• We focus on internal architecture design and build for last vs. They focus user interfaces and usability and build to sell.
• We live with the legacy system and spend a lot of time integrating with it vs. They completely write all the logics in contemporary technologies.
• We value stability vs. They value innovation
• We limit our releases per year vs. They release much more frequently
• We only serve one customer, our business vs. They look for opportunities inside and outside the organization
• We have never been worried about “selling” our product. We assume the business would take care of it and Sales people will sell it vs. They constantly promot their idea and product to the business. They made the business realize how great their product was by doing the demo, comparing the features and showing the statistics numbers.

It’s a typical example of “fast vs. slow”. In current business environment, the one who can sense the changes ahead of others and quickly adjust itself for the changes will survive.We witness more and more triumphs of the “internal marketing” or “project marketing”. Although we ARE the internal development department, we have to realize the business has choices nowadays. We cannot assume that we are the sole service providers available to the business and they will eat whatever we feed them. Those old golden days for IT are gone!

Business is business. If you would like to survive, you have to go out there and compete. You must build the best product in the market, and constantly market your service to the business leaders. If you don’t do that, the competitors from any corner of the world can come and replace you. So, wake up you old development! Time to act like a young startup now!