Some experiences and lessons from the transformation of. NET system architecture

10 年前的一篇文章（2013 年，原文robbinfan.com已经无法访问，找到最早的还是在博客园这篇），现在读来还是感慨颇深。

Wen/Fan Kai

In the Internet industry, Unix/Linux-based website system architecture is undoubtedly the mainstream architectural solution today. This is not only because Linux itself is open enough, but also because there are a large number of mature open source solutions around the traditional Unix/Linux community, covering all aspects of website application expansion.

I remember that in the era of the first wave of the Internet more than ten years ago, large websites using the Windows/. NET architecture were very popular, but nowadays, well-known websites using the. NET architecture are rare. Especially in addition to Microsoft's own websites MSN and Hotmail, many other large websites using the. NET architecture are facing architectural expansion problems, making the scalability of the. NET architecture a relatively controversial issue:

For example, the foreign SNS website MySpace website has faced serious performance expansion problems, and the domestic e-commerce website Jingdong has more than once suffered from performance bottlenecks caused by architecture expansion. Therefore, some websites based on the. NET architecture have to consider "de-NET", abandon. NET, and fully migrate to a Java-based architecture.

However, the architecture migration of large websites is itself a heartbreaking matter, and the risk is extremely high. Success cases are rare and failure cases are everywhere. This is because:

1. Architecture migration is carried out on existing business systems. It is not a leisurely development of new products. There is enough time to test and improve it. It is equivalent to replacing the engine of a passenger plane flying at high altitude. The switch must be successful once and there will be no second chance. If there is any mistake, the plane will be destroyed and people will die. And we all know that when a newly developed large-scale system is developed, it cannot be 100% perfect at once without being run-in and improved.

2. Architecture migration means that the old R & D team will be eliminated. Often, the old team system has gained enough say as the company grows, and the team with the new architecture is unstable in the company. Out of the instinct to safeguard its own interests, it is easy for the old and new teams to break out between them and exclude each other, leading to the failure of the migration.

5173 Lessons from "De-NET"

The 5173 website started as a game equipment trading. It once ushered in a period of explosive business growth during the golden age of client-side online games development. At this time, the website developed using the. NET architecture was already overwhelmed. Since the existing. NET R & D team had been unable to solve the website's performance problems for a long time, 5173 decided to fully migrate the website from the. NET architecture to a Java-based architecture. To this end, 5173 paid a lot of money, hired many engineers from Taobao and SUN companies, and formed a Java architecture research and development department with sixty to seventy people.

The new Java R & D department develops a new 5173 website, while the old. NET R & D department maintains the existing 5173 website. The two departments work in parallel, and the two versions of the website run in parallel. This brings fierce political struggle within the company. The newly developed Java version of the 5173 website has never been officially launched. The old. NET R & D team, in the face of serious survival threats, Efforts were made to resolve some core usability and stability issues. At the same time, with the end of the golden age of end-game, website performance issues have gradually become less important.

On the issue of whether the new version of the website should officially replace the old version of the website, various interest parties have been arguing and engaged in repeated tug-of-war. However, the female CTO who came in did not belong to any faction and had an ambiguous attitude. In the end, the. NET interests defeated the Java interests, and the Java R & D department was purged.

My experience and lessons from "de-. NETization"

Three years ago, when I first took over CSDN's products and R & D team, about 2/3 of CSDN's core systems were. NET architecture and 1/3 were LAMP architecture. There were very few R & D staff at the time: there were only two. NET programmers, three PHP programmers, and later a Ruby programmer I brought here. The plan at that time was: the overall architecture of the website would be transformed towards the Linux architecture and gradually replace the existing. NET system. Therefore, there is no plan to continue to recruit and replenish. NET programmers. Existing. NET programmers are responsible for maintaining old core systems.

But the two remaining. NET programmers left in less than half a year: one. NET programmer followed Microsoft executives to start a business, and the other. NET programmer jumped to work as a front-end engineer at Baidu. The reason is also very simple: since the company is going to de-integrate. NET, then. NET engineers will worry about whether they will still be valuable in the company after the. NET system is replaced in the future, and won't they be completely marginalized?

Of course, when I formulated the architecture migration plan, I also took this into account: I formulated a training plan for the more senior. NET engineer for the chief architect of the entire company, and the. NET engineer who is proficient in JS formulated a training plan for future front-end team leaders. However, uncertain promises are always less urgent than real threats, so I can particularly understand the loss of. NET engineers.

At this time, I was in a dilemma:

• If we continue to follow the "de-NET" plan in the future, even if we re-recruit and build the. NET R & D team, since. NET is destined to be replaced in the company, it is impossible for the new. NET team to be stable. It is very likely that we will stay for a month or two, and when the situation goes wrong, we will leave immediately. At that time, the core system of. NET accounted for a high proportion of the entire website and was extremely complex. Once something went wrong, there was nothing to do at all. Someone had to take charge of maintaining it. At that time, I had already begun to review the. NET code of the core system and was ready to do it myself.

• If the plan to "de-. NET" is abandoned in the future, it may solve some troublesome system maintenance problems in the short term, but it will bring many unfavorable aspects to the long-term development of the entire website: for example, website security issues, website architecture expansion issues, cost issues of comprehensive legalization of website server software, etc. If we had not made up our mind to. NET at that time, the price would only become higher and higher if we wanted to do this again in the future.

At that time, my initial idea was to hire two. NET programmers who were at a decent level, but the key was that they were not too motivated, who could be content with the status quo and work down-to-earth to maintain the old. NET core system; at the same time, recruit and build a Ruby R & D team to gain time and rewrite the old. NET core system one by one with the amazing development efficiency of websites I developed using Ruby in the past. But doing so is also very risky:

1. I didn't come to CSDN for a long time. At that time, there were many and miscellaneous products on the CSDN line, with hundreds of them, and I didn't know what they were doing with many systems;
2. Company leaders didn't support me so quickly. They were even worried that my drastic transformation of the website would completely shock the already fragile website at that time;
3. After I came to Beijing, I only brought one Ruby programmer with me. Building a Ruby team, training the team, and developing the core system was not a matter overnight. It was difficult to hurry up;

Fortunately, during the recruitment process, I interviewed two quite good. NET engineers who were highly motivated and had good programming skills and understanding. Although it didn't meet the criteria I used to recruit engineers who wanted to be content with their status quo, I didn't really want to miss out on good talents, so I temporarily changed my mind, recruited them, and formed a new. NET team.

In order to avoid the loss of the previous. NET team and create opportunities and space for the new. NET team to develop in the company, I thought about it and decided to adopt a compromise plan: to retain and use the. NET programming language and framework, but the overall structure of the website is still de-. NET-oriented. In summary:

1. The data layer abandons the SQL Server database and stored procedures and migrates them all to the MySQL database on the Linux platform;
2. Caching no longer relies on the caching mechanism provided by. NET itself and is migrated to distributed Redis deployed on the Linux platform;
3. Calls between services should be avoided using. NET's own proprietary protocol and replaced with Restful HTTP Web API calls;
4. Static resource requests are no longer handled by IIS itself, but are separated from Nginx on the Linux platform for processing;
5. The file system that needs to be read is also changed to access the distributed file system on the Linux platform;
6. The Windows server that deploys. NET code is placed behind the LVS, and the LVS is used for Load Balancer and failover.

Simply put, it is simply to let. NET be the application layer programming language and framework, and the rest is handed over to the Linux platform open source solution. The. NET framework purely serves as the application layer. Whether the development efficiency of ASP.NET MVC or the running efficiency of the. NET CLR virtual machine is very good, at present, we run millions of dynamic requests on a single Windows server without pressure, and the application layer architecture can be scaled horizontally: if the request load is very high, you just need to add more Windows servers. In short, we have achieved the best strengths and avoided weaknesses.

In addition, I also pay more attention to exchanges between different programming language research and development teams, encourage. NET engineers to familiarize themselves with the Linux operating system, and cultivate. NET engineers 'overall architectural awareness. Our current main. NET backbone told me that I feel that the biggest technical improvement since I came here is that my horizons have been opened up.

During the entire website renovation process in the next two years, this approach also proved to be quite successful:

1. The. NET team has continued steadily and has become a team with outstanding performance in the entire R & D department;
2. The entire system transformation process is very stable and smooth, and no risks have been encountered;
3. The impact on website users is very small. Basically, the renovation of the entire website product is completed unknowingly;
4. It has not had any impact on the company's online business, and as the system continues to be transformed, the support for the business is getting better and better;

When the website architecture is fully Linuxized and all architectural solutions are unified, it is actually not an important matter what programming language to use to write at the application level. Our current application level product lines are both.NET and written in PHP and Ruby, but the architecture is unified, and the programming language to use mainly depends on the allocation of resources of the R & D team.

In short, in my experience, if a website that has traditionally relied heavily on the architecture of Microsoft solutions is to undergo architectural transformation, migrate to the Linux platform, or even rewrite it in other programming languages, it is never a purely technical issue. It involves at least the following aspects:

1. How to protect the interests of the R & D team of the old system, how to encourage the old team to participate in structural transformation and share success. This is the most important and often the most fatal problem in architecture migration. If architecture transformation is destined to sacrifice the old team without considering and protecting their interests at all, I think there will be a cruel political struggle and ultimately failure;
2. The issue of how to maintain normal operation and smooth transition of existing business systems will definitely die if the architectural transformation affects the business;
3. How to ensure a smooth transition and improvement of the user experience of the website, if the architectural transformation affects the basic user experience, it will definitely be stopped;
4. The issue of leaders 'tolerance for risks arising during structural transformation, and the issue of leaders' patience after the structural transformation cycle is lengthened;

A little digression

I feel that there is a bad phenomenon in our Internet industry: some websites are paralyzed during the promotion period, and some websites often have unstable access. Company bosses like to come to Weibo and make harsh remarks, invite subordinates to tea, or rush to the rules and clamor for a million-dollar salary to recruit CTO. This is like a person who has poor daily living habits, spends a lot of time drinking, and never pays attention to health preservation. As a result, after many years, he finally falls ill. At this time, I waved a check fiercely and shouted, which famous doctor can give me medicine until my illness is cured, and who will I give a million yuan reward.

Therefore, when a website has serious technical problems, the root cause is often not a simple technical problem, nor can the disease be cured simply by changing a CTO. It is necessary to reflect on whether the company's corporate culture has never attached importance to research and development. Is the motivation of the technical team in place? Do you pay attention to the opinions of the architect? Has there been long-term R & D investment in technology thresholds that may be faced in the future?

Regarding this phenomenon, I remember Fenng said a very incisive saying: "Technical problems are always overestimated in the short term and underestimated in the long term." I also want to add: "When there is a problem with technology, it is never simply a problem caused by technology."

来自: robbinfan.com