In the past few months, it happened to be the time for the launch of major features. Before moving on to another project, there wasn't much development work, mainly fixing some small bugs and improving existing features. Because the development was not as tight, there was more time this quarter to focus on process improvement.

Background

Let's talk about the situations encountered during development. Each team and organization faces different circumstances, so understanding the background and the problems to be improved is crucial. Several situations have clearly caused problems in this development:

Involvement of multiple personnel in the organization

I am currently involved in financial-related development, including FX (Foreign Exchange), stocks, investment trusts, and other products. Although larger features are handled separately (e.g., developing a separate app for FX), most of the functionality is implemented on the same website and handled by the development teams responsible for each project. This has led to:

• Whenever shared modules are modified, it involves multiple teams, causing a high mental burden. Fearful that modifying the code will break something, the tendency is to use workarounds rather than addressing the problem directly.
• Parallel development of multiple projects can lead to conflicts. For example, in Project A, a desktop version has been implemented, but in Project B, which is being developed simultaneously, it has not. When merging the two projects, conflicts are likely to occur.

Inability to control all resources within the team

For example, during QA, we need to coordinate the schedules of all projects, but QA resources are limited. Sometimes, even though the functionality has been implemented long ago, we have to wait for a month or two for QA. This incurs significant costs:

• Developers forget implementation details after one or two months and need time to recall them.
• The entire iteration process becomes considerably long when issues are raised by QA and then fixed.
• Certain APIs or implementations require other teams to make corrections, and we cannot control their schedules, which can hinder the team's development progress.

Tedious QA environment

Ideally, there should only be one QA environment.

However, when projects are developed in parallel, there are limited testing environments, and setting up new testing environments is costly due to architectural constraints. It is possible that QA is conducted simultaneously in two environments. When there is a shortage of testing environments, development and QA sometimes share the same environment, leading to some problems:

• QA encounters issues and is unsure whether they are actual bugs or just testing new features.
• The lack of QA environments results in significant communication costs. With a large team, there are numerous channels to coordinate.
• The preparation process for testing accounts is cumbersome, so almost all environments have incomplete or unusable accounts.

Incomplete Sprints

Although the team is currently using Scrum for development, I have noticed that Scrum has become quite ineffective since leaving the previous project. Possible reasons could be:

• Engineers within the team are responsible for different projects, making the Sprint Goal somewhat illusory.
• There are no QA resources to schedule, making testing impossible.
• Progress tracking is not sufficiently implemented, often resulting in situations where priorities are set on A, but colleagues are working on B.
• There is no clear goal and iteration.

From the above situations, it seems that the team itself has not met the necessary conditions for an ideal Sprint. In a situation where many resources are constrained elsewhere, Sprint may not be the best choice.

Improvement Process

Although most of the problems cannot be solved solely on the development side, I believe there are some actions that can be taken in the development environment.

When making improvements, it is essential to first consider the motivation and desired goals. So, the first step was preparing a document that briefly described the current problems encountered and possible solutions.

The main objectives are as follows:

• Development teams can freely switch between different branch codes. This is crucial because our development teams are responsible for different projects, and there are often several ongoing projects. Thus, situations like "Can I borrow the environment for testing? I'll return it after I'm done" can occur.
• QA and development teams do not share the same environment. This way, problems can be confirmed without confusion.

After preparing this document, I started discussing it with the team. This discussion had several purposes:

• Confirming that everyone has similar problems, ensuring that everyone has the same understanding of the situation.
• Colleagues have different considerations, and these considerations are critical to the successful implementation of subsequent improvements.
• Brainstorming, as my ideas may have flaws, and gathering everyone's thoughts may lead to better solutions.

Although the focus of this process was more on the overall improvement rather than implementation, I will briefly describe my approach here:

• Since we are running a single-page application (SPA), as long as we find a way to upload the built JavaScript, we can freely switch between different branch codes. This way, development does not need to share the environment with QA, and different development teams can switch environments as needed.
• Switching between different environments using query parameters. However, it is troublesome to always maintain the query parameters, especially when there is logic like redirects (e.g., redirecting after payment).
• Implementing a functionality similar to deploy preview (Netlify), where after submitting a pull request, the development team can click on a URL to see a demo.

After clarifying these approaches, I realized that there were several issues:

• When deploying, we also had to implement a separate Node.js server on the frontend, serving as a simple authentication server and providing some APIs for convenience. Therefore, it is not purely frontend, and the approach of file upload can be misleading. Additionally, modifying the codebase is more extensive and can lead to resistance.
• Deploy preview often requires dynamic DNS assistance, but our company uses a private cloud, and it is uncertain whether there is an API to achieve that. Furthermore, certain functionalities like SSO redirect to other pages have domain restrictions, so the domain name generated by deploy preview is rejected, resulting in login failures. Although it may work for functionalities that do not require login, the effectiveness is limited and does not justify the cost.

During this process, another engineer also experienced similar issues and voluntarily joined the improvement process (I initially thought I would have to fight alone until the end), and we started working together to clarify the problems. First, I explained the concept of Deploy Preview to him, and after understanding it, he proposed a new idea for the existing solution.

"The cost of dynamically adjusting DNS is high. What if we use Nginx for redirection?"

This means following the current deployment approach and creating a new machine (a few clicks in the internal private cloud), and then adding a redirection logic to the existing machine's Nginx.

If a certain cookie value is present, it redirects to the machine used for development; otherwise, it remains the same. This way, not only does the original codebase need no changes (at most modifying Jenkins scripts and Ansible scripts), but the existing deployment approach can also be reused, significantly reducing development costs.

After finalizing the solution, I sought the opinions of other teams, and for the most part, everyone had a positive and proactive attitude. Finally, I checked with the SRE team, and there were no issues. The implementation process came to a close.

After the Improvement

Honestly, it is difficult for people to perceive tangible results from just words on paper, even if they see a demo. The real feeling only comes when the new mechanism is actually used. Once everyone started using the new environment switching mechanism, the feedback was overwhelmingly positive, indicating that the process had indeed been improved.

Conclusion

This improvement process took a total of over three months, as it involved multiple teams, which took a bit longer. It is important not to be impatient at the beginning, as it can easily lead to a lack of understanding from the team members. I learned quite a few things during this process.

1. Start with writing documentation

Writing documentation is the first step in establishing team consensus. When writing documentation, it is important not to focus excessively on implementation details but rather on problem-solving and possible solutions. This allows the team to come up with better solutions through brainstorming. Another benefit of this approach is that the team feels that this is not just your idea alone but something the team has collectively discussed.

2. Approach from a collective perspective, not an individual one

One of the reasons why this improvement received positive evaluations is that it was not solely my idea but something many people in the team experienced. When making improvements, it is important not to base decisions solely on personal preferences like "this approach is more comfortable for me" or "I am more familiar with this technology." Instead, consider whether these improvements truly bring benefits. This way, people will support the changes, and you will have people on your side.

3. Do not excessively focus on technology

Honestly, this improvement did not require much technical complexity; it was just adding a conditional statement. However, the overall benefits were significant. Many engineers become overly fixated on the technology itself but often overlook the problem they are trying to solve. Technology is meant to serve humans, and if this fact is ignored, the technology loses its purpose.

4. Do not fear other teams

This improvement required assistance from other teams, which often discourages people, as most find it troublesome and prefer to avoid getting involved. However, as long as you can accurately explain the problems and proposed solutions, hitting pain points of others, many times people will be on your side. Sometimes, the concerns of the other team may sound like obstacles, but there is no need to over-speculate.

5. Find like-minded partners

I believe one of the significant factors contributing to the success of this improvement was the fact that another engineer also had an interest in this matter. Coincidentally, he was a backend engineer with a good understanding of architecture, which led to the subsequent nginx solution. However, finding like-minded partners is not easy, so when you encounter one, cherish them.