Setting up a server in a company is not as easy as I imagined.

Motivation

During the development of this project, a need arose on the planning side for a dynamic update of data through API calls on what was previously a purely static Landing Page. As the interactivity of the page increased, the original static setup using pug+webpack+jQuery became inadequate. Therefore, we decided to integrate next.js in the development of the new version.

Let’s first discuss the architecture of the original Landing Page. During CI runs, a static HTML page would be packaged and uploaded to a CDN along with CSS, JavaScript, and images. The HTML itself was placed directly on the server, with nginx acting as a reverse proxy in front.

Huh? If everything is specifically hosted on the CDN, why is the HTML placed on the server?

This relates to historical factors. The company uses a self-built CDN that has limited support for custom domains. Therefore, if we uploaded the HTML to the CDN, the domain name would change, which is not ideal for the planning team.

Additionally, aside from the Landing Page created by front-end developers, the planning side sometimes uses a template generator to create pages, which requires a fixed domain that cannot be altered. Thus, to accommodate both types (template Landing Page and the one developed by the team), placing an nginx as a proxy was a reasonable compromise.

However, a problem arose: next.js was incompatible with the original development architecture. Rewriting the entire Landing Page in React seemed impractical, so we decided to set up a new server specifically for developing Landing Pages with next.js, and then route through nginx to ensure the same domain.

Issues

Before continuing with the main content, let’s outline the problems with the original static server and the improvements that followed the addition of the new server.

1. SEO

The biggest consideration was SEO. Although the original static server could achieve SEO, as mentioned earlier, to fetch data via an API or meet requirements like blog articles, the data was generated at build time. This meant the API would have to be called from the front-end JavaScript using ajax or fetch, which is ineffective for SEO. Another concern was the CORS issue; since the API server and Landing Page had different domains, CORS problems were inevitable.

2. Performance

While the amount of data used on the Landing Page was not large, implementing SSR (Server-Side Rendering) could enhance performance.

3. Advantages of the Server

With node.js as the backend server, we could more flexibly implement the planning team's requirements in the future. Next.js also supports various building methods, such as getStaticProps for static file generation and getServerSideProps for SSR. Having a node.js backend also makes it easier to implement caching or database access needs.

Given these reasons, we decided to establish a new server. However, little did we know that the real challenges were only beginning.

Struggles

Coordinating with SRE

To set up the server, we needed to coordinate with SRE and explain the issues involved. Perhaps due to the SRE's inherently cautious nature, they had many concerns and were responsible for other projects, which prolonged the communication process. However, our motivation was clear, and I had already clarified the entire architecture, so approval was granted relatively quickly.

Our cloud environment is a private cloud, and in the alpha environment, each developer can create their own machines. Thus, the alpha setup was completed swiftly. However, both beta and production environments required relatively complex procedures, including submitting ACL requests and relying on SRE to set things up. While creating machines was straightforward, installing various packages was troublesome. Thankfully, with SRE's assistance, packages like monitoring tools, node.js, and nginx were successfully installed.

The Historical Nginx Configuration

Setting up the server was simple, but what truly troubled me was the ancient nginx configuration file. It contained various redirection rules from multiple sources, handling maintenance mode, and special treatments for certain specific paths.

Deployment via Ansible Playbook

SRE assisted with the environment installation, but the next step was deployment. This part needed to be clarified independently. The company primarily uses an ansible playbook along with awx for deployment (AWX is a GUI that allows executing playbooks via API on the web). Fortunately, I had collaborated with colleagues on other projects, gaining some experience with playbooks; otherwise, the daunting amount of YAML could have been overwhelming.

Docker

For convenience, the recent deployment approach involved creating a machine to install Docker and then running a systemctl service within it to manage the Docker image.

I initially thought packaging would be straightforward, but various issues emerged during implementation. This related back to the project's history; we initially used a lerna+mono repo structure, where each subfolder referenced the root files, making it challenging to package just a specific subfolder.

Lerna has two main features:

• Package installations hoist dependencies used across projects to the root node_modules.
• Within sub-projects, you can reference functions from other sub-projects using import a from 'sub-project' (this works by adding a symbolic link in the node_modules).

The resultant image size reached a staggering 1GB, further highlighting the daunting nature of node_modules. The symbolic link issue also consumed a lot of debugging time; without creating the link, npm would attempt to fetch the sub-project package from the internet, which was bound to fail.

Jenkins Integration

The company uses Jenkins for integration, so Docker images are also built and uploaded to the company's internal Docker hub via Jenkins. Later, I discovered that Jenkins sometimes encountered inexplicable errors during the build process, but after a few retries, it would return to normal.

Risk Assessment

With the establishment of a new server, executing a Risk Assessment was necessary as part of the procedure.

Security Check

Although there is a Security Check for major projects, the new server setup also required the Security Team to examine security issues and make necessary corrections.

Misestimation of Timeline

The introduction of next.js was not initiated by our team; rather, colleagues from other offices integrated it into their projects. I had assumed the server setup was complete, and that we could simply build upon it for developing the Landing Page. However, it turned out that we only utilized next.js's SSG (Static Site Generation) capabilities and merely uploaded the generated static files to the original Landing Page server.

At that moment, I was somewhat taken aback, but with over a month until QA testing and since it was just layering next.js's server functionality onto the existing setup, I thought we would be just fine. However, by that time, some features had already entered QA testing, and the development of the Landing Page itself had not yet begun. Although the page was completed in just a week, handling QA, communicating with other departments, designing architecture, and the entire process consumed a month without me realizing it. Moreover, new projects had entered the evaluation phase, and with deadlines looming, I found myself overwhelmed without anyone available to assist.

During this period, my stress levels were quite high, and I nearly burned out working overtime on weekends.

Reflection

For unexpected demands like this, it's often necessary to allocate more time for preparation. Honestly, purposes like improving SEO are not particularly attractive... and likely won't yield significant performance metrics. Sigh, as I grow older, I need to rethink my working methods.

1. I Became the Bottleneck

In setting up this server, so many aspects required knowledge beyond front-end development, and one needed to be quite familiar with the company’s internal tools to succeed. Many things I figured out step by step by examining documentation, which had little relevance to front-end work, making it difficult for team members to assist. I've been pondering better approaches, but once you give up, the result is proposals left untouched, with no one willing to dig into the details. Nonetheless, this is not a good situation.

2. Underestimating the Complexity of Server Setup

Rather than saying server setup is inherently difficult, the cross-department communication and subsequent preparation work inevitably prolonged the development cycle. Frankly, I didn't realize how complex it would be until I went through the process. Next time, I can consolidate this experience into documentation to make future endeavors easier.

3. The Department is Primarily Japanese

Other than our team, most other departments are primarily composed of Japanese personnel. Therefore, my insufficient Japanese skills can easily lead to communication failures. At times, I become the bridge for communication, or I find myself needing to participate in many discussions, which is not ideal. I’m still contemplating how to improve this situation.

4. Inadequate Preliminary Preparation

Initially, I failed to recognize that lacking a server was due to insufficient preliminary research, which is something I need to reflect upon. In the future, if similar server setups are needed, it would be better to allocate more time to prevent unforeseen circumstances.

5. Multiple Timelines Colliding

In addition to the server setup, I was also juggling many overlapping timelines, such as fixing existing QA issues, developing the Landing Page, conducting code reviews, communicating with SRE, and discussing changes in requirements. This constant context switching further compressed the time I could dedicate to focusing on the server setup.