Thoughts After Five Years at LINE Fukuoka (Now LY Corporation)
Introduction
The end of December 2024 was my last day coming into the office at LINE, and I used up my remaining paid leave until the end of January 2025. This article is mainly based on how I feel and what I experienced now, so there may be differences from the current situation. All views are from my own position and based on my own experiences, so they may differ from the actual circumstances.
Why Japan?
I mentioned my motivation for learning Japanese and how I got there in another blog. If you’re interested, please take a look:
I first came up with the idea of living in Japan when I was in university. However, my financial situation wasn’t good at the time, so I didn’t have enough savings to afford a working holiday, and my grades weren’t good enough for an exchange program either.
Because I had to extend my studies and also serve in the military, I couldn’t easily go abroad. After receiving my draft notice in September 2018, I was sent to serve four months.
At that time, I had never even thought about working in Japan, let alone becoming a software engineer there. It was Denny who told me that rather than coming to Japan on a working holiday, I should find a full-time job instead, with regular weekends off and paid leave I could actually use. That made sense, so after finishing military service, I decided to start looking for work in Japan.
There were several reasons I chose LINE as well:
- I wanted to work with developers from all over the world
- I wanted to challenge myself in a larger organization
- There were no fixed clock-in/clock-out times
- It didn’t have the atmosphere of a traditional Japanese company (JTC)
Preparing for Interviews in Japan
Because I really love Japanese, before coming to Japan I ran an email newsletter called “Japanese Hundred Shop”. I wrote about 50 or 60 issues before eventually discontinuing it. During that time, I also took the N3 and N2 exams, and while I was in the military, I passed N1. Compared with many people who start learning Japanese only after arriving in Japan, that order felt quite different to me.
Before coming to Japan, I had already accumulated several years of development experience, and I was also writing technical articles regularly. Given my background at the time, that more or less helped me get the chance to interview at LINE. Around 2018 to 2019, LINE Fukuoka was heavily recruiting in Taiwan, and luckily Denny was working there, so after I finished my four-month military service I asked him for a referral.
A side note here: I believe the development of AI has already started rewriting the rules of software engineering interviews, so my interview experience back then may be completely different from what it is now. Please treat this as reference only.
As for salary, the stated minimum annual salary was 6 million yen (the exchange rate back then was still 0.3! So that was 1.8 million TWD). For my situation at the time, that was pretty good. Also, if you were applying for a Backend role, there was a chance to negotiate above 10 million yen.
Why Fukuoka?
My first introduction to Kyushu was through the anime Sakamichi no Apollon. Later, I traveled to Fukuoka twice in 2018, and I really liked the atmosphere here. On top of that, LINE Fukuoka was in Fukuoka.
For more impressions, you can refer to my thoughts after living in Fukuoka for a year.
Five years later, I had already bought a home in Fukuoka, so I can say with more confidence that Fukuoka really is wonderful. Almost every Japanese colleague I met who had previously lived in Tokyo spoke very highly of Fukuoka. Tokyo has more resources, more job opportunities, more people, more expensive, and better transportation. That’s true too.
A Short Commuting Life
I moved to Fukuoka in July 2019, just one year before the pandemic broke out, so for about half a year I commuted to the office by subway. One of the benefits of working here was not having fixed clock-in and clock-out times.
I still remember that around February 2020, when COVID broke out, the company announced indefinite remote work. Japan was basically closed off, and it took two or three years before tourism reopened. So in a sense, arriving in Japan during that period was quite fortunate.
(Here’s a photo of the plane on my way back to Taiwan at the end of 2021, with fewer than ten people on board)
What I Did at LINE
Here I’ll summarize some of the more interesting things I did at LINE. Of course, this also includes a lot of day-to-day work like filing tickets, fixing bugs, building features, and so on. I’ll just pick a few more interesting ones to talk about.
LINE’s infrastructure uses a private cloud called Verda. You can think of it as AWS: it doesn’t have all of AWS’s features, but it has enough components to build a complete web service. For example:
- Computing
- Load Balancer
- Lambda function
- K8S
- Storage and CDN
- Database
There were two services I especially liked, so I’ll briefly mention them here:
- A solution similar to Cloudflare Images: web and app services inevitably need multimedia operations, such as crop, resize, converting files to .webp, blur, and so on. If you do these directly in the application, it often becomes a bottleneck once the scale grows, and LINE was exactly a platform of that scale. This tool was really great, and it saved backend engineers a lot of work.
- Central Dogma: this is a distributed configuration management solution. Its main goals are to avoid having to restart machines or redeploy after updating config files, and to provide high availability. On the backend, Central Dogma has git-like operations for version control, and changes can also be broadcast to servers via pub/sub. It looks simple, but it’s extremely useful.
I won’t go into LINE’s tech stack too much. Broadly speaking, it was basically the full Apache buffet. Kafka, HBase, Cassandra, Airflow—different technologies were chosen depending on the characteristics of each service.
Slackbot
In my first project, because our deployment mechanism was based on updating backend tags, and the company also provided a PaaS-like internal tool that could integrate directly with internal GitHub for one-click deployment, I wrote a Slackbot for release management. Through Slack, we could trigger deployment directly and simplify what used to be a multi-step process into a single Slack command.
That was still the era of Hubot, but for some reason I’ve forgotten, Hubot became unusable, so I wrote my own Slackbot with TypeScript support. Slack messages are assembled as JSON, so describing them with a JSX-like syntax was actually very intuitive. For that part, I referred to my former colleague Kai’s slack-blockx.
Basically, what it did was condense a release process that originally had to go through several systems into a single Slack command:
@hubot release beta my-org my-repo main v1.2.0
Behind the scenes, it would validate the version number, create a tag, open a release, and then post the commit diff from that release back to the channel as a changelog. Another point was that the command itself was designed as a plug-in system: each command was just an object, and even “who can use it” and “which channel can use it” were defined declaratively, so anything not allowed was blocked uniformly before dispatch:
const deploy: Command = {
name: 'deploy',
command: /deploy (alpha|beta) ([^ ]+) ([^ ]+) ([^ ]+)/,
isAuthedUser: isMember, // only members can trigger it
enableChannels: channelIsValid, // only works in specific channels
action: async (matches, message, client) => { /* ... */ },
};This framework was also intentionally modularized, with each command split into a separate file instead of one giant function full of if/else blocks. I also cared a lot about testability—Slackbots are annoying because every test requires actually connecting to Slack and sending messages, so feedback is slow. I wrapped all Slack communication in a class, kept the command logic completely independent of the Slack SDK, and even built a terminal adapter so I could test commands locally by typing them directly in the terminal without opening Slack at all.
Looking back now, it feels nostalgic, because with the same architecture and code, AI could probably get it done in a day. In fact, instead of constraining an AI agent with code, maybe you should just tell it what you want done directly.
Improving the Deployment Environment
At the time, our service had several features under development at once. Team members were assigned by project, but because we shared the same codebase, QA testing easily conflicted when happening at the same time. We also only had a fixed number of environments. So another colleague and I used nginx cookies to determine which branch to route to, and forwarded traffic to the corresponding environment. During that period, I learned a lot about Ansible from a Taiwanese colleague I worked with, and I’m very grateful that he patiently taught me hands-on.
However, switching via cookies was cumbersome and unintuitive. Later, another colleague made a big leap forward by using Envoy with Docker and integrating GitHub so that pushes automatically updated deployments and domains were assigned dynamically. That kind of thing feels natural on a PaaS like Vercel, but in a private cloud it’s not easy to integrate at all.
Landing Page SSR
A financial service was about to launch a customized landing page, but at the time the internal tools could basically only host images (a lot of Japanese services are just full-page images made for mobile). Other teams in the same project had already adopted Next.js, but we were still using a pure SPA plus a Node.js server. Because of the architecture, we could only place static files and couldn’t run it as a server, so I started pushing for SSR adoption.
Looking back now, trying to push SSR in a large organization without Vercel is basically a nightmare. Just getting machines meant negotiating with SRE for resources, and at the time I simply wasn’t strong enough to push the project through on my own. My former manager, an Engineering Manager, helped me a lot. I wrote out the architecture, plan, and purpose clearly, communicated with SRE, and was able to get machines for production. I handled development and QA environments myself, including grinding through Ansible.
That was my busiest year, and also the year I learned the most. We later reused that SSR mechanism for other features too. Development was still labor-intensive at first, but as the number of components grew, the time savings became bigger and bigger, and the SEO exposure the planning team had been emphasizing was a real gain.
Sentry Automation
This product was FinTech, and because resources were abundant, we spent a lot of time on performance tuning. Around that time, Sentry had introduced Core Web Vitals, and in the dashboard you could see several performance metrics for the website. So I integrated Sentry’s API to build an automatic notification feature that shared the results to Slack every day.
Internal Tool K8S
There were far too many workflows in our development process that we wanted to automate. LINE had a standard development process, but it still varied depending on the service and team. For example:
- Who is responsible for hosting the Daily meeting this week
- Automatically updating JIRA status after a PR is merged
- Slackbot
The problem was that every time we deployed an internal tool, we depended on that developer, and it was hard for others to step in and improve it. At the time, the internal platform could already create K8S clusters, so one colleague took the lead and built a platform for everyone. As long as it was an internal tool, it could be integrated into K8S, and developers only had to commit—the CI handled deployment.
LINE Securities
I mainly served as the frontend Tech Lead for Tsumitate NISA, as well as working on the integration of credit card investing (クレカ投資) with NISA. The frontend team went through staffing changes, and two Japanese colleagues and one French colleague joined. They were all very strong engineers and helped me a lot. The French colleague had a very distinctive personality, liked gaming and Japanese anime, and there was always plenty to talk about.
When I joined, the team was mostly made up of new grads much younger than me, but they were amazingly capable, and there were also some people deeply involved in open source contributions. Every week we also had regular technical discussions with each other.
Even though it was a financial service, the team actively adopted new technologies like React and Kotlin on the backend. You could tell everyone genuinely loved technology, rather than treating it like just another finance job.
Unfortunately, the service didn’t make it to the end. LINE Securities’ securities business was fully transferred to Nomura Securities in August 2024, effectively bringing it to a close. Seeing something I had worked on get shut down was still a complicated feeling.
Impression-based feed ranking
In the later period at LINE, I had the chance to move to the backend and take over an impression-ranking mechanism for a feed, which recorded which content users actually saw in the feed. It sounds simple, but the service had tens of millions of MAUs, and once the scale got that large, every naive approach became a landmine that could crush the backend. The feed used server-driven UI, meaning the screen wasn’t hardcoded into the app; instead, the backend returned a JSON contract for the client to render. The trade-off was that every impression had to be sent back to the backend.
The most intuitive solution would be “send one API call every time the user sees one item,” but at this scale, impressions alone would be enough to destroy the backend. So we changed it so the client sent Kafka events directly, fully separating the write path from the serving path. Deduplication and ranking were handled by Redis Sorted Sets, with scores based on recency and a 24-hour TTL so expired data would fall out automatically. Unfortunately, this design never made it to production—I left while it was still in testing.
Hackathons
The company held hackathons almost every year, and I joined several of them. I’d like to share a few.
Carbon Dioxide Monitoring
For details, you can refer to this project I built together with a Dutch colleague who was on the same project at the time.
The main idea was to measure carbon dioxide concentration using a CO2 sensor, connect an ESP32 to MQTT over Wi-Fi, have the server receive the data, and then display it with Grafana.
You can read more about the principles in this article:
IoT Pomodoro Timer
Well… it was also IoT. I found Swift’s BLE API pretty nice to use, so I built a prototype for a physical Pomodoro timer that connected via Bluetooth, let you start and pause it from a computer, and included statistics functionality. I thought this feature was great, if I may say so myself, but since we only had a day and a half of work time, I didn’t continue with it.
Bluetooth Meeting Display
I discovered that in macOS, built-in calendar events can be monitored, and one of my favorite small tools, MeetingBar, is based on this idea. (Download it quickly so you don’t get so focused on development that you forget to attend meetings.)
That means I could also write an app to broadcast those events over Bluetooth and have a display show them. The same concept can be extended indefinitely—as long as you can broadcast the thing you want to monitor.
Manager A and Manager B
I’d like to spend some time talking about two managers who helped me grow a lot.
Manager A at LINE was Japanese and led the entire frontend department. I thought he was excellent at communication, team leadership, and dealing with different types of developers. Even though he was very humble and often said he wasn’t as technically strong as everyone else, he was actually constantly keeping up with technical trends and following everyone’s progress.
After talking more deeply with him, I found out he had basically no worries left in life 😂 He was already a father of two, his wife ran a café, his savings were bottomless, and work was just a place to make friends.
Manager A helped me grow a great deal in my career. We followed each other on Twitter, and he would proactively check in on what I was up to in a rather subtle way. Whether or not to share social media accounts with coworkers is a matter of personal preference; for me, if I get along well with someone, I usually ask. After all, if we can work at the same company together, that’s already a kind of fate.
When I first joined, the environment here was very different from the workplaces I’d known before. I was relatively picky about technical details within the team, but my way of expressing myself was poor, and I often nitpicked my colleagues’ proposals, which made me seem very forceful and hard to work with.
Later, a colleague gave me a very harsh negative evaluation, saying that my behavior seriously affected team morale. He wrote directly that my behavior had pissed him off. Manager A also seriously warned me that in the workplace, what matters more is influence: my approach wasn’t helping the project move forward, and was in fact obstructing it.
At first, I was very unconvinced by this feedback and often thought, “Why does everyone seem so unconcerned with code quality?” while also receiving negative evaluations.
But Manager A gradually and patiently guided me to who I am now. He himself read quite a lot, and recommended that I read 人を動かす, translated into Chinese as How to Win Friends and Influence People. I can say my change in mindset at the time was all because of that book—it was the first time I truly realized that there was more to life than technology. Because of that, while I was working there, I also read many books on management in addition to technical books.
The other manager, Manager B, was an Engineering Manager and a manager on the same project. He was also very good at team-building. Although we only worked together for one year, Manager B helped with many things on the project that required back-and-forth communication, coordination, or even cross-department requests for resources.
I got along well with Manager B. I told him I was translating my old articles into Japanese, and he immediately helped me proofread and correct them. And after I solved a project issue, Manager B even bought me a Nezuko figurine.
Personally, I’m not especially fond of using rewards to motivate people, but it is indeed an effective way to boost individual motivation.
Later, I occasionally played Splatoon 2 with Manager B and his daughter. When Manager B found out I wanted to replace my desk, he even drove me to the home center to buy lumber.
After the project changed, I realized just how much difference there can be between managers.
Looking back now, most of the managers I met in my career were excellent, and they gave me a lot of growth and stimulation. I’m truly grateful to all the managers I met at work.
Some Fun Things
Happy Friday
Back then, on the last Friday of every month, there would be Happy Friday. The company would prepare simple snacks and drinks, and everyone could chat in the café space. After COVID started, it stopped being held, and I really miss that period.
There was also one afternoon each week when the company held a general meeting. After the announcements, the remaining time was usually used for:
- New colleagues to introduce themselves
- Volunteering to give a technical talk, on any topic
This was a chance to hear how problems in other projects were solved, and to interact with colleagues. On Fridays there were also sharing sessions within the frontend team.
Business Trip to Korea
At the end of 2019, the company held the UIT Global Workshop, bringing frontend engineers from Japan, Thailand, Taiwan, Korea, and other places together for an event at NAVER’s Connect One in Korea.
Connect One is located in Chuncheon, South Korea, and is a training center specially built by NAVER for employees. It has all kinds of office spaces, meeting rooms, and an auditorium, and even the dormitory is quite luxurious.
The main agenda of the event was mostly sharing technical cases actually applied in product development. The rest of the time was mostly eating, networking, and getting to know engineers from different countries.
LINE Developer Day 2019
Right after that came another conference: a business trip to Tokyo on the company’s budget to attend LINE Developer Day 2019. My impressions of that event were written in these two articles: Part 1, Part 2: Slack Rewrote Search Services with Armeria.
What I gained most from that conference was:
- I learned that LINE had an open-source microservice framework called Armeria, and that Slack was using it
- Real-world applications of BERT
Why I Quit
The reason I left wasn’t a single event. It was more like several lines gradually converging into one conclusion.
At the most fundamental level, it’s hard to step outside the framework when working at a large company. Climbing upward is certainly possible, but at the time I didn’t understand enough yet and hadn’t figured out the rules of the game. At the same time, I felt my career had already hit a plateau and a bottleneck; on top of that, the two services I had participated in back-to-back both ended up being terminated, which in terms of results looked like I had achieved nothing.
Managers were another reason. The managers I met later felt very much like NPCs: when I had problems, their advice was usually very general, and I also didn’t think they cared enough about the team. 1-on-1s felt more like something done to get through daily work. This became even more obvious after I moved to backend, where I often had disagreements and friction with one of my managers.
When I was assigned to the last project, the product itself was more business-driven, and it often felt like we spent several weeks just to produce one small feature. The interesting part was that I had a chance to participate in backend development. After moving to backend, I learned a lot about what to pay attention to when handling large-scale traffic, and I was also responsible for the recommendation system design. Unfortunately, it was still in testing and hadn’t launched before I left. And this final project often required us to optimize for increasing ad impressions, which made me feel more and more lost.
What bothered me even more was the logic behind promotion-driven evaluations.
I saw a lot of things with little real value being packaged as if they had major impact. For example, the company once promoted standardization and created a configuration repository for everyone to use. But after the buzz died down, it was simply left there, and that config repo aged poorly and didn’t adapt well to different projects, so in the end each project just configured things separately anyway.
But later I came to understand it. Even if something has no value, you still need the ability to explain clearly why it has no value. And many things, once packaged properly, can go from 60 points to 100 points. That also means if you think something is worth 100 points, you have to package it to 200 points before it can actually be seen.
In a large organization, many metrics simply can’t be quantified. Managers can only look at numbers and presentations—when a proxy metric becomes the goal, it stops being a good proxy metric. Of course, my inability to play the game well within the established rules was entirely my own problem. I couldn’t change the rules, and I also failed to achieve a ranking within those rules.
The final straw was AI. 2024 happened to be the year when LLMs were flourishing everywhere. In a large company, introducing AI inevitably involves all kinds of compliance, security, and data retention issues, so it’s not easy to use state-of-the-art models.
I believe AI will be a major trend in the coming years, and software development will change dramatically as a result, so I wanted to explore it more freely. I agonized over it for a long time, but in the end I decided to quit. Making that decision was difficult for me because it meant facing all kinds of uncertainty, but looking back now, I’m glad I made it.
What I Regret
My relationships with others are deeply influenced by my upbringing. I’m not very proactive in approaching people, I’m not good at socializing, and I even prefer being alone.
That meant I didn’t actively expand my network beyond the people on my project, whether that was people from non-engineering departments or developers from other projects. I hadn’t realized how important that was before.
After the pandemic, remote work made it hard to create opportunities to interact, which was a shame for me. As a result, I left very alone, and handing back all my equipment still felt quite empty and uncomfortable.
A few miscellaneous things to mention here:
- There was a colleague from Australia. He had mentioned this at the beginning, but I mistakenly thought he was from the UK.
- There was a French colleague, and I asked him, “So, do you know how to use chopsticks now?” Looking back, I think that question was quite rude, because I had assumed that because he wasn’t Asian, he naturally wouldn’t know how to use chopsticks.
One other thing stuck with me more deeply. There was a conflict between colleagues once, and I chose to stay silent. In my view, the incident itself wasn’t that serious, but my thoughts were never conveyed to the person involved.
Looking back, I think the emotional pressure he had to deal with was huge. I’ve faced many uncomfortable situations in the same way. But if you want to keep moving up, having uncomfortable conversations is unavoidable; in fact, many times, uncomfortable conversations are exactly what you have to face. That happened a long time ago, but I’ve never forgotten it.
Later, I had dinner with the colleagues I had worked with the longest. I’m very grateful to them.
What I Learned
I truly did gain unprecedented growth opportunities at the company—whether it was designing an appropriate architecture for high traffic, building servers from scratch, communicating with developers from many countries, working with cross-functional teams, or leading a team to deliver several features. These were all invaluable experiences, and only at this level of traffic and scale can you see things you would never have even thought about before.
- Technology is a weapon, but it is not everything. Especially during the project process, because many of LINE’s workflows were standardized. But that may simply have been the result of managers or higher-ups helping to smooth things over, not my own achievement.
- If you want to exert influence, you have to deal with many people-related issues. I learned a lot at LINE about processes, documentation, and how to develop software, but when it comes to “people,” I still have a lot to learn.
Afterword
In the past, whenever people talked about working in Japan, there were often objections, such as low salaries, high taxes, or Japanese corporate culture. Every company has its imperfections, and LINE certainly did too. But I still recommend that if you have the chance, you should try working at a company of this scale.
I personally really like Japan. Of course, Taiwan itself is also a very comfortable place to live. In terms of work environment alone, Japan may still be looked down on by many people, who feel Silicon Valley is the better choice—I actually agree with that. But I just want to stay in Japan.
I’ve talked with many colleagues from Europe, the US, and even India. The fact that you can leave your computer at Starbucks and go to the bathroom, and come back without it being stolen, or walk around with your phone out without it being snatched away—those things are hard to imagine happening in Europe or the US. Among developed countries, Japan maintains a high level of social order and relatively low prices.
Be kind, and help others as much as you can within your abilities. That sounds obvious, but I’ve seen too many people use technology as a means of gatekeeping and become more arrogant as they climb. The technical barrier will eventually be leveled by AI, and when that happens, the people who remain will be the ones willing to lend a hand, not the ones standing tallest on their pedestal.
That’s why I also prefer engineers with solid fundamentals. This person wasn’t mentioned anywhere else in the article, so I’ll add him here. There are three things about him that left a particularly deep impression on me.
The first happened during the carbon dioxide monitoring hackathon mentioned earlier. At the time, I bought a CO2 sensor that came with an Arduino SDK, and I originally wanted to use it directly. But he said that since we were doing a hackathon, we should learn something from it, and if we used only ready-made kits, wouldn’t that just be the same as what we normally do—calling APIs? I thought that made a lot of sense, so we opened the sensor’s datasheet together and implemented the communication protocol from scratch.
The second was when he got stuck writing tests for form submission (multipart/form-data) and came to ask me how to simulate a form request. He knew that a form was just a somewhat special kind of HTTP request, but he couldn’t make the library work no matter how long he tried. I took a look and found that the form boundary hadn’t been set properly. Once I fixed that, it worked.
The third was when we were talking about our favorite YouTube channels, and he recommended Ben Eater to me. For anyone who enjoys digging into how computers work, it’s an absolute treasure. To explain how networks work, he starts from the signals on transmission lines (with an oscilloscope) and goes all the way up to the application layer; he has also built a simple 8-bit CPU from scratch on a breadboard.
I’ve found that in an era of rapid AI advancement, this kind of ability is becoming increasingly rare. But I think that no matter the era, it is a truly valuable quality.
Finally, I’ll end with a bunch of photos from life in Japan. I hope this article gives readers some ideas. And if you ever come to Fukuoka, feel free to grab a coffee with me!
Related Posts
- The Basket and the Spear This comes from one of the books I loved reading as a child — a story from *The Fifth Discipline*. As a kid, of course I didn’t understand anything about organizations or teams; I just liked the story itself. Only when I grew up did I realize that the story in *The Fifth Discipline* is really centered on teamwork.
- 15. A Life Filled with Gratitude When I was young, my family was not well off, and life was not going smoothly. Without financial support, the choices available to me became quite limited. Yet in such an environment, I still ...
- 14. 1,000 True Fans This is a business strategy for creators such as artists, writers, musicians, and other creators. Kelly argues that creators do not need to become global superstars or have millions of followers; they only need a group of loyal supporters to sustainably make a living and keep creating.
- 13. Embracing Uncertainty Nassim, the author of *The Black Swan Effect*, once proposed a concept: things that seem completely impossible can happen at unexpected moments, and most of the time we cannot predict them, yet they can have an enormous impact. The best example is COVID-19 in 2020. Under the influence of the pandemic, people’s lives and work patterns were completely changed, and many shops shut down or went bankrupt because they could not hold on; it was not until 2023 that things began to improve.