A Debugging Story — Connection Refused

Introduction

Starting about a few months ago, my Google Chrome frequently ran into inexplicable “connection refused” errors. Every now and then, certain websites just refused to load, yet they opened completely fine on other devices. Although it felt weird, I figured it was probably a server configuration issue on their end, and since it wasn’t causing much trouble, I just kept using my machine as is.

Setting Up a Custom Domain

Later on, after casually purchasing a .dev domain, things got really strange. I configured the CNAME record and verified that the SSL certificate was active, but no matter how many times I refreshed Chrome, it kept showing CONNECTION_REFUSED. Usually, there are a few common causes for this issue: either the server didn’t open the port, or there was a firewall misconfiguration. I checked both settings and found no problems whatsoever. Just to be safe, I ran dig xxx.dev +nostats +nocomments +nocmd to verify whether the CNAME was resolved correctly. It looked completely fine.

Next, I investigated potential SSL issues. I used SSL checker to check if the SSL certificate was set up properly—it was.

Alright, maybe the DNS changes just needed some time to propagate. I waited 10 minutes and checked again. Still nothing; Chrome stubbornly displayed a bleak CONNECTION_REFUSED. Next, I tried visiting the page on my phone and found that it loaded successfully.

At this point, I began suspecting that something might be misconfigured on my machine. I started digging into the relevant settings, first running curl blog.kalan.dev to see what happened. An error popped up:

curl: (7) Failed to connect to blog.kalan.dev port 80: Connection refused

Well, curl returned the exact same result as Chrome, confirming my suspicion that the issue lay within my machine’s own configuration. Next, I used wget to see what was going on under the hood:

wget https://blog.kalan.dev
--2019-03-18 23:11:46--  https://blog.kalan.dev/
Resolving blog.kalan.dev (blog.kalan.dev)... 127.0.0.1, 0.0.0.1
Connecting to blog.kalan.dev (blog.kalan.dev)|127.0.0.1|:443... failed: Connection refused.
Connecting to blog.kalan.dev (blog.kalan.dev)|0.0.0.1|:443... failed: No route to host.

Wait, resolving the host led to 127.0.0.1? Does that mean I added this domain to my hosts file? I had zero recollection of ever doing that, but holding onto a glimmer of hope, I checked both /etc/hosts and /private/etc/hosts.

The result was the same: I hadn’t added this domain to my local hosts file. But at least I knew the culprit symptom now—some configuration was routing blog.kalan.dev to 127.0.0.1. At this point, I had no clue what could possibly be causing it. But while tinkering around, I noticed something strange: any xxx.dev domain exhibited the exact same behavior, while other domains had no issues at all.

Suddenly, a memory sparked. About four years ago, while learning Ruby on Rails, I installed powder. In short, it automatically boots up a local server whenever you access a site—super convenient! So I happily installed it back then.

However, taking a closer look at the official documentation, section 3.1 states:

The POW_DOMAINS environment variable specifies a comma-separated list of top-level domains for which Pow will serve DNS queries and HTTP requests. The default value for this list are the two test,dev domains, meaning Pow will configure your system to resolve *.test and *.dev to 127.0.0.1 and serve apps in ~/.pow under the .test and .dev domains.

Aha! So you were the one hijacking my IPs! Because it resolved to 127.0.0.1 and my local machine wasn’t listening on port 443, it threw CONNECTION_REFUSED. Those websites I couldn’t open before? They were probably all using .dev domains.

A few takeaways worth noting:

• When encountering CONNECTION_REFUSED, immediately use wget to see if host resolution and connection routing are correct.
• Alternatively, run curl --ipv4 -v "your.site.com" to inspect the host connection details.