How to Collect and Centralize Logs in Golang Applications

This article summarizes my takeaways after reading How to collect, standardize, and centralize Golang logs | Datadog.

When handling logging, there are typically a few approaches to consider:

1. Pass the logger as a parameter whenever logging is needed.
2. Uniformly encapsulate it within the context, retrieving it when needed.
3. Wrap the logger in a package and expose a variable for other packages to use.

Obviously, the first approach quickly runs into issues: your function parameters will start looking rather strange. For example:

func (p *Post) CreatePost(name string, content string, logger *log.Logger) {
  ///
  if err != nil {
    logger.Fatal("error!")
  }
}

Having to pass a logger into the business logic of CreatePost easily pollutes your function signature, and you might end up passing the logger down through multiple layers. The biggest advantage of this approach, however, is that dependencies are explicit—you can clearly tell from the function signature that a logger is used inside, making it very easy to mock the logger during testing.

We can adopt the second approach: explicitly pass a context into functions and store the logger inside it. When needed, retrieve it with log.Logger.FromContext(ctx). I haven’t thought of any particularly obvious drawbacks to this yet.

const (
  ContextKeyLogger = "logger"
)
func main() {

  ctx := context.Background()
  loggerCtx := context.WithValue(ctx, ContextKeyLogger, log.Logger)
  
}

The third approach is what I currently use: wrap the logger into a package and expose a variable for other packages to consume. The biggest benefit of this is convenience—it’s plug-and-play. The downside is that testing becomes less straightforward. Because it isn’t parameterized, the logger is hard to mock. (Perhaps there are other good solutions?)

package logging

var Logger *log.Logger

func init() {
  Logger = &log.Logger{
    // your setting
  }
}

Next, let’s look at the recommendations from the article:

1. Recommended to use logrus. Logrus is a great logging package that seamlessly supports Go’s standard log, making migration painless. With its hook mechanism, you can easily integrate with third-party reporting platforms (like Datadog, Sentry, etc.).
2. Use JSON as the log format: JSON is easy to parse, supported across all major languages, and straightforward for third-party platforms to analyze. It is highly recommended. In logrus, you can directly set the output using SetFormatter(&logrus.JSONFormatter{}).
3. Use a unified interface.
4. Avoid calling the logger inside goroutines whenever possible. Aside from concurrency concerns, the logger’s internal implementation might also spawn goroutines, making the overall behavior harder to predict and manage.
5. Save logs to local files even if you use third-party platforms. This prevents logs from being lost due to network issues and ensures you can always retrieve them.
6. If using a cluster-like architecture, you might need something like syslog to centralize logs onto a dedicated log server.
7. If using container services like Docker, you may need to capture Docker’s STDOUT.

Miscellaneous

Regarding log.Fatal() and log.Panic(): Fatal quietly calls os.Exit(1) under the hood (alright, I know it’s in the docs, but many people probably haven’t read them), while Panic quietly calls panic. Be especially careful when using them.