This week, I worked on something very important for CodeAtlas:
👉 a logging system that is clean, stable, and easy to understand.

At first, I tried many different methods.
Some ideas worked, some failed, some broke the logs completely.
But after testing different approaches, I finally created a logging pipeline that works perfectly.

Why I Needed a Better Logging System

In Week 1, I tried storing logs directly in files.
But I faced many problems:

• Log files became too big

• Some logs disappeared or overwrote

• Searching inside log files was difficult

• Hard to monitor what was happening inside services

So in Week 2, I decided to build a proper logging system step by step and follow this structure through out the all the micro-servicer application

Step 1: Structured Logging With Winston

The first upgrade I made was using Winston, a logging library for Node.js.

✔ What Winston Does For Me

• Writes logs in JSON format (clean and easy to read by machines)

• Saves logs inside folders based on each service

• Creates 3 different files:

  • info.log

  • warn.log

  • error.log

    

Request Metadata (Very Important!)

I also created a middleware that adds useful details to every log:

• request_id (unique ID to trace the request)

• ip address

• path

• userAgent

Now every log line tells the full story of what happened.

 {
      timestamp,
      service_name,
      level,
      message,
      request_id,
      metadata: {
        ip,
        path,
        userAgent,
      },
    };

Step 2: Saving Logs into MySQL

Reading logs from files is okay, but searching inside files is hard.
SQL is much better for searching and filtering.

So I created a custom Winston transport that saves every log inside MySQL.

✔ MySQL Table Columns

• timestamp

• service_name

• level

• message

• request_id

• ip

• path

• userAgent

4. Step 3: Real-Time Log Watcher

After trying many ideas, the log watcher became the best solution.

It solves most of the problems:

• Big file sizes

• Lost logs

• Duplicate logs

• Delayed updates

✔ What the log watcher does

• Watches log files continuously

• Tracks how many lines were already processed (using pointers.json)

• Stores logs into MySQL in small batches

• Clears the file safely after processing

• Works even after restart

• No log is lost

• File size stays small

This made the whole system feel stable and predictable.

5. Final Data Flow: How Everything Works

Here is the simple flow:

Incoming Request 
        ↓
Middleware adds metadata
        ↓
Winston writes JSON logs → info.log / warn.log / error.log
        ↓
Log Watcher reads logs in real time
        ↓
Log Watcher inserts logs into MySQL
        ↓
Log Watcher clears file safely to keep size small
        ↓
MySQL stores all logs for analytics and debugging

What I Learned This Week

This week taught me something important:

A good logging system is not just writing logs.
It’s making sure logs are safe, searchable, and never lost.

By trying different ideas (many failed 😅), I finally built a system that works smoothly.

What’s Coming Next

Next week I will work on:

• Better GitHub OAuth scopes

• Routes to search GitHub repositories

• Adding monitoring to more services

• Connecting logs to Grafana dashboards

🤝 Contributions

Ideas, improvements, and suggestions are always welcome.
You’re encouraged to submit issues or pull requests to help evolve the platform.

Github repo