This week in CodeAtlas was about moving from “it works” to “it survives real-world usage.” I focused on GitHub rate limits, backend graph exploration, logging, data integrity, and—most importantly—running a hard production readiness review that exposed some uncomfortable but necessary truths.

Lots of fixes, some architectural rewrites, and a big step toward scalability. To track down the root causes of bottlenecks, I leaned heavily on Google + anti-gravity thinking—turning obscure errors into actionable fixes.

1. GitHub API Rate Limiting (Fixing 403 & 429 Errors)

As repository size grew, GitHub started pushing back hard.

The Problem

• Recursive GitHub API calls caused:

  • 403 Forbidden

  • 429 Too Many Requests

• Large repos + aggressive traversal = instant throttling

• Memory spikes from dumping entire directory trees at once

The Solution

I implemented a multi-layered defense:

Backoff Strategy

• Detects 403 / 429 responses

• Respects Retry-After headers

• Falls back to exponential backoff (default: 2s)

Request Throttling

• Added a 350ms delay between API calls

• Smoothed traffic instead of bursting

User Token Priority

• Always prefer the authenticated user’s githubAccessToken
  → 5,000 requests/hour

• Server token is no longer the default path

Lazy Directory Traversal

• Replaced “fetch everything” with BFS directory-by-directory traversal

• Lower memory usage

• Safer for large repositories

✅ Result: Stable GitHub sync even on large codebases

2. Graph Explorer Backend APIs (Neo4j)

To support frontend visualization, I built a set of lightweight, UI-friendly APIs for graph exploration.

New Endpoints

GET /api/graph/start

• Loads the initial graph

• Hard limit: 30 nodes

• Entry point for visualization

GET /api/graph/expand

• Fetches depth-1 neighbors for a selected node

• Prevents UI freezes

• Enables progressive exploration

GET /api/graph/node

• Returns full metadata (labels, properties)

• Powers the side panel

GET /api/graph/filter

• Search nodes by path or type
  (e.g., src/, File)

🎯 Result: The graph is now explorable, fast, and frontend-safe.

3. Structured Logging (Goodbye console.log)

Scattered logs were becoming unmanageable.

What Changed

• Replaced all ad-hoc console.log calls with Winston

• Unified logging across services

Updated Areas

• Repo Parser:

  • importGraphData.js

  • astController.js

  • graphController.js

• Git Auth:

  • server.js

  • githubController.js

Benefits

• Centralized logs

• File transport enabled

• Ready for MySQL / Kafka ingestion later

📊 Observability is no longer an afterthought.

4. Neo4j & Data Integrity Improvements

Incremental Sync

• AST generation now processes only new or updated files

• Files are selected where sorted_content IS NULL

• Full rebuild only happens when explicitly forced

Delete Graph API

Added:

DELETE /delete-graph

• Wipes repo data from Neo4j

• Resets MySQL status to pending

• Enables clean reprocessing

User Model Enhancements

getRepositoryFiles now:

• Updates the user’s repo list in MongoDB

• Stores:

  • stars

  • forks

  • description

• Tracks AST state flags:

  • isAst

  • isexport_graph

5. Incremental Sync: Detecting Repo Changes

This was a big one.

How It Works Now

When you call:

POST /generate-ast

Check Phase

• Fetch latest GitHub tree

• Lightweight, fast call

Diff Phase

• Compare file SHAs with MySQL records

Sync Phase

• If SHA unchanged → skip

• If new or updated → fetch content via:

    http://localhost:5000/file
  

Process Phase

• AST generation runs only on changed files

✅ Result:
No re-downloading.
No re-parsing unchanged files.
Massive performance improvement.

6. Production Readiness Review (Reality Check ⚠️)

This week also included a full Production Readiness Review—and it exposed several serious issues. To uncover these, I relied heavily on Google + anti-gravity thinking, tracing bottlenecks from surface symptoms down to root causes that weren’t obvious at first glance.

🚨 Critical Findings

Privilege Escalation via Token Fallback

• System token was used when user token was missing

• Allowed unauthorized access to private repos

Fix

• User routes now strictly require a user token

• No silent fallback

Database Connection Pool Misuse

• .end() was called inside request handlers

• Pool wasn’t a singleton

Fix

• Singleton DB pool

• Never terminate pools per request

Kafka Message Size Limits

• Entire file contents were sent in messages

• Kafka defaults to 1MB max

Fix

• Store content in DB/Object storage

• Kafka messages now pass references only

Other Risks Identified

• Dynamic SQL table creation per repo (scalability risk)

• In-memory loading of entire GitHub trees

• Kafka topic creation race conditions

• Hardcoded ports and secrets

📌 These findings directly shaped next week’s roadmap.

Key Takeaways

• GitHub API limits demand backoff, throttling, and lazy traversal

• Graph UIs must load incrementally

• Structured logging is mandatory at scale

• Incremental sync is non-negotiable for large repos

• Production reviews uncover issues dev testing never will

• Google + anti-gravity thinking = critical for finding hidden bottlenecks 🕵️‍♂️✨

What’s Coming Next

Database Refactor

• Single repository_files table

• Proper indexing & scalability

Kafka Pipeline Cleanup

• Reference-only messages

• Safer large-repo handling

Frontend Graph Visualization

• Interactive Neo4j explorer

• File → function → dependency navigation

🤝 Contributions

Ideas, improvements, and suggestions are always welcome.
You’re encouraged to submit issues or pull requests to help evolve the platform.

Github repo