Understanding a large or unfamiliar codebase is one of the hardest challenges for developers, students, and open-source contributors. Modern applications often span hundreds of files, multiple languages, and deeply intertwined module dependencies. Manually piecing together the structure of such projects is slow, error-prone, and overwhelming.

CodeAtlas was built to solve this problem.

It is a Code Intelligence Platform that automatically analyzes any GitHub repository using Static Abstract Syntax Trees (ASTs), converts the findings into a structured knowledge graph, and visualizes the entire codebase through an interactive frontend.

This makes onboarding dramatically easier and provides a clear, intuitive view of how a project is organized without executing a single line of user code.

🧩 What CodeAtlas Does

At its core, CodeAtlas performs three major functions:

1. Fetches and inspects every file in a GitHub repository

2. Generates static ASTs for each file and extracts functions, classes, imports, calls, and structural elements

3. Builds a visual graph representing all relationships inside the project

Whether you're a contributor joining an open-source project, a student analyzing source code, or a developer exploring new codebases, CodeAtlas helps you understand the entire repository in minutes rather than hours.

🎯 Key Goals of the Platform

• Provide full awareness of repository structure

• Visualize file-level and project-level relationships

• Generate static ASTs for deep code understanding

• Enable semantic search through AST embeddings

• Support incremental and scalable parsing through SHAs

• Offer cloud-ready, microservices-based architecture

• Make onboarding fast for any codebase

🏛️ High-Level Architecture

CodeAtlas is built as a collection of microservices, each responsible for a core piece of the system.

1. Authentication Service

Handles GitHub OAuth, manages access tokens, and retrieves user repositories.

2. Repository Intake Service

Fetches:

• Repository metadata

• File structure

• Individual file contents

File metadata is streamed into Kafka for downstream processing.

3. AST Parsing Service

Consumes file messages from Kafka and performs:

• Language detection

• Static AST generation

• Extraction of imports, exports, classes, functions, and calls

• AST normalization

• Metadata storage in PostgreSQL

• Emission of relationship data to the graph builder

4. Embedding / Indexing Service (Optional)

Generates vector embeddings for symbols or AST summaries using Ollama or other LLM services.

5. Visualization Frontend

Interactive UI built using Next.js and graph libraries such as Cytoscape.js or React Flow.

6. Monitoring Layer

Prometheus, Grafana, and Loki ensure complete observability.

🗃️ Data Storage Strategy

Different databases serve different roles:

MongoDB

• User profiles

• Session data

• Repo usage history

Redis

• Short-lived OAuth tokens

• Temporary caches

PostgreSQL

• File metadata

• SHAs

• AST summary data

📦 Kafka Event Streams

Kafka powers the distributed processing pipeline through three topics:

This enables horizontal scaling—multiple AST workers can process large repositories in parallel.

🔍 Static AST Processing Pipeline

The pipeline carefully processes each file:

1. File received from Kafka

2. Language identification

3. Parser selection

4. Static AST generation

5. AST cleanup & normalization

6. Extraction of key elements:

   • imports

   • exports

   • classes

   • functions

   • variables

   • call expressions

7. Store AST + metadata

8. Emit relationships for graph building

This entire workflow is 100% static—no runtime execution of user code.

🧠 Understanding the Knowledge Graph

The graph contains:

Nodes

• Files

• Functions

• Classes

• Modules

Edges

• IMPORTS

• CONTAINS

• CALLS

• EXTENDS

• DEPENDS_ON

This graph becomes the backbone of CodeAtlas’s visual exploration experience.

🎨 The Interactive Visualization Layer

The frontend provides:

• Zoomable graph canvas

• Clickable nodes with side-panel details

• Syntax-aware search across files, functions, or symbols

• Repo tree view

• Filtering (files only, imports only, functions only)

• Light/dark theme

This allows developers to visually and intuitively explore entire repositories.

🛠️ Technology Stack

📈 Monitoring & Observability

• Prometheus for service metrics

• Loki for centralized logs

• Grafana dashboards for system insights

The platform is designed for production-ready deployments.

🔐 Security Considerations

• GitHub tokens stored only in Redis (short TTL)

• No code execution—static AST only

• Sensitive data excluded from logs

• Encrypted inter-service communication

📚 Future Enhancements

• Dynamic AST (runtime tracing)

🧠 Core Concepts in Depth

1. AST Extraction

Every file is converted into an AST, which becomes the basis for:

• code analysis

• dependency mapping

• function/component discovery

• graph building

2. Incremental Updates Using SHAs

GitHub assigns every file a unique SHA.
CodeAtlas stores the SHA and only reprocesses files when their SHA changes.

This means:

• unchanged files are skipped

• large repos update instantly

• processing loads stay minimal

🔁 End-to-End Processing Workflow

1. User enters a repository URL

2. System fetches all file paths + SHAs

3. New/modified files → parsed into AST

4. Unchanged files → skipped

5. All ASTs combined into a “Sorted AST” graph

6. Visualization becomes available in the frontend

🤝 Contributions

Ideas, improvements, and suggestions are always welcome.
You’re encouraged to submit issues or pull requests to help evolve the platform.

Github repo