Accelerating IntelliJ indexing for a large Scala monorepo

The challenge

IntelliJ indexes project files to enable features such as code navigation, search, and refactoring. However, in large repositories, this process can be slow. The IDE detects significant source file changes—such as switching branches or pulling upstream updates—and initiates indexing. During this time, most IDE features are unavailable, forcing developers to wait before resuming their work.

At scale, these delays had a substantial impact on the investment bank. Developers frequently waited for indexing to complete, resulting in 1,270 hours of lost productivity per month across the organization. This figure came from both full and incremental indexing.

Full Indexing

Full indexing occurs in scenarios such as:

• Setting up a new workspace
• Regenerating/cleaning existing workspace for troubleshooting
• Cleaning indexes during troubleshooting

Before optimization, full indexing was a major pain point:

• 25% of users performed a full reindex 1–2 times per week
• Each operation took up to 30 minutes
• Even after upgrading to IntelliJ 2023.3, full indexing still required almost 14 minutes per refresh

This led to an average productivity loss of 20 minutes per developer each week.

Incremental Indexing

Incremental indexing occurs during routine tasks such as Git pulls or checkouts.

• ~50,000 file changes: 1 minute, 10 seconds
• ~500,000 file changes: Increased delays, scaling with file volume

As the monorepo grew, indexing time became a growing challenge.

Diverse JDK requirements and lack of detailed metrics

On top of the delays, the project required seven different JDK versions, which added indexing overhead. Our client also lacked telemetry on indexing performance. Without comprehensive data, it was difficult for them to diagnose delays and prioritize improvements.

The solution

Our engineers took a two-step approach to the problem:

1. Optimized IntelliJ’s existing indexing process to reduce unnecessary workload.
2. Implemented shared indexes to minimize local processing.

Step 1: Indexing optimizations without shared indexes

Before implementing new methods, our engineers identified bottlenecks and optimized IntelliJ’s indexing behavior.

Enhanced telemetry and monitoring

• Extended IntelliJ’s logging to capture indexing performance data.
• Integrated performance insights into a centralized dashboard.
• Analyzed reports to identify and resolve bottlenecks.

Selective file filtering

Our engineers introduced a configurable filtering mechanism to reduce redundant indexing in three categories:

• By file size – Excluded large, rarely used files.
• By file location – Skipped unnecessary files inside JARs.
• By file type – Omitted non-essential formats (e.g., large XMLs, map files, test data).

Also, they provided two configurations:

• Balanced configuration – Optimized indexing without disrupting workflows.
• Extreme configuration – A more aggressive option for maximum performance. This setting can improve indexing speeds further but may limit certain functionalities, such as project-wide text search in potential XML configuration.

This gives our client’s developers the ability to fine-tune settings based on their team’s requirements.

Step 2: Implementing shared indexes

To further accelerate indexing, our engineers developed an automated system to pre-build and distribute indexes for JDKs and the project codebase.

Comprehensive index generation

• JDK Indexes – Pre-built for all seven JDK versions.
• Project source code and dependencies – Pre-indexing libraries, sources, and other assets.

Optimized Index Distribution

• Pre-built indexes are stored in a geo-distributed storage system with synchronized data across multiple regions. Users are directed to the appropriate regional server based on their location to ensure efficient access.
• A high-performance proxy server handles index retrieval, optimizing transfer speeds and reducing latency for developers.

Automated CI Integration & Maintenance

• Indexes are generated periodically on Continuous Integration (CI) infrastructure and automatically cleaned up when outdated.
• Developers do not need to manually configure or manage these indexes, making adoption seamless.

This solution is fully integrated with IntelliJ Idea Ultimate, ensuring developers benefit from dramatically faster indexing times without additional overhead.

The results

Our optimizations significantly reduced indexing times, minimizing disruption for our client.

Improvements in full indexing 

• Reduced indexing time from 14 minutes to under 6 minutes, which depending on the conditions, translates to 50–65% reduction.

Improvements in incremental indexing

• In scenarios where ~50,000 files change, incremental indexing now takes approximately 45 seconds, a 35% improvement over previous times.
• The larger the incremental indexing operation, the greater the speedup—reaching up to 50% or more for high-volume file changes (e.g., 500,000+ files).

Real-time monitoring provides ongoing insights to track and fine-tune indexing performance.

Business impact

Based on time saved alone we estimate that by reducing indexing delays, our client saved about $420,000 annually. This excludes additional benefits such as:

• Reduced cost of opportunity.
• Faster time to market.