Garbage Collection and Memory Management for High-Density Cassandra Nodes

• Tue, Jul 15, 2025
• 6-minute read

This is the fifth post in my series on optimizing Apache Cassandra for maximum cost efficiency through increased node density. In previous posts, we covered streaming operations, compaction strategies, repair processes, and query throughput optimization. Now, we’ll tackle one of the most critical yet often misunderstood aspects of Cassandra performance: garbage collection and memory management.

At a high level, these are the leading factors that impact node density:

• Streaming Throughput
• Compaction Throughput and Strategies
• Various Aspects of Repair
• Query Throughput
• Garbage Collection and Memory Management (this post)
• Efficient Disk Access
• Compression Performance and Ratio
• Linearly Scaling Subsystems with CPU Core Count and Memory

Why Memory Management Matters for Node Density

Cassandra is a JVM-based application, which means it’s subject to the constraints and behaviors of Java’s memory management system. As node density increases, efficient memory usage becomes increasingly critical for several reasons:

1. Heap Pressure: More data means larger indices, more metadata, and potentially more in-memory buffers
2. GC Pause Impact: With higher workloads, GC pauses have a more significant impact on overall performance
3. Off-Heap Memory: Many Cassandra components use off-heap memory, which scales with data volume
4. Resource Competition: Memory must be balanced between Cassandra, the OS page cache, and other processes

Poor memory management is often the first bottleneck encountered when increasing node density, and it typically manifests as increased latency, reduced throughput, and in severe cases, OutOfMemoryError crashes.

Understanding Cassandra’s Memory Usage

Before diving into optimization, it’s crucial to understand how Cassandra uses memory:

On-Heap Memory Components

1. Memtables: In-memory structures that store recent writes before flushing to disk
2. Row Cache: Caches entire rows (if enabled, which is rare)
3. Bloom Filters: Probabilistic data structures that help determine if data might be in an SSTable
4. Partition Summary: Samples from the partition index to speed up lookups
5. JVM Overhead: Internal JVM structures, thread stacks, and code

Off-Heap Memory Components

1. Netty Direct Memory: Used for network communication
2. Compression Metadata: Information about compressed chunks in SSTables
3. Off-Heap Memtables: When configured to use off-heap memory
4. Bloom Filter Off-Heap: Portions of bloom filters stored outside the heap
5. Linux Page Cache: OS-level caching of frequently accessed data

As node density increases, both on-heap and off-heap components grow, requiring careful tuning to prevent resource exhaustion.

Diagnosing Memory-Related Issues

Before optimizing, you need to identify where your memory bottlenecks are. Here are the essential diagnostic tools:

1. JVM Heap Usage

Monitor heap usage using nodetool:

nodetool info

Look for “Heap Memory” metrics. Aim to keep heap usage below 75% during normal operations.

2. GC Logs Analysis

Enable detailed GC logging:

-Xloggc:/var/log/cassandra/gc.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=10M
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+PrintGCCause
-XX:+PrintPromotionFailure
-XX:+PrintClassHistogramBeforeFullGC
-XX:+PrintClassHistogramAfterFullGC

Analyze GC logs using tools like GCeasy or GCViewer.

3. Off-Heap Memory Tracking

Track off-heap memory usage with:

nodetool sjk mx -b org.apache.cassandra.metrics:type=Storage,name=TotalSSTablesOffHeap

4. Heap Dump Analysis

For detailed investigation, capture heap dumps during issues:

jmap -dump:format=b,file=heap.hprof <pid>

Analyze with tools like VisualVM or Eclipse MAT.

Key Memory Optimizations for High-Density Nodes

Now that we know how to diagnose issues, let’s look at specific optimizations for high-density environments:

1. Heap Size Tuning

For high-density nodes, the default heap sizes are often insufficient:

# In jvm-server.options
-Xms12G
-Xmx12G

General recommendations:

• For nodes ≤ 8TB: 8-12GB heap
• For nodes 8-16TB: 12-16GB heap
• For nodes > 16TB: 16-24GB heap

Never exceed 32GB heap due to JVM limitations with compressed object pointers.

2. Garbage Collection Tuning

For read-heavy or mixed workloads:

# In jvm-server.options
-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:MaxGCPauseMillis=300
-XX:InitiatingHeapOccupancyPercent=70
-XX:G1HeapRegionSize=16m
-XX:SurvivorRatio=4
-XX:MaxTenuringThreshold=6

For write-heavy workloads:

# In jvm-server.options
-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=10
-XX:MaxGCPauseMillis=500
-XX:InitiatingHeapOccupancyPercent=65
-XX:G1HeapRegionSize=16m
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1

3. Memtable Configuration

For high-density nodes, control memtable memory usage:

# In cassandra.yaml
memtable_allocation_type: offheap_objects
memtable_heap_space_in_mb: 2048
memtable_offheap_space_in_mb: 2048

Using off-heap memtables reduces GC pressure during writes and flushes.

4. Cache Sizing

Adjust caches based on your workload:

# In cassandra.yaml
key_cache_size_in_mb: 512
key_cache_save_period: 14400
row_cache_size_in_mb: 0  # Generally disable for high-density nodes
counter_cache_size_in_mb: 128

For high-density nodes, prioritize the key cache over other caches, as it provides the best performance improvement per MB of memory.

5. Index Summary Scaling

Control how much memory is used for partition index summaries:

# In cassandra.yaml
index_summary_capacity_in_mb: 256
index_summary_resize_interval_in_minutes: 60

With more data, index summaries consume more memory. These settings control the maximum memory used and how often it’s recalculated.

Advanced Memory Management Techniques

For pushing the limits of node density beyond 20TB per node, consider these advanced techniques:

1. Linux Transparent Huge Pages

Disable Transparent Huge Pages, which can cause latency spikes:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

2. Linux Swappiness

Reduce swappiness to prevent unnecessary swapping:

sysctl -w vm.swappiness=1

Add to /etc/sysctl.conf for persistence.

3. Memory Reservations for Critical Operations

Reserve memory for compaction and other critical operations:

# In cassandra.yaml
compaction_large_partition_warning_threshold_mb: 100
min_free_space_per_drive_in_mb: 5120

4. Off-Heap Objects for Large Text Fields

For tables with large text fields, consider:

memtable_allocation_type: offheap_objects

This moves large objects out of the JVM heap, reducing GC pressure.

Real-World Memory Optimization Example

Let me share a case study from a production environment where we optimized memory management for high-density nodes:

Before Optimization:

• 8TB per node with 8GB heap
• Average GC pause: 700ms
• p99 write latency: 85ms
• Frequent GC pressure during peak hours

After Optimization:

• 16TB per node with 16GB heap
• Average GC pause: 250ms
• p99 write latency: 42ms
• Stable GC patterns even during peak hours

The key changes were:

1. Increased heap size from 8GB to 16GB
2. Optimized G1GC parameters for the workload
3. Moved memtables to off-heap allocation
4. Adjusted cache sizes based on access patterns
5. Implemented Linux kernel-level optimizations

Monitoring and Maintenance

For high-density nodes, ongoing monitoring is crucial:

1. GC Metrics: Track pause times, frequency, and patterns
2. Memory Usage: Monitor both heap and off-heap memory
3. Application Metrics: Watch for correlations between memory issues and application performance
4. Regular Tuning: As data volume grows, memory settings need periodic adjustment

Consider implementing automated alerts for:

• GC pauses exceeding thresholds
• High sustained heap usage
• Rapid increases in off-heap memory usage

Conclusion

Efficient garbage collection and memory management are fundamental to achieving high node density in Cassandra clusters. By implementing the strategies outlined in this post, you can significantly reduce GC-related performance issues while increasing the amount of data each node can efficiently handle.

Remember that memory optimization is highly workload-dependent. What works for one cluster may not be optimal for another. Always test changes in a staging environment before applying them to production, and monitor closely after implementation.

In our next post, we’ll explore how efficient disk access strategies enable higher node density and reduce operational costs.