ClickHouse Broken Parts: Causes, Detection & Fix

Broken parts are the most critical signal in ClickHouse monitoring. A broken part means checksums don't match or files are corrupt — ClickHouse will refuse to serve queries from affected tables. Zero broken parts is the only healthy value. Any broken part requires immediate investigation.

What Causes Broken Parts?

1. Disk hardware failures — The most common cause. A failing disk produces corrupted writes that fail ClickHouse's checksum verification.

2. Interrupted merges or inserts — A server crash during a merge or large insert can leave a part in an inconsistent state.

3. Network issues during replication — Data partially transferred during replication can produce a broken part on the receiving replica.

4. Bug in ClickHouse version — Rare, but known to occur in specific versions. Check the ClickHouse release notes if you see broken parts after an upgrade.

Detecting Broken Parts

-- Count broken parts by table
SELECT database, table, count() AS broken_count
FROM system.parts
WHERE broken = 1
GROUP BY database, table
ORDER BY broken_count DESC;
 
-- Get detail on broken parts
SELECT database, table, name, path, data_version
FROM system.parts
WHERE broken = 1;

Clustersight monitors this automatically and alerts you the moment any broken parts appear, with the fix command included.

How to Fix Broken Parts

Fix 1: Sync from healthy replica (replicated tables)

-- Force replica to sync from source
SYSTEM SYNC REPLICA database.table_name;
 
-- Check replication queue
SELECT * FROM system.replication_queue
WHERE table = 'table_name' AND type = 'REPAIR_PARTS';

Fix 2: Detach and re-attach the broken part

-- Detach the specific broken part
ALTER TABLE database.table_name DETACH PART 'part_name';
 
-- Verify it's gone from active parts
SELECT name FROM system.parts
WHERE table = 'table_name' AND broken = 1;

Fix 3: Restore from backup (non-replicated or total loss)

If the table is not replicated and you have no healthy replica, restore from your last known good backup.

Prevention

• Monitor system.parts continuously for broken = 1
• Use replicated tables in production (ReplicatedMergeTree family)
• Maintain regular backups with BACKUP TABLE
• Alert immediately on any broken parts — threshold: > 0

Read more: How to Monitor ClickHouse in Production