When Database Corruption Would Not Go Away

Back

This story follows the investigation from recurring symptoms through failed conventional fixes to the storage-level change that finally stabilized the affected table. 

Recurring database corruption is one of the hardest problems to resolve because a successful repair does not prove that the cause has been removed. 

In this case, the corruption kept returning even after integrity repairs, index maintenance, hardware diagnostics, SQL Server patches, and application-level changes. 

We encountered exactly this situation in a customer database that stored product-related data and images. One specific table repeatedly triggered SQL Server consistency errors such as Msg 8961, which pointed to logical inconsistencies in the off-row LOB data. 

What made the issue difficult was not only the initial corruption, but the fact that every apparent fix proved temporary. 

Hardware diagnostics, database repairs, index maintenance, synchronization adjustments, and other troubleshooting steps were all explored. Yet none of them provided a permanent solution. 

The real challenge was not simply repairing the damaged data. 

It was understanding why the same problem kept coming back. 

 

The Initial Symptoms 

Automated integrity checks repeatedly failed on the affected table. 

The reported errors pointed to inconsistencies in off row LOB data, including messages similar to: 

Msg 8961, Sev 16, State 1, Line 1 : Table error: Object ID 1469964313, index ID 1, partition ID 72057866713300992, alloc unit ID 72057594087735296 (type LOB data). The off-row data node at page (1:1043389), slot 0, text ID 405064680210432 does not match its reference from page (1:3294073), slot 2. [SQLSTATE 42000] 
Msg 8961, Sev 16, State 1, Line 1 : Table error: Object ID 1469964313, index ID 1, partition ID 72057866713300992, alloc unit ID 72057594087735296 (type LOB data). The off-row data node at page (1:1043391), slot 0, text ID 405064680210432 does not match its reference from page (1:3294073), slot 2. [SQLSTATE 42000] 
Msg 8961, Sev 16, State 1, Line 1 : Table error: Object ID 1469964313, index ID 1, partition ID 72057866713300992, alloc unit ID 72057594087735296 (type LOB data). The off-row data node at page (1:1043390), slot 0, text ID 405064680210432 does not match its reference from page (1:3294073), slot 2. [SQLSTATE 42000] 
Msg 8961, Sev 16, State 1, Line 1 : Table error: Object ID 1469964313, index ID 1, partition ID 72057866713300992, alloc unit ID 72057594087735296 (type LOB data). The off-row data node at page (1:7920527), slot 1, text ID 405064680210432 does not match its reference from page (1:3294073), slot 2. [SQLSTATE 42000] 
Msg 8929, Sev 16, State 1, Line 1 : Object ID 1469964313, index ID 1, partition ID 72057866713300992, alloc unit ID 72057873151361024 (type In-row data): Errors found in off-row data with ID 123589703499776 owned by data record identified by RID = (1:3294073:2) [SQLSTATE 42000] 

Msg 8958, Sev 16, State 1, Line 1 : repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKTABLE 

 

In SQL Server, large object data such as images can be stored separately from the main row structure. When the reference from the row to that off-row data becomes inconsistent, integrity checks can report corruption even though the affected business record may still appear to exist at the application level. 

 

The database contained product information and images, while the affected table included LOB data that was being modified frequently. 

Each time corruption was detected, the situation became increasingly disruptive. 

Additional consistency checks placed significant pressure on the SQL Server instance. In some cases, the extra load coincided with momentary interruptions to high availability and availability group failovers. 

The immediate challenge was therefore not only to repair the corruption. 

It was to understand why it kept coming back. 

 

First Suspect: Hardware 

Persistent corruption naturally raises concerns about the underlying infrastructure. 

The customer therefore performed an extensive hardware investigation on the affected server. 

The environment was systematically tested, including: 

• RAM 
• SSD drives 
• storage firmware 
• RAID controllers
• BIOS 
• network connectivity

 The network infrastructure was also upgraded as part of the investigation. 

The tests completed successfully without identifying a hardware fault. 

At the same time, attention turned to the application workload. 

Because the affected database contained product images, synchronization processes between environments were investigated. The customer temporarily experimented with limiting image synchronization, including scenarios where image related data was excluded or only modified images were transferred. 

The suspicion was that massive bulk inserts or updates could be contributing to the problem. 

But the corruption continued. 

 

Looking at What Changed 

The investigation then moved closer to the write activity itself. 

Using Change Data Capture (CDC), our team reviewed the write activity affecting the problematic table and looked for patterns in the rows and LOB values being changed. 

A pattern emerged around high frequency write activity involving a table containing LOB data. 

The behavior also showed strong similarities to a known SQL Server issue documented by Microsoft KB3213240, describing database corruption associated with specific large update operations. 

This shifted the investigation away from a simple infrastructure failure. 

The problem appeared to be related to the way the affected data was being written and maintained. 

 

Repairing the Damage Was Not Solving the Problem 

Whenever an integrity check detected inconsistencies, the database had to be repaired. 

The recovery process included: 

DBCC CHECKTABLE 

With the only viable option: 

REPAIR_ALLOW_DATA_LOSS 

This was a necessary but costly operation. 

The corrupted rows were removed, after which missing records had to be restored manually. 

Additional maintenance was also performed using automated index optimization procedures. 

Several other approaches were tested: 

• regular index rebuilds 
• changes to LOB data storage
• moving large value data out of row 
• synchronization adjustments 
• infrastructure and firmware updates 

Some of these measures helped mitigate parts of the problem. 

None of them eliminated the recurring corruption permanently. 

The same question remained: 

Why was the same table repeatedly becoming inconsistent? 

 

The Architectural Approach 

After the standard troubleshooting options had been exhausted, our team proposed a more fundamental change. Instead of continuing to modify indexes, synchronization behavior, or LOB storage settings, the affected table would be physically separated from its existing storage structure to a new one - to see if the problem persists. 

The table was migrated (rebuilt) to a new, dedicated filegroup on the same volume. 

After the rebuild into new filegroup, the consistency check completed successfully. 

More importantly, the corruption has never returned again. 

 

The Result 

The migration established that rebuilding the table into a new filegroup eliminated the recurring corruption, but it did not conclusively identify the original root cause. The result points to a problem associated with the table’s previous physical storage structures or allocation state, potentially exposed by the high-frequency updates to its LOB data.  

Rebuilding the table created new allocation structures and rewrote the affected data, removing the condition in which the corruption had been recurring.  

Because the investigation did not isolate a specific damaged page, storage component, SQL Server defect, or application operation as the definitive trigger, the dedicated filegroup should be described as the successful mitigation rather than proof that filegroup placement itself was the cause. 

Automated consistency checks continued to run without detecting another error. The environment remained stable throughout the following months, and the integrity check for the overall database was eventually returned to its standard nightly schedule. 

 

Key Takeaway 

Database corruption investigations can easily become focused on the usual suspects: 

• failing hardware 
• storage issues 
• memory problems 
• problematic queries
• index fragmentation
• application bugs 

Those areas should absolutely be investigated. 

But when corruption repeatedly affects the same object and survives multiple repair and maintenance strategies, it may be necessary to reconsider the physical architecture of that object. 

In this case, separating the affected table containing a significant amount of LOB data into its own filegroup provided the stability that repeated repairs, rebuilds, infrastructure testing, and synchronization changes could not.

 

SMT Stories Takeaway

Recurring database corruption requires looking beyond repairing the damaged data and understanding why the problem keeps returning. When standard maintenance is not enough, changing the physical architecture can provide the stability that repeated repairs cannot.