1. Six Canonical Dimensions of Data Quality

Enterprise data engineering evaluates datasets against six rigorous structural dimensions:

  1. Completeness: Proportion of non-missing mandatory attributes across observational rows.
  2. Consistency: Logical agreement across cross-table relationships (balance_table.total == sum(transaction_table.amount)).
  3. Accuracy: Euclidean or Manhattan closeness to verified physical ground truth.
  4. Validity: Compliance with domain constraints (e.g., ISO currency codes, non-negative prices).
  5. Timeliness: Latency delta between physical event occurrence and database query availability.
  6. Uniqueness: Absence of entity deduplication failures across compound primary keys.

2. Little & Rubin Taxonomy of Missing Data

Let $Y = (Y_{\text{obs}}, Y_{\text{mis}})$ be the complete dataset and $M \in \{0, 1\}^N$ be the missingness indicator matrix ($M_{ij} = 1$ if $Y_{ij}$ is missing):

1. Missing Completely At Random (MCAR)

Missingness is independent of both observed and unobserved values:

$$P(M \mid Y_{\text{obs}}, Y_{\text{mis}}, \phi) = P(M \mid \phi)$$

2. Missing At Random (MAR)

Missingness depends systematically on observed covariates $Y_{\text{obs}}$, but is independent of unobserved target values $Y_{\text{mis}}$:

$$P(M \mid Y_{\text{obs}}, Y_{\text{mis}}, \phi) = P(M \mid Y_{\text{obs}}, \phi)$$

3. Missing Not At Random (MNAR)

Missingness depends directly on the unobserved missing values themselves ($Y_{\text{mis}}$), introducing severe self-selection bias if dropped.

3. Rubin’s Multiple Imputation (MI) Pooling Theory

To preserve variance structure under MAR missingness, Multiple Imputation generates $m \ge 5$ independent imputed datasets.

Given parameter estimate $\hat{Q}_k$ and estimated variance $U_k$ from imputed dataset $k$:

  • Pooled Point Estimate:
$$\bar{Q} = \frac{1}{m} \sum_{k=1}^m \hat{Q}_k$$
  • Within-Imputation Variance ($W$):
$$W = \frac{1}{m} \sum_{k=1}^m U_k$$
  • Between-Imputation Variance ($B$):
$$B = \frac{1}{m - 1} \sum_{k=1}^m \left( \hat{Q}_k - \bar{Q} \right)^2$$
  • Total Rubin Variance ($T$):
$$T = W + \left( 1 + \frac{1}{m} \right) B$$

4. Python Verification: Rubin’s Multiple Imputation Variance Pooling

import numpy as np

def pool_rubin_imputations(estimates: np.ndarray, variances: np.ndarray):
    """Rigorous calculation of Rubin's Multiple Imputation total variance and standard error."""
    m = len(estimates)
    assert m > 1 and len(variances) == m
    
    Q_bar = np.mean(estimates)
    W = np.mean(variances)
    B = np.var(estimates, ddof=1)
    
    T = W + (1.0 + 1.0 / m) * B
    se_total = np.sqrt(T)
    return float(Q_bar), float(T), float(se_total)

if __name__ == "__main__":
    # 5 imputed dataset point estimates and internal standard errors squared
    imputed_means = np.array([10.2, 10.5, 9.8, 10.1, 10.4])
    imputed_vars  = np.array([0.45, 0.48, 0.44, 0.46, 0.47])
    
    q_pool, total_var, total_se = pool_rubin_imputations(imputed_means, imputed_vars)
    print(f"Pooled Parameter Estimate Q_bar: {q_pool:.4f}")
    print(f"Total Rubin Variance T:          {total_var:.4f}")
    print(f"Total Standard Error SE:         {total_se:.4f}")