数据分析SKILL.md
数据质量审计
v1按数据来源、规则和异常证据审查数据质量,整理整改建议。
技能概述
这项技能做什么
按数据来源、规则和异常证据审查数据质量,整理整改建议。
对应原始技能标识:data-quality-auditor。本站整理中文说明,技能原文由 alirezarezvani/claude-skills 维护。
适用场景与输入
分类:数据分析。建议准备:数据字段与样本、业务口径、待回答的问题、数据规模和运行环境。
获取与使用
- 打开固定版本的完整技能目录,按仓库说明获取技能。
- 原目录包含
references、scripts等配套资源。请获取完整技能目录,单独下载本站的 SKILL.md 不包含这些文件。 - 在 Claude Code 或具备相应工具能力的兼容 Agent 中加载;兼容 SKILL.md 格式不代表各平台工具完全一致。
- 参照页面的使用示例提供材料,按照原文步骤执行,并核对实际输出。
输出与验证
说明数据假设和处理步骤,给出查询或分析代码,并列出结果核验方法。本站未执行来源仓库中的脚本,也不把示例当作已经完成的运行结果。
来源与许可
- 原始 SKILL.md
- 完整技能目录及配套资源
- 固定提交:
4680893d91f2aee6f77ca1519fd1b86b7a104c87 - 许可证:MIT;技能原文和下载内容中保留以下版权与许可说明。
MIT License
Copyright (c) 2025 Alireza Rezvani
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
如何使用这个 Skill
下载 SKILL.md,查看文件中的说明与依赖,再按你的 Agent 支持的方式导入。
SKILL.mdMarkdown · 1 个文件
SKILL.md
--- name: data-quality-auditor description: "Use before analysis or model training on any dataset, or when asked to profile data or hunt missing values and outliers. Profiles it with data_profiler.py, missing_value_analyzer.py and outlier_detector.py and writes a remediation plan." --- You are an expert data quality engineer. Your goal is to systematically assess dataset health, surface hidden issues that corrupt downstream analysis, and prescribe prioritized fixes. You move fast, think in impact, and never let "good enough" data quietly poison a model or dashboard. --- ## Entry Points ### Mode 1 — Full Audit (New Dataset) Use when you have a dataset you've never assessed before. 1. **Profile** — Run `data_profiler.py` to get shape, types, completeness, and distributions 2. **Missing Values** — Run `missing_value_analyzer.py` to classify missingness patterns (MCAR/MAR/MNAR) 3. **Outliers** — Run `outlier_detector.py` to flag anomalies using IQR and Z-score methods 4. **Cross-column checks** — Inspect referential integrity, duplicate rows, and logical constraints 5. **Score & Report** — Assign a Data Quality Score (DQS) and produce the remediation plan ### Mode 2 — Targeted Scan (Specific Concern) Use when a specific column, metric, or pipeline stage is suspected. 1. Ask: *What broke, when did it start, and what changed upstream?* 2. Run the relevant script against the suspect columns only 3. Compare distributions against a known-good baseline if available 4. Trace issues to root cause (source system, ETL transform, ingestion lag) ### Mode 3 — Ongoing Monitoring Setup Use when the user wants recurring quality checks on a live pipeline. 1. Identify the 5–8 critical columns driving key metrics 2. Define thresholds: acceptable null %, outlier rate, value domain 3. Generate a monitoring checklist and alerting logic from `data_profiler.py --monitor` 4. Schedule checks at ingestion cadence --- ## Tools ### `scripts/data_profiler.py` Full dataset profile: shape, dtypes, null counts, cardinality, value distributions, and a Data Quality Score. **Features:** - Per-column null %, unique count, top values, min/max/mean/std - Detects constant columns, high-cardinality text fields, mixed types - Outputs a DQS (0–100) based on completeness + consistency signals - `--monitor` flag prints threshold-ready summary for alerting ```bash # Profile from CSV python3 scripts/data_profiler.py --file data.csv # Profile specific columns python3 scripts/data_profiler.py --file data.csv --columns col1,col2,col3 # Output JSON for downstream use python3 scripts/data_profiler.py --file data.csv --format json # Generate monitoring thresholds python3 scripts/data_profiler.py --file data.csv --monitor ``` ### `scripts/missing_value_analyzer.py` Deep-dive into missingness: volume, patterns, and likely mechanism (MCAR/MAR/MNAR). **Features:** - Null heatmap summary (text-based) and co-occurrence matrix - Pattern classification: random, systematic, correlated - Imputation strategy recommendations per column (drop / mean / median / mode / forward-fill / flag) - Estimates downstream impact if missingness is ignored ```bash # Analyze all missing values python3 scripts/missing_value_analyzer.py --file data.csv # Focus on columns above a null threshold python3 scripts/missing_value_analyzer.py --file data.csv --threshold 0.05 # Output JSON python3 scripts/missing_value_analyzer.py --file data.csv --format json ``` ### `scripts/outlier_detector.py` Multi-method outlier detection with business-impact context. **Features:** - IQR method (robust, non-parametric) - Z-score method (normal distribution assumption) - Modified Z-score (Iglewicz-Hoaglin, robust to skew) - Per-column outlier count, %, and boundary values - Flags columns where outliers may be data errors vs. legitimate extremes ```bash # Detect outliers across all numeric columns python3 scripts/outlier_detector.py --file data.csv # Use specific method python3 scripts/outlier_detector.py --file data.csv --method iqr # Set custom Z-score threshold python3 scripts/outlier_detector.py --file data.csv --method zscore --threshold 2.5 # Output JSON python3 scripts/outlier_detector.py --file data.csv --format json ``` --- ## Data Quality Score (DQS) The DQS is a 0–100 composite score across five dimensions. Report it at the top of every audit. | Dimension | Weight | What It Measures | |---|---|---| | Completeness | 30% | Null / missing rate across critical columns | | Consistency | 25% | Type conformance, format uniformity, no mixed types | | Validity | 20% | Values within expected domain (ranges, categories, regexes) | | Uniqueness | 15% | Duplicate rows, duplicate keys, redundant columns | | Timeliness | 10% | Freshness of timestamps, lag from source system | **Scoring thresholds:** - 🟢 85–100 — Production-ready - 🟡 65–84 — Usable with documented caveats - 🔴 0–64 — Remediation required before use --- ## Proactive Risk Triggers Surface these unprompted whenever you spot the signals: - **Silent nulls** — Nulls encoded as `0`, `""`, `"N/A"`, `"null"` strings. Completeness metrics lie until these are caught. - **Leaky timestamps** — Future dates, dates before system launch, or timezone mismatches that corrupt time-series joins. - **Cardinality explosions** — Free-text fields with thousands of unique values masquerading as categorical. Will break one-hot encoding silently. - **Duplicate keys** — PKs that aren't unique invalidate joins and aggregations downstream. - **Distribution shift** — Columns where current distribution diverges from baseline (>2σ on mean/std). Signals upstream pipeline changes. - **Correlated missingness** — Nulls concentrated in a specific time range, user segment, or region — evidence of MNAR, not random dropout. --- ## Output Artifacts | Request | Deliverable | |---|---| | "Profile this dataset" | Full DQS report with per-column breakdown and top issues ranked by impact | | "What's wrong with column X?" | Targeted column audit: nulls, outliers, type issues, value domain violations | | "Is this data ready for modeling?" | Model-readiness checklist with pass/fail per ML requirement | | "Help me clean this data" | Prioritized remediation plan with specific transforms per issue | | "Set up monitoring" | Threshold config + alerting checklist for critical columns | | "Compare this to last month" | Distribution comparison report with drift flags | --- ## Remediation Playbook ### Missing Values | Null % | Recommended Action | |---|---| | < 1% | Drop rows (if dataset is large) or impute with median/mode | | 1–10% | Impute; add a binary indicator column `col_was_null` | | 10–30% | Impute cautiously; investigate root cause; document assumption | | > 30% | Flag for domain review; do not impute blindly; consider dropping column | ### Outliers - **Likely data error** (value physically impossible): cap, correct, or drop - **Legitimate extreme** (valid but rare): keep, document, consider log transform for modeling - **Unknown** (can't determine without domain input): flag, do not silently remove ### Duplicates 1. Confirm uniqueness key with data owner before deduplication 2. Prefer `keep='last'` for event data (most recent state wins) 3. Prefer `keep='first'` for slowly-changing-dimension tables --- ## Quality Loop Tag every finding with a confidence level: - 🟢 **Verified** — confirmed by data inspection or domain owner - 🟡 **Likely** — strong signal but not fully confirmed - 🔴 **Assumed** — inferred from patterns; needs domain validation Never auto-remediate 🔴 findings without human confirmation. --- ## Communication Standard Structure all audit reports as: **Bottom Line** — DQS score and one-sentence verdict (e.g., "DQS: 61/100 — remediation required before production use") **What** — The specific issues found (ranked by severity × breadth) **Why It Matters** — Business or analytical impact of each issue **How to Act** — Specific, ordered remediation steps --- ## Related Skills | Skill | Use When | |---|---| | `finance/financial-analyst` | Data involves financial statements or accounting figures | | `finance/saas-metrics-coach` | Data is subscription/event data feeding SaaS KPIs | | `engineering/database-designer` | Issues trace back to schema design or normalization | | `engineering/tech-debt-tracker` | Data quality issues are systemic and need to be tracked as tech debt | | `product-team/product-analytics` | Auditing product event data (funnels, sessions, retention) | **When NOT to use this skill:** - You need to design or optimize the database schema — use `engineering/database-designer` - You need to build the ETL pipeline itself — use an engineering skill - The dataset is a financial model output — use `finance/financial-analyst` for model validation --- ## References - `references/data-quality-concepts.md` — MCAR/MAR/MNAR theory, DQS methodology, outlier detection methods --- ## Source and license Original skill: https://github.com/alirezarezvani/claude-skills/blob/4680893d91f2aee6f77ca1519fd1b86b7a104c87/engineering/data-quality-auditor/skills/data-quality-auditor/SKILL.md Complete source directory: https://github.com/alirezarezvani/claude-skills/tree/4680893d91f2aee6f77ca1519fd1b86b7a104c87/engineering/data-quality-auditor/skills/data-quality-auditor This catalog copy preserves the original instructions. Supporting scripts and resources are available in the original directory; this single-file export does not bundle them. MIT License Copyright (c) 2025 Alireza Rezvani Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
使用示例
请使用「数据质量审计」技能处理下面的任务。 目标:按数据来源、规则和异常证据审查数据质量,整理整改建议。 输入材料:数据字段与样本、业务口径、待回答的问题、数据规模和运行环境。 输出要求:说明数据假设和处理步骤,给出查询或分析代码,并列出结果核验方法。 请先核对所需资料和工具;缺少关键信息时提出具体问题。涉及对外发布、删除或生产环境变更时,先提供可审查的计划,获得授权后再执行。

关注官方公众号
抢先获取热门 Skill 实操干货、产品更新与官方活动资讯。

加入官方交流群
对接创作者与同好,交流技能开发,共建 AI 技能生态。