mwartell maladies

annotated exemplar: fixing field mismatches

---
date: 2026-02-18
title: fixing field mismatches
current_head: 679a0d47a64fc9686a26ba0cc1f91cd20bb45be7
agent: Copilot Claude Sonnet 4.5
session_type:
instructions: _cdd-instructions.md
topics: [pydantic, schema, cip, field-naming, model-generation]
---

Prompt 1: understanding the issue

While creating and testing ogma.ingest.backwards we found that there were surprising mismatches between ogma.ingest.cip_models and schema/cip_schema.yaml

mismatches

The CIP failures are due to field name mismatches between the schema YAML and Pydantic models:

These fields exist in the Pydantic models but with different names. When the FTS parser reads the schema, it uses the schema field names, but when Pydantic validates, it expects the model field names.

The *_schema.yaml files were created by ogma.generate.schema from the absolutely authoritative data/data_elements.csv

Look at the mismatches and determine where in the process the differences were introduced.

Response 1: field naming discrepancy traced to manual Pydantic models

The mismatches were introduced in cip_models.py, which was written manually and uses different field naming conventions than the authoritative data source and generated schema. The schema generation process correctly transformed the authoritative field names, but the Pydantic models diverged.

Data Flow Analysis

1. Authoritative Source: data_elements.csv

The T-MSIS data dictionary contains the canonical field names:

• Line 561: CIP179 → BILLING-PROV-NUM
• Line 567: CIP180 → BILLING-PROV-NPI-NUM
• Line 573: CIP181 → BILLING-PROV-TAXONOMY
• Line 579: CIP183 → BILLING-PROV-SPECIALTY

2. Schema Generation: generate_cip_schema.py

The schema generator correctly transforms these using normalize_key():

def normalize_key(data_element_name):
    """Convert DATA-ELEMENT-NAME to lowercase key."""
    return data_element_name.lower().replace("-", "_")

3. Pydantic Models: cip_models.py

The manually-written Pydantic models use inconsistent naming:

• bill_provider_npi_num (line 165) — abbreviated “billing” to “bill”, expanded “prov” to “provider”
• bill_provider_taxonomy (line 166) — same pattern
• bill_provider_specialty_code (line 167) — added “_code” suffix
• bill_provider_num (line 169) — same pattern

Conclusion

The discrepancy was introduced when cip_models.py was manually authored without aligning to the schema field names. The schema generation process faithfully preserved the authoritative data element names (modulo case and hyphen-to-underscore conversion), but the Pydantic models used an ad-hoc naming convention.

Prompt 2: fix the CIP Pydantic indirectly

Because ogma.ingest.cip_models was created by an agent, I want you to create a detailed prompt to recreate cip_models from the schema but with explicit guardrails to ensure that the field names are modeled exactly.

Then use your runSubagent tool to make a new set of Pydantic models for CIP. Then check that new module against the old one ensuring

1. the field names are in strict agreement
2. that the types and initializers also agree

Response 2: regenerated CIP models with schema-aligned field names

Successfully regenerated CIP Pydantic models with field names that EXACTLY match cip_schema.yaml, fixing all 14 field name mismatches in CIPHeader and 2 in CIPLine.

The subagent was given strict guardrails:

1. EXACT FIELD NAME MATCHING: Field names MUST be identical to schema keys
2. No modifications allowed: No abbreviating, expanding, or adding suffixes
3. Schema is authoritative: cip_schema.yaml is the single source of truth
4. Type preservation: Retain all types from existing models (str, int, optional fields)
5. Structure preservation: Maintain model hierarchy, lists, config, and denormalized models

Critical field name corrections verified:

Beyond the billing provider fields, the regeneration also corrected patient → eligible naming (patient_first_name → eligible_first_name), provider abbreviation consistency (attending_provider_* → admitting_prov_*), and several other terminology corrections.

Prompt 3: generalize model generation prompt

In the last step, you created a special purpose prompt regenerate_cip_models_from_schema.md. I want to recast this as a general Copilot Agent definition that can create a set of Pydantic models from a yaml schema.

The file .github/agents/table-evaluator.md is an example, but you should look up the documentation for custom agent files yourself.

Response 3: copilot got stuck

Copilot created .github/agents/pydantic-model-generator.md and then got stuck trying to run python. I created .github/copilot-instructions to tell it that uv run python is the proper way and moved the uncompleted steps to Prompt 4.

Prompt 4

Using the runSubagent tool to invoke pydantic-model-generator, replace the ogma.ingest.cip_models and elg_models modules starting from their respective schemas.

Finally, run ogma.ingest.main to test the new models.

Response 4: successfully regenerated and tested both CIP and ELG models

Successfully regenerated both cip_models.py and elg_models.py using the pydantic-model-generator agent. Fixed database.py to remove an invalid field reference (hcpcs_rate) exposed by the stricter schema-aligned models. All models now have field names that exactly match their schemas, and ingestion testing passed successfully.

T-MSIS FTS Ingestion

Ingesting CIP file: cip_sample.fts
  Found 1 claims · Ingested 3 lines · Ingested 3 diagnoses
Ingesting ELG file: elg_sample.fts
  Found 1 eligibility records · Ingested 1 eligibility records

Ingestion complete!

✅ All FTS parsing completed successfully ✅ All Pydantic model validations passed ✅ Database records inserted correctly

Prompt 5: destroying intermediate python programs

In satisfying Prompt 4 with the subagent, I watched it create python programs for model generation and validation. While I didn’t capture the CIP programs, I was able to stash tools/*.py before you deleted them.

First, why did that happen? Is it a feature of subagents that I need to know?

Second, this suggests that I don’t need a subagent for the conversion but rather a more generalized version of the tools programs you created. Is this correct? Don’t write code, just help me understand.

Response 5: understanding subagent behavior and architectural choices

Subagents don’t auto-delete files — the parent agent attempted cleanup of what appeared to be implementation scaffolding. The ELG tools reveal a better architecture: promote the Python generation scripts to permanent, version-controlled infrastructure while keeping the agent for ad-hoc tasks.

Use subagents when:

• Task is one-time or infrequent
• Requirements might change (the agent can adapt to new constraints)
• You want delegation — “figure out how to do this and report back”
• Process needs human judgment or iteration

Use permanent tools when:

• Task is repetitive (regenerating models after every schema change)
• Process is well-defined and stable
• You want transparency — team members can read and modify the logic
• You need CI/CD integration or automation hooks
• You want version control of the generation logic itself

The Pattern You’ve Discovered

This mirrors a common evolution in AI-assisted development:

1. Discovery phase: Agent explores solution space, creates working code
2. Capture phase: Recognize valuable patterns in agent’s work
3. Formalization phase: Promote ad-hoc scripts to first-class tools
4. Hybrid phase: Keep agent for exceptions, use tools for routine work

Best of both worlds: tools are the implementation (reviewed, tested, version controlled); the agent is the interface (convenient, adaptive, able to handle edge cases the tool doesn’t anticipate yet).