AI Lessons Learned

Developing the government reference models involved thousands of related AI design and generation tasks. We learned where AI helps, where it makes mistakes, and how to make the process more reliable.

 

The best results came from using AI where interpretation was needed and deterministic software where we could apply a rule. How we organized the pipeline mattered as much as which model we used.

 

This reduced unnecessary AI work and made results easier to repeat. Cost still depends on the model, input size, retries, and how much work the pipeline does.

 

1. Keep Prompts Clear and Succinct
Give the agent the context it needs, clear instructions, the required input, and a way to judge the result. Extra information can distract it. Include only what helps it do the task.

 

2. Use Deterministic Processes Where Possible
Use AI for interpretation, judgment, and generation. Use deterministic processing for calculations, transformations, comparisons, data movement, and checks that follow clear rules. This reduces costs and increases the quality and consistency of your results.

​

3. Match the Model to the Task
Designing an enterprise model may need more reasoning than generating test data for an established schema. Choose the model and reasoning level for the work at each stage. The most capable model is not needed for every task.

 

4. Allow for More Than One Model
A pipeline that supports multiple models lets us compare results and choose a suitable model for each task. It also lets us use new models as they become available.

 

5. Keep Useful Diagnostics
Keep enough information to understand a run: the prompt, response, token usage, and execution time where these help diagnose problems. Use it to find missing context, unnecessary input, and work that consumes resources without improving the result.

 

6. Check the Results
A completed generation run is not proof that its output is correct. Build deterministic checks where possible. Compare the results with the requirements and check completeness, consistency, and conformance throughout the pipeline.

 

7. Expect Different Answers
An agent can give different answers to the same task. Differences early in a long pipeline can produce very different results later.

 

We need to control that variation. Keep prompts clear, define the inputs, use deterministic processing where possible, validate the results, and make it clear where AI may exercise judgment.