Guide
Structured decisions: format, quality and human review
A valid category can still be incorrect. This guide separates output format, decision quality and the conditions for automation.
The contract
Start with allowed outputs
A support workflow often needs a destination category. A contract can restrict responses to billing, access, technical or other. That fixed vocabulary simplifies validation and gives each output a meaning. Decide whether multiple categories are allowed, whether a category is missing and how unreadable input should be reported. These are business choices. They determine which examples need annotation and which errors need monitoring. They also help the application handle new or unfamiliar requests without inventing a destination.
The format
Valid responses can still be wrong
A schema checks whether fields are present and values match the expected types. It does not check whether a ticket was understood. An invoice can be classified as a contract in a perfectly valid format. Evaluation must therefore separate format validity from classification quality. This distinction matters regardless of the method used to constrain outputs. A format guarantee does not establish that a decision is correct, nor does it establish that the surrounding workflow will take an appropriate action.
The models
Compare several approaches
A rules engine, a specialised classifier or a language model with constrained outputs can address parts of the problem. The choice depends on available data, category count and variation in wording. Start by comparing performance on real cases against existing rules. Measure errors that change the workflow, requests sent to review and integration costs. A larger model is not sufficient evidence of a better result. A simpler system may be easier to evaluate or maintain for a tightly defined decision.
Uncertainty
Test the reported probabilities
Probabilities express uncertainty over possible choices. Their value depends on how they relate to observed outcomes. If decisions reported at 90% confidence are correct much less often, the automation threshold becomes misleading. A calibration study needs examples separate from those used to tune the system. It should track the populations being processed and changes over time. Probabilities displayed in a commercial demo do not establish calibration. Both distribution shifts and rare categories can change how useful an estimate is.
Review
Define the cost of an error
Sending a request to the wrong team causes delay. Rejecting a complete file can block a customer. These consequences justify different handling rules. The review process needs an owner, enough context and a way to feed corrections into evaluation data. Human review rate is part of the pilot outcome. A higher threshold may reduce automatic errors while increasing workload for your team. Measure both effects before deciding whether an automation policy is useful in practice.
The reference
Understanding the System One interface
TypeSafe describes Jev as a model receiving state and returning typed decisions with probabilities. That interface inspires the Deyqo project. TypeSafe’s published results, prices, format guarantees and training methods concern its own service. Deyqo does not claim those properties. Our project still needs to turn its intended interface into an evaluated, documented service operated in Europe. The distinction matters when comparing an available product with a project that is still defining its implementation and suppliers.
Read TypeSafe’s announcementGetting started
Build an observable pilot
Define one decision, gather annotated examples and reserve an independent test set. Analyse errors by category, then choose a review policy. Observe the service before granting it irreversible actions. Publish the measurement conditions: data, version, period, cost and response time. That evidence helps establish whether the system improves the workflow and when your team needs to stay involved. Continue collecting difficult cases after launch so that later decisions are based on the traffic the service actually handles.
Test data
Leave room for difficult cases
An evaluation set needs short messages, incomplete documents and requests carrying several intents. Include out-of-scope examples where none of the offered choices is appropriate. Keep disagreements between annotators: they may reveal an unclear business rule. For each error, distinguish missing context, a poorly defined category and an incorrect prediction. This helps you change the contract or the model in the right place. Difficult cases also help verify that the human review process receives enough information to resolve uncertainty.
Describe your use case
Collect the expected categories, volumes and your team’s role in a pilot brief.
Plan a pilot