Tag: Agent Evaluation Framework

  • Job-ready agents like Casey and Hunter: evaluating a prebuilt agent against a custom build

    Salesforce ships Casey and Hunter as finished employees, not building blocks. Casey handles service conversations, Hunter chases sales leads, and the demo makes both look ready to drop straight into a contact centre or a pipeline review. The pitch is simple: skip the months of topic design and action wiring, hire the agent, point it at your data, done. For a lot of orgs that pitch is close enough to true. For plenty of others it isn’t, and the gap between those two groups is where I spend most of my Agentforce conversations lately.

    What “job-ready” actually means

    Job-ready doesn’t mean zero configuration. It means Salesforce has already made the decisions you’d otherwise spend weeks making yourself: which topics exist, what instructions sit under each one, which standard actions get called, and roughly what tone the responses take. You’re still connecting data, setting guardrails, and testing against your own cases. What you’re not doing is designing the agent’s reasoning structure from a blank canvas.

    That’s a real head start. I’ve sat through enough early Agentforce builds to know that topic design is where teams lose the most time, not model tuning, not UI work. Getting the boundaries between topics right, so the agent doesn’t try to handle a billing dispute through the shipping-status topic, takes iteration most teams underestimate. Casey and Hunter arrive with that iteration already done, for the scenarios Salesforce built them around.

    Where the prebuilt path earns its keep

    If your service org runs reasonably standard Case and Knowledge processes, Casey will get you to a usable pilot faster than a custom build, often by weeks. Same goes for Hunter against a fairly standard Lead and Opportunity motion. Three situations where I’d lean toward the prebuilt option without much hesitation:

    • You need a working agent in front of stakeholders inside a sprint or two, and the use case is close to the standard pattern.
    • Your data model hasn’t drifted far from Salesforce’s defaults, so the standard actions actually find what they’re looking for.
    • You want a reference implementation to learn against before committing to a bigger custom investment.

    That third one matters more than people give it credit for. Standing up Casey against a sandbox, even one you know you’ll outgrow, teaches your team how topics, instructions and actions actually behave together. It’s cheaper training than a six-week discovery workshop.

    Where it hits a wall

    The wall shows up fast once your process deviates from what the agent assumes. A contact-centre team I worked with routes cases through four separate queues based on entitlement tier, language, and a product-specific escalation flag that lives on a custom object. Casey’s standard case-handling topic has no concept of that flag. You can extend it, Salesforce does let you add custom topics and actions alongside the prebuilt ones, but at that point you’re doing custom build work anyway, just inside someone else’s scaffolding.

    The same thing happens with Hunter wherever territory rules, custom scoring, or a non-standard opportunity stage model sit between a lead and a qualified deal. The agent isn’t wrong. It’s just reasoning against a model of your sales process that your actual sales process doesn’t match.

    The real evaluation question

    Most teams frame this as prebuilt versus custom, as if it’s a single irreversible choice made in a steering committee meeting. It isn’t. The honest question is narrower: how much of this specific use case matches the standard pattern Salesforce designed for, and how much is the customisation that made your org worth customising in the first place?

    I run four questions before recommending either direction, and I’d encourage any technical lead to do the same rather than taking the vendor comparison at face value.

    Four questions worth asking before you commit

    • Does the core object model match? If cases, leads, and the fields that drive routing are close to standard, the prebuilt agent’s assumptions hold. If you’ve heavily customised the objects those topics depend on, you’re fighting the agent’s model of reality from day one.
    • Who owns the topic boundaries long term? Prebuilt topics are maintained by Salesforce, which means they change on Salesforce’s schedule, not yours. A custom build is entirely yours to extend, which costs more upfront and less in year two if your process keeps moving.
    • What’s the actual cost of a wrong answer here? A billing agent that mishandles a refund calculation needs tighter guardrails than one recommending a knowledge article. Prebuilt agents come with sensible defaults, not your specific risk tolerance.
    • How much of the value is in the exception handling? If 80% of your volume is the standard case and 20% is the reason you built a bespoke process, ask whether that 20% is actually the part worth automating. Sometimes it’s the part that should stay with a human.

    A framework I actually use

    In practice I treat Casey and Hunter as a starting position, not a final answer. Stand up the prebuilt agent against a slice of real traffic, maybe 10-15% of inbound service cases or a single sales segment, and measure where it succeeds unassisted versus where it hands off or gets the answer wrong. That gives you an actual number instead of a guess: if the standard topics resolve 70% of volume cleanly, custom work only needs to cover the remaining 30%, which is a much smaller and more honest scope than “replace the whole process.”

    If that number comes back low, under 40% say, that’s useful information too. It tells you your process is customised enough that a ground-up build, with topics designed around your actual case types rather than Salesforce’s assumed ones, will pay for itself faster than extending something that was never shaped for your data in the first place.

    Either way, don’t skip the measurement step and jump straight to a build decision based on how the demo looked. Demos are built on clean data and the exact scenario the agent was designed to handle. Your production queue is neither.

    What this means for the technical lead in the room

    The job here isn’t picking a side in the prebuilt-versus-custom debate before it starts. It’s running a short, honest test against real cases, reading the result without bias toward whichever option looked more impressive in a vendor session, and scoping the custom work to the gap that’s actually there. Most orgs end up somewhere in between: a prebuilt agent carrying the standard flow, with a handful of custom topics bolted on for the handling that made their process worth building in the first place. That’s not a compromise. For most Salesforce orgs with any operating history, it’s the correct answer.