Is task-specific fine-tuning worth it? A review of eight company case studies

AI-generated illustration.

It's the first Saturday of October, and I'm driving to a birthday party for my friend's son at a farm in Maryland. You pay to get in, and there are goats to feed, rabbits to see, tractor rides, and a zip line. Fun stuff.

On the drive, I'm thinking about something that keeps coming up in conversations with friends and on AI podcasts: enterprises want to depend less on frontier models. The proposed alternative is usually some version of “Use open-source models and host them yourself.”

I don't disagree with reducing that dependence. But I'm still left with three questions: Which model, doing what work, and deployed where? If you're considering fine-tuning for your own product, these examples can help you decide which tasks are worth testing and how to compare the results with what you use today.

Before I get out of the car, I record a voice note on my phone so I can come back to these questions and write about them later. That note became the starting point for this post.

For this post, I looked for companies that had adapted a model for a particular job, put it into a product, and reported what changed.

There are useful examples here, from extracting a customer's issue to reviewing a payment for fraud. Most describe production systems; Harvey's newer work shows promising research with a less settled path to customer deployment. The results come from company and partner reports, and I haven't independently reproduced their experiments.

Link to A few terms before the company namesA few terms before the company names

Open weights also don't mean the company releases its adapted model, or owns the machines serving it. Those choices show up differently across these companies.

First, though, there has to be a job small enough to describe.

Link to 1. Intercom: extract the customer's issue1. Intercom: extract the customer's issue

Before Fin can answer a support question, it needs a usable account of what the customer is asking. A conversation can include a correction, a greeting, and a reply to an earlier answer. Searching for all of that at once gives the next step more to untangle.

Intercom's September 11, 2025 report describes adapting Qwen3-14B for issue extraction with LoRA, a method that trains a small set of changes to the model. It uses 60,000 training examples and 10,000 validation examples from consenting customers with US accounts and English-locale conversations, with personal information redacted.

The downstream pipeline's offline answer rate reaches 63.4%, compared with 63.3% using GPT-4.1 and 18% using untuned Qwen. That measures whether extraction helps the rest of Fin answer the question.

In a one-week production A/B test, transaction cost falls 12.5%, answer rate falls 0.1 percentage points, and customer satisfaction stays unchanged. Median end-to-end latency gets 100 milliseconds worse.

Training only on clearly resolved conversations makes answer rate worse too. The authors find that model reluctant to extract an issue from an ambiguous conversation, which is an inconvenient thing for a support model to be reluctant about.

Intercom later expands the scope. Its March 2026 Apex announcement reports nearly all English chat and email conversations running Apex as the core answering model, replacing Sonnet 4.0.

Fergal Reid's account describes an unnamed open-weight starting model, supervised training on examples, and reinforcement learning using reward signals. Intercom runs inference on EC2 for more than a million conversations a week, while work such as Procedures still uses frontier models including Sonnet. The unnamed Apex base can't be assumed to be the earlier Qwen model.

There's a defined task and a production comparison. The ambiguous conversations also point to the next problem: what goes into the training examples?

Link to 2. Shopify: turn messy products and failed tasks into training data2. Shopify: turn messy products and failed tasks into training data

A merchant's title, description, and images can each contain part of what Shopify needs to classify a product. The output needs to fit consistent product types and attributes, even when the input doesn't.

Shopify's July 2025 catalogue report describes fine-tuned models that handle text and images, including Qwen2VL 7B, in production. Its infrastructure handles 40 million calls and roughly 16 billion tokens a day (tokens are the pieces of text models process).

Multiple models produce labels, an arbitrator reconciles them, and people review evaluation labels. Uncertain predictions and production failures become material for the next training round.

Shopify also changes how much the models have to write. Training them to predict only the fields that need filling cuts median latency from two seconds to 500 milliseconds and GPU usage by 40%. The report describes Triton managing requests across its GPU fleet and a streaming pipeline carrying results back into the system; the gains belong to that training-and-serving change, rather than an isolated test of fine-tuning.

The same need for useful failures appears in Sidekick's August 2026 report. Its GraphQL agent writes and runs queries against a merchant's store data, then turns the results into an answer. GraphQL is the query language it uses to ask for that data.

Low-scoring conversations become repair candidates. Frontier models propose fixes, an automated evaluator calibrated against human judgments scores them, and experts handle difficult cases. Successful sequences of actions become training examples.

Shopify estimates annual serving costs of about $1 million for the specialized model against $27 million for frontier inference at that traffic level. Those are projections, with no full accounting of training and maintaining the loop. The report also leaves the base model unnamed, so the catalogue remains the clearer open-weight example.

The company has a way to collect failures and train against them. I'd still want to know whether the result makes someone's work easier, beyond producing a better-looking model score.

Link to 3. Airbnb: check the work around the model3. Airbnb: check the work around the model

Airbnb's support agents need to pick up a conversation without reading every message and call transcript that came before them. A summary at the end helps the next person; notes that stay current can help the person already handling the case.

Its 2025 paper describes a Mixtral-8×7B model producing incremental notes, with a separate classifier filtering trivial bullets. Initial training uses about 30,000 examples for supervised learning and preference optimization, which teaches the model to favor better outputs, followed by about 8,000 examples drawn from agent edits.

The deployed summarizer runs through Triton and TensorRT-LLM on two A100 GPUs per instance. Inference latency is 600 milliseconds at the median and two seconds at the 95th percentile (95% of requests finish within that time). The paper doesn't specify a cloud or private-network boundary.

Its rollout study reports 3% lower handling time, reaching up to 9% for cases involving three or more agents. The comparison sites already use bulk summaries from a fine-tuned Mixtral model.

The study compares changes before and after rollout across sites, a method called difference-in-differences. The result belongs to the progressive-note workflow, including timing, filtering, prompts, and model adaptation; it doesn't separate each contribution.

Link to 4. LinkedIn: assess candidates against a role4. LinkedIn: assess candidates against a role

LinkedIn has another repeated step: assessing a candidate against a role. Its December 18, 2024 report describes EON-8B, adapted from Llama 3.1 8B, powering Hiring Assistant's evaluation agent. That step accounts for nearly 90% of Hiring Assistant's LLM calls.

It reads candidate information, recruiter notes, and job requirements, then returns an assessment with an explanation. LinkedIn describes on-premises training and serving, and reports four percentage points higher matching accuracy than GPT-4o mini without additional fine-tuning on the matching task.

That is a dated result on a matching test, rather than evidence of better hiring outcomes or a before-and-after migration. Alongside Airbnb, it shows why I need the name of the measured thing as much as the size of the improvement.

Coinbase makes that separation especially clear, because it reports an initial workflow experiment, a model benchmark, and production serving results separately.

Link to 5. Coinbase: evaluate the risk decision, then change the model5. Coinbase: evaluate the risk decision, then change the model

Brian Armstrong's October 11 post describes fine-tuning Qwen3.5-9B on Coinbase's internal data for fraud prevention. The engineering team's October 6–8 series explains what the model does inside Onramp, its product for buying crypto through partner apps.

Part one starts with a Claude Opus agent reviewing selected transactions after the existing machine-learning model and rules. Application code prepares recent transaction history and applies the decision policy. The agent can flag more risk, but can't override an upstream block.

The online experiment reports 30% fewer fraudulent transactions and 22% less fraud value compared with the existing stack alone. Those are results from adding the Claude-powered review workflow.

Then comes model selection. Part two uses 16,140 historical transactions across 7,293 users, including 813 confirmed fraud cases. The data predates the agent rollout so the agent's own blocking decisions don't change which outcomes can be observed.

With the context and decision policy held fixed, newer versions of Opus, Sonnet, and GPT have lower recall and F1 on this benchmark. Recall measures how much fraud the agent catches; precision measures how many of its fraud flags are correct. F1 combines the two, so catching more fraud by flagging everything doesn't automatically earn a good score.

Part three describes adapting Qwen3.5-9B with SkyRL on Anyscale. It uses GRPO, a reinforcement-learning method that learns from groups of scored responses, with LoRA updates. Rewards are checked against known fraud outcomes, and the benchmark compares single-pass decision agents with adaptive rules and reflection disabled.

Against Opus 4.5, Coinbase reports gains of 7.5 percentage points in precision, 12.0 in recall, 9.6 in F1, and 35.4 in dollar-weighted recall. That last measure is the share of fraudulent transaction value identified. The report's figure specifies percentage-point differences despite the percent signs in its prose.

The adapted model serves production traffic through Ray Serve LLM and vLLM, software for running and distributing model requests, on autoscaled L40S GPUs. Median end-to-end LLM-request latency is 0.683 seconds against 1.515 seconds for Opus 4.5, a 55% reduction.

Detection quality and latency come from separate evaluations. The report establishes production serving, but doesn't isolate a live fraud-loss reduction from replacing Opus with tuned Qwen.

That gives me something concrete to compare: the decision, the test, and the serving requirement. It also raises the practical question behind “host it yourself”: who is running the machines?

Link to 6. Impel: decide who operates the model6. Impel: decide who operates the model

Impel makes Sales AI for dealerships, including replies, follow-ups, and conversation summaries. Its June 2025 account with AWS describes replacing the third-party model behind those functions with a LoRA-adapted Llama model on autoscaled SageMaker endpoints in its own AWS account.

Human-review scores rise from 73% to 86% for personalized replies, 70% to 83% for summaries, and 59% to 92% for follow-ups. The report leaves the exact Llama variant, sample size, and detailed scoring rules undisclosed. It provides output-review results, with no measured vehicle-sales lift or numerical cost savings.

Link to 7. Cresta: use a serving partner7. Cresta: use a serving partner

Cresta's Knowledge Assist uses a live support conversation and retrieved company knowledge to suggest an answer for the human agent. Its engineering report confirms tuned Mistral models live with customers, with its Ocean training framework adapting models by industry, customer, and task.

Fireworks' deployment account describes a shared base model with customer-specific LoRA adapters on provider-managed dedicated instances. Cresta customizes the models while Fireworks operates the serving infrastructure.

Impel deploys in its AWS account, Cresta uses a serving partner, and LinkedIn describes on-premises serving. All three adapt models; the infrastructure responsibilities differ. Owning the recipe doesn't require owning the stove.

Link to Where to deployWhere to deploy

There are plenty of places to deploy a fine-tuned model, and Nebius Token Factory is one option to consider. Its Dedicated Endpoints reserve GPU capacity for your organization. You choose the region, the GPU setup, and the minimum and maximum capacity as traffic changes.

For eligible models, you can deploy your own fine-tuned weights while Nebius runs the serving infrastructure. You pay for GPU time, so I'd compare the cost and response times using the traffic I expect the product to handle. You can browse the endpoints console, but you'll need to sign in to deploy.

But even with a place to train and serve a model, a good research result still has to make it into the product.

Link to 8. Harvey: keep research and rollout separate8. Harvey: keep research and rollout separate

Harvey's August 20, 2026 Tenet announcement describes its first post-trained open-weight model, based on Kimi K3 and trained with Fireworks for legal tasks that take many steps. It reports gains of nine percentage points on its Legal Agent Benchmark and two on LAB Contracts in all-pass rate, meaning every evaluation criterion is satisfied.

Training uses synthetic, public legal, and human-expert data, with no customer data in its post-training efforts. Tenet is presented as a research preview.

Another August report with Applied Compute trains GLM-5.2 for Review Table, where lawyers ask questions across documents and receive a grid of answers with citations. Its per-cell score combines correctness and citation quality on a 0–1 scale, with models evaluated at minimal reasoning effort.

The reported score is 0.903, against 0.867 for Fable 5 and 0.857 for GPT-5.6-Sol. Average cost per cell is 54.8% below Sonnet 5 on the same tasks. Applied Compute describes public legal documents, frontier-generated questions, and expert checks behind the synthetic training material.

In September work with Baseten, a root agent divides a document room among sub-agents and assembles their findings into a diligence memo. Reinforcement learning on the Qwen3.5-122B-A10B root, with Qwen3.6-35B-A3B sub-agents held fixed, raises the fraction of evaluation criteria passed from 29.9% to 63.0% on 50 synthetic lab diligence rooms held out of training.

These results measure the reported tasks. The diligence work is initial research, and training inside Review Table's production setup doesn't establish customer traffic on the reported checkpoint. None of these announcements establishes publicly downloadable adapted weights, a public license for them, or a public model endpoint.

Harvey's earlier work with OpenAI belongs to separate proprietary-model history. An initial GPT-3 experiment used 100 landlord-and-tenant questions from Reddit, with attorneys saying they would send 86 answers without edits.

Later, lawyers at ten firms preferred a custom case-law model over GPT-4 97% of the time in side-by-side comparisons. Longer, more complete answers were among the reasons given. Preference is what that number measures.

Then, in a May 2025 BigLaw Bench comparison, Harvey reports newer models optimized within its systems surpassing its originally benchmarked system. Specialization still needs a fresh comparison when the alternatives change.

Link to The job descriptionThe job description

I'd start with the work that repeats: what comes in, what should come back, what makes an answer unusable, and who deals with it when it's wrong. That gives the evaluation a shape before a model has been chosen.

Then compare the existing workflow with the proposed one. Keep the ambiguous cases in the examples, measure the outcome the people doing the work care about, and count training, evaluation, engineering, monitoring, and serving alongside whatever still goes to a frontier model.

These are selected company reports, so they can't tell me the typical return or how many other attempts failed. They can give me a more useful place to start the next conversation.

On the way to the birthday party, I had a question about replacing models. Now I'd ask for a job description first. “Open source” has put in an application; it still needs to say what it's there to do.