Fadhilmch.
← All posts

7 min read

A serving SDK data scientists actually use

An SDK is an interface, so design it like one. What we hid, what we kept visible, and why a narrow path to production gets used.

ML platformProduct design

An SDK is an interface. The people calling it are data scientists instead of other engineers, but the same rules apply: a small surface, sensible defaults, and errors that say what to do next. If every model author has to learn cluster setup, authentication, tracing and deployment conventions before shipping a model, the platform is exposing its machinery instead of helping with the task.

I built a serving engine and SDK for an ML platform. It supported approximately 40 model deployments, and the teams behind them did not have to configure the underlying infrastructure. This post covers how I thought about the design: what the interface looks like and which decisions I’d make again. Some of it is general practice rather than what I built, and I’ll flag which is which.

From notebook function to production service

A trained model in a notebook is a function: features in, prediction out. A model in production is a service, and most of what separates the two has little to do with machine learning. Someone has to:

  • put the model behind an HTTP or gRPC endpoint
  • authenticate callers
  • emit traces and metrics so the model can be debugged when it misbehaves
  • package it and run it on Kubernetes with sensible resource settings
  • handle concurrent requests, including any parallel execution the model needs
  • keep the same conventions as every other service in the company

Each of these is learnable. All of them together is a lot of unfamiliar ground for someone whose expertise is modelling. A data scientist can learn them, but each one who does spends days on work that isn’t their job, and each ends up with slightly different choices. When ten teams each ship a model by hand, you end up with ten ways of doing authentication and ten sets of dashboards.

Path from notebook to endpoint, before and after an SDK Illustrative. Without a shared serving path, the model author handles six steps: the model, a web app, authentication, tracing, Kubernetes deployment, and metrics and alerts. With the SDK, the author does two things, wrap the model and deploy, and the platform provides the rest. WITHOUT A SHARED PATH · THE AUTHOR OWNS EVERY STEP Notebookthe model Web appa server Authwho calls Tracingrequests DeployKubernetes Metricsand alerts five steps that are not modelling, each a chance to diverge from the last team WITH THE SDK · THE AUTHOR OWNS THE MODEL, THE PLATFORM OWNS THE REST Notebookthe model Wrap with SDKa class and a config Deployed endpointauth, traces, metrics included two steps for the author, and the same steps for every model Illustrative: the steps a typical hand-rolled deployment needs, not a measured timeline.
The point of a serving SDK is to shrink the author's path to the parts that are about the model. The steps still happen. They just happen once, in the platform.

What I put behind the interface

The engine took over the parts that are the same for every model. It checked callers the same way on every endpoint, handled tracing, ran the model on the cluster, took care of parallel execution, and emitted metrics and logs in one format. Model authors didn’t write auth code, add instrumentation calls or touch Kubernetes config.

The design test I used for each item was simple: would two different data scientists ever have a good reason to make a different choice here? For authentication and tracing the answer is almost always no, so they belong in the platform. For anything where the answer is yes, they belong in the interface, as an option with a default.

What the author writes and what the platform handles The model author writes model loading, a predict function, input and output schemas, and a small config. The SDK sits between, and the platform provides authentication, tracing, Kubernetes orchestration, parallel execution and observability. THE AUTHOR WRITES THE PLATFORM HANDLES load the model predict() input and output schema resources and options Python, in the tools they know SDK one contract defaults clear errors any framework authentication tracing Kubernetes orchestration parallel execution observability Illustrative split, following the design described in the text.
The author's side is small and stays in Python. The right-hand column is the same for every deployment, which is exactly why it belongs in one place.

What the author still writes

An SDK is easy to get wrong by hiding too much. If the only way to use it is a magic decorator that reads the notebook’s global state, nobody can reason about what it does. I wanted the author’s side to be a few explicit things:

  1. How to load the model (files, weights, whatever the framework needs).
  2. What “predict” does for one request.
  3. What goes in and what comes out, as a schema.
  4. A short config for the things that do vary, such as resource requests.

Here is the shape of that, as a sketch rather than the real API:

from serving_sdk import Model, Schema, Config

class ChurnModel(Model):
    input = Schema(user_id=str, features=list[float])
    output = Schema(score=float)

    def load(self, path):
        self.model = load_my_model(path)   # any framework

    def predict(self, request):
        return {"score": self.model.score(request.features)}

config = Config(cpu="1", memory="2Gi", replicas=2)

Two things about this interface matter more than its length. First, the class has exactly the three methods and attributes a model needs, so there is little to misuse. Second, nothing in it refers to the framework. The engine calls load and predict, and what happens inside is the author’s business. That’s what keeps the engine framework-agnostic. The contract only talks about requests and responses, so it doesn’t care whether the model is scikit-learn, PyTorch or something else.

Design choices I’d defend

Make the right thing the default. Settings that matter for safety or operability, like authentication and tracing, should need no configuration to be on. Turning something off should be deliberate. A default the author must remember to enable will be missed by some teams.

Keep it narrow. It is tempting to offer hooks for every possible need. Each hook is a promise you have to keep as the platform changes, and each one lets teams drift back into bespoke setups. I would rather have a smaller interface that covers most models cleanly, and add an option only when several teams have asked for the same thing.

Write good error messages. A stack trace from deep inside the engine tells a data scientist nothing. Validation before deployment, with messages such as “the output schema says score is a float but predict returned a string”, turns a support conversation into a fix the author makes alone. When people say a platform is “easy to use”, a lot of that is error messages.

Ship working examples. A working example for each common model type does more than a reference page. Most people will copy the closest example and edit it, so in practice the examples are the interface people learn.

Document the way out. Some models will not fit. The choice is between a documented way out and a quiet fork of the platform. A documented way out, even a clumsy one, keeps those teams visible and lets you learn from them.

Where abstractions leak

The abstraction will leak somewhere, so it’s worth choosing where. A common place is performance: a model that needs a lot of memory or a slow start-up behaves differently under the platform’s scheduling from how it did in a notebook. When the layer hides Kubernetes completely, the author has no vocabulary for the problem.

The general fix I prefer is to expose the effects rather than the machinery. Let the author declare what the model needs (memory, warm-up time, concurrency) in the config, and surface what happened (a start-up that took too long, a request that timed out) in terms of those settings. They shouldn’t need to know what a pod is to fix it.

A shared path also standardises operations

Building one serving path had a second benefit that I underrated at the start. When every deployment is produced the same way, the platform can also say something consistent about all of them. Authentication, tracing and metrics come out in one form, so a single set of dashboards and alerts can cover every model. Governance questions, such as which models are running and who owns them, have one place to be answered.

We never had to ask anyone to follow a standard. Teams got it by taking the easiest route to deployment.

Build or adopt

Existing tools cover a lot of this, so it’s fair to ask whether to build anything at all. If an existing serving framework already matches your company’s authentication, tracing and deployment conventions, adopt it and spend the time elsewhere. If most of the work is fitting those conventions, then what you’re really building is the thin layer that connects a model to them, and that layer is worth owning. Either way, you still decide what your data scientists see, and the questions above still apply.

If you’re building one

  • Treat data scientists as users of an interface, and design that interface the way you would a public API.
  • Decide what belongs to the platform by asking whether any two authors would legitimately choose differently.
  • Keep the author’s side explicit and small, and keep the contract independent of any modelling framework.
  • Invest in defaults, examples and error messages. They are most of what people experience.
  • Expect leaks, and choose in advance where they show up.

The part I was happiest with was that teams could take a model from a notebook to a running endpoint without becoming infrastructure engineers, and that we only had one path to keep secure and observable.

References

  1. Sculley, D. et al. Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015. https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  2. OpenTelemetry. Traces. https://opentelemetry.io/docs/concepts/signals/traces/. Spans, trace IDs and context propagation.
  3. Kubernetes. Resource Management for Pods and Containers. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/. CPU and memory requests and limits.
  4. KServe. KServe documentation. https://kserve.github.io/website/. An open-source model serving platform on Kubernetes.
  5. BentoML. BentoML documentation. https://docs.bentoml.com/en/latest/. A Python framework for packaging and serving models.