From Prototype to Production: Taking GenAI Live Without Losing Control

00:00:00: Hello and welcome to the next episode of Data Sciences, Milken Sugar.

00:00:04: Today we have again an English version of our podcast.

00:00:08: We have two guests.

00:00:10: today It's a first time you have two Guests in our podcast.

00:00:15: it's Somia and Thomas both AI experts data engineer data scientist at Datamix company Both self experiences in industry projects Delivered already AI products into production.

00:00:32: So happy to invite you!

00:00:38: Thank You Rene,

00:00:40: really happy that you're here.

00:00:41: let's jump in.

00:00:42: today we will talk about how to deploy AI projects and production.

00:00:47: I read today also from Gardner study there in two thousand twenty six sixty percent of AI prototypes may be removed than another study from rent showing that eighty percent won't go in production of use cases.

00:01:04: I think it's a very critical point and the very important topic, so i'm really happy to hear you here on this podcast.

00:01:10: first question would be then everybody is building now AI prototypes.

00:01:15: some maybe moved on somewhere little bit uh... development phase but then deploying them as really a nightmare.

00:01:23: what does your experiences sound in regard to

00:01:26: that.

00:01:27: Yes, so... That is so true.

00:01:29: now everyone's building.

00:01:31: it's taking one day of wipe coding maxed get your rags and agents running.

00:01:37: but the critical part here how do you deploy?

00:01:41: How do scale it?

00:01:42: How make production ready will put this prototype that you're wipe coding in front customers?

00:01:51: yes no thats a thing to answer.

00:01:54: And for that reason, it is also becoming very critical to understand how the infrastructure is built.

00:02:00: It's like the boring part people want to go away from but this is critical.

00:02:05: in big companies.

00:02:07: usually you get a budget for the POC.

00:02:10: You know once your POC is through you get budgeted for the entire thing and then you don't know where to go next.

00:02:17: How do you build?

00:02:21: So this is where I think what we have also managed to create as AIObskit would be very helpful.

00:02:30: To get the kickstart, build an infrastructure give a baseline to start on and then just deploy your agents in drags.

00:02:40: so a kickstart and customizable solution that you can support through AIOBSkit.

00:02:50: Yes, it's a good answer.

00:02:51: So let's talk maybe about the kickstart.

00:02:55: how can you kick start?

00:02:56: because there are many tools out.

00:02:58: There is quite complex Different tunings, guardrails and a lot of passwords Out there.

00:03:04: people are generating some prototypes often And then How I hope when your kick starts in AI projects.

00:03:11: so if have some documents or some feature?

00:03:17: Well, it really depends on what are you building.

00:03:21: If there's a RAG which most of the companies want under their hood when they call and AI to work for them.

00:03:28: So if we talk about RAG agents then first off all You need framework to work on.

00:03:37: so these Lambra, FlamChain, Linux whatever kind of framework.

00:03:44: Secondly, you need vector database to store your vectors.

00:03:49: You need a pipeline so that the data you're putting into your vector database is chunked properly and then sent there.

00:03:59: I'm just talking on very high-level overview here, not going in two details And once you have to figure out how are you gonna deploy it or scale it.

00:04:10: So we'll need a scalable solution also doing some sort of evaluation and tracing before making it go all the way to customers.

00:04:22: So these are some important pillars when you're deciding kickstart your GNI project.

00:04:31: And also at the end, you'll need some sort of guard rails and policies.

00:04:36: So what tool do we use in all these setups is very critical?

00:04:39: What kind of vector DB you choose or framework that you chose?

00:04:43: What kinds of LLM model did you choose?

00:04:48: Eventually observability monitoring tracing... These days the cost blew up with token usage so your infrastructure needs to be monitored

00:04:59: as well.

00:05:00: So all these things and pillars are very critical, just like any other machine learning project that people were doing earlier before the AI rack became really cool buzzwords.

00:05:19: It's the same paradigm we have been working on since long.

00:05:24: it is just names changed.

00:05:25: new tools come in so you need to make sure these things.

00:05:33: Before we jump over to Thomas, do you have any examples of all tools?

00:05:37: Newtools that you mentioned or Guardroy is talking usage.

00:05:40: how to prevent this?

00:05:41: Any tips for the listeners?

00:05:46: All right.

00:05:47: So one of the stack of my choice when your are still starting up which was easy to integrate and work with would be To use a luncheon frameworks line.

00:06:00: graph is pretty cool out there to start with.

00:06:03: It has very nice integration, a lot of different tools out there.

00:06:08: Qdranth as an vector database is really nice especially for the filtering and metadata part.

00:06:17: it's really robust.

00:06:19: next in terms for observability.

00:06:26: Langfuse is doing pretty good job also by langchain, they have langsmith which is all so good a little bit pricey but still does the job.

00:06:38: and then on the guardrails we have used some tool.

00:06:41: that's Coralogix Which had different policies you can incorporate into them.

00:06:46: So yeah this something I really like as a tool stack.

00:06:50: It was working

00:06:52: quite well All right, so now you mentioned a lot of components.

00:06:59: A little bit complexity then Thomas.

00:07:02: how I could imagine if he deployed these tools all together.

00:07:05: they have to communicate?

00:07:07: They have to be integrated.

00:07:09: maybe it's a chaos or How can you prevent the total chaos that we are work together If you want to deploy this and kick-start?

00:07:19: So i think most important The decision is to go off rate from any manual deployment

00:07:28: and

00:07:29: go towards infrastructure as code with tools like Terraform, because then you can just redeploy it.

00:07:38: And also destroy if you want.

00:07:41: You always have the same configuration and same pods in containers in place which are very important because otherwise you will end up with configuration drift and several problems, basically the chaos that was mentioned.

00:07:59: And what are the challenges then?

00:08:01: Probably also updates now.

00:08:02: if some security patches need to be quick update libraries maybe new features rolled out.

00:08:13: how can we handle this?

00:08:16: Basically, that's the beauty of Instructure as Code because you define it once and you can deploy to all your environments.

00:08:24: And if you add like automated CI CD pipelines with GitHub actions so on You get a high level of automation.

00:08:35: So save time.

00:08:37: Long would then take to kickstart project?

00:08:42: One week

00:08:43: automation and everything, so depending of course on the complexity.

00:08:48: And then you have a full running project probably with guardrails quality checks in production from test phase once the data is there.

00:08:59: Do we any example maybe use case which would like to bring production?

00:09:07: Yeah So we played around this travel assistant for our demo system And we added API that could retrieve information from Deutsche Bahn for train connections.

00:09:23: We could deploy this really quick, and I think it's a nice case because you can also show that you can connect to other APIs.

00:09:34: so you have like an intelligent agent who can do things not only retrieved from internal documents like basic reg.

00:09:47: All right, thanks for that insight and I'll make back to Somia a little bit more about strategies.

00:09:57: Like you mentioned these tools Langfuse and so on if companies they face maybe make or buy dilemmas So should they buy products that are ready?

00:10:08: Or may be build the use cases by themselves?

00:10:11: What do you think in that regard?

00:10:15: I mean, it depends on the kind of team we are dealing with.

00:10:18: If that's like one of the biggest decision teams face you buy something and it costs you more but then you don't have any operational overhead.

00:10:29: everything is working fine seamless But you do not know where your data is And when they're locking.

00:10:36: Now if you dont' buy a tool or an orchestration platform any sophisticated set of tools, then you have to manage everything on your own.

00:10:48: Also the infrastructure and Then it might take longer time for you to deploy.

00:10:54: So instead of doing that we have a developed AI Ops kit which could help You.

00:11:00: just in fact don't have to buy an be-in-the-wender lock-in.

00:11:03: It is like tool agnostic infrastructure Provisioning kit Which would deploying your GNI application to production.

00:11:16: So within one week you have everything set up, all tools connected, everything up and running.

00:11:24: basically it can also run within thirty minutes but we just say a one-week to avoid any sort of complications or problems there.

00:11:34: so in terms Build versus buy depends on the team.

00:11:39: If you really want to, if you have a smaller team and you want to be faster without managing operational overheads then you don't have money by sophisticated tools.

00:11:50: The best answer would be to use AI Opskit in that matter where You get all of the tools And also plug-and-play like LEGO.

00:11:59: Which tool do you want?

00:12:00: Which tool will you eliminate?

00:12:05: Just deploy it on your cloud or on-prem?

00:12:06: All right, thanks for that.

00:12:07: So I think what we discussed it's more LMM or a little bit of impression.

00:12:12: if you have an LMM, you have a REX system and build it out the box but now also the trends is going in direction of the loops has been on conference recently And then some kind audience told yeah You can loop it to just look at this like solution?

00:12:26: Maybe It's going this direction.

00:12:28: Also there are some startup.

00:12:29: they wanted to make self-trained AI component that AIs telling AI What To Do and then self-build.

00:12:37: Do you also have some ideas in this direction, in agents?

00:12:40: What are the tools out there?

00:12:42: what is the architecture to consider their... You mentioned already the token usage which might increase maybe to limit this.

00:12:49: do you know any solutions or approaches in that regard?

00:12:54: For the loop yes it's getting a lot of traction.

00:12:57: I haven't played with as much but you would still, regardless need some sort of observability there to manage your cost and token usage.

00:13:07: So any sort of Observability in trace management will be very well required such as Langviews or Longsmith To Manage that.

00:13:18: In terms of architecture You Would Still Need a Human-in-the-Luke to put the stop at some point.

00:13:26: so it is not entirely automated to run forever on its own, especially with the cost of token.

00:13:34: You can maybe use a cheaper LLM model here... ...to test it out how it's working.

00:13:41: what is the initial token usage and What kind of results?

00:13:46: It is producing And then eventually grow on top of that.

00:13:53: All right!

00:13:54: Thomas what do you think in this regard may be also direction of agents And to get the control of costs, like I heard at OpenClaw for example.

00:14:04: The agent system is burning a lot of tokens and you also set up an agent system in Toolkit.

00:14:14: How can you control then the token?

00:14:16: or we have different approach?

00:14:17: how do see that?

00:14:21: Of course You can click token usage with tools mentioned, length use etc.

00:14:28: But I think you should go like one step ahead and Think in the first place of what models.

00:14:36: You want to use?

00:14:37: And what they are cost because it's a big different if your user like GPT five four mini or GPT for all.

00:14:48: So really thing about the use case, and if you really need like the big model Maybe any future we will also see like a combination of models, maybe there's some determination for this task.

00:15:03: I will use the smaller model to save cost.

00:15:06: and now okay?

00:15:06: This is our complex tasks which automatically...

00:15:11: All right!

00:15:11: To see then these components that you need.

00:15:16: so as also I have seen in the translation probably You don't need to take the largest model but Yeah, even

00:15:30: yeah maybe you can also say you host some small model on your own and then for like less compute intensive tasks.

00:15:38: Like all problems go to your local model so that would be a big saving.

00:15:45: I guess

00:15:47: All right.

00:15:48: And another topic is probably security data privacy.

00:15:53: how to handle this in kick-starting projects?

00:15:57: Let's say you need one week to start a project, but you also need to consider security.

00:16:02: that data is not leaked.

00:16:05: The big consulting companies they hacked the data it's leaked and so on.

00:16:10: So what are important things about security or data privacy?

00:16:15: I

00:16:17: think most important one is PII.

00:16:20: in any form.

00:16:21: we're not taking the PII personally identified information of your customers.

00:16:28: Even if you're taking that up, You have some policy to remove this data within certain life cycle of time.

00:16:37: So That is very important.

00:16:39: firstly When your deploying it on cloud don't connect To any mcp server that you dont know off and try put in production Because a lot breaches are coming out based because This as well connected connecting to unknown mcp servers.

00:16:57: even in your staging environment.

00:17:00: that can lead to catastrophic sometimes also distracting your entire platform and infrastructure.

00:17:10: Thirdly, for the customer side important to have guardrails making sure any critical information kind of queries are asked are rejected right out there instead of going back to your LLM model.

00:17:26: So some sort of guard railing on top, so these three things are quite important in my opinion?

00:17:35: Yes you mentioned guard railings and that's also or it has been the hot topic.

00:17:41: everybody talked about hallucinations.

00:17:43: I don't know.

00:17:44: is this still a problem?

00:17:49: If yes, how can it be handled or reduced?

00:17:54: Or completely

00:17:55: removed.

00:17:57: Complete removal of hallucination I think is still debatable.

00:18:04: but you could reduce in a way that you categorize the kind questions being asked to your deployed agent and RAG system And once you categorize this, on what category of questions there are.

00:18:23: hallucinations firstly.

00:18:24: So important is to find out where and what kind of question the hallucination is happening.

00:18:35: but still it is possible that even given the right context The model hallucinates.

00:18:41: so must use some sort of metrics and evaluations pre-deployment to make sure they are reduced on your system.

00:18:50: So it's an entire process of GNI evaluation, I would say and where you...I can like talk in depth about the GNI Evaluation but yeah starting from Granto creation metric creation hallucination as one of the metrics and then deploy it.

00:19:16: So those are different, that is a difference stream altogether to detect.

00:19:22: Yeah so we're back into the circle as you mentioned at beginning.

00:19:25: like mm we do same like with before.

00:19:28: this.

00:19:29: classical machine learning seems like we labeling The possible answers or questions And can measure quality of model Can also use AI or to train test AI?

00:19:48: Yeah, definitely.

00:19:49: That's what most of the people are doing because they're using LLM as judge for doing these evaluations at the moment.

00:19:58: so imagine you ask a question uh...to your um air fryer that has a rag?

00:20:06: and then you asked how do i I don't know make pizza in my air fryer, say for example and then it blurts out some sort of answer to that.

00:20:19: You need to know what category of answer again?

00:20:22: It should go... Should I go to user manual to figure that out or should i internet to figure this out?

00:20:31: exactly?

00:20:32: which tool?

00:20:33: do you have?

00:20:34: a mcp within your air frier?

00:20:39: select certain program within your air fryer, so it depends on that.

00:20:44: So for every question you need to really know what kind of answer It should blur out or What kind of tool call?

00:20:50: Which is basically machine learning classical labeling.

00:20:55: where do we have a Question like categorize this as cat or dog and then you'll have a label.

00:21:01: right The same thing here when you ask the question which would be there probably will answer.

00:21:07: once You Have That sort of thing, you can always use it as a validation data set analogous to machine learning.

00:21:17: again and before deploying into the customer.

00:21:21: Always check on those set off fifty-to-hundred questions that you have whether changing your LLM model?

00:21:30: Whether changing your chunking strategy?

00:21:34: Whether change in your prompt.

00:21:36: how is this affecting any of your metrics?

00:21:39: for helpfulness, faithfulness.

00:21:42: For hallucination checking answer correctness and bunch of others?

00:21:46: you can have your own different metrics as well.

00:21:50: So yeah it's a complete setup doing evaluations in my opinion And its very in-depth but very analogous to machine learning labeling using validation test Validation and test data set before deploying into the customer.

00:22:07: When you say, okay my machine learning model is ninety percent accurate it's good to go.

00:22:13: It's generalizing enough and I will put into production.

00:22:17: but when you have a rag in an agent how do you say that?

00:22:20: Like at what point did they say this was good enough for me sent the production And uh... This where we need some sort of metrics measurable metrics basically.

00:22:32: So there may be helpfulness increased by from eighty to ninety percent now and it's looking good with my new prompt change.

00:22:41: I'm happy to deploy in production, whereas as i don't know its working fine on these five questions let's deploy.

00:22:47: that's different

00:22:51: okay.

00:22:51: so yeah you mentioned a lot mcp and agents.

00:22:56: could you please explain to the listeners what is this?

00:23:00: maybe not everybody knows.

00:23:06: It stands for model context protocol, it's an open source standard which was created by Anthropic that allows AI models to communicate securely with external data sources file systems and tools etc.

00:23:21: So imagine people were using APIs in their backend through a reach out to the external system from there deployed back end.

00:23:35: But now with the JNAI rags and agents, what you can do is to reach out an external system instead of using APIs.

00:23:45: Now you use MCP servers where your agent acts as a MCP client and get some information.

00:23:59: So, for example if we are living in Germany And if Deutsche Bahn opens a MCP server For booking system on the Deutsche Bahn tickets Then what can do is create our agent.

00:24:13: We connect to the MCP Server of Deutsche Bahn Definitely with some token and API keys there.

00:24:23: Once you do that make use of all the capabilities, booking Deutsche Bank tickets via Agent.

00:24:33: So now your agent has extra capability.

00:24:35: you can go to it and say hey book me a ticket from Munich Berlin And then knows which API to access.

00:24:48: how exactly do that?

00:24:51: booking for you come back with payment gateway So you eliminate that entire task.

00:25:00: That's a gist of it, I hope they explained it

00:25:06: well!

00:25:09: All right thanks.

00:25:10: i think this is good explanation.

00:25:12: so we made quite the story from the kids start at the beginning about guardrails vectors how to deploy everything How get chaos out-of-the way and have a smooth start in security and quality.

00:25:28: And now finally, even agents and MCP is in the depth.

00:25:31: so I would wrap up a little bit and we go over to your advice.

00:25:38: or what may start with Thomas?

00:25:41: What advice do you give an engineering team that's just starting their GNAI journey?

00:25:49: So my advice will be not even start manual deployments automate from the start, not later if you need to scale or redeploy and just treat infrastructure like software.

00:26:11: All

00:26:14: right.

00:26:18: And what is your advice?

00:26:27: for people who are just starting.

00:26:30: And learn what a tracing engineer is, that's also very critical because you don't know What kind of question You're asking and what trace it is generating?

00:26:43: Is going to the right tool?

00:26:45: how much cost token usage?

00:26:47: That is happening.

00:26:48: How do you analyze those traces?

00:26:51: so In traditional machine learning, there were data scientists who are analyzing all the data and coming up with a model.

00:26:59: But now for the GNI The same data scientist could be very well part of Trace analysis to figure out.

00:27:08: hey our model or our GNI application is working really bad in category X versus and performing really good in category Y so trace analysis is going to be really critical.

00:27:23: All right, very good advices, trace quality also mentioned vibe coding.

00:27:28: it's also a critical point these days but this will be another podcast topic probably too fill up so I don't want the deep dive in there But its also a topic to take care.

00:27:40: Thanks both of you for your time and advice.

00:27:43: for listeners if they would like to start with GNAI project bring them into

00:27:49: production.

00:27:50: Thanks both of you!

00:27:53: Thank You.