If you think about it, LLM providers are distributed systems on steroids. Building reliable systems with them, comes with particular challenges. A modern day developer should have proper skills of using LLMs in systems at scale.

Some of the problems one may face when building AI agents using model providers could be:

• Unreliable output of the LLM providers specially with regards to the requested format in the prompt.
• Running into rate limits boundaries of the provider.
• Model quality of output dropping or an outage on the provider side.
• Huge cost of tokens when using llms at scale.

My goal in this blog post is to give the builder a mind frame on how to build cool stuff with these systems in a reliable and scalable fashion.

Reliability​

When it comes to reliability, there are a few needs that a developer has to address right off the bat. First and foremost, is LLM providers’ rate limit boundaries. For this purpose, I used technologies such as LLM gateways where one needs to keep track of the token usage by a model and switch providers if the limits are within reach. I’ve used [Portkey] (https://portkey.ai/) in the past for this purpose. It’s open source and easy to use, you just need to pass in a config file and you are good to go.

The second important reliability issue, is the format of the output returned by the provider. Imagine your prompt asked for an LLM to return its output in json format, but on the 6th try, a string output is received. Fortunately for us, the providers have a field called response_type where one can pass the type of expected response in the header of the request. The format of this header varies based on specific providers. Please refer the documentation of the provider used for the specific format.

For providers unexpected outage purposes, there two solutions I’ve used in the past. The first solution is using default fallback cases with try catch phrases (i.e default object definitions), so your objects don’t become null suddenly and ur system doesn’t crash.

The second is having another provider as a backup in case one fails in your logic.

Saving Cost​

The fastest way to have cost saving, is enabling caching mechanism before the user input reaches the provider. If you have a cache hit, boom, you don’t need to pay the provider. This is particularly good when you are testing with the same data multiple times during development. That way you don’t have to pay for tokens while testing. Portkey also provides simple caching system on top of your LLM calls.

You can go nuts here and use a vector database as well. It’s a pretty cool system for building memory for agents and create a converging “history” systems. In the past I’ve used Pinecone for vector database purposes, however there are many other opensource alternatives that I would like to play around with as well. Trychroma is a notable one. If you haven’t worked with vector databases before, they are basically databases that store flat data in vector formats, so you can capture higher dimensional relationships between them. If you are interested read this document for deeper understanding of vector dbs: what is a vector database.

Another cost saving mechanism is fixing the output’s token count with a header variable you pass to the provider. Again each llm provider have their own variable for this and you would have to refer to their documentation. Only risk for token output fixing is getting a truncated output. Be aware of this and handle it with grace if you decide to implement token truncating.

Reaching for Better Quality​

We all know how ai can be slopish and at times very spikey. That’s why I write all these blogs by my hand with zero ai involved 😁. Personally I have used multiple layers for this purpose and put a supervisor llm at the end where it determines the quality of the previous layers’ output. You can also use more deterministic systems such as large data sets formed into a vector format and check the llm’s output at different stages against it to verify some sort of quality of response.

Final Thought​

I hope this article came helpful for developers to build a mind frame around building with these systems. Its really fun to play around with these systems as a developer. If you like to work with me, shoot me an email, or join our Discord Community of builders and build amazing things together. It’s an amazing time to be a builder and hope you all are as excited as I am.

When I first started building integrations to a huge system, I thought it would be a piece of cake. As with building systems in general, I realized they are not necessarily as easy as I thought they would be. There are issues that reveal themselves to you once you are deep into the process. Issues such as:

• Which system has the latest version of data? What is your tell?
• How do you synchronize data between the two systems to preserve the latest most correct data.
• How do you manage the data so every time you synchronize both systems, you don’t have to rewrite all the data. What are the boundaries of the system you are integrating to? What do they allow you to do and what do they not?

As with building systems go, once you work at a problem long enough, the answers reveal themselves to you. So I will be sharing some of my approaches to these problems. My goal is to set a mind frame on how to approach huge integrations from the beginning.

Before Begining the Integration​

When you start integrating into another major provider, I recommend spending some time in their documentation, understanding their apis and database design decisions. I understand AI is great at coding these days, and you can push stuff pretty far with it, but you are still the driver. It will save you time if you drive the machine right. Focus on understanding the boundaries of the external systems as well. What are some of the things that they don’t allow you to do. This is specifically important to understand early on. It holds you back from chasing dead ends and solving impossible problems.

Studying big players APIs is a great place to start when approaching to design a software in a specific vertical. For example, if you want to build an accounting software, I recommend digging deep into Xero platform’s api. This will show you abstractions that you might be missing in the beginning, in your DB design. It will also give you a very mature view of a system ahead of time.

What an integration is consistent of​

An integration is formed from three major things. A sync, a resync and export. Syncing takes the data from the external system to your system. Resync is how your system matches its data with the external system after the initial Sync. Finally, export is how your system sends data from your system to the external system. I recommend implementing in three separate stages (move vertically). Finishing Sync completely, enables you to make all your mistakes and not having to fix it in three different places. I say this because I tried doing sync, resync and export at the same time for each object (moving horizontally). This caused a lot of pain, because if a mistake was made about the third party’s object I had to go back and fix it in three different places.

Part 1: Sync​

I recommend starting with Sync, it allows you to deeply understand the third parties system and be prepared for next stages of the integration. Store the external objects id in the database, so you can find it later on, this is important. Recording time when a synchronization happens between two systems is important. At the end of the day, time can be your tell for what is the latest version of the data. Try to not complicate things, if something has already been exported to the third party, don’t allow the user to edit that field again on your platform, this avoids complications of finding who is the source of truth, and constantly syncing stuff.

Part 2: Export​

Then move to Export. Make sure you have a tell in your database of what object has already been exported to the third party platform. Another important factor about export is making sure you are not writing old data to the third party platform. This is where last sync time comes in handy. Also keep your updated_at field up to date when users change an object on your system, this will save you a lot of headaches later on.

Part 3: Resync​

Resync is a very important piece. Even tho when you get to resync you are tired from implementing the Sync and Export logic, don’t take it lightly. A method I chose for resync was called modified since. The third party applications usually accept a time variable in their header. It will return objects that have been modified after that specific time. In that way you can make sure the data you have on your application are synced with the latest data on the third party platform. Finally, make sure you cover all the CRUD cases, specially deleted objects on the third party. Sometimes the modified since doesn’t cover deleted objects on the third party causing you to face some weird problems when you are in production.

I recommend wrapping your Resync function in a cron job. The resync cron job that runs for example every day, makes sure your system and the third party system don’t diverge from each other too far. Its also not a bad idea to run your resync function before each export, just to make sure your data is not out dated between the last resync and export times.

Final Thought​

Integrations are a challenging problem, and are unique to each system. There will be weird stuff that will come up as you build the integration. My aim in this blog post was just to give you an overview and a thinking mind frame when approaching this problem. Maybe you can feed this blog post to your coding agent when you are starting to code a big integration. We are agent friendly, we don’t block coding agents from accessing our website.

Hi everyone!

I've been building an app for a client I can't mention the app yet because she hasn't released it yet, but essentially it's linktr.ee for people who host events. It's fairly common app so I figured it'd be cool to share my tech stack and a short blurb on why I chose the tool.

I wanted to share my mistakes to show you that building a SaaS can be messy, especially when you're still learning. These were short-term errors, but I quickly learned from each experience and became a better developer because of it. I hope this inspires you guys to go out there and build something!

Right now, the app brings in roughly $800 per month, with most of that money coming in as profit. I'm putting it on maintenance mode for now while I focus on my next venture: helping developers become founders.

This is my guide on the fastest way to get paid with Stripe and Flutter. I'll walk through step-by-step instructions on how I set up stripe integration, how I associate purchases (subscriptions & one-time payments) with users, and how I keep track of all of that within Supabase. Afterwards, I'll talk about the review process and how to tip-toe around the strict AppStore rules.

TLDR; Worked as a CTO of a YC-backed company but broke up with my co-founder due to a difference in values. Now I'm helping developers build their startups.

There seems to be a lot of confusion with Riverpod and the way it is used. Admittedly the documentation is lacking. And for someone getting started, there are many decisions to be made like:

• Should I use code-generation?
• How many providers should I create?
• What should be contained in each provider?

Because of this adaptability, it can become very confusing for someone just getting started. I'm creating this blog post to lay some ground rules that I set for myself when using riverpod. If you're getting started with riverpod, following these rules will be a good starting point.

For the past 2 years I've been building startups and for my most recent startup I was able to reach 50k users in under 6 months. Throughout my journey, I've been consuming startup content like a fiend. These are the resources that helped me get to where I am today.

Flutter for web has evolved significantly in the past few years and in this post I wanted to give a comprehensive comparison between using Flutter vs React for developing web apps specifically. I've used both Flutter and React for startups so I have a good sense of both.

Anyways, the most important thing in startups is iteration speed. The ability to quickly build a product, get customer feedback, and iterate is the thing that sets apart the good startups and the dead startups. Now in my opinion, a good framework (for startups), is one that enables you to iterate as fast as possible. With that knowledge, let's dive into why I think Flutter wins in almost all aspects.

Hey, I'm Matt, once the CTO of a startup backed by YC. Today, I'm here to unravel the steps I took to reach 50,000 users and how you can replicate this success!