# It Depends > Kislay's personal blog and newsletter Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### About me URL: https://kislayverma.com/about/ Last updated: 2026-07-23T13:57:43.000Z For most of my 24 hours, I'm the co-founder and CTO at [Shoffr](https://shoffr.in/?ref=kislayverma.com). When I can sneak in some time, I vibe-code fun stuff like [Back Again](https://backagain.in/?ref=kislayverma.com) (a retro game arcade), [The Museum of Everything](https://themuseumofeverything.in/?ref=kislayverma.com) (explore world history and any other topic, museum style), and [Altmove](https://altmove.in/?ref=kislayverma.com) (a chess learning system). For a while, I published a weekly [newsletter](https://kislayverma.com/tag/it-depends/) and podcast ([Spotify](https://open.spotify.com/show/5gY1eGUE0RmNHj5t1HKpJI?ref=kislayverma.com), [Apple Podcasts](https://podcasts.apple.com/us/podcast/it-depends/id1571969334?ref=kislayverma.com)) called *It Depends*. I'd like to restart that, but we shall see. [![](https://kislayverma.com/content/images/2026/07/Screenshot-2026-07-23-at-7.25.41---PM.png)](https://themuseumofeverything.in/?ref=kislayverma.com) The Museum of Everything: Explore history, and everything else [![](https://kislayverma.com/content/images/2026/07/Screenshot-2026-07-23-at-7.21.57---PM.png)](https://backagain.in/?ref=kislayverma.com) Back Again: A retro game arcade ![](https://kislayverma.com/content/images/2026/07/Screenshot-2026-07-23-at-7.27.02---PM.png) Altmove: Learn chess ## Posts ### Launch: Shoffr MCP server URL: https://kislayverma.com/launch-shoffr-mcp-server/ Last updated: 2026-08-22T13:51:32.000Z For those folllowing along, I'd been working mostly on operations tech in [Shoffr](https://shoffr.in/?ref=kislayverma.com) for the last few months. But as most of it went into ground implementation and monitoring phases, I took a little side quest to build an [MCP server](https://modelcontextprotocol.io/docs/2026-07-28/learn/server-concepts?ref=kislayverma.com) for agents to be able to book a Shoffr and manage the ride. This is now live and was so much fun and out of my usual zone of familiarity that I am compelled to write on the Shoffr blog after years. I offered the project completely open ended to Fable 5 with just the core spec - build an MCP server to allow people to deputize their agents to make bookings for themselves with the ability to reschedule and cancel also via the agent. As usual, it came back with an overblown plan with all kinds of bells and whistles which I didn't cvare for in an MVP. After a lot of negotiation and over ruling and "move this to phase 2", the [core architecture](https://kislayverma.com/shoffr-system-architecture/) I settled on was to build a new spring boot service to encapsulate the "agentic" complexity which will then interact with the core service stack over REST APIs to achieve the end to end functionality. Let's see it in action first. I've tested this primarily with Claude, but I imagine similar things should happen with other tools as well. 1. Go to Settings -> Connectors -> Add custom connector. 2. Name the connector "Shoffr" (or whatever, doesn't matter) and configure the MCP URL - [https://mcp.shoffr.in/mcp](https://mcp.shoffr.in/mcp?ref=kislayverma.com). Click Add. 3. You should see this. Click on Connect. 4. This will take you to the Shoffr MCP auth page. Enter your phone number (with country code) to receive OTP. 5. Enter the OTP if signing in, enter your name as well if signing up. 6. Provide consent for using the OAuth scopes. 7. Done! Now you can configure tool permmissions to control how much agency you want to give your agent. For my fellow CLI-ers: 1. To add Shoffr MCP server to claude, run:claude mcp add --transport http --scope user shoffr [https://mcp.shoffr.in/mcp](https://mcp.shoffr.in/mcp?ref=kislayverma.com) 2. Now run: claude 3. Then signin: /mcp - this will give a list of installed MCP servers. Select shoffr and follow along with the instructions. All set up. Now booking a Shoffr is as simple as saying "book a city ride for me from Shoffr HQ to the HSR club for tomorrow 6pm". The agent will load the Shoffr tools, get a quote for you, make a booking pending payment, and fetch you a payment link. Once you make the payment, the ride is confirmed and all is well. ![](https://kislayverma.com/content/images/2026/08/Screenshot-2026-08-20-at-1.26.00---PM.png) Ask your agent to get a Shoffr price quote ![](https://kislayverma.com/content/images/2026/08/Screenshot-2026-08-22-at-7.15.33---PM.png) Once confirmed, your agent creates a Shoffr booking pending payment ![](https://kislayverma.com/content/images/2026/08/Screenshot-2026-08-22-at-7.15.40---PM.png) The booking gets confirmed post payment The manual payment part is the one that really bugs me. Solutions around UPI auto-debit, limited mandate etc are upcoming and that is the direction we are looking to push towards. For now, this MVP just wraps the core functionality and makes it agentically accessible. ## Engineering the MCP Server The two main things to build were auth and, interestingly enough, geocoding! ### Auth In Shoffr we use token auth everywhere but the agentically preferred way is OAuth. I didn't want to complicate the core mechanisms for what is just an MVP for now. So how to bridge the gap? We added the OTP mechanism as a provider for the OAuth layer. The agent triggers the OAuth flow which internally takes it to a place where it can enter the guest's phone number, the guest uses the OTP to signin, and the MCP service then traslates this to a valid OAuth response. While this is not ideal, it triggers rarely (once on signup, and then when token expires in many months). The MCP service maintains the client list, authorization, consent, signing keys etc in a separate store so that all the OAuth complexity is abstracted away from the core services. In the core service, we introduced a few new endpoint and agent specific workflows which are completely isolated from the rest of the code and serve primarily as a translation layer between the edge of the stack and the internal core modules. ### Geocoding The trip booking experience everywhere on Shoffr is driven by Google autocomplete. With agents, though, it has to be a lot more NLP friendly. Guests can ask their agent to book a drop to the airport from their home (their agent presumably knows this - HSR Layout, Salarpuria Cadenza, etc) to terminal 2\. We now have to figure out the exact map locations from these ambiguous descriptions. What happens if there are two apartments with same name in different cities () So the MCP server exposes a geocoding capability to discover service areas and resolve location using which the agent can disambiguate what it wants. Obviously, the greater context the agent has, the faster and easier this becomes. The geocoding ability is built as part of our core geo platform and gets exposed to the agents via specific API calls. Call it oversight on my part, but building geocoding was not top of mind when I thought about this. but, we added a core platform ability cleanly in the process of building a business thing - always the best way to do it. --- So there's that. A fun little side project that took about 2 days and opened up an interesting booking stream for the future. I will be pushing on the payments and deeper integrations (add pets, tolls etc - basically everything you can do as a human should be possible for the agent) over the next some weeks. ### Tech at Shoffr URL: https://kislayverma.com/tech-at-shoffr/ Last updated: 2026-08-22T13:03:38.000Z Originally published on the [Shoffr blog](https://shoffr.in/blog/tech-at-shoffr?ref=kislayverma.com). Vikas and I love Bob's Bar in Indiranagar. Quiet enough to talk, and buzzy enough to have fun while talking, it is a great place to discuss "how would you do this" with friends. So it was that after I decided to join [Shoffr](https://shoffr.in/?ref=kislayverma.com) as the tech guy, we sat there one evening and talked about how we would build the technology that powers Shoffr. That technology would power it was not in doubt. Today all companies are software companies. Much of any company's customer experience is delivered via technology touch-points like apps. Everyone has technology running to optimize and scale operations. But what specific principles would guide us in building our tech stack? So we went back and forth over it and came up with some guidelines. - The technology should be super flexible so that we can quickly adapt it as we grow. Especially in the early days, this is critical since no one knows how things will really pan out. - Buy rather than build as far as possible. I knew that in the beginning, most of our tech problems would be mundane. Since many of the problems are mundane, we will try to buy as much off-the-shelf software as we can. This gives us time to solve the problems that are core to us instead of building everything by hand. - Use as boring/familiar a tech stack as possible. This would mean that everyone on the team would be well-versed with the tech stack and so more productive, and help would be easily available online should we falter somewhere. - Hire *extremely* reluctantly. Overlarge teams are less productive than small teams working with familiar tools. - A side effect of hiring reluctantly is that we only build the most important things at any point. The meat of our operations is the car and the driver and not two dozen features on the customer app. The idea was to slow down, hear from customers and operations, and then solve the most pressing problems. These ideas have played out in interesting ways over the last few months. - There were off-the-shelf tools available for end-to-end adoption. But we had two reservations about them. - The no-code ones were too difficult to customize as needed. I tried a couple and found that I was spending more time figuring out the tool than building what I had to. - We wanted to be free to evolve our customer touchpoints regardless of how we ran our backend operations. Most of the tools out there were all or nothing (or too expensive). - So we decided that we will build the external interfaces and get them to talk to our own backend. This backend will then integrate against whatever other tools are required. This is a fairly standard design but listening to all the no-code hoopla, I had hoped to do far less by hand than this. - At this early stage, an app also didn't seem justified — even our most power-users wouldn't use it more than 1–2 times *per week*, given the nature of airport travel. A minimalistic + responsive website would serve the purpose, and take much less resources. - Since I was the only developer in the beginning, I started building the backend and the website in Spring Boot + Thymeleaf + MySQL which is both boring and what I am most familiar with. - Then we got Saurabh Thakur to work with us for some months and he was disgusted by the server-rendered/client-rendered hodge-podge I had made of the frontend. So he convinced me that we should write the website in Next.js. I managed to convince him that we will not put the typical backend-of-frontend Node.js layer between React and the core backend. So today the customer website is rendered in React/Next and calls the backend directly. - The backend integrates a bunch of other tools like Gupshup (for SMSes), Amazon (for S3), Google (for maps and sheets APIs), Slack (for trip notification messages to the operations team), and Paytm (customer payments), and Zoho integration is ongoing. We have been able to quickly tweak all these integrations multiple times in the last month alone. - I was very worried about changing requirements when I started writing the backend, so I specifically chose to separate the [workflows](https://kislayverma.com/architecture-pattern-orchestration-via-workflows/) from core APIs to maintain the flexibility of the code. I will share more low-level details on this in a follow-up post, but this has paid out handsomely so far. More on the tech-product side of Shoffr in upcoming posts. ### Altmove: Learn to play Chess URL: https://kislayverma.com/altmove-learn-to-play-chess/ Last updated: 2026-07-29T01:31:39.000Z I am addicted to playing [chess](https://themuseumofeverything.in/explore?ask=chess&ref=kislayverma.com) on chess.com. Part of it is their streak system (I have a 193 day streak at this time), and the other part of it is a kind of sunk cost fallacy - I haven't been getting better but still keep playing in the blind hope that I will get better automatically. But the rational part of me knows that nothing happens unless you do it intentionally. So some time ago, I built [Altmove](https://altmove.in/?ref=kislayverma.com) to learn chess in a structured manner. My only learning tool till then was the game review functionality in chess.com. However, for me, a post-facto review of moves and mistakes isn't very intuitive. It is too late, too detached, too I-already-lost-and-I-don't-care. My ideal learning tool was a real time tutor which highlights my mistakes as I make them and points me to a better move. Choosing this new move now splits this game into a new parallel universe where I made the better move. I can then see what happens from here on out. The game itself evolves with real time tips. So that's the USP of Altmove. In-game coaching (identifies mistakes as you make them) and the possibility of pursuing an alternate, better move (hence the name). When you make a mistake, the game shows you a better move and allows you to branch into a new board with that move so that you can explore the what-ifs. A new player's jouney has 6 steps of progression from "Newcomer" to "Expert" tied to their ELO scores. Progression to a higher level is based on two criteria: 1. Minimum number of games 2. Consistent ELO score above the current level's cutoff in those games Both these are aimed at preventing promotion based on a lucky streak. You have to play better consistently to get promoted. However, once promoted, you won't get demoted (since that would demotivate players). This is to bring in rigour and milestones to what can otherwise seem like an endless journey of seeking mastery. --- Altmove is built with AI but does not use an AI engine for gameplay. Chess is a deterministic thing - I don't want the AI playing random moves or giving incorrect advice. The computer uses the [Stockfish chess engine](https://stockfishchess.org/?ref=kislayverma.com) at different levels of difficulty (depending upon the player's level) to train you. This gives a far more crisp, precise experience than an LLM could give. And it keeps the system free to run. Even so, I have added an option to drop in your own Claude API key to make the interaction with the computer more chatty and human-like. If you add your key, each move is analyzed both by stockfish as well as the LLM (which will add cost on the key and makes the system a little bit slower). The choice of good/bad move is still determined by Stockfish, but the LLM gives much more humanlike feedback on the move. It's more fun, but at the added expense of the LLM. The API key is encrypted and stored on the backend so that it can be used if you play from multiple devices. --- The intent for building Altmove was never to build a rival for chess.com or lichess. These are superb platforms for playing Chess. Altmove focusses on learning because that is where I felt these platform were not focussing much. I intend it to be the FIIT-JEE to their IIT entrance exam. To that end, you can import your chess.com or lichess games into Altmove. Altmove will analyze them, show you your mistakes, and offer an alternate move in that place. More fun is the ability to replay (restart) that game from any move, and now with the beenefit of hindsight, play it out with new moves against Altmove as a sort of alternate history to see, i.e. what could have happened if that one move was different. --- Altmove is entirely vibe-coded using Claude Opus 4.5 for the most part. I have seen very little of its code, though I have played it enough myself and reviewed things enough to be satisfied with latency and bugs. Thats said, drop a comment if you find an issue. For most of my time on chess.com, I was stuck at a rapid score of \~600\. I have been playing both altmove and chess.com for the last months or so now. Still playing casually and indifferently, but my score has now lurks around 750\. Most of this because the interrupt-me style experience of Altmove has made more conscious of my usual mistakes like hanging pieces and standard checkmate gotchas. Check it out and let me know what you think. I'd love to hear more about how I can it better as a learning tool. ### Launch: Shoffr Android App URL: https://kislayverma.com/launch-shoffr-android-app/ Last updated: 2026-08-22T13:35:00.000Z We are excited to announce the launch of our Android app on the Google Play Store. Booking and tracking your [Shoffr](https://shoffr.in/?ref=kislayverma.com) rides just became even more convenient! ### Why? Apps are pretty much a standard these days, so the why is kind of obvious. But beyond that, we had one very specific reason to build an app — trip notifications for guests. We have written here before of the anxiety guests have about cars being on time etc. We send SMS and email notifications containing driver details. But occasionally, we have to reassign the driver for a trip to keep it on track. We send messages for this too. But guests often get confused between the original message and the new one and call the old driver who is doing something else now. This creates a lot of tension. We wanted our guests to have a convenient way of accessing this type of information. People have a much stronger affinity towards opening an app versus opening a website. So our new plan is to send push notifications for all trip updates which will take guests to the [upcoming trip widget](https://kislayverma.com/launch-upcoming-ride-widget/), which always has the latest information about driver, pick-up time etc. #### Why Android first? Our customer base is slightly skewed towards iOS users. So why not build the iOS app first? For the somewhat embarrassing reason that Apple doesn't allow testing push notifications in a simulator and I didn't have a test phone at that time. This has since been remedied and we should have an iOS app very soon. ### How? Given that Shoffr is a one-man engineering shop, we thought hard about what the best way would be to build and evolve an app. We wanted to keep the app and website as close to each other as possible, while also giving the little native experiences that are only possible via apps. Eventually, we decided that we will wrap the website in a webview and that will be the heart of the app. Then [Param Aggarwal](https://www.linkedin.com/in/paramaggarwal/?ref=kislayverma.com) gave a genius idea. We already had a driver app which was quite lightweight and did pretty much the same thing (webview in app shell) for our drivers. He suggested that instead of building a new app, we repurpose the same one to load different things based on the user's roles and access. This would reduce two new "properties" (1 Android and 1 iOS) that we would have to maintain otherwise. We made it so, and this is why if you notice the package name of our app it is `com.shoffrDriverApp`. We have used [React Native](https://reactnative.dev/?ref=kislayverma.com) to build the app because it had a shorter learning curve and the app is not heavy enough to merit extreme performance considerations. On top of the webview wrapper, we started building the loading and the push notification integrations (using [Firebase Cloud Messaging](https://firebase.google.com/docs/cloud-messaging?ref=kislayverma.com)). Many `npm install`s and `run android`s later — here we are! ### Evolution We expect that the app will slowly become the main way people interact with Shoffr. This should add more traction to features like [live car tracking](https://kislayverma.com/launch-live-car-tracking/) and the upcoming trip widget. In terms of new features, the sky is the limit, of course. We have a long list of features that we want to bring to the website (and consequently to the app now), but there are quite a few things we want to do with the app alone (e.g. rich, sticky notifications on the home page tracking the trip). If you are an Android user, go give the app a whirl and let us know what we can do to make it more delightful. Bug/annoyance/WTF reports are most welcome. --- Originally posted on the [Shoffr blog](https://shoffr.in/blog/launch-shoffr-android-app?ref=kislayverma.com). ### Launch: Live Car Tracking URL: https://kislayverma.com/launch-live-car-tracking/ Last updated: 2026-08-22T13:32:16.000Z ### Why? One of the biggest anxieties in booking a trip is whether the car will come in time or not. An interesting incident a few months ago was when a guest booked a trip with us several days in advance, then panicked and cancelled because "there was no communication" from our side. That was our first lesson in non-transactional customer communication — we started sending an "You have an upcoming trip with Shoffr" kind of message 1, 2, and 3 days before the trip to tell the guests that we hadn't forgotten about it. However, this anxiety is at fever pitch close to the pick-up time. We get a lot of calls and WhatsApp pings just enquiring where their Shoffr is and when it will reach. Our problem was keeping customers informed that their vehicles are on time (or a little late on the rare occasion). This makes for a happier customer, and reduced call volumes for us to handle. ### How? The accepted solution to this problem is showing vehicle tracking. And while this is ubiquitous, it is neither easy nor cheap to implement. So while we wanted to implement it, we also wanted to keep our pockets burn-free. #### OSS to the rescue! [OpenStreetMap (OSM)](https://www.openstreetmap.org/?ref=kislayverma.com) is a project which maintains map information via volunteer contributions and makes it publicly available via publicly hosted APIs. [Leaflet.js](https://leafletjs.com/?ref=kislayverma.com) (and its React.js wrapper [react-leaflet](https://react-leaflet.js.org/?ref=kislayverma.com)) is a handy map rendering library. With these tools, we went hand-rolling our own little tracker. We decided to merge the GPS information from our cars with data from OSM to show live tracking to our customers. And while showing a car on a map was the big guest-facing feature, there was a bunch of stuff the backend had to do before this became feasible. #### Getting the data While we have used telematics and GPS data for operational monitoring since the beginning of Shoffr, this would be the first use of this data in powering a customer experience feature. This caused a few changes in the way we pulled this information from [Intellicar](https://intellicar.in/?ref=kislayverma.com) (our telematics provider). We already had a polling setup which ran every 10 minutes to pull the location of each car into our system. But to show a noticeable tracking experience, this data had to be pulled in much more frequently. We could simply increase polling frequency, but that would put too much pressure on the Intellicar API as well as our database. So we worked with Intellicar to implement a push mechanism where they would push GPS info to our system every 10 seconds. This removed the polling pressure. To reduce the pressure on our database, we implemented a mechanism where the latest pushed location would always be put in a cache to be read, and we [pull and store it into our database every 5 minutes](https://kislayverma.com/architecture-patterns-caching-part-1/). While this means that we do not have the values inside that 5 minute window stored in our transactional database, that's fine since we store it all in our data lake anyway. #### Rendering the map Once the data was in place, we went ahead with merging it with map data from a publicly available OSM API and rendering it via react-leaflet. To avoid overloading this free infra (free is great, but we wanted to be responsible about using it), we decided to limit how and when live tracking was shown. We would show it starting 30 mins prior to the trip start time (when the car should really be close by), and stop showing it once the trip starts. Leaflet.js works directly with the DOM and doesn't support server side rendering so a few tweaks were needed to make it work with our Next.js setup. *Debugging tip — if you do everything right with the library but do not explicitly specify the size of the `MapContainer`, the map WILL NOT render. Hope it helps — I lost several hours to this.* ### Impact We have been guiding our guests to this feature and the feedback has been great so far. We realized that there's no such thing as "too many ways" when it comes to relieving our guests of the fundamental fear of getting picked up on time. Additionally, we also learnt that a huge hurdle to adoption of tracking is guests not opening the website repeatedly in the first place. Opening websites just isn't the default user behaviour anymore. We hope to address this with our upcoming app launch. ### Evolving While the limited tracking we show to our guests today is useful, we want to build a better and more comprehensive tracking experience, not just for the traveller but perhaps also for their parent, friend etc. However, it is clear that it cannot be built at scale on public infrastructure. So we need to evaluate our choices in terms of commercial operators or explore self-hosting OSM data. --- Originally posted on the [Shoffr blog](https://shoffr.in/blog/launch-live-car-tracking?ref=kislayverma.com). ### Guest profile at Shoffr URL: https://kislayverma.com/guest-profile-at-shoffr/ Last updated: 2026-08-22T13:28:13.000Z If you have booked a few rides with [Shoffr](https://shoffr.in/?ref=kislayverma.com) and come back for the next, you might notice that the "Use Toll" checkbox on the checkout pop-up is already set to your favourite setting. This is enabled by the user profile module we launched last week. ### Why? The "Use Toll" option in our booking flow has had an interesting life. It started out as a way to reduce customer queries about whether or not we will take the road — the guests could choose themselves and allow us to price the trip transparently. We built it like an automation feature but it turned out to be a delight instead — people love the fact that they can choose whether or not they pay the extra toll amount. The obvious next step was to "remember" which guests like to use the toll and which not, and to personalize their checkout experience. Generalizing this thought process led us to the construct of a user profile where we can store what we know or can derive about a guest. The user profile can then serve as a central knowledge bank about our guests and a springboard for all kinds of personalization. ### How? The initial design for the user profile was pretty simple. We would model the profile as an internal sub-domain of the User domain, calculate user profile attributes as part of various user workflows and expose the profile via an API. However, we soon realized that treating profile as an internal component of the user domain would limit its power. User is, after all, a central entity and many other domains can derive information about a user. e.g. The payment system can determine a user's favourite payment method. So we decided to apply [The Golden Rule of Platforms](https://kislayverma.com/the-golden-rule-of-platforms/) to this and split it into two parts. The core profile module would simply allow CRUD over profile attributes — it would not define what the allowed attributes are or understand what they mean. That meant no enums or static definitions of any kind. External modules like user or payments or trip would define their own attributes and manage them. The profile attribute thus remains a part of the user domain, but now exposes its capabilities for user and other domains to use. At implementation level, this was fairly simple. Just two new APIs — `getProfile(userId)` and `saveProfileAttribute(string, string)` — and a table to store profile attributes. Note how the string parameters in `saveProfileAttribute` allow us to put in "whatever". With the profile platform in place, we moved to the product engineering bit and wrote the plumbing to populate the toll preference. Every time a customer makes a booking, we trigger an asynchronous flow to see whether the majority of the user's bookings in the last 60 days use toll or not — and populate the preference accordingly. We also wrote a job to backfill this data for all customers. Now it was a simple matter to load the user profile on the checkout screen and populate the toll checkbox according to the value stored in it. ### Rollout The rollout happened piecemeal in the order of execution described above. The backend pieces were all completely decoupled so could be released without any need for a feature gate. The website change was rolled out with a feature gate because "what if", but the gate was quickly enabled without any problems. ### Evolving A more evolved version of the profile attributes would make external modules "define" the data type and other parameters of the attribute they want to populate and apply some validation using this definition. However, we are holding on to this thought to see how many use-cases truly emerge to require this. The algorithm for determining the toll preference can also be evolved to be more weighted towards recency, or to determine two preferences — one for business travel and another for personal travel. Again, we felt it would be overkill to do this now. --- Originally posted on the [Shoffr blog](https://shoffr.in/blog/guest-profile-at-shoffr?ref=kislayverma.com). ### The Life of a Shoffr Trip URL: https://kislayverma.com/the-life-of-a-shoffr-trip/ Last updated: 2026-08-22T13:24:43.000Z The "trip" aka booking, or ride, or duty (as our Shoffrs call it) is the fundamental construct at Shoffr. Everything here is built to deliver the perfect trip experience for our guests. But how does a trip actually work? Here's the under-the-hood view. A trip, like most other entities, has a life-cycle. ### Booking the trip The build-up towards a trip starts when a potential guest comes to our website and enters their pick-up and drop locations. At Shoffr, one of these is always the Bangalore airport, the other could be anywhere in Bangalore city. We take these two points and calculate the distance and time (as predicted by Google APIs) this trip will take. This is then used to calculate the fare and show the checkout pop-up on the website (where guests can choose the use of toll, enter their details etc). Once the customer confirms the booking by entering the booking OTP, the booking creation workflow takes over. The early stages of a trip are closely linked to the order entity ([here's how that happened](https://kislayverma.com/launch-book-return-trip-along-with-onward-trip/)). First we create an order and an order item (each of which corresponds to a trip), and then create a trip — all in PAYMENT\_PENDING status. This means that a booking request has been initiated, but payment has not yet been received. This is pretty standard design for order management systems. The intent of this design is to confirm the order when payment confirmation is received from some payment gateway for online payments. While we built the system with this in mind, Shoffr does not have online payments today. So as part of the same "book" action, we trigger the confirm order workflow as well. This marks the order and order item as CONFIRMED. The effect on the trip is much more interesting. "Confirmed" has a different meaning for a trip than it has for an order. Once an order is confirmed, we move the trip(s) underlying it to CONFIRMATION\_PENDING state. ### Confirming the trip Now we run our confirmation engine. While a lot of Shoffr tech is "simple", the confirmation engine is one of the few "secret sauce" pieces which play a critical role in the business. The engine takes all the bookings we have at that time and tries to evaluate if we can serve the new trip without impacting the others. To do this, it marries our shift and car data, geo-spatial information from Google, and some statistical information from trip behaviour in the past (e.g. median delay in starting airport pick-up due to luggage coming early/late). Essentially, we are kind of solving the Travelling Salesman problem with "n" salesmen (number of cars) and real world unpredictability. The engine can say one of two things: confirm / don't confirm. If it says the former, we move the trip to BOOKED state and show a success screen to the guest. If the latter, we keep the trip in CONFIRMATION\_PENDING state and tell the guest that we will confirm the trip in 45 minutes. Behind the scenes, this creates a task for our Ops team to review and they take a judgement call on whether to accept the trip or not. The original idea was that there should be no manual process involved. But we soon realized that the input to the confirmation engine was not accurate (incl. Google Maps ETAs) especially during peak traffic hours, rains etc, and so while we could judge some trips conservatively, we could not judge all trips precisely. Hence this manual process was adopted. The ops team can now do one of two things: approve the task, which then moves the trip to BOOKED state, or decline the task which marks the trip DECLINED. The latter is a terminal state much like CANCELLED (which we use for cancellation post confirmation). All of these transitions trigger appropriate comms (SMS/Email/Slack) to our guests and internal teams. ### Serving the trip Now we move to the operational parts of the trip life cycle. Our allocation engine picks up trips 2–3 hours before their pick-up time and tries to allocate them to the best possible driver. Like the confirmation engine, the allocation engine is a key part of our tech. It is meant to optimize for multiple variables like vehicle arrival time, vehicle travel time, and vehicle utilization simultaneously. In a way, it is a stricter, more real-time version of the confirmation engine, because not only does it determine if a vehicle can go to do a given trip based on more data like telematics, it also determines the best suited vehicle for it. - Once a vehicle is allocated to a trip, it moves to the ALLOCATED state and the allocated Shoffr can see it on his app. The next set of transitions are driven by the driver. - When the Shoffr starts towards the guest's pick-up location, he marks the trip ON\_THE\_WAY on his app. - When he reaches the pick-up location, he marks it REACHED. - When the guest has got into the car and is en route to the destination, the Shoffr marks the trip as STARTED. - When the guest has been dropped, the Shoffr marks the trip as COMPLETED. ![](https://kislayverma.com/content/images/2026/08/driver-app.png) ### Evolution The trip is such a central entity that its evolution is guided by pretty much everything else happening in the business. That makes the trip domain very fast moving and a lot of fun. There are a few parts of the trip life-cycle that we are actively working on developing further. The first is the communications aspect. We want to share a lot more information with our guests about their designated Shoffr which is context-sensitive and not spammy. This involves some geo-spatial intelligence and driving a lot of driver rigour on the ground. Another is extending the trip life-cycle beyond servicing the guest and into the financial settlement world. We want to manage the financial settlement and reconciliation of the trip using trip states instead of doing it offline or elsewhere. --- Originally posted on the [Shoffr blog](https://shoffr.in/blog/the-life-of-a-shoffr-trip?ref=kislayverma.com). ### Shoffr System Architecture URL: https://kislayverma.com/shoffr-system-architecture/ Last updated: 2026-08-22T13:20:37.000Z Early architecture is a critical part of an early-stage, tech-enabled company. It has to enable many, often conflicting things. Agility of feature delivery, stability under rapid growth, easy evolution towards higher scalability, are all critical and the early design choices should power them all. Earlier we wrote about the [tech principles we apply at Shoffr](https://kislayverma.com/tech-at-shoffr/) to build our systems. Today we want to showcase the high level architecture to which these technical requirements and our principles have led us. Behold the Shoffr monolith in the image above! ### Why monolith? Distributed systems are a pain to operate, and we would rather not deal with that complexity unless we must. Our monolith is fairly well-designed and modular (even if I say so myself), and has enabled quick iterations. Also, Shoffr's scale is not enough at the moment to justify a separation of service for reason of differential scale. So a majestic monolith our system will remain for the foreseeable future. ### External touchpoints We decided that we will build all touch-points where we want a differentiated experience. These would be ALL guest touch-points and some internal ones. The blue boxes on the left side indicate the experiences we have chosen to control — the website for our guests (and mobile apps etc in the future), the driver app for our drivers to manage their work, and the internal admin portal for the operations team. We chose to outsource some capabilities where good/affordable tools already exist and we have no need to re-invent the wheel. These are the blue boxes on the right side. These are the many external tools we use to run the business — Google Maps for calculating fares, Zoho for invoicing and accounting, Gupshup for sending SMSes, Slack for internal comms etc. Communication between all these touch-points flows via our backend. ### Edges of the backend The backend is a 3-edged system — incoming traffic, outgoing traffic, and database interaction. Incoming traffic from all user facing properties enters via auth-protected REST layers. Each end product like website or admin UI has a custom set of end-points (to create modularity if we ever have to split them into separate pieces). So while the logical architecture above has backend-of-frontend like boxes, in reality these are just Java packages in the codebase. The second edge is where we talk to our database. We use MySQL as our datastore and do some amount of in-mem caching. Again, we did not want to complicate things by adding external caches like Redis etc and our data fits in quite comfortably. The third edge is all the external integrations that we have. All of these integrations are routed via respective "gateway" packages so that it is easy to find what functionality of these tools is being used and how. These "gateways" also help localise any changes to these integrations. ### Domain / core services These services are the bottom of our abstraction pyramid. They encapsulate the core domain models and business logic/invariants pertaining to them. Each of them manages one (logical) entity and exposes APIs for other parts of the system to interact with that entity. We decided that domain services will never know about each other. This helps us avoid the typical mess of every service calling every other service and maintains decoupling between them. Any orchestration between them is done externally. Since integrations are essentially libraries, any service is allowed to use them, though we try to avoid it. e.g. User Service uses Zoho integrations to maintain user info in both places. ### Workflows Most high level actions (e.g. book a trip, create an account) are made up of steps across multiple core entities. Since these entities cannot talk to each other, we use [workflows to model high level actions](https://kislayverma.com/architecture-pattern-orchestration-via-workflows/) and stitch together actions across multiple domain services. Additionally, similar actions taken from different sources may involve a slightly different set of steps. e.g. A guest cancellation triggers a different SMS than a Shoffr-initiated cancellation. So we have separate workflow components for our website, our admin panel, our driver app, and so on. In cases where behaviour is the same, they all converge onto the same internal implementation. Adopting the isolated domain service orchestrated via workflows paradigm has allowed us to be extremely flexible in how our system behaves. We can trivially move around and change individual workflows to build new behaviours and features, and since each workflow typically caters to a single high level action, any errors are unlikely to cascade and testing is very easy. All together, we can iterate very fast. ### Technology choices We decided that we will use the tools which are suitable, and among all the suitable ones we will prefer the tools we are most familiar with. The backend is built in Spring Boot, the customer website is built in React.js, and the driver app is built in React Native — all choices made using the above criteria. The Admin UI is built using Thymeleaf and is bundled along with the backend — a somewhat unusual choice in today's world of client side frameworks. There are two reasons for this. We expected that we will be showing/processing lots of data on it and so wanted it to be rendered completely on server-side for efficiency. While the current version isn't entirely server rendered, most of it is and it performs quite well — so that perspective has played out nicely so far. The second reason was simply that I knew how to use Thymeleaf but not React. So even the first version of the website was built in Thymeleaf in the semi-SSR style. We moved to React on the website somewhat later. --- Originally published on the [Shoffr blog](https://shoffr.in/blog/shoffr-system-architecture?ref=kislayverma.com). ### State of Shoffr Tech: Feb '23 URL: https://kislayverma.com/state-of-shoffr-tech-feb-23/ Last updated: 2026-08-22T13:16:41.000Z The last 7 days have been absolutely insane at Shoffr Tech. We shipped out a huge amount of stuff, some of which will make our guest experience better and the rest will make our operations more efficient. I will share details on most of these individually — but here's the high level recap of everything that got done. - **Website Redesign** — Our website went through a complete redesign to better reflect our brand aesthetic. This had been in the works for a long time, and both Vikas and I spent a lot of time to get this right. It isn't perfect — but it still looks gorgeous. - **Trip Edit** — Guests can now edit their booking on the website. So far this was only possible via our customer support — inconvenient for customers and time-consuming for us. - **Invoices on the website** — Invoices for all trips are now available for download on the website. Getting invoices for reimbursement is a key problem for many of our loyal guests, and again this was done over phone and email by our customer support. The full story of this involves a Zoho integration and driver app/SOP changes, all of which manifests in a simple "invoice" link for our guests. - **Fixed pricing for cusp locations** — We set our prices by distance buckets. For places at the edges of these ranges, Google Maps can sometimes show different routes leading to different prices. We identified such places among our most frequent guest locations and made sure the price remains stable going forward. - **Payment method capture** — Drivers can record the payment method used by the customer (Cash/UPI) in their app now. This helps in payment reconciliation later on. All this and a bunch of smaller stuff, bug fixes, and so on. This marks a nice little milestone in our tech-product journey. Actually, I think of this as the second milestone. The first was in early December when we launched our website. Shoffr was a 2-car shop running exclusively on phone and Excel sheets till then, and guests were just beginning to love us. The first version of the website and the internal operations portal provided the preliminary customer experience like booking and messaging that everyone takes for granted these days. But the way people took to it showed us that Shoffr was headed the right way. Two months of intense work from mid-Dec to mid-Feb have brought us from that formative stage to what I think of as the foundational stage. Our customer experience is stable and "sufficient", and our front-line operators (drivers and managers) have the tools they need to manage the foreseeable scale efficiently. Of course, everything has plenty of room for improvement — a customer app, more messaging options for customers, better driver allocation automation for ops come to mind readily. But the foundation is laid. Over the coming weeks, tech takes a bit of a back-seat as Vikas and I will spend time focusing on growing both supply (via funding and partnerships) and demand (via promotions, tie-ups and whatever else works). The next phase of tech work will focus on scaling and refining the customer and ops experience. More on it as we get to it. --- Originally published on the [Shoffr blog](https://shoffr.in/blog/state-of-shoffr-tech-feb-23?ref=kislayverma.com). ### Launch: Upcoming ride widget URL: https://kislayverma.com/launch-upcoming-ride-widget/ Last updated: 2026-08-22T13:13:38.000Z Last week we shipped an upcoming rides widget on the [Shoffr website](https://shoffr.in/?ref=kislayverma.com). Guests who are logged in can now see the details of their upcoming ride the first thing on the home page. With driver details etc covered there, they do not have to rely only on our SMS updates for this information. ### Why The main way we communicate with our guests today is via SMSes. We send SMSes for login, booking, cancellation, driver allocation etc etc. However, SMSes are not always reliable or timely. We have had a few guests reach out to us on the phone to resolve various situations they were in (e.g. unable to book because the OTP isn't reaching, the message with driver details in it didn't reach them etc). One way to address this is to add more modes of communication like email, WhatsApp etc. We are working on this. But it also pointed us to a big gap on our website — it was completely transactional and stateless. Sure, you could go to the account section and see your trips etc if you were logged in, but the home page had no special behaviour regardless of a guest being logged in or not. So we took the first little step in making the website a little more context sensitive by adding a widget showing the details of a guest's upcoming trip (if any in the next 3 days) along with driver details etc as a driver is assigned. ### How Technically, building this was simple enough. Just call an API to get the details of the next ride if the guest is logged in and render the details. ### Impact We expect that this will reduce the anxiety about ride details most guests have and also give them a reason to engage more actively with the website. For us, this should mean fewer guests calling on the phone and hence more time to do other awesome stuff. So if you have booked a Shoffr for the coming few days, you can use this widget and not worry about missing SMSes. ### Evolving This is a small start towards making our website sensitive to the customer context. This is going to be a key focus area for us in the coming weeks. Ideally we would like to be able to offer our guests the most pertinent information at any time in their Shoffr journey. e.g. Car tracking when pickup time is imminent, fare/payment details when the ride is about to end, feedback after the ride, and so on. --- *Originally published on the* [*Shoffr blog*](https://shoffr.in/blog/launch-upcoming-ride-widget?ref=kislayverma.com)*.* ### Launch: Book return trip along with onward trip URL: https://kislayverma.com/launch-book-return-trip-along-with-onward-trip/ Last updated: 2026-08-22T13:09:59.000Z Yesterday we launched the "Book return" functionality on [Shoffr](https://shoffr.in/?ref=kislayverma.com) which will make it easy for our guests to book both onward and return trips in one go. They can now click on the "Book Return" checkbox, and select the pickup time for the return. We will create two trips for them with the origin and destination reversed and the two pick-up times selected. With the upcoming Republic Day weekend, this is a great way to book a Shoffr both ways easily. ### Why? We have seen a fair number of our guests booking trips from their homes to the airport and back again, often on the same day, when travelling for work or for dropping off friends/family. People travelling into Bangalore for work do this even more often since their schedule is predictable. The obvious thing to do after seeing this behaviour was to build the functionality to book return trips (like most travel sites do). This would save the guest the hassle of making two bookings and this convenience should give us more trips. ### How? There were two main problems in building this. - We thought about letting guests choose both the pick-up point and time for the return journey. But adding all this info would make our booking form very large and confusing — potentially leading to drop-offs. - Managing the order lifecycle now that it could contain multiple trips. What happens when we could confirm one trip immediately but not the other? How do we do invoicing if one trip is cancelled? And so on. #### Taming the booking form In the beginning, we were asking guests to enter their name, phone number, email, location, landmark, and pickup time before we let them see the fare for the ride. Though we pre-fill these if the guest is logged in, it is still quite a lot. Adding returns to this would add at least two more fields. We decided to trim down the booking form before we built return booking. I moved all the personal info fields to the ride confirmation pop-up and left only location, pickup, and landmark fields on the home page. This freed up a lot of real estate on that page. Once this change looked stable and we confirmed that there was no change in the number of incoming rides, I looked at the data of the customers booking returns. It turns out that most of them were booking onward and return rides from the same two points. This meant that for most guests, selecting another location would be unnecessary. Hence we decided to add one checkbox to indicate that the guest wanted to book a return, and a date-time picker for selecting return pick-up time. This way, even with the added functionality, we had fewer fields on the main page, leading to lesser cognitive load on our guests. #### Multiple trips per order I've lost count of the times I have "learnt" that cutting corners in core tech hurts, yet it happens. I had decided early on that our Order entity will directly map to one Trip entity. This was challenged when we added the "Use Toll" option since toll would now have to be another item on the invoice, but I managed by adding a toll amount field in the order itself. Bundling returns in the same order completely broke this since one "order" could now contain multiple trips. I could have hacked a little more by creating two orders and then linking them via some field, but disgust finally overcame hustle. I rewrote the order management system, the website, and the internal admin portal (which our ops team uses) such that an "order" can contain multiple "order items", each of which maps to one "trip". Orders and order items are highly coupled as they are both part of the order management domain. They share identical life-cycles, and impact each other when operations like cancellations, refunds etc happen. This core change obviously led to a large fallout everywhere else in the system, most importantly in our website and in the admin panel used by our Ops team. Any code dealing with amounts had to be re-evaluated and changed if needed. ### Rollout I started with removing the customer details fields. This was the least risky change since it didn't change anything in the core order creation integrations, and the booking form could use a cleanup anyway. Then we did the backend rewrite to introduce order items. The trick here was to add a schema transformation layer in our existing order API so that external systems like the website could continue calling them without being aware of the internal data model change. This was a little extra work but allowed me to deploy continuously instead of having to do everything in one big push. Then just to be extra sure of everything, I added the "Book return" functionality to our admin portal. This would make life easier for our CX/Ops folks, but also let us test stuff in production without rolling it out to customers at large — a 10% rollout, if you will. Then, I built and rolled out the feature on the website. This is available to all customers. ### Evolving There are a few rough edges on the feature right now which we will fix very soon. - Currently the return trip is always created as unconfirmed (though it gets confirmed within 30 mins) because running the allocation engine twice added a lot of latency to the booking flow. This is obviously not ideal. - Also in terms of communication, if one of the trips is confirmed but the other is not, customers get two SMSes with different language which is a little confusing. We will clean this up very soon. Have you booked your [Shoffr](https://shoffr.in/?ref=kislayverma.com) yet? Come give us a spin. ### Evolving Software: SOLID principles as a continuum URL: https://kislayverma.com/evolving-software-solid-principles-as-a-continuum/ Last updated: 2026-07-22T12:46:19.000Z SOLID principles are powerful tools for building a system with low coupling between its components. A quick recap on these principles: 1. SRP: Single Responsibility Principle 2. OCP: Open Closed principle 3. Liskov Substitution Principle 4. Interface Segregation 5. Dependency Inversion If you don't know what these terms mean, I recommend[ this primer](https://www.baeldung.com/solid-principles?ref=kislayverma.com). Go check it out and then read the rest of this article. Here, I want to talk about how all the SOLID principles are interlinked. They all apply simultaneously in any situation. Breaking one will also break multiple others. In my opinion, they should be read as a continuum, rather than a set of independent principles - one always needs some of them to achieve the other(s). Personally, I start by saying that I don't want to modify the existing code. Who knows how I might break it? I want to just inject my new logic into the currently running system in the specific places where I need to. So SRP is kind of my favourite principle. But all other principles come in to uphold this one. Let's consider the windows machine example from the primer I linked above. Here’s the class for reference. ![](https://kislayverma.com/content/images/2022/04/applying-solid-principles.png) The machine has a Keyboard, a CPU and a Monitor. In the example, the machine has the responsibility of creating these objects. It needs to not only know how it does its own functions but also how to create these subcomponents. This breaks SRP. How can we remove the knowledge of building the monitor from the machine? The easiest way is to let someone else build the monitor object and give it to the machine class during construction (much as it happens in the real world). This is the dependency inversion principle - we are using one SOLID principle to achieve another. But if anyone can construct a monitor and pass it to a machine, the machine needs to be sure that the monitor is compatible with the implementation of the machine. Otherwise, the machine has to handle the differences between various types of monitors. This again breaches SRP. The machine wants others to create monitors, but all monitors must do exactly what the machine expects, regardless of how they do it. How do we get this? Enter Liskov Substitution Principle, which requires subclasses to do exactly this. The machine class exposes some interface or base class that comprises the "specs" of the monitor. Every monitor must do exactly that, and nothing else. This ensures that the machine can be given any implementation of the base monitor class to work with and nothing in the machine code needs to change. Yet again, we see how one principle supports another. To the extent that the behaviour of the machine is controlled by the behaviour of the monitor, we have already achieved OCP because we can pass in different monitor implementations to do the same thing in different ways. But how can we modify the behaviour of the machine itself? One way would be to open up the machine's code and add some conditional or additional business logic there to cater to our new requirements. But if we do this, the final artefact still breaches SRP in a way since it changes for multiple behavioural reasons now. To prevent this, we must redesign the machine component in a way that allows others to subclass it and override it with new behaviour. The old code is still in play for existing use cases, but wrappers can now be built around it to support new use cases. Here, OCP is helping maintain SRP in the long run. Let's look at the monitors themselves. In isolation, they can have many many attributes. Monitor manufacturers deal with tons of complexity. But all of that is not relevant to the machine. So the machine-facing part of the monitor implements a much narrower set of specifications that the manufacturing-facing part. This is the interface segregation principle, where the monitor object implements two different sets of interfaces for two different use-cases. As this example shows, SOLID principles cannot be applied one by one. They have to be applied all at the same time to achieve the decoupling we want in our systems. ## An evolutionary take An interesting way to look at this is in terms of system evolution. Everywhere in the world, evolving systems develop greater degrees of specialization for every type of component, and they develop a rich collection of different types of components. The cross-play of both these axes results in the immense diversity of living systems we see around us. I have written before about the [mechanics of software evolution](https://kislayverma.com/the-mechanics-of-software-evolution/). That article painted a higher-level picture of system evolution. Let’s consider how we can guide this evolution inside our components. Software programs are living, growing systems. SOLID principles are the guiding forces that let the system evolve specialization and diversity in a healthy way, instead of collapsing into a mess of chaos. ![](https://kislayverma.com/content/images/2022/04/solid-pre-evolution.jpg) SRP and OCP create specialization. SRP is the restriction that prevents a component from becoming too muddled internally. But due to external business pressures, the same component MUST do different things. The pressure builds, and OCP relieves it by allowing a deeper subtree of more and more specialized subclasses which can satisfy business needs. We saw in the above machine example how the concept of monitor evolved from inside the concept of the machine due to SRP. Similar things can happen with CPU and mouse and so on. Here too, SRP is the forcing constraint, and the Liskov Substitution Principle, Interface Segregation, and Dependency injection jump in to satisfy the constraint by creating diverse types of components, most of which can work with each other. Once outside, monitor and the other concepts take on a life of their own, each developing its own hierarchy of specialized subclasses. And hence the cycle repeats, creating an increasingly large but consistently decoupled system. ### Building robust distributed systems URL: https://kislayverma.com/building-robust-distributed-systems/ Last updated: 2026-07-22T12:46:21.000Z I have written before on this blog about [what distributed systems are](https://kislayverma.com/for-the-layman-ep-1-what-is-a-distributed-system/) and how they can give us [tremendous scalability](https://kislayverma.com/on-scalable-software/) at the cost of having to deal with a more complicated system design. Let's discuss how we can make a distributed system resilient to random failures which get more common as the system gets larger. [Systems theory](https://kislayverma.com/book-review-thinking-in-systems-a-primer/) tells us that the more interconnected parts of a system are, the more the likelihood of large failures. So to build a resilient system, we need to reduce the number of connections. Where this cannot be done, we need to implement ways to "temporarily" sever connections to failing parts so that errors do not cascade to other parts. ![](https://kislayverma.com/content/images/2022/03/connected-components.jpg) Every component has to assume that every other component will fail at some point and decide what it will do when such failures happen. Lastly, we need to build some buffers in the system - some ways to relax, if not remove the demands placed on it so that there is slack to handle unexpected conditions. ### Minimize inter-component dependencies Components of a distributed system communicate with each other for data or functionality. In both cases, we can reduce the requirement of connectivity by pushing the data/functionality into the calling component instead of being accessed remotely. Building a high scale distributed system forces us to abandon a lot of the "best practices" of standard software engineering. The key thing to remember is as we adopt the complexity of distributed systems to attain scalability, we also need to keep the "distribution" in check as much as we can. #### Duplicate Data If we access some data from another component frequently, we can duplicate it in our component to not have to retrieve it at run time. This can massively reduce runtime dependency and help improve latency on our component. Data that is frequently accessed but changes with some regularity can be cached temporarily with periodic cache refreshes. Data that changes even less frequently or never (e.g. Name of a customer) can be stored in our component directly. We might have to do some extra work if/when this data does change, but this added small overhead is usually worth it for the increased resilience. #### Denormalize Data Denormalization is a special form of duplication that happens within a component. If we are using relational data stores, we can reduce the cost of looking across multiple entities by duplicating data in the main entity. The principle of localizing scattered data for better performance applies here as well. #### Libraries To mitigate functional dependency of another component, we can package the remote component as a library and embed it within our component. This is not always possible (it might be written in some other language or be too large to be a library) and comes with its own set of problems (change in functionality requires library upgrades across multiple components), but if the functionality is critical and frequently accessed at a high scale, this is a viable way of breaking the inter-component connection and making it local. ### Isolate errors Error isolation is important for two reasons. One is that individual errors are more common in distributed systems (simple function of lots of moving parts). The other is that if we cannot prevent errors from cascading throughout the system, then we lose the very reason for building a complex in the first place. The primary construct of error isolation is SLA. Every component declares some quality parameters it will honour in performing a function. these parameters can include latency, error rate, concurrency, and others. Beyond this SLA, components invoking it assume it to have failed and need to take suitable action on their own. If the component itself detects that it is unable to maintain its SLA, it can preemptively tell its callers to back away and come back later. To [maintain overall system health](https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/), it is better to fail fast rather than succeed with breached SLA. Both components (the one invoked and the one invoking) must put in mechanisms for this. #### Protecting the caller **Timeouts**: If the called component doesn't respond within its SLA, the caller must timeout (give up) and use some fallback mechanism instead (even if it is throwing an error) to maintain its own SLA and prevent a cascade of SLA breaches. **Retries**: Since the network is unreliable, many errors in a distributed system are just random. The caller can retry the operation if its own SLA permits it to do so. The prerequisite for retries is the idempotency of the operation. i.e. it should not change state or do it only once even if it is invoked twice. **Circuit Breakers**: If calls to a component are failing continuously, the caller can sever the connection and stop calling it for some time by "opening the circuit". Since the caller already has some backup behaviour for error scenarios, this saves the caller precious resources which would have been wasted. Stopping the calls also reduces the load on the called component and give it some breathing room to recover. Circuit breaker libraries have mechanisms to poll the troubled component periodically and restart the call flow if its performance seems to have returned to normal. #### Protecting the called **Random Backoffs**: While retries reduce errors, a small performance blip in a heavily used component can cause all of its callers to retry at once. This "retry storm" can create spikes in load and prevent this component from recovering. To prevent this, retries should be done with a random time gap between them so that load is staggered. **Backpressure**: If a component detects itself under too much load and about to breach its SLA, it can preemptively start dropping new requests till its performance comes under control. This is much better than accepting requests which it knows it can't serve within SLA or without the risk of a complete crash. ### Build buffers in the system #### Asynchronous communication [Asynchronous communication](https://kislayverma.com/content/files/2026/07/asynchronous-programming.html) channels like message buses allow remote components to be invoked without a very tight SLA dependency. By letting the messages be consumed when the called component is ready instead of right away, the system becomes a little more elastic to the demand of increased workload. #### Elastic provisioning Scalability eventually boils down to making the best use of available hardware. But a simple way of giving the system room to breathe can be to allocate more hardware if see the scale growing. While this is only feasible up to the extent of the cost we can bear, it gives us the last line of defence against unpredicted variations in load. You can read about some low-level details of using these techniques in my articles on [code review](https://kislayverma.com/code-review-checklist-for-distributed-systems/) and [design review](https://kislayverma.com/design-review-checklist-for-distributed-systems/) for distributed systems. ### On building scalable systems URL: https://kislayverma.com/on-scalable-software/ Last updated: 2026-07-22T12:46:25.000Z In software engineering, scalability is the idea that a system should be able to handle an increase in workload by employing more computing resources without significant changes to its design . ### Why don't systems scale Software, though "virtual", needs physical machines to run. And physical machines are bound by the law of physics. The speed of light limits how fast a CPU can read data from memory. The information-carrying capacity of wires determines how much data can be moved from one part of a system to another. Material sciences dictate how fast a hard disk can spin to let data be read/written. Collectively, all this means that there are hard limits to how fast our programs can run or how much work they can do. We can be very efficient within these limits, but we cannot breach them. Therefore, we are forced to use software or hardware design tricks to get our programs to do more work. The problem of scalability is one of designing a system that can bypass the physical limits of our current hardware to serve greater workloads. Systems don't scale because either they use the given hardware poorly, or because they cannot use all the hardware available to them e.g. programs written for CPUs won't run on GPUs. To build a scalable system, we must analyze how the software plays with hardware. Scalability lives at the intersection of the real and the virtual. ### The key axes of scalability #### Latency This is the time taken to fulfil a single request of the workload. The lower the latency, the higher the net output of the system can be since we can process many more requests per unit time by finishing each one fast. Improving latency can be understood in terms of "speed up" (processing each unit of workload faster) and is typically achieved by splitting up the workload into multiple parts which execute in parallel. #### Throughput The other axis of scalability is how much work can be done per unit time. If our system can serve only one request at a time, then [throughput](https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/) \= latency X time. But most modern CPUs are multicore. What if we use all of them at once? In this way (and others), we can increase the total of "[concurrent](https://www.youtube.com/watch?v=oV9rvDllKEg&ref=kislayverma.com)" requests a system can handle. Along with latency, this defines the total things happening in a system at any point in time. This can be thought of in terms of "scale up" or concurrency. [Little's law](https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/) gives a powerful formulation of this which lets us analyze how a system and its subsystems will function under increasing load. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.58.12-AM.png) If the number of items in the system's work queue keeps growing, it will eventually be overwhelmed. #### Capacity This is the theoretical maximum amount of work the system can handle. Any more load and the system starts to fail either completely or for individual requests. ### Performance is not Scalability A system with high performance is not necessarily a scalable system. Ignoring scalability concerns can often result in a much simpler system which is very efficient for a given scale but will fail completely if the workload increases. An example of a performant yet non-scalable system is a file parser that can run on a single server and process a file up to a few GBs in a few minutes. This system is simple and serves well enough for files that will fit in the memory of one machine. A scalable version of this may be a Spark job that can read many TBs of data stored across many servers and process it using many compute nodes. If we know that the workload is going to increase, we should go for a scalable design up-front. But if we are not sure of what the future looks like, a performant but non-scalable solution is a good enough starting point. ### Quantifying scalability #### Amdahl's Law Other than the problems with implementations, there are theoretical limitations to how much faster a program can become with the addition of more resources.[ Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s%5Flaw?ref=kislayverma.com) (presented by[ Gene Amdahl](https://en.wikipedia.org/wiki/Gene%5FAmdahl?ref=kislayverma.com) in 1967) is a key law that defines them. It states that every program contains part(s) that can be made to execute in parallel given extra resources, and part(s) that can only run serially (called serial fraction). As a result, there is a limit on how much faster a program can become regardless of how many resources are available to it. The total speedup is the sum of time taken to run the serial part plus the time taken to run the parallel part. The serial part, therefore, creates an upper bound on how fast a program can run with more resources. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.49.44-AM.png) This means that by [analyzing our program structure](https://www.youtube.com/watch?v=EfOXY5XY9s8&ref=kislayverma.com), we can determine the maximum amount of resources that it makes sense to dedicate to speed it up - any more would be useless. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.49.56-AM.png) #### Universal Scalability Law (USL) While Amdahl's law defines the maximum amount of extra resources which will improve a program's performance by allowing parts of the program to run in parallel, it ignores a key overhead in adding more resources - the communication overhead in distributing and managing work among all the new processors/machines. [USL](https://wso2.com/blog/research/scalability-modeling-using-universal-scalability-law/?ref=kislayverma.com) formalizes this by adding another factor to Ahmdahl's Law which incorporates the cost of communication. This further reduces the net gain we can get from the addition of resources and provides a more realistic measure of how much a program can be sped up. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.50.36-AM.png) Real-world tests show that in the worst case, these communication overheads build up exponentially as each new resource is added. Program performance improves in the beginning due to more resources, but this improvement is eventually overwhelmed by the communication overhead. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.50.19-AM.png) ### Strategies for scaling systems #### Vertical scalability Vertical scalability says that if our computer is not powerful enough to run a program, we simply procure a computer that is. This is the simplest approach because we don't have to make any change to the system itself, just the hardware it runs on. Supercomputers are a manifestation of this strategy of scalability. This is essentially throwing money at the problem to avoid design complexity. We can build more powerful computers, but the cost of doing so gets exponentially larger. And there are limits to even that. We certainly can't build a computer powerful enough to run the entirety of Google. So the vertical scalability strategy can take us a good way, but it is not enough in the face of most modern scale requirements. To scale further, we need to make fundamental changes to our program itself. #### Horizontal scalability The simplest computer program is one that runs on one computer. The limits of vertical scalability indicate that the fundamental bottleneck in serving web-scale workloads is that our programs are bound to one computer. Horizontal scalability is the process of designing systems that can utilize multiple computers to achieve a single task. If this can be achieved, then scaling the system is a simple matter of adding more and more computers instead of being forced to build a single extra-large computer. The challenges of "distributing" a program can be far harder than the actual logic of the program itself. We must now deal not just with computers, but with the wires between those computers. The[ fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies%5Fof%5Fdistributed%5Fcomputing?ref=kislayverma.com) are very real, and demand that horizontal scalability be baked into the very fabric of the program instead of being tacked on from above. ### Distributed systems In embracing horizontal scalability, we embrace [distributed systems](https://kislayverma.com/content/files/2026/07/distributed-systems.html) \- systems in which various computing resources like CPU, storage, and memory are located across multiple physical machines. These are complicated [architectures](https://kislayverma.com/category/software-architecture/), so let's go through some main approaches. #### Distributing data Stored data typically represents the state of the system as it would be if there were no active processing going on. To store web-scale data, we have no choice but to split its storage across many machines. While this means that we have no storage limitation, the problem now is how to locate on which server is the specific data point located. e.g. If I store millions of songs across hundreds of hard disks, how do I find one particular song? Various techniques are used to solve this problem. Some of them are based on selecting which server to use in a smart, predetermined way so that the same logic can be applied while reading the data. These are simple techniques but somewhat brittle because the apriori logic needs to be updated constantly as the number of storage servers increases or decreases. Shard id or modulo based implementations are an example of this approach. ![](https://kislayverma.com/content/images/2022/03/scalability-data-sharding.jpg) Some other techniques are based on building a shared index to locate data across the set of servers. Servers talk to each other to find and return the data required while the actual reading program is unaware of how many servers there are. Cassandra's peer to peer approach is an example of this. ![](https://kislayverma.com/content/images/2022/03/scalability-data-clustering1.jpg) #### Distributing Compute A computation or the running of a program typically modified the data owned by the system and therefore changes its state. Being able to leverage the CPU cores of multiple machines means that we have far more computing power to run our programs than with just one machine. But if all our CPUs are not located in one place, then we need a mechanism to distribute work among them. Such a mechanism, by definition, is not part of the "business logic" of the program, but we may be forced to modify how the business logic is implemented so that we can split it into parts and run it on different CPUs. ![](https://kislayverma.com/content/images/2022/03/scalability-distributed-compute.jpg) Two situations are possible here - the CPUs may simultaneously work on the same piece of data (shared memory), or they may be completely independent (shared-nothing). In the former, we not only have to distribute the compute to multiple servers but also have to control how these multiple servers access and modify the same pieces of data. Similar to multi-threaded programming on a single server, such architecture imposes expensive coordination constraints via distributed locking and transaction management techniques (e.g. Paxos). The shared-nothing architecture is far more scalable because any given piece of data is only being processed in one place at a point in time and there is no danger of overlapping or conflicting changes. The problem becomes one of ensuring that such data localization happens and in finding where this piece of compute is running. #### Replicating data ![](https://kislayverma.com/content/images/2022/03/scalability-data-replication.jpg) This is a hybrid scenario where even though our data fit on one or more machines, we deliberately replicate it across multiple machines simply because the current servers are not able to bear the compute load of reading and writing this data. Essentially there is so much processing going on that to be able to distribute compute, we are forced to distribute data as well (or at least copies of it). Using read replicas of databases to scale read-heavy systems, or [using caches](https://kislayverma.com/architecture-patterns-caching-part-1/) are an example of this strategy. ### Considerations in distributed computing When we build a distributed system, we should be clear about what we expect to achieve. We should also be clear about what we will NOT get. Let's consider both of these things while assuming that we are designing the system well. #### What we won't get ##### Consistency [Eric Brewer defined the CAP theorem](https://kislayverma.com/working-around-the-cap-theorem/) which says that in the face of a network partition (the network breaking down and making some machines of the system inaccessible), a system can choose to maintain either availability (continuing to function) or consistency (maintaining information parity across all parts of the system). This can be intuitively understood by considering that if some machines are inaccessible, either the other should stop working (become unavailable) because they cannot modify data on the inaccessible servers, or continue to function at the risk of not updating the data on the missing machines (becoming inconsistent). Most modern systems choose to be inconsistent rather than fail altogether so that at least parts of the systems can function. The inconsistency is later reconciled by using techniques like[ CRDTs](https://crdt.tech/?ref=kislayverma.com). ##### Simplicity A distributed system design is inevitably more complex at all levels from the networking layer upwards than a single-server architecture. So we should expect complexity and try to tackle it with good design and evolved tools. ##### Reduction in errors A direct side effect of a more complicated architecture is an increase in the number of errors. Having more servers, more inter-server connections, and just more load on this scalable system is bound to result in more system errors. This can look (sometimes correctly) like system instability, but a good design should ensure that these errors are fewer per unit workload and that they remain isolated. #### What we must get ##### Scalability This is obvious in the context of this article. We are building distributed systems to achieve scalability, so we must get this. ##### Failure Isolation This is not an outcome but an important guardrail in designing a distributed system. If we fail to isolate the increasing number of errors, the system will be brittle with large parts failing at once. Ideal distributed system design isolates errors in specific workflows so that other parts can function properly. ### Why are distributed systems hard? In a word - coupling. While there are many types of coupling in software engineering, there are two which play a major role in hindering system scalability. Both of them derive from a single server program's assumption of "global, consistent system state" which can be modified "reliably". Consistent state means that all parts of a program seem the same data. Reliable modification of the system’s state means that all parts of the system are always available and can be reached/invoked to modify it. But as we have seen, the CAP theorem explicitly outlaws the consistency-availability-invocability in a distributed system. This makes the leap from single server architecture to distributed architecture very difficult. Let's look at both these types of coupling. #### Location Coupling Location coupling is when a program assumes that something is available at a known, fixed location. e.g. A file parsing program assumes that the file is located on its local file system. or a service assuming its database is available at a given fixed location. or a subpart of a system assuming that another subpart is part of the same runtime. ![](https://kislayverma.com/content/images/2022/03/scalability-location-coupling.jpg) It is difficult to horizontally scale such systems because they do not understand "not here" or "multiple". Additionally, their implementation might assume that reaching out to these other components is cheap/fast. In distributed systems, both aspects are critical. A subcomponent doing a part of the computations may be running on some other server entirely and therefore difficult to find and expensive to communicate with. A database may be many servers working as a sharded cluster. Location coupling is therefore a key problem in being able to horizontal scalability because it directly prevents resources from being added "elsewhere". ##### Breaking Location coupling ![](https://kislayverma.com/content/images/2022/03/scalability-breaking-location-coupling.jpg) The trick to breaking location coupling lies in abstracting the specifics of accessing another part of the system (file system, database, subcomponent) from the part which wants to access it behind an interface. This means different things in different scenarios. e.g. at the network layer, we can use DNS to mask the specific IP addresses of remote servers. Load balancing techniques can hide that there are multiple instances of some particular system are running to service high workloads. Smart clients can hide the details of database/cache clusters. An interesting way of becoming agnostic to the called system physical location is by not trying to locate them but by leaving all commands in a common, well-known place (like a message broker) from where they can up the commands and execute them. This, of course, creates location coupling with the well-known location but ideally, this is smaller in magnitude than having all parts of the system being coupled to all others. #### Temporal Coupling This is the situation where a part of a system expects all other parts on which it depends to serve its needs instantaneously ([synchronously](https://kislayverma.com/content/files/2026/07/asynchronous-programming-1.html)) when invoked. In the context of scalability, the problem with temporal coupling is that all parts must now be "scaled up" at the same time because if one fails, all its dependent systems will also fail. This makes the overall architecture sensitive to local spikes in workload - any change in load on any part of the system and the whole system can crash, thereby removing much of the benefits of horizontal scaling. ##### Breaking temporal coupling The most common approach to breaking temporal coupling is the use of message queues. Instead of invoking other parts of a system "synchronously" (invoking and waiting till the output appears), the calling system puts a request on a message bus and the other system consumes this You can read more about [messaging concepts](https://kislayverma.com/defining-messaging-terms-precisely/) and how [events can be used to build evolutionary architectures](https://kislayverma.com/using-events-to-build-evolutionary-architectures/). The event/message-driven architecture can massively increase both the scalability and resilience of a distributed system. ### Mini Essay #4 - The other side URL: https://kislayverma.com/mini-essay-4-the-other-side/ Last updated: 2022-03-04T05:55:28.000Z **Read Next**: [More mini essays](https://kislayverma.com/category/mini-essay/) ### Mini Essay #3 - Say "no" URL: https://kislayverma.com/mini-essay-3-say-no/ Last updated: 2022-02-24T17:03:38.000Z ![](https://kislayverma.com/content/images/2022/02/Say-No-1.png) --- **Read next**: [More mini essays](https://kislayverma.com/category/mini-essay/) ### Architecture Patterns: Caching (Part-2) URL: https://kislayverma.com/architecture-patterns-caching-part-2/ Last updated: 2026-07-22T12:46:26.000Z In [part 1 of this series](https://kislayverma.com/architecture-patterns-caching-part-1/), we looked at the different types of caches and the various ways they can be used to scale up applications. Now let's look at some nuances of using caching. --- ### Scaling caches Like any other part of the system design, caches come under load as the scale of the application increases. External caches are servers like any other and can buckle under the read/write traffic being sent their way. Even in-memory caches can suffer degraded performance due to read locking if too many application threads try to access them, although this is much harder to get to and easier to mitigate. let's look at some problems and solutions scaling caches. #### Scaling to more traffic A cache is essentially a data store, and the problem of scaling for traffic is a well-known one in the database domain. Rising traffic can cause scalability problems by increasing the CPU usage or by choking the network bandwidth available to a server. The most straightforward way to scale for an increase in traffic is to have multiple servers which can serve the traffic. In databases, we typically configure a master-slave (aka leader-follower) topology where all writes go to a single server which replicates them across all the other servers. This way, all servers have all the data and the application can connect to any of them to read them. This reduces the load on each server by a factor of the number of servers. ![](https://kislayverma.com/content/images/2022/02/caching-read-replicas-for-scaling.jpg) Since writes to cache data are much rarer than reads, the overhead of replicating the writes to all slave servers is usually acceptable. Both [Twemproxy](https://blog.twitter.com/developer/en%5Fus/a/2012/twemproxy?ref=kislayverma.com) and [Redis Sentinel](https://redis.io/topics/sentinel?ref=kislayverma.com) are examples of implementation which use redundancies to scale caches. #### Scaling to larger data size We have already discussed this under external distributed caches. If we want to store more data, we really don't have a choice but to distribute it across more than one server. This directly brings the cache into the world of distributed systems with all its attendant pros and cons. ![](https://kislayverma.com/content/images/2022/02/caching-clustered-cache.jpg) Note that distributing data across multiple servers solves for both data and traffic scaling since no single server faces as much traffic as in the case of a single cache instance. However, we can look at only doing redundant deployments if our problem is just traffic and not data volume. Data replication is a simpler problem than data distribution. Both [Twemproxy](https://blog.twitter.com/developer/en%5Fus/a/2012/twemproxy?ref=kislayverma.com) and [Redis Cluster](https://redis.io/topics/cluster-tutorial?ref=kislayverma.com) are examples of distributing data to scale to larger data volumes. ### Cache Stampede In high throughput systems, a scenario emerges in the use of caches which, if not handled properly, can bring down the entire system. This has happened to pretty much every large scale company like Facebook, Doordash, Instagram etc. I have personally encountered it while building the Promise Engine at Myntra. Let's say that we are using a read-through cache where application processes first try to read data from a cache, and if not found (cache miss) they tried to load it from the source database/system. If this is a high throughput system involving a large number of concurrent accesses to the cache, then even if a single key is missing, a large number of processes will try to access the database to read this data at the same time. This now triggers a flood of traffic to the database, which may now collapse under it since it always expects a cache to sit in front of it and is not designed to take such a heavy spike of traffic. This means that a single heavily accessed key being missing from the cache can trigger a complete system collapse. Such is the fine line of high scalability! ### Prevent cache stampede Before we try to solve the problem, please evaluate if this is a problem for your system. If your cache were to vanish and the traffic were to hit your database at once, would the database hold up with perhaps only a temporary spike in latency? If so, you do not need to worry about a cache stampede. It is worthwhile doing a simulation in your production environment to verify this. If you think this is going to be a problem for you. there are a few approaches you can take. #### Keep cache always full This approach treats the cache as the source of truth. The idea is that since a cache stampede is triggered by a cache miss, if you can load all the data into the cache using a refresh ahead strategy, then a cache miss will never happen and hence cache stampede will be avoided. While this is possible for small datasets, keeping large data set fully loaded at all times is not always feasible for cost reasons. #### Nominate one process to fill the cache Let's say a cache miss occurs when 1000 processes are trying to access a key concurrently. To prevent all of them from rushing to the database, we can implement a mutex lock/leader election mechanism to elect one process which goes to the database to get the data and refresh the cache. The other processes can either wait for the cache to be filled before trying again (leading to temporarily increased latency) or they can all throw an error to their respective callers (leading to a brief spike of errors). Obtaining locks in a distributed environment is a complicated but solved problem. Zookeeper and Redis both offer convenient ways of doing this, but you can roll your own using lock entries in a table if you think it is easier (it isn't). #### Probabilistic early expiration This is a smarter solution than most teams require, but the idea is to balance the above two approaches. If we cannot load all the data in the cache but still want to prevent cache misses, then the only way is to reload the keys intelligently before it expires from the cache. [This paper](https://cseweb.ucsd.edu/~avattani/papers/cache%5Fstampede.pdf?ref=kislayverma.com) outlines one of the strategies for preemptive loading of keys before they expire. #### TL;DR Cache stampede is one instance of the new failure modes introduced into an architecture that uses caching. In designing high scale applications, this outcome of cache miss should be carefully considered before using caches. **Read Next**: More articles on [architecture patterns](https://kislayverma.com/content/files/2026/07/architecture-pattern.html). ### Architecture Patterns: Caching (Part-1) URL: https://kislayverma.com/architecture-patterns-caching-part-1/ Last updated: 2026-07-22T12:46:28.000Z Performance has always been a key feature of technical systems. Today on the internet, sub-second latencies are the norm. It costs companies money if their pages load slowly because potential customers won't wait longer than that. On the other hand, there is more and more data from many different sources which have to be loaded into a rich user experience (think of the number of things going on a typical FB page). This data gathering problem is further exacerbated by the trend towards microservices. Given all this, how are we to build super-fast systems? ### What is caching? Caching is the general term used for storing some frequently read data temporarily in a place from where it can be read much faster than reading it from the source (database, file system, service whatever). This reduction in data reading time reduces the system's[ latency](https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/). This also increases the system's overall throughput because requests are being served faster and hence more requests can be served per unit time. ![](https://kislayverma.com/content/images/2022/02/caching-why-use-cache.jpg) Microprocessor architectures have long employed this technique to make programs run faster. Instead of reading all data from the file system all the time, microprocessors employ multiple levels of cache where it is slower to read from lower levels than from the higher levels. The game is now one of making sure that the data read most often is in the highest level cache, and so on. Getting this right can make a dramatic difference to the speed of a running program. Typically, caches hold a copy of the source data for some time (called expiration time or time-to-live (TTL)) after which the data is "evicted" from the cache. As more data is loaded into a cache of finite capacity, some strategies may be applied to decide which data is retained and which is evicted. ### Where can we employ caching 1. **The system must be read-heavy** \- As should be clear by now that caching is only a solution for scaling the reading of data. So if you are building a system that reads data significantly more times than it writes data, caching can be a powerful technique for you. Write or compute-heavy systems have relatively less to gain from caching. 2. **Tolerance to stale data** \- Caching can only be applied if we can tolerate reading stale data (at least for some time). Since the cache is a copy of the source data, it is possible that the source data changes and the cache doesn't know of it. Some systems can tolerate this e.g. the likes counter on your Instagram reel can be a little behind the actual count for some time. Other systems like accounting systems cannot tolerate working with stale data. If working with stale data is not acceptable in our system, then caching is not a viable option for scaling. 3. **Data doesn't change frequently** \- A corollary to the above is that caching works best for data that doesn't change frequently. This will invalidate the cache and push the system's tolerance for data staleness. There are strategies to cache frequently changing data (like write-through caching discussed below), but they are usually more expensive to execute. 4. **Limited to the amount of data that cache can hold** \- In most large scale use cases it is not possible to store all of the data that you are using into the cache. e.g. You may not be able to load the profile data of all your customers into your cache because then the user data cache would be as large as the user database and that can get expensive. In these situations, we need to be smart about what is cached (the most frequently used data points) and what is retrieved from the source. ### Levels of Caching As I said earlier, caching is a general concept, not restricted to use only in web architectures. We can also employ it at various levels of the system architecture. Some of these are done by application developers explicitly, others are done behind the screens by tools and frameworks being used. I have called out some of the most commonly seen levels of caching in internet architectures, but there may be several more in between. 1. **Microprocessor** \- We have already covered this above. Microprocessors and Operating systems work together to cache data into registers and other caches to make our programs run faster behind the screens. 2. **Databases** \- Most databases employ some sort of internal caching mechanism to keep "hot" data in memory. e.g. MySQL loads small but frequently accessed tables entirely in memory. This is also typically hidden from developers, but understanding these mechanisms can be helpful in debugging database performance issues under high load. 3. **Application** \- Applications typically cache the data they own in cache tools of their choice. This is a developer-driven activity and one where we can exert the most control. 4. **Scatter-gather** \- This is a flavour of application-level caching but for applications that gather data from multiple sources (each of whom may have their internal caches). This level of caching is widely used on the internet, typically in backend-for-frontend type applications and has a significant effect on the end-user experience. 5. **CDN** \- This is "internet level caching" where we cache entire pages and distribute them across geographies so that they be read from servers very close to the end-user. ### Caching Strategies Depending on the type of application, the type of data, and the expectation of failure, there are [several caching strategies](https://docs.oracle.com/cd/E15357%5F01/coh.360/e15723/cache%5Frtwtwbra.htm?ref=kislayverma.com#COHDG5177) that can be applied for caching. #### Read through ![](https://kislayverma.com/content/images/2022/02/caching-read-through-cache.jpg) This is the simplest and most commonly used strategy. The application tries to read the data from the cache. If it finds the data (known as "cache hit"), all is well. If it doesn't find the data in the cache (known as a "cache miss"), it goes to the source to fetch it and loads it in the cache. If the cache is full, then some policy based on the nature of the data (e.g. Least frequently used, most recently used) is used to identify the data which should be removed from the cache to make room for the incoming data. In this strategy, tasks that encounter a cache miss will have a higher latency than those that get a cache hit. #### Write through ![](https://kislayverma.com/content/images/2022/02/caching-write-through-cache.jpg) In applications where the cache can hold all the data and is expected to be always fresh, we can use the write-through pattern. In this, every write is first done to the cache, and then to the source. This means that the cache is always in sync with the source. The cache becomes the source of truth for the application and it never reads the data from the source. On the flip side, this requires the full data to be loaded in the cache at the outset. It also introduces higher latencies on the write operations, and higher write load on the cache system, which may, in turn, impact its read performance. #### Write Behind ![](https://kislayverma.com/content/images/2022/02/caching-write-behind-cache.jpg) Similar to the write-through strategy, the application first new data to the cache. But after that, the application process returns to its main duties. The cache itself or some other process runs periodically and batch-writes the cache data into the source. This is an effective strategy for cases where we do not want to bear the latency cost of writing to the source in the main application process and the cache is reliable enough that we are sure of not losing the data before it is pushed to the source. This strategy requires that writes to the source never fail while dumping data from the cache, or there be a resolution mechanism to resolve inconsistencies. #### Refresh Ahead ![](https://kislayverma.com/content/images/2022/02/caching-refresh-ahead-cache.jpg) In this strategy, we pre-emptively refresh all or part of the data of a cache as it is reaching its expiry time. How to decide what to reload is up to the application. Note that the application may still face cache misses if not all data can be reloaded, and there this technique is usually combined with "read-through caching but with the idea that reloading process should make cache misses rarer. This is not a very common technique because it requires setting up a process to identify expiring data and reloading it based on some smart logic. This is usually not needed by applications. ### Choosing a cache implementation Now that we know of the various caching strategies, let's consider what kind of cache implementation to actually use. While technically any data structure/medium that is faster to access than its source version can be used for caching, typical cache implementations are key-value stores of some sort. Three types of cache implementations are popular. #### In-memory ![](https://kislayverma.com/content/images/2022/02/caching-in-mem-cache.jpg) This is the case where the reading application loads the data into its main memory (as a hash table or map) and uses it as the cache. This makes for the fastest possible access since the data is available literally like a program variable. It is also the simplest possible implementation since it does not introduce any new elements into the system architecture. Many libraries are available to abstract the implementation details of caching/eviction etc from user code. There are also several downsides to this style. The cache lives inside the application, so if the application goes down, the cache vanishes and has to be rebuilt while launching the application. The memory footprint of the application increases and the amount of data that can be cached is limited by that. This type of cache is also local to the application server. If you have multiple instances of the applications running, each of them will have its own cache (waste of memory) and these may be temporarily out of sync with each other if one instance reloads its cache while the others still haven't. #### External ![](https://kislayverma.com/content/images/2022/02/caching-external-single-server-cache.jpg) We can use a standalone system like Redis or Memcached as an external cache. It is essentially like having an external server that all nodes of an application talk to and which stores the hash table instead of storing it inside the application. This introduces a new element to manage in the architecture but creates a central cache that is durable and ensures that all instances of an application see the same cache value. This type of cache has the problem of failure tolerance. If the cache server crashes for any reason, the application will fail. This is solved by some implementations by having redundant caches which are kept in sync with a "leader server" but that can step in if the leader fails (Redis Sentinel uses this mechanism). This gives failure tolerance at the cost of design complications. #### External Distributed ![](https://kislayverma.com/content/images/2022/02/caching-external-ditributed-cache.jpg) Both the in-memory cache and the external cache suffer from some scale problems. The amount of data that can be stored in either of them is limited to the memory size of a single server. As large scale systems emerge and the volume of data to be cached increases, this becomes a bottleneck. To overcome this, we can use an external yet distributed cache implementation e.g. Redis cluster. In this architecture, the data is distributed across multiple instances of the cache servers. More servers can be added to this "cluster" as data size grows, making this architecture horizontally scalable. Data distribution among the servers is typically managed by the cache implementation itself. The reading application can continue to treat the cluster as a single entity when reading data. This is a full scale distributed architecture and comes with all the associated problems like node failure, split-brain, data redistribution etc. It is also the only feasible architecture at the highest web scale. ### TL;DR Caching is a powerful scalability technique that can be used in many different scenarios and in many different flavours to speed up the performance of our applications. In the [next part of this series](https://kislayverma.com/architecture-patterns-caching-part-2/), we will look some more nuance in the use of caches and a specific but deadly failure pattern in systems that rely on caching. ### Mini Essay #2 - Failure is a Feature URL: https://kislayverma.com/mini-essay-2-failure-is-a-feature/ Last updated: 2026-07-22T12:46:28.000Z **Read Next**: [More mini essays](https://kislayverma.com/content/files/2026/07/mini-essays.html) ### Mini Essay #1: Mindful Actions URL: https://kislayverma.com/mini-essay-1-mindful-actions/ Last updated: 2022-02-17T17:07:16.000Z **Read Next**: [More mini essays](https://kislayverma.com/mini-essay/) ### The pentagon of entity models URL: https://kislayverma.com/the-pentagon-of-entity-models/ Last updated: 2026-07-22T12:46:29.000Z I recently read an article by Matt Ricard about the "[Heptagon of Configuration](https://matt-rickard.com/heptagon-of-configuration/?ref=kislayverma.com)" in which he discussed how configurations evolve in a cycle. It struck me that there's another thing that follows a similar routine - entity attributes. Entity attributes often grow into complete systems in their own right. - Tax number grows into tax records and engines. - boolean flags grow into multi-valued state machines etc ### The entity evolution cycle ![](https://kislayverma.com/content/images/2022/02/pentagon-of-entity-models.jpg) This is the transition as I've seen it go in my experience. 1. Some new, urgently required properties of an entity are modelled as a hack in a configuration somewhere. 2. This configuration is then pulled into the entity's main data mode, typically as a boolean attribute. 3. Boolean attributes evolve into named/enumerated types, sometimes with more associated data. 4. Enumerated types become full-fledged entities linked to their previous host entity 5. Entities evolve into complete business domains with their systems, process, almost organizations. ### Why? This iterative process happens because as we explore more use-cases, an increasingly complex domain model emerges. The pentagon of entity model evolution is a sign of developers trying to keep up with a deeper understanding of the business but not yet knowing enough to model its nuances fully. [I interpret technical debt in a similar manner](https://kislayverma.com/uncertainty-and-learning-as-tech-debt/) \- as wisdom in hindsight. The system's implementation is essentially just catching up with the developer's understanding of the problem domain. The pentagon is a perfectly safe, natural way for systems to evolve. But if you want to crank the wheel a little faster, the way to move fast but stay on track is to [spend more time understanding the business domain](https://kislayverma.com/the-mechanics-of-software-evolution/) upfront. Understanding the domain better can help us predict future needs and build the necessary extensibility, if not the actual capabilities in our designs right away. **Read Next:** [Focus on getting faster rather than being quick here and now](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/) ### The Mechanics of Software Evolution URL: https://kislayverma.com/the-mechanics-of-software-evolution/ Last updated: 2026-07-22T12:46:31.000Z Have you ever heard engineers in your team complain about only building "business features" and never doing any "tech work"? There are some ways in which this complain is legitimate, but I feel that there is an underlying unity to both these things. I'd like to explain this by applying a evolutionary lens to changes in software. Let's start by identifying the kind of change requests we typically see in software. 1. Most of the time, changes come as feature requests for enhancing whatever capabilities already exist. 2. Once in a while, changing business needs aggregate into large change requests which require new things that the system has never had before. In the latter case, the need for ground-up thinking is obvious. New capabilities or abstractions are being explicitly requested by the external world, and we must deliver. While such large changes are tricky to deliver on, they are also the more straightforward ones (in terms of what the output is expected to be). There is an explicit directive to "evolve" the system. ### Changes add up However, the former case often also contains the seem need, though it is embedded a little deeper. Each small feature is innocent in isolation, but applying a[ systemic lens](https://kislayverma.com/book-review-thinking-in-systems-a-primer/) to it can sometimes reveal a more fundamental gap in technical capabilities. This gives us the opportunity to devise a more holistic solution that not only addresses the current requirement but also add some new fundamental capability to the system. This is what all the advice around ["understand requirements clearly"](https://kislayverma.com/system-design-from-one-level-up/) is talking about. We have to understand the immediate requirements properly, but we also have to read between the lines a little bit to see where the customer need is coming from and try to take the system there directly instead of traversing a morass of many small, disjointed changes feature requests. Strategy is not just the arena of business. It plays an equally important role in how technology evolves. Simon Wardley has formally adapted this relationship of strategy and evolution into his[ "Wardley Mapping Framework"](https://kislayverma.com/book-review-wardley-mapping/). As a CEO looks at market trends in aggregate and builds a strategy to evolve his company to keep up with them,[ the engineer has to look beyond the obvious requirement](https://kislayverma.com/make-it-better-every-day-of-the-week/) today to see if there is an emergent theme underneath a set of feature requests when seen in aggregate. ### Evolving software at the edge of chaos Let's phrase the design-for-the-future approach in evolutionary terms. ![](https://kislayverma.com/content/images/2022/02/systemiic-evolition-using-features.jpg) Change requests, however small, are the environmental pressure for software evolution. Teams that can identify the driving forces behind seemingly small requests and develop coherent abstractions in their systems in time will have adapted the best to this pressure. They will live to see another day. Teams that consistently fail to do this will perish, something alongside their entire organizations. Before the rallying cry of YAGNI etc starts, there is obviously a fine line to walk here. An over-engineered system is just as bad as an under-engineered when it comes to being a fit for the business landscape. We have to find a balance where we allow the emergent themes to be manifested somewhat clearly before we solve for them. Too far to one side is the chaos of hacks and piecemeal changes, too far to the other is too many useless abstractions that slow down everything else. When done well, a system grows new layers of abstraction and complexity just in time to prevent small needs from becoming big problems. A constant evolution mindset is best put into effect when designing anything, but we also have to deal with existing code that is getting outdated due to changes to the ecosystem.[ Continuous refactoring](https://kislayverma.com/saving-the-day-with-continuous-refactoring/) can be a good way to encounter this. Refactoring is a great opportunity to identify scattered yet recurring patterns in code and see if there is an opportunity to aggregate them into something more concrete. It also gives a good sense check of whether the perceived “theme" is real (something happening multiple times in code) or just wrong intuition. Large change requests are top-down evolution, but this is the kind of purposeful bottom-up evolution[ I have written about before](https://kislayverma.com/the-sense-of-purpose-in-a-complex-system/). If we don't adopt this features-as-evolutionary-pressure mindset, scattered solutions to small requests will pile up and result in[ increasing tech debt](https://kislayverma.com/uncertainty-and-learning-as-tech-debt/) or an eventual large change request to the system. This "stop-everything-and-rearchitect" scenario is expensive and risky (since such efforts are liable to fail or underachieve). ### TL;DR My recommendation is to use feature requests as a breeding ground for the next generation of the system's architecture. By continuously evaluating what we are being asked to change, we can jump the gun and get to the next level faster and often more safely. **Read Next**: [Using Agile practices to go beyond execution excellence](https://kislayverma.com/agile-for-innovation-going-beyond-execution-excellence/) ### The crypto-web's fatal flaw URL: https://kislayverma.com/the-crypto-webs-fatal-flaw/ Last updated: 2026-07-22T12:46:32.000Z > “The form of law which I propose would be as follows: In a state > which is desirous of being saved from the greatest of all plagues—not > faction, but rather distraction—there should exist among the citizens > neither extreme poverty nor, again, excessive wealth, for both are > productive of great evil . . . Now the legislator should determine what > is to be the limit of poverty or of wealth.” > > [Plato](https://gordoncstewart.com/2012/02/18/plato-on-wealth-and-poverty/?ref=kislayverma.com) I have long held that as compared to the outsized influence it enjoys in our lives, most of Wall Street actually plays a minimally important role in the world. It is the place from where the ultra-rich set the engine of the world to turn for their ideological and financial benefit. This is hardly a unique opinion and does not merit further repetition. Web3 started with the promise of breaking the control of the few by decentralization. But by now it is well acknowledged that blockchain technology has the fatal flaw of not being scalable **by design**. So for all practical purposes, try as it might, it cannot build the decentralized world for everyone that it promises. Coupled with the problem that [complex-tech-at-scale fundamentally gravitates towards centralization](https://blog.fabiomanganiello.com/article/Web-3.0-and-the-undeliverable-promise-of-decentralization?ref=kislayverma.com), what I get is the sinking feeling that the internet may never be truly decentralized. But for me and many others, there is a deeper, more fundamental problem with this eco-system. The crypto-web is irretrievably tied up with money. Nothing in it exists that is not defined or measured by money. Indeed, the fundamental construct of the crypto-world is a distributed ledger! By linking everything up to money in the form of coins/tokens (by design), and with absolutely no oversight (also by design), it invites the kind of actors that focus on the financial aspects to the exclusion of all else. The barrier to entry is only computing power so the already rich can get in easily. The more such actors come in, the more difficult it gets for people with less computing resources to get in. And so, progressively, the crypto world looks just like wall street, where people who make nothing sell everything at prices that are beyond the reach of everyone except the super-wealthy. For me, the crypto-web is the direct philosophical descendant of wall street. For all the talk about empowering creators and so on, the most money to be made in this economy is not by creating digital art but by buying and selling it via NFT (or other means too - I'm no expert). Crypto communities focus exclusively on prices instead of outcomes. Creating new things is not fundamental to this system, but continuous trading of anything possible is existential (no trading -> no mining -> dead blockchain). Also very reminiscent of financial markets. As an engineer, I think decentralization becoming synonymous with blockchain/crypto is a bit of a tragedy. It is like choosing implementation detail before thinking through any other solutions. To my mind, the only salvageable use-case of technology here is distributed identity management of some sort (public/private keys on the blockchain perhaps), most other things can be done in better ways. There are other ways of doing decentralized data, identity, and so on. The [Solid project](https://solidproject.org/?ref=kislayverma.com) is giving this a good (but extremely sluggish) shot. The biggest criticism I hear of projects like this is that there is no killer app (true), or that it will never work for the masses because [people don't want to run servers](https://moxie.org/2022/01/07/web3-first-impressions.html?ref=kislayverma.com). To this, I can only say that in what sense is crypto working for the masses? The reality, IMO, is that the crypto-web is generating money from the get-go, which makes everyone forget the original problem statement - a fairer, digital existence for everyone on the planet. Maybe the utopia of a truly decentralized internet where people own their own data will never come to pass (due to any number of reasons). But reaching the promised land of \[\[Crypto\]\] would be far, far worse. In that dystopia of pure finance, the technology will no do what it claimed to do, it will be far more profitable to trade than to produce, nothing you buy will really exist, and everything will exist only to be sold. Not my idea of heaven. **Read Next** \- [The Marketplace Scam](https://kislayverma.com/the-marketplace-scam-sellers-beware/) ### The Marketplace Scam (sellers beware!) URL: https://kislayverma.com/the-marketplace-scam-sellers-beware/ Last updated: 2026-07-22T12:46:32.000Z > “Every great magic trick consists of three parts or acts. The first part is called "The Pledge". The magician shows you something ordinary: a deck of cards, a bird, or a man. He shows you this object. Perhaps he asks you to inspect it to see if it is indeed real, unaltered, normal. But of course... it probably isn't. The second act is called "The Turn". The magician takes the ordinary something and makes it do something extraordinary. Now you're looking for the secret... but you won't find it, because of course you're not really looking. You don't really want to know. You want to be fooled. But you wouldn't clap yet. Because making something disappear isn't enough; you have to bring it back. That's why every magic trick has a third act, the hardest part, the part we call "The Prestige"." > > [Christopher Priest (The prestige)](https://www.goodreads.com/author/quotes/23419.Christopher%5FPriest?ref=kislayverma.com) [Marketplaces](https://kislayverma.com/marketplaces-are-not-platforms/) are everywhere on the internet. And they all typically start like this. 1. Find a fragmented market for service/product where the service providers are difficult to discover and compare for the customer. 2. Build a product that brings a large number of them under a single place to facilitate discovery and order. 3. Convince service providers that they will make more money by joining you. 4. Convince customers that this is a much better way of shopping, often by giving crazy discounts. 5. ...you know how it goes. The advantages to both the service providers and the marketplace owner are obvious. The former gets more demand, and the latter gets a share of the transactions. The customer gets a super convenient experience, and very often marketplaces have some kind of provider rating mechanisms that help the customer make a more informed choice. So far, so good. Everyone seems to be winning. ### The scam emerges But this is only the first of the marketplace magic trick, the pledge if you will. Because while providers are enjoying the increased revenue coming via the marketplace, the marketplace itself has already moved on to the next step of the playbook - identifying customer needs. Because it controls the entire supply of customer data (behavioural, transactional, everything), it can now build a very deep understanding of where the customers are, what they want, and when they want it. Initially, the data is used for targeted pricing, advertisements, better-informed product decisions, etc. This stage sees intense competition among marketplaces. Many marketplaces do not come out of this alive. Those that do typically command a tremendous scale of both demand and supply. This stage is typically too good to be true for both customers and service providers. Service providers haven't changed their business much but are making a lot more money. Customers are getting insane discounts and near-instant gratification (as compared to the past). Unfortunately for the service providers, this is when marketplaces bring in the next devastating piece of the strategy - replacing the service provider. The thinking goes that why should the marketplace owner share anything with service providers when it can itself become a service provider? Providing the service is, after all, just a skill that anyone with time and resources can acquire (fairly easily in most domains). So fashion marketplaces like Myntra start their own clothing brands, food marketplaces like Swiggy start their own kitchens, convenience/delivery marketplaces like Dunzo start their own stores and warehouses, travel marketplace like Uber try their hand at self-driving cars, and so on. Now the scam is beginning to fully reveal itself in the power relationship between the marketplace and the service providers. While the providers were needed in the early days of the platform to solve the marketplace cold-start problem (demand needs supply, supply needs demand), they become a cost as the marketplace builds its own service capabilities. So the marketplace now tries to get a larger and larger share of the proceedings from the providers. Theoretically, providers that don't like this could walk away, but by now the whole system has become too dependent on the marketplaces by now. Customers are no longer loyal to stores and brands but are addicted to the convenience of the marketplace (e.g. I almost completely stopped going to my local grocery store and started ordering from Dunzo, Grofers etc). The marketplaces have captures the demand, and are now moving to capture an increasing share of the supply. Sure they can't capture all of the supply, but they don't need to as long as they capture the most lucrative bits. This playbook has now been repeated and perfected by so many marketplaces that it is now the obvious, logical path for any such business. There is no reason to believe that any marketplace will "not" follow it in the future. In my view, this is not a way of running an economy that can be sustained in the long run. There is a whole socio-economic commentary to be done here about the transfer of wealth and power from SMEs to a smaller elite, but I will abstain from that here. Instead, I want to focus on a separate aspect of this whole system which makes "the prestige" possible. Let's talk about ownership of data. ### Ownership of Data The core reason why marketplaces can make the shift from convenience providers to super-efficient service providers is because of the data they have about customer behaviour. This data being accessible only to the marketplace essentially means that the people providing the actual service have no way of knowing whether they are doing the right things and what else they could do. To be fair, this data at this scale did not exist before the marketplaces came along, but now that it does exist, it skews the marketplace-seller relationship tremendously with no recourse for the latter. Sure, most marketplaces offer "seller insights" or other such tools for service providers, but that is typically only the tip of the iceberg in terms of information and is typically the marketplace's view of what the providers should know. once they enter the marketplace world, providers have no leverage to dictate how the business should be run. They surrender much of the agency they had in running their business and have to become robotic followers of whatever the marketplace wants them to do. The scam eventually turns to customers as well, but spenders are harder to replace so it usually takes longer. https://twitter.com/JhanveeV/status/1484606802755219457 I don't have an answer to the scam yet, but I do know that the scam exists. End-to-end ownership of functions, in my admittedly limited understanding, is beneficial for individual players but seems to be actively reducing the diversity of actors in the economic ecosystem and creating massive centralized entities, which are eventually powerful enough to act unilaterally and arbitrarily in the system. That is not good. A more sustainable version of this economic model can "probably" be built by adopting a decentralized model of information ownership where all sale and inventory data is owned by service providers and hence can be retracted anytime, logistics information known to shippers, rating and review know to other neutral observers, and all of them willingly sharing data to build a marketplace experience for the customer. The current D2C and creator economies are a move towards this structure, but I think that the internet architecture (as it is built today) is yet to truly catch on to this. Most of the creator economy work still happens on centralized platforms, most of which are also closed from a data perspective. True empowerment of creators will require them to physically own their data, and marketplaces to receive consent for receiving this data. The current marketplace model pretends to be free and open but it is actually the reverse, creating eventually unbalanced power relationships between marketplaces, customers, and sellers. Those of us who are building on the internet can and should do better. **Read Next**: [The crypto-web's fatal flaw](https://kislayverma.com/the-crypto-webs-fatal-flaw/) ### Guidelines for writing useful libraries URL: https://kislayverma.com/guidelines-for-writing-useful-libraries/ Last updated: 2026-07-22T12:46:34.000Z ![](https://kislayverma.com/content/images/2020/07/chris-ried-ieic5Tq8YMk-unsplash.jpg) ### What is a library? > *A library is a collection of implementations of behavior, written in terms of a language, that has a well-defined* *interface by which the behavior is invoked* > > [Wikipedia](https://en.wikipedia.org/wiki/Library%5F%28computing%29?ref=kislayverma.com) So a library is an artifact containing the implementation of some functionality but hiding it behind an API. "Host" systems can use the library to achieve the functionality by simply invoking the API instead of having to understand the implementation. Libraries are created to share code between multiple systems. e.g. In the Java world, [Rulette](https://github.com/kislayverma/Rulette?ref=kislayverma.com) is a rule-engine library published to a central repository. Anyone who needs to use a rule engine in their system can pull in the library from the repository and use it. A library is different from a **framework** in that while your code calls library functions, a framework will typically call your code. e.g. I can write code that uses the MySQL-connector library like [HikariCP](https://github.com/brettwooldridge/HikariCP?ref=kislayverma.com) to connect to a database. [Spring Boot](https://spring.io/projects/spring-boot?ref=kislayverma.com), on the other hand, is a framework that provides the structure within which I must write my code. Spring Boot invokes my code, while my code may invoke the library for connecting to the database. There are a few basic characteristics of a well-designed library. It should be easy to understand and use. Its behavior should be easily modifiable where that was the intent of the library author. Behaviours not intended to be modifiable should be completely hidden from users. To these ends, here are a few guidelines that have helped me in writing libraries. ### Make it small - One of the biggest problems in using a library is the number of dependent libraries it requires. LIbraries with too many dependencies are often large in size (causing the size of the host system to bloat) and may cause clashes with other libraries being used in the host system itself (complicating the host system). - Resolving conflicting dependencies which cause errors at runtime is one of the worst debugging experiences IMO. - The fewer dependencies a library has, the easier it will be to use. ### Be opinionated - A library should do a specific thing in a specific way. - While the exact definition of "specific" is up to the author, and one can make things as "configurable" as one wants, it is usually better to build a tight, opinionated library than a large, multi-faceted monster that tries to do too many things in too many different ways. - A minimal but sufficient feature set, APIs, and configurations, all go a long way in making a library easy to select (among alternatives) and use. ### Write the user's code first - I have written about [designing from one level above](https://kislayverma.com/system-design-from-one-level-up/) before. For a library, the "one level above" is the user code which is going to use it. - So first write a few samples of how host systems might use the library and experiment with a few different scenarios. We can run this by our actual users or even sit with them and ask them to write how they might want to use the library. - This will give good insights into which APIs/configurations are necessary and which are not. ### Identify internal components and identify the extensible ones - With the external environment set, we need to design the internals. - If we want library users to be able to modify certain the behaviour of certain parts, we first need to identify these "parts" so that we can design for extension. - So having an opinion means we decide what we will allow being customized, and in the design process, we identify exactly where these customizations will go. - These components/interfaces need to be designed with special care as they will be exposed to users and therefore hard to change later on. ### Allow injection of implementations of extensible components - Since we want a library to be usable by any type of host system, it is usually a bad idea to assume a runtime environment when writing libraries. e.g Writing an HTTP client library using annotations like autowired (spring example) will make it unusable in non-spring systems. - But it is ok if you are deliberately writing a library to be used only in a specific environment. It's a fair opinion to reduce complexity by assuming runtime. - IMO, it is better to provide a builder (or other similar) pattern so that these configurations and custom implementations can be injected when the library is being set up for use. - Provide as few ways (ideally only one) of using the library. Again, opinionated design will reduce the complexity of the library API. ### An aside on platform thinking - The guiding principles of designing libraries are very similar to the high-level principles for [designing platforms](https://kislayverma.com/category/platform-thinking/). - Libraries, like platforms, are not things by themselves. They exist to allow others to build things using them. The same principle of composability and extension can be seen at high and low levels in platforms and libraries respectively. - The fact that a library is shipped as a "closed" artifact makes sure that even the owner of the library is forced to use it like any other user. This is the [Golden Law of Platforms](https://kislayverma.com/the-golden-rule-of-platforms/) \- eat your own dog food. If the library wants to allow any modifications to its behaviour via configuration/extension, it must provide well-defined hooks for this. This is [External Programmability](https://kislayverma.com/external-programmability-the-second-law-of-building-platforms/) \- the second law of building platforms. **Read Next**: [How to choose a service template for your team](https://kislayverma.com/choosing-a-service-framework/) ### Competitive programming is useless URL: https://kislayverma.com/competitive-programming-is-useless/ Last updated: 2026-07-22T12:46:34.000Z https://open.spotify.com/episode/2sodHpx6OIB0HzWdfZujN7?si=ffd000a6135a49a9 ![](https://kislayverma.com/content/images/2021/08/competitive-programming.png) Image credit: GeeksforGeeks This post is a rant. I know that this is not true of every company and or every engineering student. But it is widespread enough in my experience that I find it worth ranting about. ### tl;dr Competitive programming is a good tool for building the programming muscle. An extreme pursuit of competitive programming is worse than useless. Unfortunately, companies and students are both headed in that direction at the moment instead of looking for engineers with broad interests. ### From competency to fetish Competitive programming started out as a good thing. When I was in college, there was no [leetcode](https://leetcode.com/?ref=kislayverma.com) or equivalent websites. I think GSoC had just started out and only the "rockstar" programmers used to participate in that. Or maybe this was just the situation in my college - I don't know. By and large, my class got by without writing much code, much less code of the type one might encounter in a real job. I and a few others who genuinely enjoyed programming ended up doing a variety of projects on our own and learning things that way. The leetcodes and [geeksforgeeks](https://www.geeksforgeeks.org/?ref=kislayverma.com) of the world filled a critical gap between textbooks and hands-on exercise. They provided a convenient place to see the kinds of questions asked in interviews and practice solving them. And then something went wrong. With growing access to these questions, interviewers started asking harder and harder questions in the coding rounds in college (and generally <5 years exp.) interviews. The expectation for these rounds is currently, IMO, meaningless. We left competency behind and are now well into fetish-land. As a response, college students now pursue competitive programming obsessively to stay on top. In this weird arms race against prospective hires, companies keep asking harder and harder questions in a misguided attempt to raise the bar. The students respond by doing nothing else but solve every single available question on every single competitive programming website. The junior engineer interviewing process, as it exists today, has a systemic problem. It doesn't matter how high newbies are above a certain bar of logical, coding, and algorithmic competence - it is all the same. This is a classic case of a metric being gamed. A premium is placed on being able to solve super complex data structures questions. To meet this unnecessarily high benchmark, college students do whatever it takes. If the question had instead been "can this person become a great engineer in our company", perhaps the outcomes might have been different? Being able to solve typical data structures, algorithms problems is a signal in the larger interview process. By lowering the unduly high bar on DS type questions, organizations can make room for students to exercise their curiosity and develop their passions for something unique to them - their favourite technology, their favourite tech stack, their favourite industry. This will help them find people that can be trained, are self-motivated, and have an actual interest in technology beyond a gamified version of it. ### A diluted signal Now all this would be fine if this would help people become infinitely better programmers (it doesn't) or at least distinguish themselves from the pack. But there is no indication that this is the case. Students with 37 million zillion stars on coding ninjas, extra-super-advanced level on leetcode, or uber-coding-lord status on codeforces regularly fail the interview process because EVERYONE around them is at that same level! And in the pursuit of that level, they have ignored a lot of other fundamentals they should have learned or things they could have explored and tried out on their own. When asked what kind of technologies they find interesting, several students have told me over the last few years that they are only excited by competitive programming and have no other interest in software engineering or technology as such. I can understand students from lower rung colleges following this strategy. Assuming for a minute that students from lesser tier colleges are less smart (possibly untrue but again a discussion for another day), the better students can distinguish themselves from their peers by extreme achievements in competitive programming. But at the better colleges, everyone is doing the exact same thing, and as a result, there is no benefit for anyone. It is again the responsibility of the organizations to call this out and focus on other aspects of being an engineer. At least on other academic subjects if nothing else. But almost always, the first interview round is a super-hard data structures challenge which lesser mortals can't get through. So later rounds are always evaluating people who are competitive programming biased. This funnel will never allow a different breed of software engineers to pass through. It is an accepted idea in the industry that employee performance appraisals are subjective. Most companies make only superficial attempts at making them objective because different employees contribute in different ways. This subjectivity/uncertainty is part of evaluating anyone for any role. Unfortunately, we have reduced the fresher interview process to a computing game. This is not good for anyone, and the sooner we stop it, the quicker we might be able to find good engineers instead of human-like robots. **Read Next**: [Why programmers don't write documentation](https://kislayverma.com/why-programmers-dont-write-documentation/) ### The Rise of Edge Computing URL: https://kislayverma.com/the-rise-of-edge-computing/ Last updated: 2026-07-22T12:46:35.000Z https://open.spotify.com/episode/25EqpOJ5mFB3DXvdoGjoRZ?si=3065534038bb4e13 This article is contributed by Maaz Humayun. Maaz is a senior engineer at Amazon. He spent five years with Amazon Appstore working on high-volume web services that power search, ordering, and entitlements on first and third-party devices. More recently, he's joined the Amazon Luna team, where he's working on game-streaming tech. In his free time, he enjoys reading about SaaS platforms and new trends in software development. --- You would be hard-pressed to find an industry the internet has not yet transformed. Banking, health, publishing, entertainment; the list goes on. And we're all better off for it. Internet-enabled services are faster, cheaper, and more reliable. The only thing that's outpaced the technological progress of the internet is our expectations of it. So you want that YouTube video to stream in 4k without buffering, no pixelation, and crystal-clear audio quality? Why, yes, I'll have that, thanks. The internet has gone through massive changes to keep up with growing demands. A decade ago, companies had to maintain network infrastructure and fund an IT department to keep everything working smoothly. This all changed with cloud computing. Today, all you need is a great idea and an AWS account to create a product/website that's globally available and infinitely scalable. ### Moving to the Edge ![](https://kislayverma.com/content/images/2021/08/Edge_computing_infrastructure.png) So what do we expect will change in the future? Instead of answering the question directly, let’s ask ourselves “what will stay the same?” Users will continue to expect services to get faster and cheaper. Developers will want to iterate quicker and focus their efforts on writing core business logic instead of tinkering with infrastructure. Edge computing will help us evolve the internet to satisfy these requirements. We already use content delivery networks (CDNs) to optimize latency for applications. The concept is simple. Information can't travel faster than the speed of light. So, to reduce latency, we need to move the data closer to the user. CDNs have several points of presence (PoPs) -- also called edge locations -- deployed near concentrated population centres. A CDN will cache popular content at the edge location based on customer usage patterns. If a device requests content cached at the edge, the CDN can serve the data directly, without the request going to the origin server (which could be thousands of miles away). However, these CDNs have traditionally been very -- for the lack of a better word -- dumb. Customers are limited to configuring content-management policies -- how and when to expire data from the cache. But a new wave of CDNs led by Cloudflare and Fastly have gone a step further by adding general-purpose compute instances at these PoPs. If you're a developer, this means you can insert any code between the end device and your app server. ### Computing at the Edge In 2017, Cloudflare launched 'Cloudflare Workers', which lets customers run arbitrary code on the Cloudflare platform. Workers use Google's high-performance [V8 engine](https://v8.dev/?ref=kislayverma.com) to launch [V8 Isolates](https://developers.cloudflare.com/workers/learning/how-workers-works?ref=kislayverma.com) that execute your code. Unlike containers, Isolates are fast to spin up, which reduces cold-start time, and they are computationally cheap so you can run thousands of them on a single physical machine. To see Workers in action, watch this [YouTube video](https://www.youtube.com/watch?v=48NWaLkDcME&t=557s&ref=kislayverma.com) which shows a developer intercepting calls to his domain and modifying the response based on the URL. Fastly has taken a slightly different approach to serverless computing. Instead of building their compute platform on top of existing technology, they created an optimized WebAssembly compiler and runtime called [Lucet](https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime?ref=kislayverma.com). Fastly claims that Lucet can instantiate WebAssembly programs in under 5 microseconds using only a few kilobytes of memory. By comparison, Chromimum's V8 engine has a larger memory footprint and can take 5 milliseconds to initialize programs. Fastly released Lucet as an open-source project under the [Bytecode Alliance](https://bytecodealliance.org/?ref=kislayverma.com) so you can check out the source-code [here](https://github.com/bytecodealliance/lucet?ref=kislayverma.com). Lucet already seems to be gaining adoption as a way of executing WebAssembly outside a browser environment. Shopify uses Lucet to host partner programs, called “Apps”, on top of their infrastructure. Understandably, this saves Shopify partners considerable effort because they don't have to set up their own servers. You can read more on Shopify’s engineering [blog](https://shopify.engineering/shopify-webassembly?ref=kislayverma.com). ### Maintaining State The Achilles heel of serverless computing has been the inability to persist data between requests. In other words, “serverless” has become synonymous with “stateless”. Sure, you can connect to a database over the network, but you must reinitialize your database connection every time you bootstrap your function. More, you need to deal with networking latency because the database and function could be running in different data centres. Cloudflare is innovating on the data-storage front with a product called [Durable Objects](https://blog.cloudflare.com/introducing-workers-durable-objects/?ref=kislayverma.com). A Durable Object is attachable persistent storage for your serverless function -- just imagine someone plugs a pen drive into your serverless function in the cloud. Each Durable Object is globally unique and offers transactional guarantees. By co-locating the compute and data, we significantly cut down both latency and cold-start time. As you might imagine, this is well suited for real-time applications like gaming, chat, and online collaboration tools. In fact, [here](https://blog.cloudflare.com/building-real-time-games-using-workers-durable-objects-and-unity/?ref=kislayverma.com) is a sample application that shows how you can use Workers and Durable Objects to build a simple 3D multiplayer game. It's also interesting to note that the serverless + storage architecture forces us to rethink our application design. In our new paradigm, the data constructs we create closely mirror our business constructs. For example, each Durable Object can maintain the state of a specific context, like a chat or document. There is no need for a centralized database that hosts data across the entire user base. Best of all, this architecture lets the edge layer transparently migrate our compute instance close to the user to optimize latency. A natural next step and one that will pose some exciting challenges is coordination between edge nodes. Imagine a future where roads have intelligent traffic lights that have sensors to detect traffic flow in real-time. These lights could adjust the flow of traffic to avoid congestion or idle wait times. Such a system is most effective if each sensor constantly shares its data with all nearby sensors in the network. But how would we orchestrate such a system? We wouldn't want to send all the raw data back to a central server that's hundreds of miles away. Instead, we want all the decision making to happen at the edge. Perhaps each node is listening to data updates from all nearby nodes and making decisions independently? Alternatively, a group of nodes might elect a leader that orchestrates traffic between them, and leaders might communicate with each other through a similar mechanism. I don't know how we'll solve this problem, but I know it will help improve traffic. ### Global Regions Working in the cloud, we've grown accustomed to the idea of discrete geographical regions. Developers have to balance tradeoffs like cost and latency to decide where to deploy their applications. Want to expand to a new country? You need to deploy the entire stack to the closest region. A region-aware architecture forces developers to make decisions about geography even if their product doesn't require it. However, the edge inverts this problem. There are no regions insofar as there is just one -- "Earth". When you deploy your code, it is deployed globally in minutes. The application is fast everywhere from day one, and you don't pay for unused servers. In the future, developers will have to balance yet another constraint when architecting their applications -- politics. Several countries are writing laws governing data flow. For example, China has mandated that data of Chinese citizens cannot leave the mainland. Typically, this translates to companies hosting a "China stack" siloed from the rest of the world. It doesn't require a giant leap to imagine that other countries may someday follow suit. Of course, it would be cost-prohibitive for companies to launch a new stack for each country. In a regionalized architecture, the onus is on the developer to manage the flow of data in compliance with each county's laws. Counter-intuitively, a global architecture helps developers because we can set jurisdictional boundaries at the object level. For instance, Cloudflare allows you to set [jurisdictional restrictions](https://blog.cloudflare.com/supporting-jurisdictional-restrictions-for-durable-objects/?ref=kislayverma.com) on Durable Objects that control where your data is stored. Remarkably, all of this is accomplished by specifying the jurisdictional restriction as a string, like so: ``` let id = OBJECT_NAMESPACE.newUniqueId({jurisdiction: "eu"}) ``` ### Where we are going? We keep hearing about how much new data we generate each year. But let's think about the directional flow of said data. Most data today flows from the inside-out, i.e. from the cloud to the edge. Billions of people use YouTube, Netflix, Instagram. However, most bits flow to customers consuming content. With the proliferation of IoT devices, wearables, autonomous cars, the flow of data will invert. Eventually, we'll start to see most data flow from the edge to the cloud. Because most data will be machine-generated, it won't all be useful. Instead of sending back terabytes of raw data to our application server, it will be more efficient to process data at the edge and only send post-processed data. Not only does this improve latency, because we're sending less data across the network, it will also reduce costs because we're using less network bandwidth. As edge computing becomes more mainstream, our edge devices can become smaller and cheaper. We won't need to ship devices with powerful hardware because the edge can do the heavy lifting. For example, a smart speaker can send raw audio to the edge server, which will strip out unnecessary bytes before sending the byte-stream to the app server. Cloudflare recently announced a [partnership](https://www.cloudflare.com/nvidia-workers/?ref=kislayverma.com) with Nvidia where they plan to introduce AI/ML at the edge. For use-cases like autonomous driving, the edge creates an optimized network for cars to communicate with each other. Imagine a road with hundreds of vehicles that need to talk to each other. It would be highly inefficient for the data to flow all the way back to a centralized server, only to be received by a vehicle a few feet ahead. With an edge network, data will only travel to the closest edge node, significantly reducing latency. ### Add not Subtract The rise of edge computing and programmable networks does not mean the death of the cloud as we know it. There will always be use-cases inappropriate for the edge, like training complex ML models, storing shared user data. Both paradigms will co-exist and work in tandem, much like SQL and NoSQL today. The future of edge computing looks promising and exciting. Already, we're starting to see several edge computing startups try to capitalize on the coming revolution. While we can imagine all the unique ways edge computing will change the world, I suspect that the reality will be far more surprising. ### References 1. [https://blog.cloudflare.com/serverless-performance-comparison-workers-lambda/](https://blog.cloudflare.com/serverless-performance-comparison-workers-lambda/?ref=kislayverma.com) 2. [https://www.youtube.com/watch?v=48NWaLkDcME&t=557s](https://www.youtube.com/watch?v=48NWaLkDcME&t=557s&ref=kislayverma.com) 3. [https://blog.cloudflare.com/introducing-workers-durable-objects/](https://blog.cloudflare.com/introducing-workers-durable-objects/?ref=kislayverma.com) 4. [https://en.wikipedia.org/wiki/Software-defined\_networking](https://en.wikipedia.org/wiki/Software-defined%5Fnetworking?ref=kislayverma.com) 5. [https://stratechery.com/2021/cloudflare-on-the-edge/](https://stratechery.com/2021/cloudflare-on-the-edge/?ref=kislayverma.com) 6. [https://www.cloudflare.com/en-in/press-releases/2021/cloudflare-partners-with-nvidia/](https://www.cloudflare.com/en-in/press-releases/2021/cloudflare-partners-with-nvidia/?ref=kislayverma.com) 7. [https://blog.cloudflare.com/cloudflare-workers-unleashed/](https://blog.cloudflare.com/cloudflare-workers-unleashed/?ref=kislayverma.com) 8. [https://softwarestackinvesting.com/decentralization-effects/](https://softwarestackinvesting.com/decentralization-effects/?ref=kislayverma.com) 9. [https://shopify.engineering/shopify-webassembly](https://shopify.engineering/shopify-webassembly?ref=kislayverma.com) 10. [https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime](https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime?ref=kislayverma.com) 11. [https://github.com/bytecodealliance/lucet](https://github.com/bytecodealliance/lucet?ref=kislayverma.com) 12. [https://hhhypergrowth.com/what-are-edge-networks/](https://hhhypergrowth.com/what-are-edge-networks/?ref=kislayverma.com) 13. [https://www.youtube.com/watch?v=QdWaQOgvd-g](https://www.youtube.com/watch?v=QdWaQOgvd-g&ref=kislayverma.com) **Read Next**: [The AI revolution will be unsupervised](https://kislayverma.com/the-revolution-will-be-unsupervised/) ### Uncertainty and Learning as Tech Debt URL: https://kislayverma.com/uncertainty-and-learning-as-tech-debt/ Last updated: 2026-07-22T12:46:36.000Z https://open.spotify.com/episode/046HxQEczLDgPeeZBLdY6K?si=RazsJCqNTvSPfAxxaQgYug&dl\_branch=1 ![](https://kislayverma.com/content/images/2021/07/tech-debt.png) Tech debt represents an accumulation of conscious or subconscious decisions which can now be identified as bad decisions. Basically, a shoddy job that makes taking the next steps harder. The more tech debt you have, [the harder you have to work to make new changes](https://adlrocha.substack.com/p/adlrocha-the-risks-of-technical-debt?ref=kislayverma.com). My approach to system architecture and evolution is via continuous learning and improvement. I believe that all things change all the time. So all parts of a system will carry tech debt some time or the other. The developer team's job is to continuously identify these parts, prioritize the critical deficiencies which are blocking the way forward, and consciously deepen their understanding of the system by taking on controlled tech debt if required. This article is just a few observations about this phenomenon, mostly focussed on change and uncertainty. ### Every decision is tech debt in the making - The reason we see so much tech debt, even in good teams and organizations, is usually not because of bad decisions. Decisions were probably good when they were made, but the world keeps changing. Parts of even good decisions will end up being tech debt at some point in the future. - No one designs bad systems deliberately. If a decision looks bad today, it does not matter why it was taken. Maybe the team didn't know any better, or the world changed on them. The outcome is the same either way - we need to do something about the situation. - The ideal architecture is only ideal at a point in time. Perhaps not even then because there are almost always things we do not know that might lead us to design our systems differently (Conway's Law). ### Tech debt is great when taken deliberately - A lot of discussion around technical debt is only about technology quality. It does not talk about learning. - Deliberately taking on tech debt allows us to learn faster by shipping faster. - The problem is that we forget that we made the feature as a learning step. The context of that deliberate decision is lost. So when the next iteration comes around, the decision looks like a bad decision rather than a feature in process of growing. Too often, engineers think on the lines of "we built it badly and need to fix it" and not like "which part of this system should be evolved next". - The more we remember as a team, the more we can think of tech debt in terms of WIP features rather than an already finished piece that was designed badly. ### The location of tech debt matters - The place where technical debt is seen in the overall system matters. - At the edges of the system, where the most amount of learning is happening (like customer-facing product teams), tech debt is tolerable, even somewhat desirable (as mentioned above in the context of learning). It is less tolerable in the deeper layers of the system because the deeper in the tech stack a system is, the more dependencies it has, and therefore the more damage a bad decision can do. [The more stable the core systems are, the more fearlessly we can mess about in the other places](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/). - [Platforms architecture](https://kislayverma.com/category/platform-thinking/) actually encourages this type of dual-thinking. It [establishes standards for the platform components and allows product components to do whatever they want](https://kislayverma.com/control-and-chaos-in-platform-systems/), however they want it. - This is a point in time argument - there is no "core" as such. Since all systems are always evolving, even core components will change. When that happens, we should [apply this argument to them too](https://kislayverma.com/platforms-and-dogfood-everywhere/). ### Tech debt is a balance - The trade-off isn't between speed and quality. The tradeoff is learning and executing on that learning in the long run. - I wrote earlier about [ditching the urgency](https://kislayverma.com/ditch-the-urgency/) to execute in the learning phase. This is where it is okay to take on tech debt. Build something small and fast to see what the users do with it. - Now while the user feedback is coming in and we are trying to understand it, [clean up the most undesirable of the bad decisions](https://kislayverma.com/saving-the-day-with-continuous-refactoring/) you have in the system. - BOTH THE ABOVE STEPS ARE CRITICAL. It is inevitable that we will go back and forth to some extent. ### The origins of tech debt are important - I mentioned earlier in this article that why we have a bad decision right now doesn't matter. What matters is to fix it. This is true from an operational perspective but not from a growth perspective. - For the engineering team, identifying the origin of the tech debt is a critical part of the learning process. - Looking back on their decisions, can they identify the bad decisions that they then thought were good? What led to those decisions? This will typically reveal some sort of information gap - not having enough technical skill, not having enough knowledge about the product or the customer, not understanding the direction of the organization etc. - These gaps can then be filled actively. - This is hard because teams are biased against their former selves due to the new knowledge they have gained since a decision was made. **Read Next**: [Sidestep architectural -ilities to deliver business value](https://kislayverma.com/sidestep-architectural-ilities-and-deliver-business-value/) ### Book Review and Highlights: "Accelerate" URL: https://kislayverma.com/book-review-and-highlights-accelerate/ Last updated: 2026-07-22T12:46:38.000Z Accelerate has consistently been described as one of the best books when it comes to DevOps and building technical agility in organizations. I finally got around to reading it and it was every bit as good as I had expected it to be. The book is essentially presents the conclusions of a multi-year research program and contains a whole section on why certain research methods were chosen over others. I have been reading and writing a fair bit about [agility](https://kislayverma.com/content/files/2026/07/agility.html) and [moving fast](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/), so most of the things in the book were not new, but the solid research backing it means that “CI/CD is a must-have” is not just an opinion anymore. The authors have shown it to be a demonstrable trait of successful teams in the wider industry. Accelerate is a short-ish read, but it is dense with information. I am sharing my highlights from it below so that you can get a taste of what the book is like. If you like these semi-organized snippets, you should definitely read the book. ### Preface improvements in software delivery are possible for every team and in every company, as long as leadership provides consistent support — including time, actions, and resources — demonstrating a true commitment to improvement, and as long as team members commit themselves to the work. ### Chapter 1 - Accelerate 1. Small teams that work in short cycles and measure feedback from users to build products and services that delight their customers and rapidly deliver value to their organizations. 2. [DevOps](https://kislayverma.com/content/files/2026/07/devops.html) emerged from a small number of organizations facing a wicked problem: how to build secure, resilient, rapidly evolving distributed systems at scale. 3. The Forrester report states that DevOps is accelerating technology, but that organizations often overestimate their progress (Klavens et al. 2017). Furthermore, the report points out that executives are especially prone to overestimating their progress when compared to those who are actually doing the work. 4. The key to successful change is measuring and understanding the right things with a focus on capabilities — not on maturity. 5. Maturity models are not the appropriate tool to use or mindset to have. Instead, shifting to a capabilities model of measurement is essential for organizations wanting to accelerate software delivery. 6. Three reasons capability models are better than maturity models: 1. Maturity models focus on helping an organization “ arrive ” at a mature state and then declare themselves done. Capability models focus on helping an organization continually improve and progress, realizing that the technological and business landscape is ever-changing. 2. Maturity models are quite often a “lock-step” or linear formula, prescribing a similar set of technologies, tooling, or capabilities for every set of teams and organizations to progress through. Capability models are multidimensional and dynamic, allowing different parts of the organization to take a customized approach to improvement, and focus on capabilities that will give them the most benefit based on their current context. 3. Capability models focus on key outcomes and how the capabilities, or levers, drive improvement in those outcomes — that is, they are outcome-based. Most maturity models simply measure the technical proficiency or tooling install base in an organization without tying it to outcomes. 7. Maturity models define a static level of technological, process, and organizational abilities to achieve. In contrast, capability models allow for dynamically changing environments and allow teams and organizations to focus on developing the skills and capabilities needed to remain competitive. ### Chapter 2 - Measuring Performance 1. Velocity, lines of code, and other typical technical measures focus on outputs rather than outcomes. Second, they focus on individual or local measures rather than [team or global ones](https://kislayverma.com/managing-developer-identities-in-autonomous-teams/). 2. Ideally, we should reward developers for solving business problems with the minimum amount of code — and it’s even better if we can solve a problem without writing code at all or by deleting code (perhaps by a business process change). 3. A successful measure of performance should have two key characteristics. First, it should focus on a global outcome to ensure teams aren’t pitted against each other. 4. Second, our measure should focus on outcomes not output: it shouldn’t reward people for putting in large amounts of busywork that doesn’t actually help achieve organizational goals. 5. In our search for measures of delivery performance that meet these criteria, we settled on four: 1. Delivery lead time 2. Deployment frequency 3. Time to restore service 4. Change fail rate. 6. Lead time 1. This is the time it takes to go from a customer making a request to the request being satisfied. 2. There are two parts to lead time: the time it takes to design and validate a product or feature, and the time to deliver the feature to customers. 3. In the design part of the lead time, it’s often unclear when to start the clock, and often there is high variability. 4. However, the delivery part of the lead time — the time it takes for work to be implemented, tested, and delivered — is easier to measure and has a lower variability. 7. Deployment Frequency 1. Reducing batch size is another central element of the Lean paradigm. 2. We settled on deployment frequency as a proxy for batch size since it is easy to measure and typically has low variability. By “ deployment ” we mean a software deployment to production or to an app store. 8. Delivery lead times and deployment frequency are both measures of software delivery performance tempo. The key question becomes: How quickly can service be restored (if something goes wrong)? 9. A key metric when making changes to systems is what percentage of changes to production ( including, for example, software releases and infrastructure configuration changes ) fail. In the context of Lean, this is the same as percent complete and accurate for the product delivery process, and is a key quality metric. 10. The ability to take an experimental approach to product development is highly correlated with the technical practices that contribute to continuous delivery. ### Chapter 3 - Measuring and Changing Culture 1. Organizational culture can exist at three levels in organizations: basic assumptions, values, and artifacts (Schein 1985). 2. Basic assumptions are formed over time as members of a group or organization make sense of relationships, events, and activities. 3. The second level of organizational culture are values, which are more “visible” to group members as these collective values and norms can be discussed and even debated by those who are aware of them. 4. The third level of organizational culture is the most visible and can be observed in artifacts. These artifacts can include written mission statements or creeds, technology, formal procedures, or even heroes and rituals. 5. Type of organization as defined by Ron Westrum- https://cloud.google.com/architecture/devops/devops-culture-westrum-organizational-culture 6. Culture enables information processing through three mechanisms. 1. First, in organizations with a generative culture, people collaborate more effectively and there is a higher level of trust both across the organization and up and down the hierarchy. 2. Culture emphasizes the mission, an emphasis that allows people involved to put aside their personal issues and also the departmental issues that are so evident in bureaucratic organizations. The mission is primary. 3. And third, generativity encourages a ‘level playing field’, in which hierarchy plays less of a role”. 7. We should emphasize that bureaucracy is not necessarily bad. 8. Westrum’s theory posits that organizations with better information flow function more effectively. 1. A good culture requires trust and cooperation between people across the organization, so it reflects the level of collaboration and trust inside the organization. 2. Better organizational culture can indicate higher-quality decision-making. In a team with this type of culture, not only is better information available for making decisions, but those decisions are more easily reversed. 3. Finally, teams with these cultural norms are likely to do a better job with their people, since problems are more rapidly discovered and addressed. 9. Failure in complex systems is, like other types of behavior in such systems, emergent (Perrow 2011). 10. Following the theory developed by the Lean and Agile movements, implementing the practices of these movements can have an effect on culture. You can act your way to a better culture by implementing these practices in technology organizations, just as you can in manufacturing. ### Chapter 4 - Technical Practices 1. [Continuous delivery](https://kislayverma.com/how-to-speed-up-software-delivery/) is a set of capabilities that enable us to get changes of all kinds into production or into the hands of users safely, quickly, and sustainably. 2. There are five key principles at the heart of continuous delivery: 1. Build quality in: “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place ” (Deming 2000). 2. Work in small batches: By splitting work up into much smaller chunks that deliver measurable business outcomes quickly for a small part of our target market, we get essential feedback on the work we are doing so that we can course correct. 3. Computers perform repetitive tasks; people solve problems. One important strategy to reduce the cost of pushing out changes is to take repetitive work that takes a long time, such as regression testing and software deployments, and invest in simplifying and automating this work. 4. Relentlessly pursue continuous improvement. 5. Everyone is responsible 3. In order to implement continuous delivery, we must create the following foundations: 1. Comprehensive configuration management. It should be possible to provision our environments and build, test, and deploy our software in a fully automated fashion purely from information stored in version control. 2. Continuous integration (CI): high - performing teams keep branches short-lived ( less than one day’s work ) and integrate them into trunk/master frequently. 3. Continuous testing: Because testing is so essential, we should be doing it all the time as an integral part of the development process. Automated unit and acceptance tests should be run against every commit. No one should be saying they are “ done ” with any work until all relevant automated tests have been written and are passing. 4. By giving developers the tools to detect problems when they occur, the time and resources to invest in their development, and the authority to fix problems straight away, we create an environment where developers accept responsibility for global outcomes such as quality and stability. 5. We discovered nine key capabilities that drive continuous delivery. 1. The comprehensive use of version control is relatively uncontroversial. 2. Configuration is normally considered a secondary concern to application code in configuration management, but our research shows that this is a misconception. 3. [Test automation](https://kislayverma.com/testing-strategies-for-agile-teams/) is a key part of continuous delivery. Having automated tests that are reliable: when the automated tests pass, teams are confident that their software is releasable. 4. Developers primarily create and maintain acceptance tests, and they can easily reproduce and fix them on their development workstations. It’s interesting to note that having automated tests primarily created and maintained either by QA or an outsourced party is not correlated with IT performance. 5. Successful teams had adequate test data to run their fully automated test suites and could acquire test data for running automated tests on demand. 6. Our research also found that developing off trunk/master rather than on long-lived feature branches was correlated with higher delivery performance. 7. High-performing teams were more likely to incorporate information security into the delivery process. Their infosec personnel provided feedback at every step of the software delivery lifecycle, from design through demos to helping with test automation. 8. A critical obstacle to implementing continuous delivery is enterprise and application architecture. ### Chapter 5 - Architecture 1. The architecture of your software and the services it depends on can be a significant barrier to increasing both the tempo and stability of the release process and the systems delivered. 2. We found that high performance is possible with all kinds of systems, provided that systems — and the teams that build and maintain them — are loosely coupled. 3. This reinforces the importance of focusing on the architectural characteristics, discussed below, rather than the implementation details of your architecture. 4. In teams that scored highly on architectural capabilities, little communication is required between delivery teams to get their work done, and the architecture of the system is designed to enable teams to test, deploy, and change their systems without dependencies on other teams. 5. Organizations should evolve their team and organizational structure to achieve the desired architecture. 6. The goal of a loosely coupled architecture is to ensure that the available communication bandwidth isn’t overwhelmed by fine-grained decision-making at the implementation level, so we can instead use that bandwidth for discussing higher-level shared goals and how to achieve them. 7. If we achieve a loosely coupled, well-encapsulated architecture with an organizational structure to match, two important things happen. First, we can achieve better delivery performance, increasing both tempo and stability while reducing the burnout and the pain of deployment. Second, we can substantially grow the size of our engineering organization and increase productivity linearly — or better than linearly — as we do so. 8. Architects should focus on engineers and outcomes, not tools or technologies. ### Chapter 6 - Integrating Infosec into the Delivery Lifecycle 1. Infosec is a vitally important function in an era where threats are ubiquitous and ongoing. However, infosec teams are often poorly staffed and they are usually only involved at the end of the software delivery lifecycle when it is often painful and expensive to make changes necessary to improve security. 2. We found that when teams “shift left” on information security — that is, when they build it into the software delivery process instead of making it a separate phase that happens downstream of the development process — this positively impacts their ability to practice continuous delivery. 3. First, security reviews are conducted for all major features, and this review process is performed in such a way that it doesn’t slow down the development process. ### Chapter 7 - Management Practices for Software 1. Limit work in progress (WIP), and use these limits to drive process improvement and increase throughput. 2. Create and maintain visual displays showing key quality and productivity metrics and the current status of work. 3. Use data from application performance and infrastructure monitoring tools to make business decisions on a daily basis. 4. WIP limits on their own do not strongly predict delivery performance. It’s only when they’re combined with the use of visual displays and have a feedback loop from production monitoring tools back to delivery teams or the business that we see a strong effect. 5. WIP limits are no good if they don’t lead to improvements that increase flow. 6. Implement a lightweight change management process. 1. We found that approval only for high-risk changes was not correlated with software delivery performance. 2. Approval by an external body ( such as a manager or CAB ) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. ### Chapter 8 - Product Development 1. The key to working in small batches is to have work decomposed into features that allow for rapid development, instead of complex features developed on branches and released infrequently. 2. The ability of teams to try out new ideas and create and update specifications during the development process, without requiring the approval of people outside the team, is an important factor in predicting organizational performance as measured in terms of profitability, productivity, and market share. ### Chapter 9 - Making Work Sustainable 1. The technical practices that improve our ability to deliver software with both speed and stability also reduce the stress and anxiety associated with pushing code to production. 2. In order to reduce [deployment pain](https://kislayverma.com/why-and-how-to-use-feature-toggles/), we should: 1. Build systems that are designed to be deployed easily into multiple environments, can detect and tolerate failures in their environments, and can have various components of the system updated independently. 2. Ensure that the state of production systems can be reproduced (with the exception of production data) in an automated fashion from information in version control. 3. Build intelligence into the application and the platform so that the deployment process can be as simple as possible. 3. Christina Maslach, a professor of psychology at the University of California at Berkeley and a pioneering researcher on job burnout, found six organizational risk factors that predict burnout (Leiter and Maslach 2008): 1. Work overload 2. Lack of control 3. Insufficient rewards 4. Breakdown of community 5. Absence of fairness 6. Value conflicts ### Chapter 10 - Employee Satisfaction, Identity, and Engagement 1. Employees in high-performing teams were 2.2 times more likely to recommend their organization to a friend as a great place to work, and 1.8 times more likely to recommend their team to a friend. 2. We found that the employee Net Promoter Score was significantly correlated with the following constructs: 1. The extent to which the organization collects customer feedback and uses it to inform the design of products and features 2. The ability of teams to visualize and understand the flow of products or features through development all the way to the customer 3. The extent to which employees identify with their organization’s values 3. investments in continuous delivery and Lean management practices, which contribute to a stronger sense of identity, may very well help reduce burnout. 4. Being able to apply one’s judgment and experience to challenging problems is a big part of what makes people satisfied with their work. ### Chapter 11 - Leaders and Managers 1. Being a leader doesn’t mean you have people reporting to you on an organizational chart — leadership is about inspiring and motivating those around you. 2. According to this model (Rafferty and Griffin 2004), the five characteristics of a transformational leader are: 1. Vision 2. Inspirational communication 3. Intellectual stimulation 4. Supportive leadership 5. Personal recognition 3. Transformational leadership means leaders inspiring and motivating followers to achieve higher performance by appealing to their values and sense of purpose, facilitating wide-scale organizational change. 4. A transformational leader’s influence is seen through their support of their teams ’ work, be that in technical practices or product management capabilities. 5. ...leaders cannot achieve goals on their own. 6. As the real value of a leader or manager is manifest in how they amplify the work of their teams, perhaps the most valuable work they can do is growing and supporting a strong organizational culture among those they serve - their teams. 7. Enable cross-functional collaboration by: 1. Building trust with your counterparts on other teams. 2. Encouraging practitioners to move between departments. 3. Actively seeking, encouraging, and rewarding work that facilitates collaboration. **Read Next**: [Summary of "97 things every software architect should know"](https://kislayverma.com/highlights-97-things-every-software-architect-should-know/) ### The sense of purpose in a complex system URL: https://kislayverma.com/the-sense-of-purpose-in-a-complex-system/ Last updated: 2026-07-22T12:46:39.000Z https://open.spotify.com/show/5gY1eGUE0RmNHj5t1HKpJI ![](https://kislayverma.com/content/images/2021/06/System-boundary-environment.jpg) A few weeks ago, I wrote about how we should [design any system from one logical level up](https://kislayverma.com/system-design-from-one-level-up/), i.e. considering the environment of our system. A Redditor offered an interesting comment that this approach was contrary to how anything natural evolved. Natural evolution is always bottoms up and that seems a lot more flexible. https://www.reddit.com/r/programming/comments/o6awmh/system\_design\_should\_consider\_the\_larger\_system/h2rxjmo This is certainly a valid observation and it got me thinking. Natural evolution happens bottom-up with everything co-evolving at the same time. Why should we not define and build the lowest levels first? ### The purpose of a system I have come to the conclusion that the difference is one of “purpose”. Purpose, in complex systems, is a decentralized thing, its shape differing from actor to actor. But the goal is always the same - an actor acts a certain way or pursues a certain goal because that, according to their limited mental model of the system, will allow them to perform better in their environment. The existence of a *purpose* implies the existence of one or more actors. For nature to have a purpose requires something to exist outside of nature. This supernatural entity would be contending with a supernatural environment. Without getting onto divine turf, a simpler conclusion is that nature has no definite purpose. Natural systems evolve without a sense of purpose or definite objective. Every element of the natural system tries to perpetuate itself in a changing environment by adapting. The system, on the whole, tries to attain a stable equilibrium regardless of what the equilibrium looks like. Any stable state will do - nature has no opinion on the quality of the outcome. It is acceptable in the natural world for entire evolutionary hierarchies to collapse if they [no longer fit in with the environment](https://pigontracks.substack.com/p/8-no-i-cant-give-you-certainty?ref=kislayverma.com). In man-made systems like large organizations or software architectures, we are not quite as generous. These systems exist to fulfill a certain role, and we intervene in them with clear intent. This is the [essence of strategy](https://kislayverma.com/book-review-wardley-mapping/) \- defining goals and the action needed to achieve these goals. Evolution in man-made systems doesn’t run rampant, it is constrained to proceed in the directions which potentially lead to the outcomes we want - at least as far as we can tell at the moment. We want to minimize large failures that would result from completely uninformed trial and error. This is why system design should be done from one level up. First, we [visualize the effect we want to create](https://kislayverma.com/so-you-want-to-privatize-a-bank/) (our purpose), and then we take an action that is likely to attain that objective. In man ade systems, this is evolution. This is what makes [building shared context in teams](https://kislayverma.com/the-problem-is-not-the-problem/) so critical. A shared understanding leads to shared motivation and intent of action. The more actors share the same mental model, the more likely it is that an action can be pulled off successfully. Building the shared context is akin to extinction in the natural world, we are pruning those paths of evolution which might lead to an unsatisfactory equilibrium for us as a team. This is the “purpose” of the team. Hence the modern insistence on aligning engineering teams with top-level business objectives. If the teams are aligned to the organization’s global “purpose”, they are less likely to be f[ocussed on *local maxima*](https://kislayverma.com/independence-autonomy-and-too-many-small-teams/) when they operate. ### Everything is bottom up This explanation, though practically useful, is philosophically misleading. Human and natural systems are not disconnected. All in all, the Redditor was right. Everything happens bottom-up. Even that is incorrect. Better to say that everything happens all at once. Every component of a system, whether above or below, reacts to changes around it. The confusion only exists because we subconsciously draw a logical boundary that defines up or down. It seems like we are building top-down because we are looking at only a part of the system to make a tactical decision. As an architect, I design from one level up because that allows me to impress my intent upon that specific neighborhood of the software system. I first define a boundary (consciously or subconsciously) inside which I want to take top-down action. Now I can be system-minded and look just outside the boundary to visualize what might be going on “outside”. The act of defining that scope gives the illusion of control, top-down action, and purpose. But seen from the outside, I am just another actor responding to my environment and motivations. This is the essence of Conway’s Law - regardless of my intentions, there are things I do not know and hence alternative actions that I cannot take. On the grand cosmic scale, we are all just doing the best we can to improve our situation in an un-opinionated universe. And that is good. **Read Next**: [Control and chaos in platform systems](https://kislayverma.com/control-and-chaos-in-platform-systems/) ### Summary: Lectures by Dr. Russell Ackoff URL: https://kislayverma.com/summary-lectures-by-dr-russell-ackoff/ Last updated: 2026-07-22T12:46:40.000Z I have recently started getting interested in [systems thinking](https://kislayverma.com/content/files/2026/07/systems-thinking.html). I got started in this field by [Donella Meadows’ “Thinking in System”](https://kislayverma.com/book-review-thinking-in-systems-a-primer/) which was great. And a few weeks ago I discovered Dr. Russell Ackoff (Hat tip to [Trond Hjorteland](https://twitter.com/trondhjort?ref=kislayverma.com) for that). His genius for telling stories and correlating these funny stories back to a systemic view of the world is as good as anything I have ever come across. There are a lot of videos of him explaining systems and management etc on Youtube, but like a lot of old-time proponents of a specific topic, many of them carry similar explanations, examples, and deductions. So I thought I will summarize and condense a few of the most “academic” of these videos to bring out the common and prominent ideas. I absolutely recommend that you listen to the original lectures because these notes do not convey the richness of thought, expression, and experience that the lectures contain. I only hope to give you a quick boost in terms of ideas before you have to decide to commit \~4 hours of time. I have spent \~20 hours over the last two weeks listening to Russell Ackoff and I do not regret it one bit. > A system is a whole that is defined by its function(s) in a larger system of which it is a part and that consists of at least 2 parts without which it cannot fulfill its defining function --- 1. **Ways of solving a problem**: 1. Absolution: Ignore the problem and hope it will go away 2. Resolution: Solving the problem based on prior experience and qualitative judgment. This is "satisficing" - doing enough that is better than nothing. But this can cause further problems which are often more complicated than the original problem. 3. Dissolution: Redesign the system to remove the problem 2. A system is a whole that is defined by its function(s) in a larger system of which it is a part and that consists of at least 2 parts without which it cannot fulfill its defining function 3. The essential parts of a system must satisfy 3 conditions: 1. Each essential part can affect the behaviour or properties of the whole 2. No essential part has an independent effect on the whole 3. Each subset of parts can have an effect on the whole but not an independent effect. 4. The essential characteristics of a system depend on how its part interact, not on how they act taken separately. 1. No part of a system taken independently can perform the function of the whole 2. The performance of a system is not necessarily improved when the performance of its parts taken separately, is. 5. We understand how the parts of a system interact by the process of *design*. Through *idealized design,* we understand how the system *ought to* behave. 6. **Why leadership courses are useless** 1. Leadership is an art and a talent - can't be taught 2. The difference in different roles when leadership is mentioned: 1. Administration: Direct others in pursuit of goal using some means where goals and means are both selected by a third party 2. Management: Direct others in pursuit of goal using some means where goals and means are both selected by the manager 3. Leadership: Guiding and encouraging others in pursuit of goal using some means where goals and means are both selected by them 3. Leadership requires the ability to bring the will of others into consonance with the will of the leader so that they follow voluntarily. It is inspiration, not persuasion. 4. The vision is the *idealized design* produced by the leader. 1. **This whole conversation is tinted with the idea of a charismatic leader instead of bottom-up leadership**. But it also somewhat aligns with what I wrote about building shared context and maybe that’s best done by some people who are the leaders. 7. **Why transformations fail** 1. Transformation requires the intelligence to identify a problem and the courage to do something about it 2. Two types of errors: 1. errors of commission: doing something wring 2. errors of omission: not doing something that should have been done. 3. In most systems of accountability, only errors of commission are registered. 4. Hence, rational people choose to not pursue change. 8. **Panaceas**: 1. The righter one does the wrong thing, the wronger you become! 2. Some of the deficiencies of Panaceas: 1. Management should be directed at what we want instead of what we don't want. 1. Focus on quality of output instead of the quality of work-life for workers 2. Ignorance of consumer wants 1. Wants have to be discovered by the process of design - **software architects need to know this** 3. Continuous improvement cannot keep up with step jumps 4. Process Re-engineering 1. Focusing on a different kind of slice of the system rather than the whole system - hence anti-systemic and ineffective 5. Downsizing 1. The purpose of an organization is to create and distribute wealth. Hence downsizing is an immoral act 2. De-bureaucratize and de-monopolize internal loss-making units 6. Benchmarking of parts: anti-systemic. Benchmarking of the whole is what competition is. 1. Benchmarking against competition -> continuous improvements -> we give up the opportunity to ideally design what we want. 2. We also set the competition as the gold standard. 9. **Creativity** 1. Every creative process has three steps: 1. Identify an assumption that is limiting the choices that can be explored 2. Remove the assumption 3. Examine and utilize the new landscape of choice now revealed 2. Principles of creativity (These are more tricks to solving problems creatively in a corporate environment) 1. Deny the "facts of the case" and find them out for yourself 2. Remove externally imposed constraints 3. Influence those who cannot be controlled 4. Enlarge the system 5. Role reversal by using the source of the problem as the solution 10. **Other thoughts about organizations** 1. Eliminate job descriptions: 1. They are limiting 2. Get in good people, put them in an area/department, and ask them to do what they think needs to be done. 3. Provide guidance and keep discussing what and how they are going to do. 2. Salary shouldn't be limited to status 1. Don't create managers for status, increase compensation as needed 2. Pay what the employee is worth, status/role is incidental 3. Fun 1. Fun = self determination 2. Let people find out what they want to do 4. Management is not a profession, it is a form of employment. 1. Professions have standards to which professionals owe their highest obligation. E.g. Hippocratic oath for doctors 2. Employees owe their highest obligation to the good of the organization 5. The mission statement of a team/organization has to be a deliberately designed expression. If the inverse of the statement is not logically viable, then the mission statement is unlikely to be instructive 1. e.g. "we want to provide superior returns to our shareholders" is meaningless because the inverse of this doesn't make sense as a goal. Hence this statement does not inform action. **Read Next**: [Summary of "Working with the CAP theorem" by Eric Brewer](https://kislayverma.com/working-around-the-cap-theorem/) ### So you want to privatize a bank? URL: https://kislayverma.com/so-you-want-to-privatize-a-bank/ Last updated: 2026-07-22T12:46:41.000Z https://open.spotify.com/episode/547qXp5SJPu1VEdi3mPP3w?si=858f3a635d8e4751 I was recently listening to my friends argue about whether to privatize govt banks or not. Some argued that this will improve efficiency, some wondered if private corporations can be trusted with all the banking in the country, some debated socialism and its pros and cons. ![](https://kislayverma.com/content/images/2021/06/dmitry-demidko-eBWzFKahEaU-unsplash.jpg) Photo by [Dmitry Demidko](https://unsplash.com/@wildbook?utm%5Fsource=unsplash&utm%5Fmedium=referral&utm%5Fcontent=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/bank?utm%5Fsource=unsplash&utm%5Fmedium=referral&utm%5Fcontent=creditCopyText) This is a common discussion pattern in both formal and informal settings. A lot of public opinions are framed this way (e.g. rapists should be summarily hung, but what about humanitarian treatment or legal rights) so is a lot of tech hype (e.g. NoSQL should be used for high scalability, but is it too complicated or do we have the tools). One thing that really struck me is that how this discussion is actually upside down. It assumes that a course of action has been determined, and now all that is left to do is to debate whether it is acceptable to everyone or not. The conversation on privatizing national banks presents an opinion like a strategy to attain an undefined goal. The argument is not about the privatization of banks at all! People talk past each other because each of them is talking about a different thing like corruption, socialism, bad loans, etc. These things just cloud the issue and keep us from the actual discussion. To actually discuss whether banks should actually be privatized or not, we would have to define the effect we are trying to create. Why have the conversation at all? All the other words are about the side effects and symptoms of an action we have presumably already taken (privatize the banks), not about "why" we took the action. The real question is "what do we want to achieve". Economics and banks are elements of a system that does something, i.e. has a purpose. We need to define this purpose before we can define if a proposed change serves this purpose. Once we understand the purpose, we can try to navigate the environment and achieve it. Without knowing the purpose, "privatize the bank" means nothing. This should be familiar from the way a lot of inter-team presentations are done. A decision is presented for discussion and critique, but with the understanding that fundamental questions are not to be raised and any feedback should be given in the context of the conclusion being presented. We look at the proposal, debate whether it is good or bad, and propose changes that may improve the proposal. But for this critique to be useful, the feedback has to consider the purpose of the system and the change first, and that sadly does not happen very often. Let's consider some technical examples of this. Decoupling is one of my favourites. The statement "decouple your components" is unlikely to engender debate, or at best engender debate on the mechanism of decoupling. It is a great idea, but the bigger question lies in the effect that we want to create? What exactly is it that I want to get out of this? Do I really want to change one of my components? The principle of decoupling isn't valuable by itself, it needs the context of motive to become so. Apple, for example, doesn't decouple. It integrates deeply. Or take autonomy in teams. At this point, the Twitter-going crowd believes that [autonomous teams](https://kislayverma.com/independence-autonomy-and-too-many-small-teams/) (two-pizza, problem-driven, etc) are the best way to organize a workforce. But why do I want this? Someone told me that engineers like this because they like "freedom". Freedom from what? Who knows. If the intended effect is to efficiently use a large, unmotivated workforce, maybe autonomy isn't the way to go. Old-school command and control might be best. The words "decouple" and "autonomy" are trying to use the perceived authority of best practices without identifying the context of applying these ideas. The point is not whether they are good or bad, but that we haven't thought about the objective, and therefore must work backward from solution to the problem and hope they fit each other. We don't even know what the problem is in the larger context. Starting from the intended effect "I want to reduce the cost of change in my software" might lead us to consider that change is faster without decoupling if deep integrations are required and swapping out components is not necessary. Starting from the effect "I want my teams to create high impact" might leads us to consider that before teams can create impact, they should understand which direction the organization wants to go in and therefore what "impact" looks like. That alignment must come before autonomy can be unleashed. Applying a strategy by looking at one aspect reverses cause and effect. Decoupling doesn't axiomatically improve software. Autonomy doesn't automatically create impact. Without knowing the context and intent, a strategy can only succeed by chance. It is more likely to cause damage via second-order effects. So if you want to take action, first identify the intended effect and the environment in which you are working. All else will follow from that. **Read next**: [Preventing go-around architecture with platform thinking](https://kislayverma.com/preventing-go-around-with-platform-thinking/) ### System design from one level up URL: https://kislayverma.com/system-design-from-one-level-up/ Last updated: 2026-07-22T12:46:42.000Z https://open.spotify.com/episode/3H1mZpqlwyYcybQF05sxAl?si=tQMHkLF0SsuIh0-U\_dBpPQ&dl\_branch=1 Given some requirements for a system, how should one start designing it? --- Before answering that question, let's first think about how one understands an existing system. A typical approach is to first understand the boundary of the system. Inside the boundary is the system, outside the boundary, is the environment. The boundary separates the system from its environment and therefore in a way, defines what the system is. Like a fence with a door, the boundary has gates that allow well-defined interactions between the environment and the system. Both the boundary and the internals need to be designed well. ![](https://kislayverma.com/content/images/2021/06/System-boundary-environment.jpg) In an existing system, we can look at the internals of the system and tell what each part does. We can say that MySQL has been chosen because transactionality is required (or the other way round - why would they use MySQL if they didn't need transactionality) and so on. But we cannot look at the edges of the system and explain "why" they do what they do. Understanding the design of the boundary requires us to understand the environment in which it was built. This is the difference between what Russell Ackoff calls [know-how and information](https://www.youtube.com/watch?v=spm2HUxgI30&t=1s&ref=kislayverma.com). I can tell how the system does what it does by looking at it in isolation, but only by looking at its environment can I tell *why* it does what it does. Documentation, ADRs, tribal knowledge, etc. are all tools for creating this environmental context. Unit and integration tests are not because they only verify the internals of the system and not its objectives. Therefore, it only makes sense to establish the environment for a system and identify its role in that environment before we figure out how to make it work. In other words, start with why. When given a system to build, I treat it like a component of a larger system and go one level higher to ask "what role does my component play in the larger system" or "who will interact with this component". Sometimes this answer is simple because it plays a minor role in the larger system. Sometimes, it plays a major role, and defining the environment (i.e. the larger system) initially lets me understand the role of my component a little better. The clearer the environment and the other subsystems are, the more confidence I can have that I have considered all the roles my component is required to play. In practical terms, documenting this process turns out like the [ narrative approach to software design](https://kislayverma.com/a-narrative-approach-to-software-design/) I have outlined elsewhere on this blog. ![](https://kislayverma.com/content/images/2021/06/system-and-components.jpg) All this talk about edges and systems is not just philosophy. It helps in establishing the degree and dimensions of uncertainty, and this, in turn, has significant consequences on the design choices we make. If the role of the component in the environment is an experimental one, we should probably not harden the boundaries of our system too much just yet. We can go for simpler implementation and design internally. We should even be ready to throw away the component as the environment evolves. If we ignore the uncertainty in the environment, I might harden the interfaces of my component too early and get stuck with an inflexible component that is difficult to evolve. On the other hand, if the environment has well-defined expectations from my component ("Requirements are clear" in developer parlance), I choose to build well-defined interfaces from the get-go and go for a more robust internal implementation. In this case, leaving ambiguity in the interfaces would only cause confusion later. Maintainers might imagine uses for these flexible interfaces which are not needed or intended in practice. So we identify the shape of the system first before we start colouring between the lines. We can now start breaking down the walls of the component to design its internal parts. In fact, this is the same process we will follow for building the component. We identify the purpose of the component and then create sub-parts that collectively serve this overarching purpose. The component is the environment and each internal piece is a system to be built. We should apply the same principle to the original requirement specification. ![](https://kislayverma.com/content/images/2021/06/system-system-system.jpg) Requirements do not arise in isolation. The very fact there are "requirements" indicates that there is a purpose, and that purpose is created by something external to or "above" the requirement. Going one step higher to understand the environment will help us design a system that is much better suited to the context of the system. **Read Next**: [More than testing, unit tests help in system design](https://kislayverma.com/more-than-testing-writing-unit-tests-for-better-design/) ### Combining rule-systems and machine learning URL: https://kislayverma.com/combining-rule-systems-and-machine-learning/ Last updated: 2026-07-22T12:46:42.000Z I was recently reading an [article by Neal Lathia](https://nlathia.github.io/2020/10/ML-and-rule-engines.html?ref=kislayverma.com) about how using machine learning is not always necessarily better than using rule-based systems. There are pros and cons to taking either approach, and you should take the one which suits the problem complexity, expected execution speed, and various other factors the best. --- I am working on a project which I and my team believe is a great fit for applying ML techniques. However, we have also achieved some measure of success in achieving our goals using a rule-based system. Without sharing the details, I can say that while we have a long way to go, one thing that is increasingly clear is that rule-based and AI/ML-based approaches to building systems are not mutually exclusive. There are various ways in which these can be applied together effectively. I want to discuss some of these ideas. The thing to remember is that every system is made up of many parts and each of them serves a different function. Most parts are simple and require little to no intelligence in the [machine learning](https://kislayverma.com/content/files/2026/07/machine-learning.html) sense. So it only makes sense to slice the problem and the overall system into smaller parts and apply the technique most suited to it. The parts that require inferences made from lots of data can be built using machine learning techniques, and other parts can be built using rules or plain old application engineering. In fact, acknowledging that these two types of parts exist in a system can have a dramatic impact on the architecture and technology choices being made. ### Rule System output as ML feature Machine Learning models work on sets of inputs called *features*. Features might already exist as first-class artifacts in some data store, or as is often the case, they are attributes derived from multiple other data points of the system. These derivations are often built using simple rules or heuristics and then consumed by ML models. Both steps of the process operate independently but play an important role in the final outcome. Think of this as lower-level staff processing raw information into reports that can be consumed by upper management to make complex decisions. ### ML Model output as input to rule engine The inverse of the above process is also common. We can have ML models use various features to come to a conclusion which is then used as one of the inputs to a rule-based system. This again works by splicing a complicated into two parts - the more intelligent/complex part uses ML to process complex data patterns. Once the data is reduced to a simple conclusion, the rule-based system can jump in to make further, simpler decisions. Think of this as upper management passing down the results of complex strategic analysis where the lower layer can make comparatively simpler decisions about how to execute things. ### Rules as elements of AI This last pattern of combining rule and AI/ML is especially fascinating for me because it blurs the lines of what is simple and what is complex. A whole class of AI-based systems called [Learning Classifier Systems](https://en.wikipedia.org/wiki/Learning%5Fclassifier%5Fsystem?ref=kislayverma.com) (probably others too, I'm no expert) use rules as the building blocks of complex evolutionary hierarchies where rules mutate and evolve in such a way that we finally end up with rules that best fit the given application environment. This is a complex way of applying either model but it shows that there is sufficient common ground between them. Think of this as a brainstorming session with all levels of the company (autonomous team? 5 person startup?) where simple ideas are churned around till strategies and execution plans emerge together. --- Since this is a real-life problem for me at work, I want to hear what you think about it and if you have used such a combination with good results. Drop a note in the comments section with your experiences. Read Next: [What does this decade's full stack team look like](https://kislayverma.com/the-full-stack-team-of-this-decade/) ? ### The problem is not the problem URL: https://kislayverma.com/the-problem-is-not-the-problem/ Last updated: 2026-07-22T12:46:43.000Z > *You see a problem at work and you think you have a solution. Some people you have spoken to in hallways agree with you. So you go ahead and try to solve it but suddenly start getting pushback from everywhere and eventually things don't work out as you had thought they would. You are left bitter and frustrated at "them".* How many of you have been in this situation? I know I have, and today I want to talk about what I have learned from those failures. This is an idealized version of how I now think about orchestrating high-impact changes. There are many "real-world" details that you will have to fill in based on yourself, your team, and your organization. As an engineer, I have often had a somewhat reductive point of view of the [organization's problems](https://kislayverma.com/category/organizations/). My mind jumps straight to what system can I build to solve a problem*.* This tech-centric perspective told me that those who couldn’t see that building this type of system will solve the problem just “didn’t get it”. Sometimes, I believed that even if I couldn’t convince “them” (my manager, my PdM, other engineers in my team, etc), I would do it my way and they would thank me later. ### DON’T DO IT! This is the single worst thing you can do in a team. Working in a team *successfully* implies having the ability to convince your team and stakeholders about why you see a problem a certain way and why you think a certain solution will work. The kind of behaviour I just spoke about is a direct refusal to engage with the system around you. If your team saw you doing this, you will never regain their trust because not only did you fail to convince them, you overrode their disagreement and did something for which they now share accountability. I’ll share some systemic reasons why you are wrong to act like this in a minute. But there are practical reasons too for why the *lone ranger* act is unlikely to work. Software engineering is a team sport. Even if you go off and do something your own way, who will make your solution work on the ground? The people you did not convince? The manager you did not get buy-in from? ### Good intentions versus Systems Thinking [Systems thinking](https://kislayverma.com/content/files/2026/07/systems-thinking-1.html) tells us that all actors in a system have different mental models of the system. Donella Meadows calls this "Bounded Rationality" in her book "[Thinking in Systems](https://kislayverma.com/book-review-thinking-in-systems-a-primer/)". Each actor has their own perspective, problems, and priorities. While the symptoms may be visible to everyone to some extent, their interpretation of underlying causes can vary wildly. This is one of the reasons "root cause analysis" is sometimes a fallacy - there may not be one root cause. Often, problems are caused by actors working on local information for local goals. Replace "actors" with departments or employees and you get a reasonable picture of how organizations work. ![](https://kislayverma.com/content/images/2021/03/blind-men-and-multiple-readmodels.jpeg) The solution, especially for large or complex problems, isn't often an objective truth that can be shown to be true or correct regardless of perspective. If a solution involves multiple actors, we are trying to change the way all of them view and interact with their world. We can do this in a command-and-control way but this takes away the agency of the actors. No one likes to be told to do something "because I said so". This can easily cause "Policy Resistance" - actors resisting or circumventing a central directive. Here's a compilation of some of the [most hilarious backfires](https://twitter.com/TrungTPhan/status/1396500898742824960?ref=kislayverma.com) of this model of working. This is what you are up against in trying to "get in and do it myself”. Let’s think about the problem. Is it even a problem? It may be, but my particular framing of it reflects only my interpretation of the aspect of the problem that is visible to me. Perhaps whatever-it-is is only a problem as perceived from a technology standpoint ("we are no longer a tech company, what should we do"?). Unless I have exposure to all viewpoints and information (practically impossible in an organization), my understanding of the problem could be biased or incomplete. Someone from the business teams could have a completely different point of view on the same visible symptom. What's ironic is that we could both be right! The other problem is that this POV assumes that all solutions are technical and I am the centre of the world. This is obviously not correct. In most organizations, technology is only one part of the landscape and often a small one. Until we surrender our vantage and ego, we are unlikely to see the full shape of the world. ### Build a shared context and worldview > *If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea* > > \-Antoine de Saint-Exupery A better way of creating change is to look at the system collaboratively from multiple actors’ perspectives and build a shared understanding/context of what it looks like. This goes beyond convincing or getting buy-in on my perspective. Invite collaboration by getting all involved actors to come together and build a common world. This new, shared worldview will likely be far richer than any single actor’s view, and the process of constructing it will start a dialog where problems and solutions are easier to discover and discuss. > The problem, therefore, is not the problem. Building a shared view of the system - that is the problem. In this process, my perspective can serve as the basis out of which the discussion grows. This suggestion is based on Tim Casasola’s suggestion of building [containers for collaboration](https://theoverlap.substack.com/p/containers?ref=kislayverma.com). We want a collective framing of the environment and the problem, but having a starting point helps. Scoping the audience in these discussions is critical. The group should be as small as possible to cover all viewpoints needed. This process is about balance - not ivory tower, and not a committee. Every member of the group should be directly involved in the resulting decision or directly impacted by it. Anyone not directly impacted is a consultant who may inform the group but has no part to play in it. e.g. If the problem is "our systems have too many outages", including marketing in the conversation has no purpose - keeping it limited to technology and operations should suffice. ### From problem to solution This is where I have made the most mistakes in my career. Even in situations where I and my team agreed on the problem, I used this agreement as a vindication of my original perspective and reverted to acting on my solution. It was "ah everyone agreed, so now let's get to work now". The obvious problem here is that the shared context is abandoned and we go back to "Let me tell you what to do" mode. Given that a collective understanding exists, it is especially regressive for me to push my perspective all over again. A far better way is to simply continue the dialog now in the direction of solutions. Very often, the process of building the shared context will reveal solutions (often multiple) organically. Essentially rinse repeat till the team converges on a solution. Of course, it can be that there is no single solution acceptable to everyone. We can use compromises, or multiple limited solutions, or continue to disagree forever. Any way we choose, the key thing is that everything from now on happens with a collective agreement. The one thing we want to avoid is going rogue if the conclusion isn't to our liking. As I started out by saying, "against the tide" efforts are unlikely to succeed and erode the team's trust forever. ### What should we do next? Nothing stands still, ever. A shared perspective is not a static thing. Every action we take changes the environment, so to maintain our situational awareness, we need to keep the dialog going. With the dialog constant, the worldview is constantly updated and new possibilities are constantly explored. The next steps become organic increments rather than big bang “quarterly plan” type efforts. Since this shared understanding is the core of how a team functions, the context IS the team. If the worldview changes significantly, a whole new type of team may be required. In any case, the key is to keep talking and learning. ### Why doesn't it happen often? This idealized process sounds good enough that you’d expect some variant of it to be attempted often enough. - The process is not very efficient as a one-time activity to deal with some specific situation. As I said before, the dialog is a flywheel. If it isn’t kept alive, then you might as well not do it. - It is extremely hard for intelligent people to surrender their POVs. Ironically, the more passionate each of them is about the problem, the harder it gets to contribute constructively to the collective. - The idea of including "everyone involved" leads to involving so many people that building a shared context goes from being difficult to being impossible. - Often, there are bad-faith actors or power-play situations. The participants are not even interested in the intent of the dialog - only in extracting some other types of outcome. **Read next**: [Reduce the need to collaborate by using good desgin](https://kislayverma.com/reduce-collaboration-by-good-design/) ### How to organize your code? URL: https://kislayverma.com/how-to-organize-your-code/ Last updated: 2026-07-22T12:46:44.000Z What is the most popular style of arranging code you have come across in enterprise codebases? The one I have seen most often groups all classes (assuming Java-land) by the layer in the tech stack. So in an MVC style system, all controllers are together, all services are together, all repositories are together, all POJOs are together etc. Let's call this convention the **"stack" style of organizing code**. ![](https://kislayverma.com/content/images/2021/06/organizing-code-stack-style.png) This is a terrible way of organizing code and I will explain why below. But first, allow me to offer the alternative. A much better way of organizing code to group it by the logical entities it represents. Let's call this the **"entity" style of organizing code**. The idea is to make sure that all classes related to a single concept stay together. By putting the logical entities first, we are optimizing for human comprehension (compilers don’t care where you put which class). By virtue of how the code presents itself, developers are nudged to make smarter choices about where the actual system boundaries lie. Not between *SomethingRepository* and *SomethingElseRepository*, but between *Something* and *SomethingElse* as concepts. ![](https://kislayverma.com/content/images/2021/06/organizing-code-entity-style.png) Now let's understand why I think the entity model is better than the stack model. ### Improper abstraction People don't read code by layers of the stack. No one ever says "show me all the APIs of this system" or "give me all the queries being fired by this system". People read code along domain boundaries. In a hotel management system, people think about rooms, and guests, and prices, and so on. Since “stack” style code is organized along technology layers, it is difficult to understand the logical model of the system from the way it lives in the repository. The boundaries the "stack" style exposes are technical layers. We cannot understand the "nouns" and the relationships between them from this code. You have to dig one level deeper for that. For a new person reading the code, this obfuscation of "logical" structure is a huge source of friction. In our hotel management example, the “entity style puts all code related to guests (regardless of the technical layer) into one package, all code related to rooms goes into another, and so on. Each of these packages can have its own internal organization in the "stack" style or just a few classes all at the same level. This makes it easy to find everything related to guest in one place. ### Poor cohesion Another common argument given for the “stack” style of arrangement is that it puts separate modules at different layers of the tech stack. e.g. Controllers are visibly separated from service, service from repositories etc. To find classes at different levels of the tech stack, you need to go to packages representing those levels. This encourages decoupling between the different layers. ![](https://kislayverma.com/content/images/2021/06/poor-cohesion-in-stack-style.png) The problem with this argument is that it focuses on coupling but disregards the other critical property - cohesion. Between what classes do we want to increase cohesion and which do we want to decrease coupling? Since all services are located together, can we say it is okay for them to be highly cohesive but decoupled from their model classes or repositories? Can we allow all repositories to become highly reliant on each other but decoupled from the business logic of the service layer? The obvious answer is NO! This kind of code would be textbook big-ball-of-mud. Refactoring such a system into smaller systems would be an absolute nightmare because you would have to decouple classes at every layer of the tech stack. It defeats the whole purpose of using an MVC style. The “entity” style, OTOH, promotes cohesion while still leaving room for tech stack style decoupling. It is okay if all hotel-related classes depend on each other (technically or conceptually) since they form a single unit of work anyway. It also makes future refactoring easier because the logical boundaries are clearer than in the "stack" style. ![](https://kislayverma.com/content/images/2021/06/high-cohesion-in-entity-style.png) ### Hard to change To make any meaningful change in a codebase organized in "stack" style, a developer has to cut across multiple packages. e.g. to add a new field to an entity and its CRUD API, all packages will be modified. This creates cognitive load because the developer has to modify many "things" rather than a single logical thing. In the “entity”, if you change a thing, you make changes only in one logical boundary. This makes changes to them easier because we are working only in a small part of the codebase if working with a single entity. If you cut across top-level packages, you are cutting across logical constructs by definition and this will alert you to potential coupling-related considerations. ### Limits design choices Since code is organized by tech stack or functionality, it limits the way people think about system design. e.g. Since business logic should go into "services", developers resist using proper design constructs and would rather shove everything inside services thereby creating nightmare classes thousands of lines long. Even when they use good design principles, the organization of the code resists them because every new "type" has to be in a unique package. If I want to use the factory pattern in different services, then I have to invent a whole new package hierarchy called *factory* and henceforth all factories should go there whether or not they have anything to do with each other. As I mentioned earlier, the “entity” makes no assumptions about how each logical package is grouped internally. It can be in the stack style, or have as many types of packages as required without influencing the choices made in another entity’s package. ![](https://kislayverma.com/content/images/2021/06/design-freedom-in-entity-style.png) One concern here can be about how to organize things that span across entities. e.g. workflows operating on multiple entities. Neither style has a neat answer to this, but IMO the “entity” style does a better job at it since it forces the creation of a new package outside all entity packages. This highlights that a workflow is a new concept and potentially a system boundary that should be developed independently. The idea is to group similar concepts together, but things not bound to a single concept can still have their own logical homes in the base. --- The modes of thinking that code organization promotes is something I feel we don't think about enough. This is similar to Conway's law at codebase level. I'd love to hear more from you about how you organize your code sand how you think it shapes developer behaviour, mental models, or efficiency. Drop a note in the comments! **Read Next**: [Unit testing is a tool for designing, not merely testing](https://kislayverma.com/more-than-testing-writing-unit-tests-for-better-design/) ### How big should a method be? URL: https://kislayverma.com/how-big-should-a-method-be/ Last updated: 2026-07-22T12:46:45.000Z Let's say there's a method in the codebase that does 4-5 things one after the other. The code for doing all those things is well-written, but the result is a somewhat large method. At what point should you break it up into multiple methods? There is the old adage about every method being fully visible without scrolling which I have always found weird. Let's try to do better. The good part about having all of the code inside a single method is that it is all in one place and in a sense, easy to absorb in one shot. The bad is part is that it makes the method large and if split, the reader will have to move in and out of multiple methods to understand what is going on. I prefer the second approach because smaller methods are easier to understand for me. But there has to be a balance in that direction too. So while the correct answer is obviously "it depends", I'll try to give a little more in the way of guiding principles. **The key to writing or refactoring software is to reduce the cognitive load on the reader/maintainer of the code**. All other considerations are subservient to that goal (of course, "it depends"). So the key to answering the question of granularity is to identify how people read and understand any piece of code. Or rather, how do we want them to read a piece of code. Typically people read code from a higher level of abstraction to a lower level of abstraction. The guiding principle of each layer of code, typically represented by methods, is that it should prevent the user from wanting to dig deeper into the next layer. Not in the sense of sending them away screaming as quickly as possible, but rather by making what must be happening underneath so obvious that the reader never feels the need to read the next layer. In the context of this article, cognitive load can manifest in two forms: engagement and curiosity. We don't want them to feel like they have to think (engagement) to understand something, and we don't want them to want to think (curiosity) about something. This is sometimes called the[ "Principle of least astonishment"](https://en.wikipedia.org/wiki/Principle%5Fof%5Fleast%5Fastonishment?ref=kislayverma.com) and applying this can help us bound the upper and lower granularity of a method. At each level, a well-written method represents a declarative (what is to be done) unit to work to its higher layers (the ones who called it) and internally contains a set of imperatively defined steps (how is this to be done). In this sense, our method in question which does 4-5 things, is in some sense a workflow. As I wrote in [my previous article on workflows](https://kislayverma.com/architecture-pattern-orchestration-via-workflows/), the place where a workflow is defined should only define what steps are to be taken, not how each of them works internally. Let's take the example of booking a doctor's appointment at a hospital. A *makeAppointment(hospitalId, doctorId, patientId, startTime, endTime)* method can do several things: 1. Check the working hours of the hospital 2. Check if the doctor has no other appointment during that time 3. Book the appointment 4. notify the doctor of the appointment 5. notify the patient of the appointment Let implement this by putting all the code in one place ( [gist](https://gist.github.com/kislayverma/69d96b7b18497e7b6d376de6114fb9d2?ref=kislayverma.com) ). This method is not so bad as some real-world code you may have seen. But the moment I reach this method, my brain has to wake up to the details of everything going on here. I wanted to understand how an appointment is booked but suddenly I've run into a bunch of API calls and other things. This doesn't exactly match my [mental model](https://kislayverma.com/how-to-write-self-documenting-code/) of how to book an appointment. This result is a high cognitive load. Therefore, this method should be broken down till it reaches the mental model "in English" as much as possible. Here's a possibility ([gist](https://gist.github.com/kislayverma/daac16e77b4542455c82335accde9bfa?ref=kislayverma.com)). This looks a lot more like the workflow I was expecting. The details of each of the steps are now hidden, which means that they can change without us knowing about that. But the biggest thing is that I don't feel like I have to go into each of the new methods to see what they do if all I want is a logical understanding of how appointments are booked. There are technical things like exceptions that still intrude upon the reading experience, but for the most part, astonishment has been eliminated by reducing the model-code gap. Now let's go one step deeper and further break down the methods here ( [gist](https://gist.github.com/kislayverma/f57d9f00efe913f99d369cabb49a18d8?ref=kislayverma.com) ). Note the *buildScheduleService* method. If I really want to understand how ScheduleService is invoked to get a doctor's schedule, this still looks all right, although just about at this stage someone will start arguing that service creation should not be done in this class or that it can be more generically for all service. That's fine, it can still stand alone as a method, if not in this class then elsewhere. But the question indicates the curiousity/astonishment quotient has started rising. Let's take it one step further ( [gist](https://gist.github.com/kislayverma/2a6c66a0602576545d74cb0a8e392ae4?ref=kislayverma.com) ). Any developer with any amount of experience in any codebase will get curious as to why we need a separate method just to throw an exception. They will try to go to the lower level to understand it, which is exactly what we set out to prevent. At this level of granularity, our method size is causing astonishment so we should abort this step. The point at which engagement or curiousity begins to rise depends on the business and technical context of a codebase. So while this is obviously a simple example, preventing cognitive load is widely applicable as a guiding principle in software engineering. When reading a piece of code, keep in mind your engagement and curiousity levels. If either increase, there may be the possibility of refactoring. **Read Next**: [Code review checklist for distributed systems](https://kislayverma.com/code-review-checklist-for-distributed-systems/) ### Ditch the Urgency URL: https://kislayverma.com/ditch-the-urgency/ Last updated: 2026-07-22T12:46:46.000Z A sense of urgency in shipping features is probably the worst result of the [agile](https://kislayverma.com/category/agile/) mindset. While it makes some sense in absolute early-stage startups where everything has to be built ground up, but in places that have a little bit of stability, this is a vestigial mindset which causes a lot of problems. ![](https://kislayverma.com/content/images/2021/05/Learn-Solve-Deliver.jpg) Delivering the right kind of product is a three step process. 1. Learn: This is where we try to identify the customer’s problem(s). 2. Solve: We identify the best ways of solving the problem. 3. Deliver: This is where the solution is built and delivered. While tools help, Learning and Solving can only be fast-tracked to a certain extent. These are the most valuable actions that a team can perform, so they should be given due importance. Unfortunately, the prevalent reading of *move fast* permeates both thinking and execution with the same level of urgency. While software engineering is an art and science by itself, seen from this level, it is operations, and operations work best when they are optimized to death. Automation and best practices create an environment where the execution pipeline can be made faster and faster. Each developer can exert a lot more technical leverage to produce a greater output. [Organizations](https://kislayverma.com/category/organizations/) have taken this to mean that more things can be shipped out with the same amount of resources. It actually can’t, as a simple application of the [Theory of Constraints](https://www.leanproduction.com/theory-of-constraints.html?ref=kislayverma.com) can show. Envisioning the process defined above, the true bottlenecks in the flow of value are learning and solving. So we have two choices. One is to widen the bottleneck by short-circuiting true ideation and pushing out all kinds of ideas down the line to be executed. The other is to not widen it at all and send only the most impactful things (as best as we can tell) out. This might mean that some execution capacity lies unused at some times, but the impact of delivery is not diminished. Guess which one gets chosen way more often than the other? So I don’t think of moving fast in the sense of delivering many changes very quickly. This is a very operations-centric view of things. Looking at the entire team and the product holistically, iterating rapidly should be about buying the team as much time as possible for identifying and solving problems. It should focus on making the actual execution boring and efficient so that the time between identifying the solution and delivering the solution becomes minimal. ![](https://kislayverma.com/content/images/2021/05/Learn-Solve-easily-deliver.jpg) I think this is a better way of looking at the excellence of a team and engineering velocity because it put the most important steps of the value addition process in the spotlight. The tech-product team now has two very clear mandates: 1. Identifying and solving the biggest problem(s). 2. Eliminate everything causing friction in getting the envisioned solution in the hands of the customer. In management terms, the first is strategic thinking, and the latter is operational excellence. At the team level, strategic thinking should come first. We should spend a lot of time figuring out where we stand and what we want to do. This phase should be deliberate, intense, and the step where the team comes together behind a shared vision. But the ability to do this hinges on making the execution process “just work”. Great teams spend time and effort populating this phase with tools and processes that remove unpredictability and turn it into a well-oiled machine. Engineering bandwidth that is not fully occupied at all times is not a bad thing. A good team will use this time to make sure their execution phase stays smooth and boring. Paying down tech debt, adopting modern operating practices and tools, adding documentation, etc keep execution friction from rising and let small teams deliver big results. But this meta-work usually gets tagged as wasteful since it is not perceived as being beneficial to the business. In its place, teams put in unimpactful busywork in the name of agility. There is a balance between thinking too much and not at all. At the moment we seem to be leaning far, far towards the latter. Organizations should stop trying to push ideas down the pipe just for the sake of cranking the wheel. OTOH, engineering teams need to take learning from operations methodologies to identify what is slowing them down when it comes to delivering code and ruthlessly eliminate these bottlenecks. ![](https://kislayverma.com/content/images/2021/05/Mark_Zuckerberg_-_Move_Fast_and_Break_Things.jpeg) **Ditch the urgency** doesn’t sound like much of a mantra to motivate your employees, which is probably why the Zuckergerian adage has caught on much more. But ditch the urgency to move for the sake of moving. Don’t short-circuit the thinking process. Give it the luxury of time by making it super smooth to put good ideas into action. In smart teams, thinking should emerge when doing is taken out of the way. Think about learning to play a musical instrument. The ideal goal is not to learn songs and melodies very quickly and ad-infinitum. It is to develop the skill of playing the instrument to an “unconscious competence” so that the main focus can be set on musicality. This is where the “effortless” playing of the true masters comes from - they don’t even think about the physical act of playing the instrument. Delivering software is not so different. **Read Next**: [Managing microservice hell with domain boundaries](https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/) ### Why programmers don't write documentation URL: https://kislayverma.com/why-programmers-dont-write-documentation/ Last updated: 2026-07-22T12:46:47.000Z I have been [writing about documenting code](https://kislayverma.com/how-to-write-self-documenting-code/) of late, so of course, my Medium recommendations threw out an article about “[the real reason why developers don’t write documentation](https://betterprogramming.pub/this-is-why-most-software-engineers-dont-write-documentation-670ceecb6a21?ref=kislayverma.com)”. The article claims that the lack of good tools for writing is the biggest culprit in discouraging software engineers from documenting their work and decisions. I usually don’t pick on specific articles, but this one triggered the hell out of me. The writer makes some okay points about diagramming tools, but the overall piece is so misleading that it obfuscates this important issue. If you are going to compare two drawing tools and claim that neither being good enough is the main reason for developers not writing docs, then either you are writing for clickbait or in bad faith. I believe that there are two main reasons software engineers don’t write documentation. Tools play their part but they are a hugely distant third. ### Writing is hard Software engineers, like everyone else, don’t write because writing clearly is very, VERY difficult. Writing is a tough, demanding task. It requires organizing our thoughts clearly, examining them critically, and expressing them clearly. While the expressing part can be simplified to some extent (depending on the quality of writing required), all three steps are taxing when done properly. In the world of programming, where “it depends” is often the best answer and everything is based on trade-offs, writing becomes that much harder. It needs to set the context, justify the decisions, and then power the low-level thinking leading into the code. This type of writing is only useful if done well, and since doing it well is tough, it often doesn’t get done at all. Bad code will still fly, bad documentation won’t. This is why a lot of people argue about the value of comments in code and the merits of self-documenting code (whatever that means). Kevlin Henney says that asking for comments around complicated code is futile because we expect the same people who could not express themselves clearly in code to now turn around and express themselves clearly in English. ### Not documenting doesn’t block shipping If a developer doesn’t write documentation, their *work* still gets done. Not writing doesn’t block shipping (at least not right away). The damage done by not documenting technical decisions is cumulative. Much like tech debt, it doesn’t cause damage in the here and now. Like I said above, writing is primarily a matter of thinking and analyzing. In most places, coding can be done by the seat of your pants. A disorganized pile of classes and methods in code may work - a pile of work of words and paragraphs won’t work. Writing HAS to be clear if it is to be of any use. Code will be accepted (to some extent) as long as it does its job. And since most organizations focus only on getting the product shipped, that which doesn’t block shipping gets ignored. Unit tests face a similar problem in many teams. To test the code we need to understand it (that takes more effort than writing it), and not having tests doesn’t block shipping. Ergo, no unit tests in code. There is also the matter of obsolescence. Even good documents go obsolete, so engineers have to keep repeating the think-analyze-express over and over again as they build out systems. So dropping off the documentation wagon is easy. So even with best intentions, documentation often happens only in spurts of writing and cleanup. ### What about the tools There is no doubt that the commonly used set of tools used for documenting software today are woefully inadequate. We don’t think in terms of documents one at a time. We think in terms of ideas and goals by pulling together multiple concepts at once. The resultant document is just one manifestation of the thought process. We need tools that can help us collate ideas across time to solve the problem at hand. Google Docs, Confluence, Markdown are all poor tools for this type of writing. However, a new generation of tools like [Notion](https://www.notion.so/?ref=kislayverma.com) and [Roam](https://roamresearch.com/?ref=kislayverma.com) are attacking this problem of harnessing networked though. Hopefully, these will work as intended and help in the thinking that goes into writing. However, the lack of a second brain cannot really be used as an excuse for not using the first one. Tools play their part, but the willingness to undertake the process is the real hurdle. ### So how to do documentation Writing software has taught me one thing. If you really want your users to do something, then doing it has to be a blocking step in their journey with your product. In the same way, tacking on documentation to written code is never going to work. Worse, it is useless. Writing is about critical thinking. It is meant to explain your thought process and intent to yourself and to your audience (e.g. your team). The thinking process is where documentation/writing adds value, not as a static record of already implemented code. Proponents of mob/pair programming and XP often disparage documentation. But barring the adoption of those techniques, the practice of writing and reviewing technical documents is the only way teams build a collective understanding of what they are trying to build. This shared world-building is what makes this process critical to the long-term health of the team and the codebase. I feel that the only way to make the process of writing documentation sustainable is to make it a blocker for software development. Make it lightweight but mandatory. It should become part of the process instead of being yet another thing to do. Some things that have worked for this in my experience. - **Write before you code**. Unless the change is trivial, every engineer writes a note about what they are going to do and runs it by the rest of the team. At the end of the discussion, the actual coding should become trivial. - **Write simply**. Don't complicate the writing, at least until it becomes second nature. Diagrams, fancy sections etc can wait. Write very simply about what you thought, what you are doing, and why. Even if the document can serve as a basic pointer to the rest of the team now and in the future, it is superbly valuable. - **Document the decision with their alternatives** \- Rather than documenting the actual implementation (which may change over time) in detail, focus on documenting the choices and why they were made. [This is what the code cannot ever explain](https://kevlinhenney.medium.com/comment-only-what-the-code-cannot-say-dfdb7b8595ac?ref=kislayverma.com) and hence writing it down adds the most valuable. Details can be documented based on the time you are willing to invest. - **Make it searchable** \- No amount of documentation will be of any use if people cannot find it. Use tools that support text searching out of the box. This is one of the reasons I don't like Google Docs for documentation. It is great for writing but just horrible for collaboration and discovery. - **Track changes**. Some organizations use version control to track changes to the system's design over time. That’s great. But if you are not there yet, keep one document per feature and keep putting dated updates on it so that evolution can be tracked in one place with minimal hassle. The hope is that as the team seems the merits of having and reviewing some documents (e.g. new members need lesser hand-holding) and writing becomes muscle memory, the practice will become self-sustaining. Till then, it should be treated like working out or dieting - painful but necessary. **Read Next**: [How to speed up software delivery](https://kislayverma.com/how-to-speed-up-software-delivery/) ### External Programmability: The second law of building platforms URL: https://kislayverma.com/external-programmability-the-second-law-of-building-platforms/ Last updated: 2026-07-22T12:46:48.000Z ### TL;DR We should not have to modify central systems/platforms to achieve variant behaviours for different use cases. We should be able to *plug in* these behaviours from the outside to customize specific parts of the overall system behaviour. This will make our system more durable by offering a powerful mix of capability and customizability. ### The Problem ![](https://kislayverma.com/content/images/2021/04/external-prog-original-design.jpg) Imagine you are building a central system that is intended to be used by multiple other teams. Depending on the kind of complexity offered by the system, one or more clients may ask for variations of the original behaviour specific to their use-cases. We can readily imagine such situations arising in B2B software where every client needs some custom variant of the original feature. ### How do we accommodate these situations? ![](https://kislayverma.com/content/images/2021/04/external-prog-build-centrally.jpg) The most obvious way, of course, is to build it! The team that built the system also builds the customizations in the features as required by any client. This makes sense if these requests are rare (so the team can easily allocate time for it) or complex (this is the ONLY team that can do it). If this is not the case, however, the original team becomes a bottleneck for multiple teams because it cannot spare the time to take care of all the incoming customization requests. The second way is to ask the client teams to get into the code base and make the changes themselves. This removes the bandwidth bottleneck. Client developers can usually make the changes given sufficient enough tools/documentation and guidance (code review etc). But over time, this almost always leads to deterioration in code quality and blurry lines of ownership. It is difficult to hold any single team accountable for the quality of the system since everyone is making changes. Depending on the nature/complexity of the change, the oversight and communication required may well be a lot. Also, this model is practically impossible if the client team is external to the organization and hence cannot be given access to the codebase. ### System Boundaries are Team Boundaries Conway’s Law, Team Topologies, and various other schools of thought have made it abundantly clear that an organization’s software architecture mirrors its communication architecture. So the problem of building customizations can be generalized to a problem of defining how client teams interact and influence the team that owns a system, thereby influencing the design of the system. If multiple teams want to use and grow the same system, we need to define a model for coordination between them. To my mind, this model must minimally achieve two objectives: - We should be able to evolve the software independently without getting bogged down in communication overhead. The first approach discussed above is ruled out on this ground because it puts the owning team on all change paths. Clients have to beg/bully/convince them into making the changes for them. - We should be able to do this without degrading the quality of the codebase. The second approach discussed above is ruled out by this. Maintaining code quality and operational excellence is almost impossible if anyone can (and is expected to) make changes to your code. So we need a way to define a system boundary and change process such that others can make changes independently without impacting our code quality. We can do this if we can allow people to “hook in” to the internal decision points of our system and modify the behaviour for their use cases. This is what Steve Yegge calls External Programmability in his[ legendary platforms rant](https://gist.github.com/kislayverma/d48b84db1ac5d737715e8319bd4dd368?ref=kislayverma.com) (you can read my redux [here](https://kislayverma.com/distilled-steve-yegge-s-platform-rant/)) and next to [*Eat your own Dogfood*](https://kislayverma.com/the-golden-rule-of-platforms/), it is the second cardinal principle of building platforms. ### External Programmability The idea of External Programmability is to identify the parts of an application that we think should be customizable, turn them into hooks for variable functionality, and then expose these hooks externally. Clients can then plugin to these hooks and trigger custom behaviour or make decisions based on custom logic without having to go into the codebase of the system. As a result, the behaviour of the system is not completely determined by the logic implemented by the owning team, but by the collective impact of the core logic and customization hooks. ![](https://kislayverma.com/content/images/2021/04/external-prog-external-prog.jpg) This style is, in a way, OCP at a multi-system level, and has distinct advantages over the modify-from-within approach. 1. Clients know exactly how to hook in custom behaviour because the design of the system makes it explicit. No risk of them going inside the system and breaking something by mistake. 2. It also makes change faster for the client, because they do not have to learn how to work inside a new codebase. They integrate from the outside along well-defined interfaces, and the customizations themselves are implemented in a technical environment of their choosing. It’s like being able to tell another microservice about which of your APIs to call at which step without having to modify its code. External Programmability transforms an internal decision of a system into an open interface that users of the system can modify as per their needs. From a system design perspective, this means that the system interface is a lot less “closed” than you would normally expect. The internal parts which got turned into externally customizable hooks transform perhaps a suite of APIs into a collaborative interplay of decisions and actions. We are deliberately exposing a lot of system internals for customization so that we don’t have to expose the entire system to invasive change. If we look at the traditional layered architecture style, control always flows from higher layers to lower layers. However, in the platform architecture which external programmability creates, control flow back and forth between upstream and downstream systems (client systems being considered upstream and the platform system downstream). The emergent collaborative system architecture is better visualized as a 3-dimensional mesh of systems rather than a two-dimensional stack. There are still upstream and downstream pieces but the boundaries between them are a lot more fluid. ### How do we get there? One way of implementing this is to externalize all the business logic (even the original business logic) into workflows outside the core application. The core thus becomes very, very lightweight and all the logic moves out into the orchestration layer. In this way, clients have complete control of what they want to do. They do whatever they want and then call the simplistic APIs of the core system as they see fit. This gives the ultimate freedom and inversion of control - instead of modifying what exists, clients can compose whatever they want. The problem here is that the core that remains usually gets stripped of all business semantics and hardly remains a product at all! Clients have to build not just customizations of some existing behaviour but the entire functionality over and over again. The domain boundary completely breaks down. There is no way to know where the logic for processing a certain kind of order is implemented because that logic lives completely outside the core and we have no way of systemically finding out what is happening where. The other way is to implement a callback-based system . The original system identifies the parts which parts of the control flow it deems to be customizable (the other parts become *core* by definition since they cannot be modified by clients) and exposes them over APIs. The APIs allow clients to define the rules under which their specific customization should be triggered and exactly how they should be triggered (execute an API call back to the client system). ![](https://kislayverma.com/content/images/2021/04/external-prog-register-custom.jpg) Once these customizations are “registered” with the main system, whenever client A invokes the feature X, it executes all non-overridden points as per default behaviour but executes the registered override to achieve an end-to-end result customized for client A by client A. ![](https://kislayverma.com/content/images/2021/04/external-prog-execute-custom.jpg) I have written a [more detailed explanation](https://kislayverma.com/platform-nuts-bolts-flexible-decision-making-with-rule-engines/) of how we can use a combination of rule systems and workflow management systems to stitch the whole experience together. In this approach, all interactions for a certain problem come to the same central system, and we can identify from that place what we want to do. Either client uses the default behaviour of the system, or they will have registered specialized hooks to custom callbacks. In either case, it becomes easy to track down the flow of control because all branching out happens from well-known points of divergence. As a result, a porous technical domain boundary remains with much of the business logic running inside the boundary, but the occasional customization going back up the stack to client systems. Our core system is still the one place where all business logic can be traced from. Note that in this approach, [we need not distinguish between internal and external teams](https://kislayverma.com/control-and-chaos-in-platform-systems/) . All client teams communicate across a porous system boundary which defines a clear interface and protocol but otherwise, both teams operate independently. The team which owns the system and the teams that use the system are in effect *co-building a much larger system* by allowing each other to reach deep into each other’s systems to create business value. **Read Next**: [How to build evolutionary architectures using an event based approach](https://kislayverma.com/programming/using-events-to-build-evolutionary-architectures/) ### Testing strategies for agile teams URL: https://kislayverma.com/testing-strategies-for-agile-teams/ Last updated: 2026-07-22T12:46:49.000Z End-to-end testing refers to the approach of testing every step of every single user flow. e.g. In the e-commerce domain, this might mean testing every API call and database write/read from the moment an order is placed to the moment it is delivered. The end-to-end testing process will validate every notification sent to the customer, every tool used by the ground force, and every third-party interaction involved. This testing paradigm validates the end-user experience by validating behaviour across multiple domain boundaries and teams. ![](https://kislayverma.com/content/images/2021/04/end-to-end-testing.jpg) Back when all systems were monolithic, development batch-y, and releases infrequent, this was the de-facto approach for testing software before it was released. It made perfect sense because if the last release was some time ago and many features were being developed, it was quite possible that many parts of the codebase had changed simultaneously. Sadly, end-to-end testing has persisted in the modern microservice architectures as well where it adds very little value and creates a massive roadblock in attaining a [high speed of development and deployment of software](https://kislayverma.com/how-to-speed-up-software-delivery/). Especially if an organization follows agile principles and ships code quickly using CI/CD etc, this approach of testing can bring the entire delivery pipeline to a standstill. Let's see how we can do better in the microservice world. There are two main technical advantages that microservice architecture gives us. The first is independent services which can be developed, deployed, and scaled independently. The second is that relationships between these services are now explicit and the dependencies trackable. The first means that at an organizational level, there is no longer a single artifact to release at a certain fixed time/cadence. All services change at their own pace. The second means that only a small subset of the overall system is impacted when any of the services deploy a change. It is easier to identify what impacts what in microservice architecture (if it is well designed). The end-to-end testing mindset focuses on the idea that since there are now many things, any of which can cause a problem, we must test all systems/features before we deploy anything. It therefore completely surrenders the second advantage of microservice architecture - identifiable dependencies. And by ceding this advantage, it becomes a blocker to agility rather than an enabler. It can be argued that this is not a microservice-monolith question but rather a slow/fast shipping question. In a slow shipping monolith/microservice, it is difficult to establish what the immediate neighbours are because many changes are being deployed together. If each deployment consists of a single change, then it is easy for devs and tester to establish what should be tested. I feel that microservices simplify this further by making the separation of concerns even more explicit or externalized. We don't need to test everything all the time. Looking at the service which changed, we can isolate all services that interact with it, and test only the interactions with them. If system boundaries are drawn well, only the immediate neighbours should be impacted by any change. So how should agile organizations test a microservice-based architecture? I prefer a two-pronged approach, both of which assume automated integration testing ability. If you are still doing full manual, you need to get in the automation game ASAP. First, we need to speed up the testing of each service or component so that our independent teams can move fast. We need the ability to identify all its immediate neighbours and verify that all those interactions are working properly after the change. If service A is changing, then immediate neighbour means every service that service A calls and every service that calls service A (or consumes events from service A). So both upstream and downstream systems constitute a service's neighborhood. Only the neighbourhood of a component has to validated in order to safely deploy changes. This reduced scope of testing makes the whole process faster. Identification of a service's neighborhood can be automated (eg. using a service mesh/request tracing to extract who calls who) or manual. Once identified, we can use techniques like CDC (Consumer-Driven Contract) Testing to verify that every interaction of the service under test is working fine. Many organizations also tag/group integration tests to be able to run subsets of integration tests for further optimization (e.g. if only API 1 is changing, only run tests related to that in the immediate neighborhood). The more widely used a service is, the larger its neighborhood will be, and hence the greater the amount of testing that needs to be done. While this means that deployments of this service won't be as fast, it also makes sense that the most used services should move slowly to prevent big outages. As long as the whole process after merging to main is automated, who cares anyway :) But the optimization in testing each service that we have done (we aren't testing everything all the time anymore) means that some unknown unknowns can cause bugs to slip through. So the second prong of our testing strategy is to continuously run integration tests on all critical customer-facing features in production . This can be automated tests against our public API, selenium style tests against the UI, or anything else which can flag any behaviour unexpected by the customer. If an anomaly is detected here, we raise a bug which the relevant dev team then takes over to investigate and fix. ![](https://kislayverma.com/content/images/2021/04/agile-testing-strategies.jpg) The latter set of tests should be put in place first - in fact, these should be considered part of a feature release. This is yet another argument for developers writing integration tests. A post facto QA team/process can't hope to keep up with a fast-moving developer team. The organization should focus on enabling dev-driven testing by giving them time to do it and setting up the tools to make writing tests easy. Dev teams can then focus on building and testing their features. Move as fast as you can, and break as little as you can help it! **Read Next**: [Events, not logs](https://kislayverma.com/publish-events-not-logs/) are the right paradigm for Observability ### How to write self-documenting code URL: https://kislayverma.com/how-to-write-self-documenting-code/ Last updated: 2026-07-22T12:46:50.000Z ### What is self-documenting code? I love documenting code and systems. Many don't. A major argument against documents is that they get outdated as the system evolves. And the faster a system evolves, the faster its documentation gets outdated. Ironically, this is the very type of system which needs the most up-to-date documentation! An argument is often made, therefore, for self-documenting code. This is ostensibly the kind of code that doesn't need separate documentation because it is designed and implemented in such a way as to be self-explanatory to the reader. How does anyone reading a codebase understand it? First, they need to know what the code is "supposed to do". Then they can graduate to figuring out how it does that. And this is where the problem of self-documenting code lies. Because reading code is essentially reading the how. "Query some things from a database table and process them into a map, match them against some other things from some other table, and return everything that does not match as a list". [Well-written code makes it simple to understand how it is doing something](https://kislayverma.com/reduce-collaboration-by-good-design/). But it doesn't tell the reader why it is doing what it is doing. And hence the reader remains confused and documentation becomes necessary to understand the intent behind the system design. So what kind of code would reveal, even in some limited way, why it does things the way it does them? I want to talk about the only way (IMO) code can explain itself to readers and hence become self-documenting. Let's discuss the model underlying the code. ### Becoming self-documenting Before code comes the conceptual model that the code is the physical manifestation of. This is the mental model of the problem and the solution. This may be the domain model or another specific way of representing the programmer's thought process and how it will achieve a programmatic solution to the problem at hand. ![](https://kislayverma.com/content/images/2021/04/model-and-code.jpg) The model is the core of low-level system design. It defines the things that we are working with, what their nature is, and what role they play in solving the problem at hand. The model must be developed first before any other aspects of the low-level design like APIs, data stores, data flows can be determined. These are physical forms of the things conceptualized in the model - ways to make the model "run". The model itself is the why, and the code is the how. > ***The only way to make the code self-documenting is to make the code reveal the model underlying it.*** This is why the only way to make the code self-documenting is to make the code reveal the model underlying it. Code that highlights the core model instead of hiding it in implementation details builds [a narrative that is much more accessible](https://kislayverma.com/a-narrative-approach-to-software-design/) than trying to infer the meaning of some lines in code. The lines can always be interpreted, but the model made evident in well-written code sets the context under which those lines of code make sense. Let's take an oversimplified version of Airbnb booking. In the object-oriented, REST-ful world, the code to update a booking might look something like this. (See [this gist](https://gist.github.com/kislayverma/a0575599d5f164091b9aa8b4f2ad5959?ref=kislayverma.com) if you have disabled JS) This type of generic update code is fairly common. To me, this doesn't explain why things are happening the way they are happening. Did we miss any cases? While this code can be refactored to a much cleaner form, but it does not reveal why things are happening in this way. Let's consider an alternative (See [this gist](https://gist.github.com/kislayverma/dfdd81fd63d6772cac33d462c9fe3fed?ref=kislayverma.com) if you have disabled JS). Or perhaps this (See [this gist](https://gist.github.com/kislayverma/5497931c93250768518d2d62a47d5791?ref=kislayverma.com) if you have disabled JS). ### What is the difference? Looking at these examples makes it obvious how we can write code that makes the conceptual model explicit and hence reduce the cognitive load on the reader. The way to do this is via abstractions. All code is likely to have some amount of abstractions, but not all abstractions surface the thought process behind the code. Often developers use the model abstractions only as data carriers without imbuing them with any behavioural or semantic significance. This is the case in the first example of the generic update API. The Booking abstraction merely carries the data, all meaning is encapsulated in the if-else conditions. Model-Driven code, on the other hand, uses model abstractions to represent the core elements and uses them as the building blocks of all other interactions in the system. They are the heart of the system and all other code only manipulates them in ways defined and controlled by the model itself. This is the case in both the second and the third examples. It is not the case that the code in the first example does not have a model underlying it. Much like it is not possible to have "no design" (there is always design, even if inadvertent and poor), it is not possible to have "no model". The solution sitting in the developer's head is the model. The first example just chooses to obscure it while the latter two make efforts to make it clear. I hope this has made clear the benefits of explicitly using a conceptual model, and building the system around it. This won't make the system (especially a large system) automatically self-evident, but it does go a long way in that direction. **Read Next**: [A comprehensive look at turbocharging your software delivery pipeline](https://kislayverma.com/how-to-speed-up-software-delivery/) ### Reduce collaboration by good design URL: https://kislayverma.com/reduce-collaboration-by-good-design/ Last updated: 2026-07-22T12:46:51.000Z The value of collaboration, talking to your team and your customers is in the brainstorming stage or in the learning phase. This is where we should put together our collective minds, bounce ideas, and figure out the next steps. Once this stage is crossed and we enter the execution phase, [collaboration becomes a dangerous overhead](https://kislayverma.com/independence-autonomy-and-too-many-small-teams/). ![](https://kislayverma.com/content/images/2021/03/good-bad-collab.jpg) If multiple teams and people must build something new together, synchronization is inevitable to some extent. However, there is a specific type of low-impact collaboration where people talk to each other to find out what capabilities exist and how to use them. This is seen when a product team tries to onboard to a platform by first setting up a meeting on how to use it. This is seen in “integration war-rooms” where two teams try to set their systems up correctly to talk to each other. This is seen when our teammates ping us about the right way to use that API we wrote the other day. This conversation is not about building new stuff, it is about figuring out how to use what exists. In my opinion, this is utterly wasteful. Developers obsess about making all business interactions self serve. In operations heavy organizations, teams spend a significant amount of bandwidth building tools for “enabling” operations teams. The same engineers, however, are perfectly okay sitting with team after team in meetings to explain to them how to use their software. Working with non-engineering teams is considered overhead, working with other engineering teams is collaboration. The deeper a team’s systems are in the architecture stack, the more they are likely to be reused. The more people want to use them, the more this team needs to answer questions around what and how. The more time they spend answering these operational questions, the lesser time they are actually building cool new stuff. A perfect vicious cycle. Teams and individuals spend an immense amount of time simply explaining how things work to others. This situation is clearly less than optimal, so how can we do better? This is not so much a problem as a shift in perspective. As engineers, we often consider all engineering as internal, and capable of being able to navigate convoluted technical processes. The thing to keep in mind is that they shouldn’t have to. Just like operations and other “business” teams, if other teams and developers have well-defined SOPs for using your code, we will have gone a long way towards removing unnecessary human collaboration. In this world of operational excellence, there are two tools that stand heads and shoulders above all other techniques - documentation and design. What if we think of all software work as building a product? My customers, who happen to be other engineers in this context, should be able to explore and understand my product, play around with it, and then finally sign up for it with minimum human intervention. For them to be able to do this, documentation is a powerful tool. Imagine if your system’s documentation was up to date, had clear guidelines on how to do what, and explained the different scenarios. Wouldn’t that be great? The world of operations practically runs on this type of documentation - detailed, precise, and explicit. The problem is that this type of documentation is hard to maintain, especially in the world of agile software. Also, no one read docs. People are far more likely to ping the author of the document for a meeting rather than go through something very detailed. This is why I consider documentation a supporting actor in this play. The single most important tool that we can bring to bear on this problem, in my opinion, is good design. When the system is exposed in such a way that the user cannot make a mistake in using it, then the very core of the problem goes away. Documentation can further bolster this good design by herding stragglers to the right place, but the entire experience of using a system or tool or API should be explicitly designed to make the user do the right thing unambiguously. This is not an outlandish idea. This is what product thinking, as applied to most customer products, is. It’s just that engineers mostly don’t apply the same thinking to developer tools. Pretty much always, customer experience > internal tools experience > developer experience. This is kind of sad because developer experience is where the entire engineering organization spends its time. Imagine what your team could achieve if every experience of using a new software thing was as smooth as your company’s customer experience. “How should I use it” indicates a failure of design and requires focussed product thinking to fix it. It is worth the effort because answering this question is the lowest form of collaborative value-addition. I hear a lot of developer feedback on the lines of “X is a great dev because he really spent a lot of time helping me use her system”. While I do not disagree with the sentiment, X and X’s team have also failed in prioritizing their time. They could have been discussing and building the next set of things. Instead, they ended up doing some grunt work that could very likely have been automated. While writing a new class or a new method, ask yourself if your teammates are going to have to ask you how to use it? If your system is to be used by other teams, will they have to ask you how to get started? Will you have to make some manual DB entries or change some configurations? Is there any step where you must “talk” to them to explain things? If so, then regardless of how useful the thing you just built, you have increased the communication overhead in your organization which you will be paying, personally, many times over the next few weeks or months or worse. I’d love to hear more from you folks about how much of your time goes into what kind of collaboration, and what value do you think it brings to the table. This article only expresses my narrow experience, and it would be instructive for me to learn from others who have different kinds of organizations than mine. Are there other tools/processes that can make a team more effective? Leave your thoughts in the comments section. **Read Next**: [Managing developer identities an autonomous team](https://kislayverma.com/managing-developer-identities-in-autonomous-teams/) ### Architecture Pattern: CQRS URL: https://kislayverma.com/architecture-pattern-cqrs/ Last updated: 2026-07-22T12:46:52.000Z Software systems serve a variety of purposes from their first day, and the requirements on them grow over time. Changing requirements may pertain to a change in business logic, scaling needs, or some other aspects of the system. To satisfy these often contradictory or overlapping requirements, engineers must make a variety of trade-offs in the design of the system. The problem in making trade-offs is that many of them are not required at the beginning and by the time the need arises, the system design has evolved in such a way that the trade-off cannot be made at all. In my opinion, the most pernicious incidences of the design getting locked in happen at the data layer. A typical application’s data model is designed by marrying domain knowledge with performance considerations. The domain knowledge dictates what the entities are and how they relate to each other logically. Performance considerations dictate how they are implemented physically (e.g. RDBMS-vs-NoSQL, primary keys, indexes, etc.). These two sets of choices together enable an application to serve its use-cases efficiently. ![](https://kislayverma.com/content/images/2021/03/blind-men-and-multiple-readmodels.jpeg) Same data, many views In large applications with a lot of data and complex entity models, some implementation details become “core” over time. This is sometimes explicitly done by engineers, but often it happens in an unstated or even inadvertent manner. In these situations, new requirements can be so far at odds with the existing implementation that they cannot be accommodated at all. This general class of problems is large with different solutions for different cases. In this article, I want to focus on problems that arise when the way data is read from an application is very different from how data is written to a system. The difference can be in terms of query patterns, output format expectations, or scale of operations. I wrote about my encounter with this situation in [this](https://kislayverma.com/asynchronous-programming-a-cautionary-tale/) post. The order management system I was working on at that time was optimized for working with entity ids (order id, item id, etc). But over time, complex read requirements emerged which the data model was unable to support. The problems were two-fold. New query patterns were emerging which were difficult to implement efficiently in the existing implementation. Far more worryingly, the readers of order data were beginning to expect a very different model of the data. E.g. sellers on the e-commerce platform wanted their slices of a larger customer to be represented a certain way, customer-facing apps wanted the data to look very similar to how it looked in the cart. This is not an uncommon occurrence, especially for systems that own the core entities of an organization. The data they encapsulate is so widely used that it is required to be available in many different formats. The system itself needs yet representation to work with its data. How can we bridge this gap? ### CQRS CQRS stands for **C**ommand **Q**uery **R**esponsibility **S**egregation. Systems built with the CQRS principle distinguish between data models used for Commands (write operations) and Queries (read operations). The command model is used to perform write/update operations efficiently while the query model is used for supporting the various read patterns effectively. The data between the two models is kept in sync by propagating the changes in the command model to the read model via domain events or any other mechanisms. ![Introduction to CQRS ](https://kislayverma.com/content/images/2021/03/cqrs-basics.jpg) If this sounds like two different microservices to you, let me point out a subtle difference. The physical implementation of these two data models can indeed be done as two separate microservices. A single command model can even be used to support multiple query models. However, a key construct of microservice architecture is that two microservices typically represent two *independent* domains. In CQRS, both the command and the query models are part of the same logical domain regardless of the runtime architecture. The query model cannot function without understanding the command model deeply. The coupling here is expected, unlike the decoupled behaviour we hope to create in two separate microservices. CQRS does not dictate how the two models are kept in sync. This may be done synchronously by updating both the models at the same time. It may also be done asynchronously by transmitting commands from the command model to the query model over a message broker like Kafka. The latter choice is the one made often because it creates a more scalable system, though it comes with the obvious tradeoff of eventual consistency between the write action and read action. ![](https://kislayverma.com/content/images/2021/03/cqrs-dual-or-async-write.jpg) ### Isn’t this just caching? A data mode dedicated only for reads sounds suspiciously like a cache. Indeed, the query model can be implemented using a caching technology like Redis. However, the purpose of applying CQRS is not just to separate the place where is written from the palace the data is read. The fundamental intent is to create multiply representations of the same data, each of which satisfies the needs of some users. A CQRS style may have many query schemas, each of which may use a different physical implementation. Some may use the same database, some may use Redis, etc. ### Why should I use CQRS? CQRS is a useful architecture pattern in a couple of different scenarios. The first one is that which I have pointed out earlier in this article. If the same data model is not able to satisfy the read and write patterns of a system effectively, then it makes sense to decouple the two schemas by applying CQRS. The resulting data models can then cater to their specific requirements. CQRS effectively unlocks the data from a single representation into any number of (read) representations all of which are kept consistent with the core representation which handles all updates made to it. The second scenario in which CQRS is helpful is in separating the read load from the write load. This may sound like cheating when I have explicitly distinguished between caching and CQRS just a couple of paragraphs above, but hear me out. CQRS doesn’t seek out caching as an objective. However, by separating the command and the query schemas, we can create the possibility of scaling one independent of the others. The query schema may live on a separate database and employ caching of its own. It may be implemented in a technology that best caters to the query patterns of a particular use case. In any of these cases, the command model is exempted from having to scale to the requirements of the query model. I would repeat here that despite all this, these are not independent systems. The coupling between them is deep and this is not a problem. ### Why should I not use CQRS? Using CQRS in a system introduces significant cognitive overhead and complexity. Instead of a single data model and technology choice, developers now have to contend with at least two data models and potentially multiple technology choices. All of this is an overhead that cannot be ignored. The next problem is keeping the command and the query data models in sync. If the choice is made to keep the updates asynchronous, the entire system is forced to deal with the fallout of eventual consistency. This can be extremely troublesome, especially if parts of the system are directly exposed to human users who expect their actions to reflect in the data immediately. Even a single requirement for consistency can imperil the whole design. On the other hand, if we choose to keep the model in a consistent state at all times, the CAP theorem and 2 phase commits come knocking around. If both the schemas are colocated on a single ACID-compliant database, we may still be able to use transactions to keep them consistent. However, this takes away much of the scaling benefit of CQRS. If more than one query model is to be supported, the write operations will continue to get slower and slower since they need to update all query models before they can succeed. Both these problems make the use of CQRS a proposition that should not be taken lightly. Judiciously applied, it can result in a highly scalable application. But supporting multiple data models is a tricky affair and should only be considered if there are no other means of satisfying the necessary query patterns. **Read Next**: Introduction to the [workflow architecture pattern](https://kislayverma.com/architecture-pattern-orchestration-via-workflows/) ### Book Review: 10% Human by Alanna Collen URL: https://kislayverma.com/book-review-10-human-by-alanna-collen/ Last updated: 2021-02-13T09:05:50.000Z *10% Human: How your body’s microbes hold the key to health and happiness* by Alana Collen is my first introduction to the hidden world of the microbes that live in our gut. This topic has been gathering momentum (especially after the Gamechangers documentary) and this book is a great introduction to many to the basic concepts and makes a great case for thinking about our body’s bacterial residents a little more holistically than we have done so far. The book is well written and engaging. There are a lot of surprising facts and new concepts that I could not help be intrigued by. However, as I find with more and more books these days, it is written in what I think of as “Malcolm Gladwell style”. A lot of the time and text is devoted to story-telling and making a captivating case for the microbiome instead of making the book more information-heavy. I think about half of the book could be edited out for a person who is looking for facts and not repeated anecdotes. This is, of course, my personal opinion. I know a lot of people whole liked reading about the many case studies and yet-unproven possibilities that the book explores. The author comes across as an evangelist of a brave new frontier, where despite scientific rigour, there is entirely too much speculation for my taste. 10% Human is full of zeal and enthusiasm. It is a powerful proponent of its core idea that our microbiome may be far more important than we have given it credit for so far. I definitely recommend reading the book if you don’t have any knowledge of this topic yet. It has definitely opened my eyes to a very different world and provided scientific backing to some things that people often say but don’t really understand. I have included my summary of the book below. This is an actual summary, not a set of highlights as I have done in some [other book reviews](https://kislayverma.com/category/books/) . I hope it conveys the essence of the arguments and knowledge of the book. If you find it interesting enough to make you read the book, or if you want to read the book anyway, I suggest you read Chapters 2,3, and 8 thoroughly since they have a lot of information. Chapter 7 also has some interesting insights. The rest of the chapters can be skimmed. ### Introduction - The human body doesn't have significantly higher genetic complexity than rats or pigs. - Human and microbes are symbiotes - The human microbiome project (similar to the Human Genome Project) maps the DNA of our microbiome. - It is convenient from an evolutionary perspective to have microbes do some functions instead of having to evolve genes for everything. - Darwin onwards, the Appendix was always thought to be useless, but it actually is a reserve of microbes and thus has evolutionary value. - Our body is a tube with skin on the outside and the digestive track ALSO on the outside (the inner, exposed layer of the tube). ### Chapter 1: 21st-century sickness - Pneumonia is actually a symptom of several difficult microbes working together. There is no single responsible microbe. - 4 major medical breakthroughs have significantly reduced death and disease: - Vaccination - Germ Theory and hygienic medical practices - Water sanitization - Antibiotics - There are diseases that we think are normal but are very recent in their scale of occurrence. They were extremely rare just two or more generations back. These are not NORMAL, nor is there anything inherently "human" about them. - Autoimmune diseases : - Allergies - Diabetes - Obesity - Autism - Why are these diseases happening now : - Genetics: Good explanation for individual cases, but does not explain the widespread increase in numbers. Our genes could not have changed that much. - Environment: Two main themes tying disparate thing together - - Immune system common between Allergies and autoimmune - Gut Dysfunction: Autistic people have chronic diarrhea - 60% of the immune system's tissue is located in the intestine since the separation between and "outside" in "inside" is only a few cells deep. ### Chapter 2: All diseases begin in the gut - Calories in - Calories out is not enough to explain obesity - Warbler birds gain a disproportionate amount of weight before they migrate. They shed it equally fast post-migration. Even those in captivity do this without migrating at all. How? - Irritable Bowel Syndrome is a microbial imbalance: - No disease as such, hence no known cure - Usually triggered by bouts of antibiotics or some other infection. - Conjectured to be an imbalance in the microbiota. Microbial populations become unstable, thereby causing "irritable bowel" - Obesity may be infectious: - Usually blamed on lifestyle or genetics. - Obese people have more of the Firmicute class of bacteria, lean people had more of the class of Bacteroidetes. - Changing the proportion can cause weight loss/gain in mice. Since this is caused by microbes, it might mean that obesity is contagious. Some statistical data supports this. - Calories in means not just what we eat, but what we absorb - Small intestines absorb whatever we easily can. - Leftovers go to large intestines, where microbes breakdown what they want further. What remains is simple enough to be absorbed again by us, hence increasing the calorie intake. - A vegetarian suddenly eating meat will not get extra calories because they don't have the microbes to break down residual meat (which carnivores already have). - Microbes can also switch specific genes on/off to control fat storage. - The gut as a complex system: - Leptin is released by fat cells to suppress appetite once we have sufficient energy stored in fat cells. - In obese people, the brain becomes resistant to Leptin. - This causes the feedback loop to breakdown and people just keep eating. - Lean people make new fat cells to store sparsely with energy. In obese people, larger cells are formed with too much fat content. These fat cells are also surrounded by immune cells as if they are an infection. This indicates dysfunction in the energy storage mechanism. - They also have high amounts of LPS (Lipopolysaccharide) in their blood. LPS causes this fat cell inflammation and also suppresses new fat cell creation, leading to existing cells being overstuffed with fat. LPS forces us to store rather than burn. - LPS gets into the blood because there isn't any *Akkermansia Muciniphila* in the gut lining of fat people, which leads to a thinner layer of mucus over the lining, and hence LPS seeps in. ### Chapter 3: Mind Control - We assume mental disorders are due to genetics or socio-environmental factors, but this assumption is baseless. - Bacteria are known to modify animal behaviours for evolutionary advantage. e.g. Cordyceps forces ant to spread their spores - Gastrointestinal symptoms are common in people with psychological disorders. - Ellen Bolte's son became autistic after multiple rounds of antibiotics. She focussed on his gastrointestinal symptoms and came up with groundbreaking insights into how autism may be caused by microbiome damage/imbalance. - Toxoplasma is a parasite that causes personality changes in humans. This is well documented. Disproportionate amounts of it are found in the bodies of people suffering from schizophrenia, OCD, and other mental disorders. - The vagus nerve connects the gut to the brain and microbes can send small electrical spikes up it to make us "happy". e.g. If we eat the food they like, they can create chemicals like Serotonin, thereby "rewarding" us with happiness. - Propionate has been known to cause autism-like behaviours among rats. It causes rats to lose the ability to "unlearn" which is essentially the process of unused synapses being cleaned up by the immune system. New connections can be formed, but the old ones never go away. It is possible that propionate (created by Short Chain Fatty Acids in our large intestine) may be a cause for autism. - Ellen Bolte's theory is that autism is caused by the bacterium C.tetani after it enters the blood directly after the protective microbiome has been damaged by antibiotics (leaky gut). ### Chapter 4: The Selfish Microbe - The most widespread microbes make us not-quite-sick-enough so that we can continue to move around and spread them. The most virulent diseases don't spread too much since they too fast. - In most people, the problem is not how to boost immunity but how to dampen it. Our immune systems confuse harmless things with dangerous ones. e.g. Allergies to common things are treated by smothering the immune system using antihistamines. - Hygiene hypothesis: Increase in allergies ties with an increase in hygiene and therefore too few infections at an early age. This could not be proved, and a strong counterargument is that if immune cells are lying idle in hygienic environments, why don't they just attack the whole microbiome? - How does the immune system identify that which is external but acceptable (food, good microbes) or internal but to be attacked (unused synapses to unlearn/forget)? - Where cooperation helps in spreading genes, groups are selected over individuals. Animals and their microbiota have always co-evolved (mitochondria are essentially very simple bacteria), and hence evolution selects for the best combination of human + microbial genes (called the *holobiont*). - The immune system has different types of cells: macrophages consume threatening bacteria, memory B cells attack specific targets, T helper cells help in communication between other cells, T-regs calm down an immune response. - Immuno response is triggered by antigens, molecules coming off the surface of invading pathogens. But pathogens and our microbiota both have antigens coming off them. - Evidence suggests that our microbes know how to increase the number of T-regs to prevent the immune system from attacking them. Each species has its own way of doing this. - The cholera pathogen *V. Cholerae* uses diarrhea as a way of spreading, just as the immune system uses diarrhea as a way of flushing out germs. It has copied the immune system mechanism to its own advantage. - Leaky Gut: When a pathogen is able to convince the body to open the protein walls of the gut lining and get in the bloodstream. When the microbiome is damaged, pathogens are able to reach the gut lining, triggering the immune response to open the cell wall, and hence cause a lot of diseases. ### Chapter 5 - Germ Warfare - The use of antibiotics has been on a steep rise since 1945\. Farmers started giving antibiotics to chickens to get them to grow fat. They may be causing the same effect in humans by disturbing the microbiome. - In the 1950s, antibiotics were successfully prescribed for premature or malnourished babies to get them to gain weight. The broader implications of this were ignored in medical research. - In the western world (and generally everywhere), antibiotics are prescribed indiscriminately even when in a large majority of cases they are useless or overkill, resulting in antibiotic resistance. - There is a statistical correlation that microbial imbalance caused by taking antibiotics can cause autism (or any other disease triggered by dysbiosis), but there is little hard evidence yet. - Broad-spectrum antibiotics kill pathogens and benevolent microbes since they cannot distinguish between them. The diversity of microbiota reduces rapidly and can take weeks or sometimes years to recover. - Antibacterial products (other than alcohol) like triclosan have no scientific basis, and can actually cause more infections by killing resident bacteria and allowing new opportunistic ones to take root. - *Streptococcus* may be the cause of OCD. It normally just causes strep throat but occasionally triggers an autoimmune response that harms basal ganglia, thereby making us unable to choose between one of the multiple possible actions. This causes "twitching" like Tourette's Syndrome. OCD patients are often obsessed with washing hands. This may be because streptococcus can survive hand washing better than other species and hence might be triggering a mental reward cycle to perpetuate itself. ### Chapter 6: You are what they eat - The Giant Panda is genetically carnivorous. It only manages a vegetarian diet with the help of microbes that break down cellulose. - It is difficult to study nutrition in isolation since reducing one component must increase some other and vice-versa. - We don't need to look at the diets of cavemen or other centuries-old people to know what we are "meant" to eat. We can look at remote tribes untouched by modernization. One of the biggest differences is fiber. Modern western diets are low in fiber and the decrease corresponds neatly with the obesity epidemic. Switching to a high fiber diet can promote *Akkerminsia Municiphila* as mentioned in chapter 2, thereby reducing obesity. - A plant-rich diet makes for a "lean" set of gut microbes. ### Chapter 7: From the very first breath - Babies are sterile in the uterus. Their microbial colonies are seeded by getting covered in vaginal and fecal matter in the process of being born. - The vagina is covered in *lactobacilli* which can kill other harmful bacteria to protect the baby. They can also breakdown milk, hence the baby can maximize energy extracted from milk. - C-section births have shot up in the last few decades. These babies are more prone to dysbiosis related diseases and other infections since they do not get the microbes from their mothers (as naturally born babies do). - Over 130 types of Oligosaccharides are found in human breast milk. They nourish not the baby directly but nourish its microbiome instead. They also prevent pathogens from taking hold by occupying the specific attachment points pathogen use to latch on to the body - this is distinctly natural selection. - The composition of breast milk changes as the baby grows older. It contains more oligosaccharides in the early and more lactose later on. - The microbial composition of breast milk also changes. Gut microbes in the mother are transported to the breasts so that the baby can consume them for a more diverse bacterial colony. There are far more microbes in the early days than there are later (when the baby's microbiota is already stabilized). - Bottle feeding takes away all of the advantages of breastfeeding. Bottle milk is mostly cow milk, and infant formula has nutrients but has no bacteria. Babies need very specific types of microbes at specific stages of growth. bottle feeding has none of the adaptations necessary for this. ### Chapter 8: Microbial Restoration - The "autointoxication" theory of diseases: Microbes rot the remains of our food in the colon and the organ produces all kinds of diseases. The widely adopted solution was to remove the colon. - **Probiotics** - Elie Metchnikoff suggested that eating bacteria (e.g. as yogurt) can cure autointoxication. These were the first Probiotics. - The line between food supplements and drugs is blurring and there isn't a lot of legislation yet. - Probiotics deliver a small amount of bacteria to the gut, but it difficult for such small numbers to set shop, or work harmoniously with existing bacteria, or not crowd out others which are also useful. The effects are therefore not very predictable. - For complex illnesses like type 1 diabetes and multiple sclerosis, probiotics are too little too late. - **Faecal Microbiota Trasplantation** - Take poop from a healthy person and put it in a sick person's gut to restore their microbiome. - Other animals also eat feces - Cure rates are as high as 95% after two rounds of treatment - Since stool is not medically regulated, there have been no formal clinical trials - a lot of doctors remain unconvinced. - Open-Biome is a non-profit stool bank that works similar to a blood bank. They screen donors for health, maintain supplies, and ship samples. - **Prebiotics** - Probiotics need constant replenishment. Prebiotics promote the right kind of microbial growth by supplying the raw materials for it. - Speculation: Can we tailor probiotics and fecal transplant like we personalize and choose at sperm banks? **Read Next** : More [book reviews](https://kislayverma.com/category/books/) on this blog ### Preventing "go-around" with Platform Thinking URL: https://kislayverma.com/preventing-go-around-with-platform-thinking/ Last updated: 2026-07-22T12:46:55.000Z Consider this fairly common scenario. A team builds a system that is meant to be used by the entire company e.g. a video management platform (VMP). This system is supposed to take care of all needs like video storage, editing, bitrate optimizations, delivery/CDN, etc so that no one else has to deal with them. You just bring your video and all else is taken care of. As a result of this overarching goal, only one interface is exposed to the user of the system to consume its capabilities - the bring-your-own-video API/UI. This is great for the major use-cases and teams start adopting this new Video-Management-Platform quickly. At the same time, new requirements start popping up which require only content distribution, or optimization, or only storage. However, none of these capabilities are available for use in the VMP. So to unblock themselves, teams start going around the entire system and building their own specific, small solutions that are just enough to meet their needs. ![](https://kislayverma.com/content/images/2021/01/go-around.jpg) End result? We are back, kind of, to square one. The overarching VMP was intended to solve all video-related needs in one place. But in the final accounting, the organization still has scattered (and duplicated) bits of video capabilities, each tailored to the needs of the team that built them. ### What went wrong The team that built the VMP took the product experience as defined at that time (just bring-your-own-video) and embedded it into the system architecture literally. They hid all system capabilities behind the *opinion* that there should only be one way to use them. As a result, when opinions changed, there was no way to leverage the existing capabilities because they can only be used in the context of bring-your-own-video. This tight coupling at the system architecture level meant that other teams that had to deal with a changing landscape (typically product-specific/vertical teams) had no choice but to bypass the entire stack and build their own things. This problem of go-around in systems that are expected to be central/platform/generic occurs often. The best designs often abstract the most, and while this is a great characteristic in small systems and end-user products, it turns out to be an expensive mistake in building large scale architectures. This is because most large software systems are composed of the ability to do multiple similar things. If abstracted behind the facade of the larger product, these capabilities become inaccessible in other scenarios. We lose agility in responding to change since we no longer have the building blocks to create new things. This has ramifications beyond just technical coupling and duplication. Products that do something well also often do it only in a certain manner. Strongly abstracted systems simplify many things, but they can lock the organization into patterns of behaviour. If there is only one way of using the system’s functionality, the organization often organizes along the same lines in behaviour (and vice-versa). Three outcomes are possible. 1. Use cases get force-fit into the product. While this is manageable if the leadership is keeping an ear to the ground, the likely outcome of this is the accumulation of tech debt in a previously solid product and all the bad things this eventually leads to. 2. Technical teams go-around the central product to reinvent the wheel in specialized ways. This is a waste of engineering resources. 3. Business teams modify their processes into sub-par versions because that is the only version of the process that the technology can support. This is bad for business beyond just the technology team. ### Scope of the go-around decision One argument against all of this is that product teams should simply have modified the product to unlock the capabilities or the team that owns VMP could have done it for them. This is possible but complicated because it requires something very difficult - [coordination between two teams](https://kislayverma.com/independence-autonomy-and-too-many-small-teams/). The first step is to determine what changes are needed in the existing system. Either team independently, or the two teams jointly need to figure out what is to be changed and what it will cost. This exercise might be time-consuming by itself in poorly documented systems. Now that we know what changes are to be done, who should do them? Should the VMP team do them by dropping some other things it was planning to do. Or should the team which wants the change do it in an unfamiliar codebase whose operational responsibility it does not have? Both are difficult decisions. In opposition to all this ambiguity, there is a simpler choice - just build something small and quick for the new requirement. It doesn’t have to be great as long as it serves a limited purpose. We can always talk about consolidation with VMP “later”. Go-around limits the scope of the decisions that the team has to make and is one of the reasons it is the route taken so often. ### Platform Thinking to the rescue Applying platform thinking to the problem offers a simple (not easy) way out - separate the capabilities of the system from the specific uses of those capabilities. Also known as the [Golden Rule of Platforms](https://kislayverma.com/the-golden-rule-of-platforms/), this allows a team to identify core building blocks of a platform that can be used to build more than one product. Thought of in this manner, building a VMP means first identifying all the capabilities that need to be present to build it, building these capabilities independent of the requirement for a VMP. The more we can build these capabilities as standalone constructs, the better it is for the architecture in the long term because these are the blocks that will be difficult to bypass. An existing, atomic system that offers a single functionality and is easy to integrate with is our best insurance against changing needs. I’m not suggesting that we build feature complete versions of all these sub-systems. What we should build is a structure that identifies them as independent, self-contained constructs that have a definite boundary and purpose. Richer functionality can emerge over time within that boundary. A video encoding system should be identified and built which does only encoding and has interfaces only for that purpose. The types of supported encodings supported can grow over time, but we should first identify the scope of this system. The plan to build a VMP on top of this has no role to play at this time. ### Perspective: product-first to platform-first One trick that makes the platform perspective possible is the inversion of the design approach from product-first to platform-first. A typical system design approach would start from product requirements and then design a system that would fulfill these requirements. The resulting design might be modular, maintainable, etc, but its structure, like the mindset of its designer, is tied to the product requirements. It only evolves as the requirements of THIS product change. ![](https://kislayverma.com/content/images/2021/01/product-first-thinking.jpg) The image above is a typical component diagram that you might expect for our VMP. The problem here is that all those modules paint a misleading picture. While they represent capabilities that the system builders think of as independent, they are actually independent “only within the context of the larger product”. The product perspective embedded in the design is likely to create co-dependent sub-components rather than independent systems. ![](https://kislayverma.com/content/images/2021/01/implicit-product-boundary.jpg) This product perspective in system design is part of the reason why coupling emerges even in microservice architectures where different microservices are expected to be independent of each other. The real problem lies not in the architecture/design pattern but rather in the design mindset. Platform-First thinking approaches this problem statement approximately bottoms up. We analyze the product requirement to [identify the underlying capabilities required to build it](https://kislayverma.com/platforms-and-dogfood-everywhere/). Once we have this list, we forget about building the bigger product and focus solely on the sub-parts and design/build them all by themselves. The resultant architecture, thus, grows outside in (all systems have their own requirements and boundaries) and bottom-up ([lower complexity systems are built first and then composed into higher complexity systems](https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/)). ![](https://kislayverma.com/content/images/2021/01/platform-first-thinking.jpg) Once these systems are ready, we switch back to building the main product. The platform components continue to operate standalone and any team with divergent requirements can use them on their own, thereby removing the notion of go-around completely. ![](https://kislayverma.com/content/images/2021/01/the-platform-solution.jpg) While this inversion of design perspective in no way ensures that independent components will emerge (developers are human after all, and it is difficult to not think of the [main deliverable and the deadlines around it](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/)), it makes it far more likely. ### Scope of the go-around decision Pre-existing platform components are a deterrent to the dreaded go-around. For one, the capability that is required for a new use case might already exist in a perfectly reusable manner. Even if there are enhancements required in the existing component, it is easier to approach them because the scope of the component will be lesser than the complete VMP. The cognitive load of understanding the system and the implementation overhead of making the changes are both likely to be much smaller - likely far lesser than building something ground up. The technical and the delivery incentives both align in favour of adopting and enriching the existing systems rather than going around them. ### Summing it up Designing a system with only the final product in mind has the flaw of fencing all aspects of the design with a boundary defined by the product experience. Any variation on the requirement becomes hard to accommodate and the system becomes less nimble. This problem can be mitigated by identifying the capabilities required for the product as standalone systems, building them independently, and then composing them into the necessary product experience. This creates versatile building blocks that can be combined in multiple ways to create new products as the need arises. **Read Next**: Designing [extendable data models for platform systems](https://kislayverma.com/platform-nuts-bolts-extendable-data-models/) ### Perhaps we shouldn't be so well connected? URL: https://kislayverma.com/perhaps-we-shouldnt-be-so-well-connected/ Last updated: 2021-01-16T05:36:46.000Z I’ve been watching the events unfolding in the US political scene with a sort of dread fascination and thinking about the role technology has played in this. While it is good to see technology companies responding in some way to the madness of Donald Trump, the timing is so suspect that it creates more mistrust than faith in my mind. FB et al are jumping in the convenient direction - all this cancelling should have been done 2, if not 4 years ago. But then, is a for-profit company obliged to have a moral imperative, high-minded mission statements notwithstanding? The far-right and far-left have always been at the forefront of anti-intellectualism in all countries. In an increasingly polarized world, since FB, Twitter, and others are identified as technology companies rather than media companies, these shenanigans further erode the trust in technology to build a better world for everyone and in fact further the “fake news” narrative instead of setting tech as a custodian of empowerment. I’ve read a few histories of the first world war and one of the things that most commentators observe is that much of the carnage resulted (at least partly) because the scale and destructive power of the weapons far outstripped the communication capabilities at the disposal of the commanders. Huge attacks would be launched without the ability to properly manage the information. To me, our current scenario feels like the exact reverse of this. The internet has allowed us to communicate and spread information/propaganda on a global scale, but I do not think that human beings have the mental/emotional capability to deal with it. Nor do we have the structural frameworks to navigate this mess of data. Our collective fictions like society, country, etc that have helped us grow as a species so far are being pulled in so many directions that they are beginning to mean completely different things for different people, effectively making them meaningless as a unifying force. Perhaps the solution is to not connect everyone on the planet. A smaller world, connected by more individual choices and technology which supports individuals rather than scale. And at the root of it all, the basic idea of loving a person for themselves, not because they are part of our favourite collective. ### Hindsight in 2020 URL: https://kislayverma.com/hindsight-in-2020/ Last updated: 2026-07-22T12:46:55.000Z 2020 was a milestone year in all our lives. From acknowledging that such a plague had visited us, to pretty much living indoors for 9 months (and counting), to hearing horror stories of death and loss of livelihood, political turmoil the world over, getting used to a continuous stream of Zoom/Team/Hangout meetings while never actually meeting your colleagues - it has all been a handful. At the end of the year, I wanted to look at some of the things I managed to achieve, and some of the things I managed to learn. These are the highlights of my 2020. #### Personal 1. Moved on from my Uber gig and got a new job at [Curefit](https://www.cure.fit/?ref=kislayverma.com). 2. Got the strongest ever in my life. Went from \~20% body fat to the 8-10% range. From a previous max of 60 kg deadlift (8 years ago) to \~150 kg in April. Lost some ground during the lockdown but not all the way. 3. Learnt how to drive a car. 4. Learnt how to cook a very basic meal. #### Writing 1. Started my personal website instead of blogging on other platforms. 2. Blogged consistently - 51 articles this year (target was 1 per week), up from 9 last year (target was 1 per month). 3. Started a weekly newsletter about technology and teams that build technology. It now has \~1200 subscribers. I published the first edition on July 24 and a total of 19 editions in 2020. 4. Built and released [Rulette Server](http://demo.rulette.org/?ref=kislayverma.com) as an easier interface for [Rulette](https://kislayverma.com/content/files/2026/07/rulette.html). 5. Just for kicks, I self-published Rulette documentation and case studies as [a book on Amazon](https://www.amazon.in/Working-Rulette-Mastering-business-management-ebook/dp/B089S7NWS6/ref=sr%5F1%5F1?dchild=1&keywords=rulette&qid=1610172147&sr=8-1&ref=kislayverma.com). A lot of people have asked me about how I get the time to stick to writing, or working out or any of the other things I mentioned above. So I thought I will put some of my learnings and realizations of the year in writing. None of this is new, but some of them have registered deeply for me this year, and others I find valuable enough to merit repeating. ### Be Consistent Having a schedule and sticking to it is the single most powerful I did - writing when I just wanted to chill with Netflix, working out despite a hectic day of work, cooking when ordering-in felt so much more convenient. Every instance of doing these things added up. A lot of research suggests that our thinking patterns change after we repeat any activity over some time. We start liking those activities. Till you reach the “liking it” stage, just show up, day after day. Trust the process. ### Learn the basics I hate it when developers say that they don’t need to know how a piece of technology works internally as long as they can use it to good effect. To me, peering inside the hood of my tech has always been important. However, I realized that I was doing the same to other aspects of life e.g. fitness. At the beginning of 2020, I had been working out irregularly for about 12 years. I had done whatever workouts a bunch of trainers told me without understanding why. And at the end of all this time, I didn’t have any remarkable physical gains to show for it. In January ‘20 I decided to learn about the fundamentals of muscle building and fitness. Following Youtube channels like [AthleanX](https://www.youtube.com/channel/UCe0TLA0EsQbE-MjuHXevj2A?ref=kislayverma.com) , [Jeff Nippard](https://www.youtube.com/user/icecream4PRs?ref=kislayverma.com) , [Shredded Sports Science](https://www.youtube.com/channel/UCXrqErU%5FTjqiHAHJkzITAvg?ref=kislayverma.com) , and others, I finally got into the fundamentals principles of fitness and bodybuilding. While this world is as much of a rabbit hole as the tech world, there are a few basic principles of biomechanics and biochemistry that underpin everything. Understanding the theory behind exercises has helped me understand what I am doing, find replacements when I can’t do my usual gym training (this was a blessing when I had to cut over to home workouts) and made the pain of training more palatable, at least intellectually. It helped me understand how much misinformation surrounds us, and how to steer clear of it. Learn the basics. It can completely change the way you operate. ### Spend time consciously Not spending time consciously is the biggest reason of not having any time to spend. To extract more out of a day, we have to deliberately choose to do some things over others. Without this, it is difficult to be consistent in the long run. I advised creating a schedule for consistency earlier. That’s great, but where most of us spend our time is in the *nothings* between two tasks. Time just goes by over one more Youtube video, 10 extra minutes over a coffee, and so on. And at the end of the day, I often feel guilty for doing all these normal things that I like doing. I felt that I had *wasted* my time. However, I realized that there the difference between *wasting* and *spending* is one of being conscious of things. Taking an active, conscious decision to do one thing over the other forces me to evaluate the decision at hand. I can choose to do what I planned to do, or I can choose to watch another episode of Star Trek. Choosing the latter, however, forces me to create an alternative plan for when I am going to write. A very conscious self-dialog. For me, this active budgeting made the trade-off clear and removed the guilt which I had felt earlier. I now try to have an active awareness of how I am spending my time, hence there is no waste, only choices that I make. I can change them if I do not like the outcome. Choose to do the things that you are doing, even if the choice is to do nothing. ### Build a personal brand 2020 was [the year of the passion economy](https://anquetil.substack.com/p/-the-passion-economy-why-now?ref=kislayverma.com). A few articles I wrote this year went viral on HN/Reddit and helped me connect with some of the best folks in the tech industry. For me, this really drove home the importance of having personal identity in a niche, how underrated this idea still is or how shallowly this is done by “hustlers”. A personal brand is about creating good odds for yourself. Extremely unlikely things are possible at scale. There is immense power in people recognizing your name and abilities beyond the specifics of your job. One viral tweet/article/anything can lead to very interesting results. And it takes effort to create a brand. We need to identify what we want to be known for, find our unique voice to express our ideas, and dive deep into the community to connect with others like us. It goes beyond writing a blog post or two. But it is absolutely worth it. ### Just do it I put this thought at the last because above all the earlier musings, this is the one I want you to leave with. If you want to do something, anything, just doing it in any way is infinitely better than planning to do it in the perfect way. Many people have told me that they have many ideas for writing and want to start a blog. However, some just keep polishing that one article, or keeping looking for the perfect platform, or making a long list of topics so that they can keep writing for some time, or keep thinking what to write about. All this planning ensure that the writing never starts. Same thing with exercise. People first want to figure out the best diet on the internet, or find a good gym, or home workout is boring, or any thousands of reasons why workouts will start the coming Monday. They never do. Write a short article. Write that documentation for your team at work. Write an email to your friend tweet. It doesn’t matter if it’s been said before. Just start writing. There is no perfect diet. There is no perfect gym. Your workout shoes are fine. Just start exercising. Just do it. Wishing all of you a great 2021! ### Why and How to use Feature Toggles URL: https://kislayverma.com/why-and-how-to-use-feature-toggles/ Last updated: 2026-07-22T12:46:56.000Z I mentioned feature toggles a couple of time in one of my recent articles on [increasing deployment speed](https://kislayverma.com/how-to-speed-up-software-delivery/), and they have been a topic of discussion in my team at work, so today I want to dig into them in some detail to see what they are, how they can be used, and what may be the problems of using them. ### What are feature toggles Feature Toggles (also sometimes called Feature Gates) are a kind of configuration used to switch specific features on and off at runtime. By *on* and *off* I mean that they control whether or not certain code paths are executed (on) or not (off). They are used by developers when they want to change some existing functionality or introduce new functionality but they want to do it in a controlled manner, rather than having the changes go into effect as soon as the code is deployed. Feature Toggles need not only be on and off, although that is how they are most widely used. Since they are a type of runtime configuration, they can be defined in any manner whatsoever. E.g. They can be used to enable certain new features only for employees (before public rollout) or only in certain geographical locations. In all cases, however, they are a means to switch behaviour i.e. whatever meets the toggle criteria gets the feature, others don’t or get the older behaviour. Some people have told me that they think of this as the same as A-B testing or canary deployments. Feature Toggles can be used for both of these, but serve a fundamentally different and far more operational purpose. We don’t want to observe behaviour of people in different groups (A-B) or do gradual roll out of new features (Canary). The objective is to be able to deploy code safely and then be able to do targeted damage control by switching off specific features if they cause trouble in production. The inverse of this is to rollback entire deployments, possibly containing multiple features, if any of those features goes bad. ### Why use Feature Toggles Feature Toggles have risen to greater and greater popularity under the agile school of thought which loves to ship code early and often. The problem with this paradigm is that shipping fast can cause bugs in production when a change is introduced. Feature Toggles are an attempt to solve this problem by decoupling deployment of the code of a change from the actual release/activation of that change. We put the new change under a feature toggle configuration (which is set to off originally), and then after testing etc merge the code to the deployment branch. This is good for multiple reasons, the primary among which is that our fellow developers get our changes quickly and can use that in their work rather than working on stale code and then having to merge branches with lots of changes conflicting with theirs. This significantly reduces the time spent in integrating other’s code. Code fast - Merge fast - deploy fast - release whenever. E.g. Let’s say that you optimized a currently inefficient method in the code. It seems to do well in testing environments, but we cannot be sure of the change unless we expose it to full production traffic. We can use a feature gate to conditionally expose this method in production. We deploy the change to production under a feature toggle set to off (Friday night deployment? No problem!). Then when we are good and ready to do so, we set up all monitoring tools and enable the optimization. Now we can monitor system behaviour and in case something bad comes up, we can disable the change and go back to fixing things. ### Implementing Feature Toggles Since Feature Toggles are a form of runtime configuration, they can be managed via any framework that a team uses for this. Below I highlight the three most popular mechanisms in my experience. #### Application Configuration Files The most common of implementing feature toggles is the application configuration file. Most application development frameworks support some version of this (JSON, YML etc), and this is most likely the first place that teams start from. This is a great start, but the problem is that in order to toggle the feature, we have to deploy a code change in these files, which might cause its own set of problems (maybe other changes have already been merged to the deployment branch). It is also a slow process. #### Application database The second best place to do this is to store all feature toggles in the application’s own database and to evaluate the toggle condition by reading from the database. In this model, every application maintains a database for runtime configuration and manages access to this via direct DB access or via some API+UI combination as they see fit and necessary. Of course optimization techniques like caching can be employed here to reduce database load. #### Central Configuration Management Service As some of you might have guessed, the logical next step in this journey is to have a central repository of all dynamic configuration. All applications fetch their configurations from this system (via push or pull model) and typically store the toggles in-memory to reduce the access time. This is a slightly more complicated architecture but it brings clear benefits beyond the obvious one of having a single system and consistent way of managing toggles across different applications. Since feature toggles and other dynamic configurations can significantly change the behaviour of a system, beyond a certain scale there arises a need to manage and audit changes made to these configurations. A central system can manage such ancillary requirements in one single place, instead of all teams having to deal with them on their own. ### Coding Time So let’s try coding this thing a little bit. We will stick to the example of having optimized a method in the code but trying to control its release in production. Originally the code might look something like this. ``` public void mainBusinessLogic() { //some business logic here unoptimizedMethod(); //Some more business logic here } private void unoptimizedMethod() { // Some shitty code here } ``` To use feature toggles using whatever method we have for keeping dynamic configuration, the first thing to do is to read the configurations and keep them refreshed at runtime. We can implement a simple thread based mechanism to periodically pull the feature toggle configurations from our data store. ``` public abstract class ConfigClient { private static final long REFRESH_INTERVAL_MS = Duration.of(5L, ChronoUnit.MINUTES).toMillis(); private Map configurationCache; private ConfigurationRefresherThread configurationRefresherThread; public ConfigClient() throws Exception { // Load the properties one time so that we can fail app startup on any problem. loadProperties(); // Start thread for periodic reloads after this ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor(); executor.scheduleAtFixedRate(this.configurationRefresherThread, 0, REFRESH_INTERVAL_MS, TimeUnit.MILLISECONDS); } public Optional getConfigValue(String property) { String props = configurationCache.get(property); return Optional.ofNullable(props); } private void loadProperties() { Map propertyMap = getProperties(); if (propertyMap != null && !propertyMap.isEmpty()) { configurationCache.putAll(propertyMap); } } // Override this method to read from were the configurations are stored protected abstract Map getProperties(); /** * This thread refreshes the property cache */ public class ConfigurationRefresherThread implements Runnable { @Override public void run() { log.debug("Refreshing configurations..."); loadProperties(); } } } ``` This class creates a thread called *ConfigurationRefresherThread* which runs every 5 minutes, loads the data from the configuration store (in the *loadProperties* method), and stores it into an in-memory map. The actual mechanism for reading the configuration store can be plugged in by extending this class and implementing the *getProperties* method. Now we have the feature toggles in memory. The change to use this is a simple if-else in the main business logic. ``` public void mainBusinessLogic() { //some business logic here if (Boolean.valueOf(configClient.getConfigValue("enable-optimized-method").orElse("false"))) { optimizedMethod(); } else { unoptimizedMethod(); } //Some more business logic here } private void unoptimizedMethod() { // Some shitty code here } private void optimizedMethod() { // Some good code here } ``` To engage the optimized code, we just set the value of the “enable-optimized-method” property in the configuration store to “*true”*. If something goes wrong, we switch it back to “*false”*. Either change will reflect in no more than 5 minutes. ### The problems While I cannot recommend the use of Feature Toggles any more strongly, there are certain drawbacks that you need to be aware of when you use them. #### Backward compatibility Since the idea of feature toggles is to, well, toggle between old and new code, you cannot just delete the old stuff and write new stuff, or modify the behaviour in-place. All changes have to be made backward compatible so that in case the new implementation has any problems, we can revert back to the old implementation. While this kind of code is ideal in principle, in practice it is far simpler to just change the code. You should expect to deal with some tricky situations when doing low level design. #### Two failure modes Similar to the previous point, since code using feature toggles has two modes of running, it also has two modes of failure. Both the toggle-on and toggle-off code paths have to be tested is we actually expect to be able to toggle between the two. #### Cleaning up the toggles If we use feature toggles as a general practice, the toggles can start accumulating in code and things can start looking messy. So it is important that once a change is stabilized, we should go back to the code and remove the toggle we used for it. This is in a sense the antidote to the two previous problems. Ideally we should not let code exist in toggle-mode for a long time. This will reduce the code complexity, testing effort, and cognitive overhead of understanding that code significantly. #### A shortcut to proper modelling Feature toggles are meant to be short term switches in code paths. If you see some toggles expressing richer system semantics than just on-off (e.g. if toggle values are pending/created/in process/complete), then it means that somewhere our domain modelling is weak and we are trying to plug that gap with ad-hoc configuration values. We should keep an eye out for such scenarios and revisit our system model when we come across them. ### Architecture Pattern: Orchestration via Workflows URL: https://kislayverma.com/architecture-pattern-orchestration-via-workflows/ Last updated: 2026-07-22T12:46:58.000Z ### The problem in modelling business processes Most business processes involve doing multiple things across multiple systems. Imagine onboarding a new vendor onto a B2B platform. When a vendor is onboarded, we might have to create an identity for it, trigger some sort of verification process, issue new API keys and credentials to it, and so on. All of these steps typically happen across multiple teams and technical modules. ![](https://kislayverma.com/content/images/2020/12/multi-step-business-process.jpg) Or imagine something with a shorter life span like an item getting ordered on an eCommerce site. An order has to be created, payment accepted, inventory blocked, and a confirmation email sent - arguably in an all-or-nothing manner. How should we implement such systems? One way to do this in the object-oriented/REST style is to identify a *primary entity* (Vendor/Vendor Service in the former example, Order/Order Service in the latter) which owns the entire operation. This primary entity invokes all the other components involved to make sure that the process runs end to end without any problems. If the process is asynchronous in some way (e.g. if verification of vendor documents is handled manually offline), the primary entity carries the state necessary to make sure that it can continue the process from where it was paused. ![](https://kislayverma.com/content/images/2020/12/primary-entity-ownss-orch.jpg) In many cases, this is a perfectly acceptable design. It keeps things simple. However, as systems get larger and business processes get more complex, some problems start emerging. One technical problem that can already be seen is that the business logic in the primary entity is very tightly coupled to all the other entities. As any part of the order taking process changes, the logic in the primary entity has to be updated continuously. The team which owns the primary entity is now forever in the path of every team which wants to modify any part of the business process. Roadmap/bandwidth negotiations abound. Another problem is imposed by microservice and other distributed architecture patterns. Not only are vendor/order services logically coupled to inventory, verification, email and other services, they are also obliged to handle the vagaries of communication over the network and the various kinds of error handling and performance overheads that [distributed systems](https://kislayverma.com/content/files/2026/07/distributed-systems-1.html) are prone to. [Orchestration of operations across multiple services](https://kislayverma.com/design-review-checklist-for-distributed-systems/) can be a very challenging task. In large scale systems, these two problems are enough overhead that the primary entity soon feels like it’s riddled with technical boilerplate and debt. It often becomes more and more dangerous to modify it, and shipping velocity suffers. We need a better way to implement long-running, multi-component processes than the object-oriented paradigm offers. Enter the **workflow** design/architecture pattern. ### The workflow pattern The workflow pattern is a powerful means of modelling business processes. We create a workflow management system/component whose primary responsibility is to model each action in the business process as a step. This series of steps constitute the entire business logic are executed one after another. A workflow can be thought of as a directed graph that invokes multiple components at each node/step to achieve the system objective. ![](https://kislayverma.com/content/images/2020/12/workflow-mgmt-for-orch.jpg) This is different from the object-oriented model where processes are abstracted as object behaviours and their specifics typically hidden behind object APIs. The workflow pattern explicitly turns this inside out by talking about the steps of the process as first-class constructs. While workflows are graphs first and logic later, in the object-oriented style, we typically encapsulate the graph inside objects and methods as an implementation detail. Using the workflow pattern means moving all orchestration responsibilities out of the core system components and into a separate component which only deals with the definition and execution of workflows. Executing all the steps, handling errors, retries and breaches of SLAs are primary system objectives of the workflow management component. The core components expose APIs offer only some core capabilities and have no/minimal context of when and from where they are being called. Most of the “business” logic goes into the workflow component, which can now be used to compose the capabilities offered by different core components in whatever manner the business demands. This is especially useful in distributed architectures because orchestration is a critical part of the system and no single component is often equipped to deal with this heavyweight activity without getting too deeply coupled with the other components. Extracting all of the orchestration (persistence, state, error handling etc) out of all the components which are getting orchestrated results in a nice layered, “smart pipes” architecture. This style is sometimes called by some other names like Saga pattern (in the context of transaction handling across multiple systems). ### Stateful Orchestration Most uses of a workflow based design are in cases where we require the workflows to be long lived and have heavy synchronization overhead. In such scenarios, it usually (though not necessarily - remember that a workflow is an abstraction layer - the implementation details are relatively less important) makes sense to make our workflow management stateful. This means that the workflow implementation itself remembers how far it had run, whether it succeeded or failed and whether things can be retried in case of failure. ![](https://kislayverma.com/content/images/2020/12/stateful-orchestration-for-builtin-tracking.jpg) By putting all of the workflow tracking information inside the workflow management system, we are making it heavier and more capable while making the core components simpler. To know whether we generated a new vendor’s API keys or not after they were successfully verified we ask neither the verification system nor the API management system, we ask the workflow management system about the state of the vendor onboarding workflow. Another benefit of having stateful workflows is that it is easy to tell how far each business process has run so far. In a way, status updates come for free! ### Mixing declarative and imperative to create layered abstractions Workflows allow us to compose declarative and imperative styles of programming in a layered manner. Seen as a sequence of steps, a workflow is an imperative expression of how a problem should be solved. Seen as a single unit, each step in the workflow is a declarative expression - “Do X”. However, doing X can be a whole workflow by itself. So a workflow-driven system weaves a combination of declarative and imperative styles into a layered architecture. ![](https://kislayverma.com/content/images/2020/12/combine-declarative-and-imperative-in-layers.jpg) Layers are good. It reduces the cognitive effort required to understand any component. If a component seems to be doing too many things, then this is often a good indicator that we are missing some kind of layering opportunity that will [separate core functionality from the use of that functionality](https://kislayverma.com/platforms-and-dogfood-everywhere/) . This is, of course, general observation and can be applied at any level of code. If we are working inside a single component that has multiple smaller modules inside it, we can apply this pattern to [compose these components](https://kislayverma.com/platform-nuts-bolts-flexible-decision-making-with-rule-engines/) effectively. At an architectural scale, we can use this same construct to build [layered domains and architectures](https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/) which can give huge productivity gains if done right. Workflow-based design, with its mix of declarative and imperative paradigms, offers us a way to do exactly this. Each layer of the emergent architecture provides higher-order functionality which is composed of the capabilities of the layers underneath. A workflow allows end-users to be hidden from the numerous steps involved in doing a logical business activity, and hence serves as an abstraction layer above the core components. As a result, what we get is not just technically layered architecture, but also layering from a business and organization perspective. Business processes can also build on top of each other - this can be a powerful tool in designing an effective organization. ### Workflows as Products As I said above, workflows are ad-hoc compositions of lower-order capabilities into higher-order ones. What drives the choices of what capabilities to compose and in what order? Obviously the requirements of the business. While lower-order capabilities like sending mail, creating order may be relatively generic, they can be composed in ways which are specific to use-cases or specific business scenarios. If we consider the superset of all the business requirements as an aggregate, what we get is a Product. A [product is a specific set of choices](https://kislayverma.com/products-are-not-platforms/) made to create a specific user experience. So a different way of looking at a workflow-based design is to say that each workflow represents a part or whole of a product, while the underlying core components \_may\_ constitute platform building blocks. In this sense, it is technically acceptable to have multiple workflows for similar things like order placement or vendor onboarding since each would represent a specific business process or use-case. From an organizational perspective, however, having different processes to do similar things might constitute wasted effort and overhead. The ability to easily identify the major workflows in the product helps us improve organizational processes. In the early days of any system, we often see a lot of changes happening because the end-user experience changes often and this seems to affect the entire stack over and over. The core methods and APIs of the system seem to be getting modified again and again with if-else conditions. While this flux is not ideal, it is actually pointing towards the fact that our system seems to have fewer layers than required to address the numerous use cases of the business. A good way to deal with this problem is to identify each of these use cases as a workflow, and then identify the common pieces one or more of them need, and so on. Each of these levels of recursive is a logical layer in our system’s design. Of course, if we can do this before we implement the system, then it will save us all a lot of refactoring. However, I often find it technically acceptable to ship quickly so that we cover a good number of real use-cases before we try to identify too many unnecessary commonalities and abstractions. ### Caveat Emptor Calling workflows a separate architectural pattern is only a change in perspective. No matter what kind of design approach we take, we will always have to do all the things that make up the business process. Focussing on the process as a first-class concept puts the definition and management of the steps in the spotlight instead of brushing it inside an API boundary and considering it an implementation detail. Like most things in software architecture, this is a trade-off. If the workflows are many and complicated, the workflow pattern can bring them out in the open and make them easy to understand and manage, while keeping core components relatively simple. On the other hand, if the workflows are short/simple/few, using workflow can introduce unnecessary overhead and make the code difficult to understand. Adopting a full-blown workflow management system like [Camunda](https://camunda.com/?ref=kislayverma.com) is overkill if you don’t have long or very complicated workflows. In my experience, the best uses of workflow style emerge as an artefact of refactoring done on object-oriented systems to reduce the accumulating “tech-debt” related to orchestration. As I have mentioned earlier on in this article, it is perfectly acceptable to start out by modelling a workflow as the behaviour of a primary entity. A lesser concern is that the workflow management system now becomes the hub of all action and we have to careful in making sure it is fault-tolerant, scalable, and resilient to problems in the downstream components. I say this is a lesser concern because these are well-understood problems and careful analysis of the design and implementation choices can help us keep them at bay. **Read Next**: [Combining rule engines and workflows in platform architectures](https://kislayverma.com/platform-nuts-bolts-flexible-decision-making-with-rule-engines/) ### Managing developer identities in autonomous teams URL: https://kislayverma.com/managing-developer-identities-in-autonomous-teams/ Last updated: 2026-07-22T12:46:58.000Z I'm trying out a new point based blog format, instead of the usual prose style thing that I and most everyone does. I compose a lot of my articles like this and then change them to the prose style later. I want to see if this makes for easier reading and conveys the ideas better, in the same form that I thought of them. Let me know what you think in the comments. Let's talk about autonomous teams, evolution of developer self-identities, and a manager's role in all this. --- 1. [Simon Wardley](https://twitter.com/swardley?ref=kislayverma.com) says that all teams and business problems need three kinds of solvers. He calls them pioneers, settlers, and town planners. Quoting from [his book](https://kislayverma.com/book-review-wardley-mapping/): 1. **Pioneers** are brilliant people. They are able to explore the never before discovered concepts, the uncharted land 2. **Settlers** are brilliant people. They can turn the half-baked thing into something useful for a larger audience. They build trust. 3. **Town Planners** are brilliant people. They are able to take something and industrialise it taking advantage of economies of scale. 2. All engineering teams start from not having a system, to building the first version of the system, to gradually maturing and standardizing the system. Each of these phases is suitable to a unique temperament, 3. Good software is a potent force multiplier. By building good products, developers are often left with only tweaking knobs to achieve tremendous output. 4. If the mandate of a team is defined by the work they do rather than the problem they solve (a "data download team" is unlikely to do anything other than download data), then as the team builds better technology to handle their job, it is also setting itself up for increased work-dissatisfaction. Good software developers and teams are always making themselves redundant in their current form. 5. If the team is defined by a system, some developers (pioneers) will start getting restless at this point. 6. The only way to satisfy them is more greenfield system building in the current or different team. This is fine if we have the bench strength of people who want to take over an existing system and make it better as well as a transition plan. If not, managers will be stuck with the current developers, who will, eventually, quit out of boredom. With them goes their domain expertise and tribal knowledge. 7. Because the entire team’s identity is based around the ownership of a system or set of components, there is a limit in terms of system maturity beyond which first the pioneers and then the settlers stop being interested in being parts of the team’s mission. Conversely, there is a limit to the chaos of experimentation that town planners and settlers are willing to tolerate. 8. We see the breaching of these limits very often in organizations - the most tell tale sign is components whose owners have all moved on to other teams (or companies). This is partly an operational problem (no single owner of the component), but most fundamentally it is the problem of tasking someone to “build a system” for some purpose. The task is finite in scope and engagement, and the builders obviously move on once their job is done. They don’t have any incentive to hang around once the current system objective is met. 9. What can we do to keep all three types of developers engaged in a team? This problem gets worse the better the team’s software is - poorly written software will keep its owners working on it for a long time plugging leaks, rearchitecting etc etc. Good, powerful software gets boring really quickly. 10. One solution is to define the purpose of a team and its members in terms of a business problem rather than in terms of systems. They are in charge of solving a business problem, and the systems they work with or build are incidental to the problem solving process. Such a team is often called an “autonomous team”. 11. An **autonomous team** is a team which is responsible for solving a specific business problem. The members of this team cut across system boundaries to solve business problems 12. Autonomous teams are a great way to engage all three types of developers since they can be different stages of evolution all at the same time. 1. There may be components which are highly evolved and are being treated as the organization’s standard or platform for something. 2. There may be other components which are becoming tried and tested and need to be stabilized now. 3. There are ways/tools of solving the problem that the team has never before explored and need to start playing around with. 13. All three types of developers can cycle through these types of components in the autonomous team. Pioneers can tinker around with experimental stuff, settlers can stabilize prototypes and town planners can drive standardization/platformization of stable products. 14. Unless a business problem ceases to be a problem for some reason, an autonomous team’s work is never done. It is forever building on top of what they built in the past to climb the value chain and attack the problem at a higher and higher level. Sure the gains become smaller and harder over time (law of diminishing returns), but the team is always on the lookout for newer tools, while it still runs the business with the systems it has already built. 15. It is critical that developers understand this distinction in how roles are defined. Unfortunately, developers in many companies are insulated enough from the business goals that business achievements or metrics fail to inspire them sufficiently. ![](https://lh4.googleusercontent.com/kmP9DuXSXXCQ4TWimwjFfJUt_TYwK2zNBn64Avr0T7JtEEz-cwyhmQhzl_u0aky9V-3ZaISr_ZaTix7DxowtbrJ3bATU1Clp1NNMXHmI3KygFw43CrRzlSZ-v0mHynu9seLi4-6P) 16. They think only in terms of tech stack or scale, and get far more fired up by technical numbers and achievements. This ties a developer’s self identity deeply with the systems they build, the tech they use, or the technical challenges they surmount rather than with the actual problem they are trying to solve. 17. A lot of attrition in companies happens when developers feel “there’s not much left to build”. When I joined Myntra, a colleague of mine said that our team’s job was nearly done. 5 years after that conversation, we were still building that problem. 18. But in an autonomous team, a developers profile cannot be locked in and MUST climb up the value chain. We are what we do, and so our self-identity also cannot remain fixed if we are not to be "bored". 19. Therefore, it is important for managers to coach their teams in this process of growth. This means that they themselves need to be looking to climb the value chain in the mission of the team. “More of the same” is easy for both managers and developers because both can operate from a place of familiarity instead of having to find new common ground of expectations in an environment with changing goals. 20. "more of the same" is far easier than honest self reflection and scanning the horizon for the next challenge. Growth is not easy and invariably some members of the team will not be comfortable with it. 21. Managers need to drive home the idea that developers are not bound by what they are doing but what they can bring to the table in terms of solving the business problem at hand. It is far better for the entire team if the developers can become champions of “what next” in this quest rather than having tasks being handed over to them. This is the essence of autonomy and purpose. 22. All that said, autonomy is not for everyone. In my experience a lot of developers (and people in general - as Erich Fromm eloquently describes in his book “Fear of Freedom”) don’t take well to autonomy. If they are not invested in the mission and engaged in the process, they will inevitably end up following what others(teammates, manager, product manager etc) want to do and will feel unmotivated and uninspired because “nothing much is left to be done”. 23. This applies in general to tech and scientific progress. It is easy to see over large time horizons but difficult to keep track in the day to day work, but it applies nonetheless. Every advancement in technology removes more and more of the automatable things from our lives or makes them very fast/trivial. This leaves us with more and more time on our hands. It is up to us to decide what to do with this time we can browse social media or pick up a new hobby. This is the personal equivalent of picking up bigger problems to solve or doing the same things over and over again and getting frustrated in the process. 24. So if you want to manage an autonomous team, you have to look out for signs of this behaviour. Get them engaged in your mission and problem statement. If this doesn’t work, don’t be surprised that they soon walk out. 25. The challenge of the manager is to get his team to engage with the problem statement, and then to constantly move them up the value chain of the problem space. 26. The ability to do this, in my opinion, the hallmark of a great manager/leader, because being able to reinvent ourselves by not getting tied to static self-identity is how we truly grow in life. Self-disruption is the key in personal growth, and someone who can induce us to it is worth working with/for. **Read Next** : [Layering microservices and domains](https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/) for manageable microservice architectures ### How to speed up software delivery URL: https://kislayverma.com/how-to-speed-up-software-delivery/ Last updated: 2026-07-22T12:47:00.000Z The whole theme of the last decade or so (maybe more) has been about agility and techniques that enable agility. CI-CD, DevOps etc have become a critical feature of this, and yet I often see a lot of friction when it comes to deploying. There is a magical aura around deployment which makes it something special, and this impacts how many of our teams work in each of delivering software. ### Why deploy fast No matter how beautiful our system architecture, how elegant our code, and how solid our test suite - the only way we get to make an impact on our business and our customer’s lives is when we actually deploy code to production. Before deployment, code is just an intellectual exercise, like an interview question. Deployment turns this intellectual property into an economic proposition. So it seems like a no-brainer that [we should be deploying code as fast as possible](https://ankit-gupta.com/blog/2020/why-frequent-deployments-matter.html?ref=kislayverma.com). And yet engineering orgs struggle with this activity. It is a documented fact that the overwhelming majority of outages are a result of new changes deployed. This makes sense - if nothing changes, then things are far less likely to break down. So deploying code has obvious risks. Most of the currently popular engineering practices are aimed at increasing the rate of software deployment while mitigating the risk of failure arising from this increased rate. Pretty much anything works at small scale and companies, but as the scale of the enterprise grows in terms of software or team size, context becomes increasingly important, and reverse engineering from development back to design/implementation can expose a lot of inefficiencies. If we agree that deploying software fast with high quality is a worthy goal, then I want to apply a pipeline-like perspective to the process of delivering software so that we can identify bottlenecks and broaden them to make the process more efficient. Applying the Theory of Constraints means that we model the software delivery flow in reverse, identify the bottlenecks, and optimize them one after another. ![](https://lh5.googleusercontent.com/fLvoJsU8bu79AvqM0Paf5yNK7oRIqSV1U6XBQotp_SQhx6YCkEuhp8JCzhsKtlBgp8_pphCXQdSmnTmIn9_BwVrIJAUliCVZkcsRXjxtN56Rfzund_PaBXT-KpKSHwVkCxgWYiWF) ### Deployment causes outages This is the most widespread argument against deploying changes to production and as I mentioned above, industry data supports this. It has therefore become the flagship argument for treating deployment as some type of sacred activity. However, there is a lot of subtext to this top-level problem that merits looking into. While this is a whole universe of topics in its own right and includes multiple disciplines, I want to call out that if fast and safe deployments are the desired objectives, then we cannot do without efficient means of : 1. **Discovering when things go wrong** \- To reduce outages, we need to be able to detect them. [Observability](https://kislayverma.com/observing-is-not-debugging-and-other-misnomers/) tools are indispensable for this, and automated tests running in production as very effective too. We need these tools regardless of whether we deploy frequently or not, so we might as well go ahead full steam and reap the benefits! Additionally, the organization should have dependable on-call support protocols (preferably manned by developers, but at least by some sort of central ops team) to respond to alerts. 2. **Mitigating the problem** \- Since we are talking about outages caused by deployments, the most powerful tool we have for mitigating the problem are canary deployments and automated rollbacks. Coupled with monitoring tools, they give a pretty solid safety net for when things go wrong. Feature gates are another extremely powerful tool to manage deployment risk. They allow us to deploy code that can later be engaged under close supervision instead of having every change going into effect as soon as it is deployed. 3. **Debugging the problem** \- Whether you like the logs-metrics-traces approach to observability or the [event-based approach](https://kislayverma.com/publish-events-not-logs/) recently gaining ground, you need tools that allow your team to rapidly nail down the cause of the problem. Without these, you will be helpless even in the face of known problems because you won’t know where they stem from and why. 4. **Fixing the problem** \- Once the problem has been identified, product and development teams need to have processes that let them determine the priority for the fix and get the fix out of the door as quickly as possible. Note that the ability to deploy a bug fix fast is contingent on our ability to deploy ANYTHING fast. ### Deployment takes time Before we dig into the specific, I would like to point out that thanks to [feature toggles](https://kislayverma.com/why-and-how-to-use-feature-toggles/), ***deployment is not release***. Code for a feature may get deployed without coming effect because it is toggled off - this will become relevant below. Let’s consider the actual process of deployment. Some teams I have worked with in the past have argued against deploying often because deployment takes up a lot of time from the team. Here again, we can look at the various steps typically involved and identify the ways in which they can be made efficient. 1. **Merge all the code to the deployment branch** \- This is often, but not always, done by the engineer who happens to be on the hook for deploying the application (for whatever reason) - and it shouldn’t be. Merging code to the deployment branch is part of the development cycle but often becomes part of the deployment cycle because any code sitting in the deployment branch becomes “active” when deployed. This dependency can be broken by using feature toggles as mentioned above. Developers should merge code with appropriate implementation of feature toggles. 2. **Trigger and monitor deployment** \- While the gold standard of all of this is CI-CD process,, a good enough compromise (IMO) is an automated deployment process which includes building the deployment branch, running automated sanity/integration tests, canary deployment with auto rollback followed by rolling/incremental deployment. While this might take time depending on the infrastructure being used, it should not be a hands on activity for the dev team. They should get actively involved only if they see failures. 3. **Validate that system is stable post deployment** \- This is essentially the activity of detecting and debugging outages in production when caused by deployment and we have already discussed techniques to make this efficient. Another aspect here is the actual release of the code that was shipped (remember - deployment is not release). Release should be the responsibility of individual feature developers and they should judge when they should use feature toggles to expose their code to its users. As such, this is not a process of the deployment cycle. ### Testing takes time When I was working in the supply chain team at Myntra, we had this practice of testing [end-to-end](https://kislayverma.com/testing-strategies-for-agile-teams/) . This meant that a feature (any feature) could be considered signed off by QA only if we could make a whole set of orders of different types all the way from the customer cart to logistics. With testing environments being broken often due to untested code, needless to say, sign-offs were a bitch, both for devs (too slow) and for testers (too painful). Widening the testing bottleneck typically means working on two fronts: 1. **Testing individual changes** \- Since we are trying to deploy fast - it means that changes will be coming in fast, and each complete feature might involve changes to multiple teams and systems. Having a QA handoff on this path is extremely inefficient. I believe testing of individual changes is part of the development process. Whether by unit testing, or automated/manual integration tests, developers should verify their changes are safe to deploy and functioning as intended (including feature toggles). [Consumer Driven Contracts](https://martinfowler.com/articles/consumerDrivenContracts.html?ref=kislayverma.com) are a powerful tool in testing small changes and individual components that I am surprised are not very popular. 2. **Testing complete features** \- This is the testing of the complete customer experience, and this is something that QA teams can own and drive using rigorous automation. While this is an important component of testing, it should be understood that this is a process parallel to the mainline software delivery path and will always run slightly (and only slightly) behind the latest production system (we had put it inside the delivery path at Myntra because, at the time, we were not doing a good job with testing individual changes). It is however, very powerful in detecting regressions and also acts as a repository of information about how the system is expected to function. Splitting these two tracks and making separate teams responsible for both significantly broaden the testing bottleneck. ### We deploy complete features Feature development often works as a batch process. Developers design for the complete feature, and then implement the complete design in one go. In a way, the complete feature becomes the unit of work for developers. This is okay for features of trivial size, but when we work on large features which touch many parts and layers of an application, this style of working creates a huge risk because multiple developers are doing the same thing. If all the changes come together just before deployment, there will be tons of conflicts and it is difficult to predict if all features are still working correctly. I’m not even talking about the agile process of identifying sprint stories - this is an even lower level than that. If a code change makes sense on its own (e.g. schema changes to create a new table in a database, core business logic that is not yet exposed via an API), what stops us from deploying it? By holding it back, we are causing two problems: 1. Our teammates working in the same areas of code haven’t seen our changes - we may be stepping on each other’s toes. Getting small changes out there quickly avoids confusion later. 2. If the change is logical and deployable, then why NOT deploy it? After all, the larger the deployment, the larger the risk of things breaking on deployment. However, there are certain prerequisites to be able to implement features in small bites. 1. **Design large to implement small**: We spoke earlier about how treating the entire feature as the unit of work can be troublesome. However, to be able to break down work into smaller units, we need to design (at least roughly) the entire feature. This is necessary to make all the small pieces fit correctly. The output can look something like this - “we need a table to store these data points, REST APIs to insert single and bulk records to this table, and integration with external service X to check for Y before we insert”. We identify coarse-grained system boundaries that the feature implementation will impact and identify the changes that will be made. Coarse-grained is a relative term here, and this is a recursive process - if the feature is very large and spans multiple systems then we have to keep applying this breakdown till we reach the code if actual code that will be written in each of those systems. 2. Test small changes deeply - Since each change is small, developers should be able to establish easily that each of them works exactly as intended. The fastest mechanism for doing this is via unit tests that mock external dependencies. However, any mechanism is fine as long as it establishes the deployment safety and intended behaviour of the change being published. Code reviews should look out for these tests. 3. Small, fast-moving code reviews: One feature need not travel as one pull request. As a result of our design exercise, we can now implement the feature in a series of very small pull/code review requests in line with the design boundaries defined above. Large pull requests are never going to get reviewed as thoroughly as small ones because they take a much larger amount of effort on the part of the reviewer. The flip side of this argument is that we should have team processes in place to get the code reviewed very quickly. I have published some [guidelines for reviewing distributed systems code](https://kislayverma.com/code-review-checklist-for-distributed-systems/) \- check them out for some pointers on things to look out for. After these changes, we are pushing very small, verified to be safe changes down the deployment pipeline, much of which is already automated and needs minimal manual intervention. But no amount of automation or tooling will help make our deployments fast or safe if we insist on pushing large changes in batches. Large change sets lead to poor testing/reviews lead to unstable systems. ### Software Delivery as Change Stream To me, this is a mental shift in terms of thinking of software delivery as a stream of small changes that are intended to compose into a feature on deployment instead of thinking in terms of moving features from the developer’s laptop to production servers. There is no such thing as a feature that is “done” - everything is always evolving and changing. So instead of viewing features as statically-bound things that we ship, we should think of them as a set of changes that we need to make within certain system boundaries. Some people call this ”Flow” of work through software organizations. This is enabled as much by the adoption of agile techniques in the development phase as it is enabled by evolving out-of-the-box infrastructure capabilities in the deployment and operations phase. ### TL;DR Deploy as frequently as you can. No matter what you think the cost of deploying is now, it will only be greater later on. Unless you are building some life-or-death related software or something which is highly regulated, my vote goes to deploying as fast as you can. Like the broken-window theory, making rapid deployment an engineering objective will directly give rise to a robust engineering culture and practices which will help your organization in the long run. **Read Next** \- [Moving faster (not just fast) as an organization imperative](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/) (aka Build momentum not velocity) ### Book Review : Wardley Mapping URL: https://kislayverma.com/book-review-wardley-mapping/ Last updated: 2026-07-22T12:47:00.000Z The name of this article is misleading because the book doesn’t actually have a name. It is written as a series of articles - so I’m just choosing to call it “Wardley Mapping”. The images here are taken from the book. --- I have been slowly going over Simon Wardley’s free book on Medium where he walks us through the problems he faced, which then led to the development of the wardley mapping technique. It’s a long book, but the 6 first six chapters cover the most important parts at a high level and provide a nice closure to the problems and proposed solutions. So I am publishing the highlights from these as a package for those who are just starting out exploring Wardley mapping technique. For the uninitiated, Wardley Mapping is a technique which plots the user needs and the capabilities a company needs to meet them on the axes of customer-visibility and stages of evolution. Things can be more or less visible to customers (Current location of Uber cars is highly visible, the map data powering this is not), and they may be nascent in nature or highly evolved (Uber’s app is highly customized, the map data powering it is fairly standardized). These two dimensions form the “map” of a business problem, and along with some concepts taken from the military, are the core of a process of using the map to decide the next steps the “general” (the person responsible for making a decision) should take. ![](https://kislayverma.com/content/images/2020/10/the-first-wardley-map.jpeg) Although Simon developed this technique as a CEO of a company and used it to drive the overall strategy, I find it extremely useful as an engineer with a more limited, engineering centric view as well. I have been trying my hand at it, and it allows me to map customer experiences to the capabilities we need throughout the ecosystem at all levels. This was always a feature of architectural discussion for me in the past, but knowledge of wardley maps has made it possible for me to bridge the gap between a product specification and architecture doc a little more formally. I can plot and discuss the entire path from roadmap to frontend needs to backend engineering to infra capabilities more crisply with product managers and other engineers. The other interesting part for me is that it allows for questioning of claims around capabilities already existing in the organization. E.g. If we claim to have a well established authentication system, then it should necessarily be very cheap for me to start using it. I can start out with an assumption that this is a commodity and then question this if I spend myself spending a lot of time on using this system. It’s a good way to keep teams and systems accountable to their claims. This is one of those “messy middle” books - one that calls for focus on continuous evaluations and evolution instead of focussing just on the method. There are no “answers” in it as such, but there is definitely a powerful tool for discovering answers for your own situation. Life isn’t simple, so why should strategy be - it is good to see this acknowledged as a first class concept in a business book. The following are my notes and learnings from the book, which is delightfully accessible to people with all levels of expertise. There is no substitute for reading the original anyway - I absolutely recommend it. ### Chapter 1 :[ On being lost](https://medium.com/wardleymaps/on-being-lost-2ef5f05eb1ec?ref=kislayverma.com) ![](https://kislayverma.com/content/images/2020/10/5-principles.jpeg) - All models are wrong but some are useful - We rarely learn from past experience especially when it belongs to others or when it conflicts with our perception of how things are. - SunTzu had described five factors that matter in competition between two opponents. Loosely speaking, these are: — Purpose, Landscape, Climate, Doctrine and Leadership. - **Purpose** is your moral imperative, it is the scope of what you are doing and why you are doing it. It is the reason why others follow you. - **Landscape** is a description of the environment that you’re competing in. It includes the position of troops, the features of the landscape and any obstacles in your way. - **Climate** describes the forces that act upon the environment. It is the patterns of the seasons and the rules of the game. These impact the landscape and you don’t get to choose them but you can discover them. It includes your competitors actions. - **Doctrine** is the training of your forces, the standard ways of operating and the techniques that you almost always apply. - **Leadership** is about the strategy that you choose considering your purpose, the landscape, the climate and your capabilities. It is to “the battle at hand”. It is context specific - There is not one but two questions of why in chess. I have the why of purpose such as the desire to win the game but I also have the why of movement as in “why this move over that?” - There existed two very different forms of why that mattered — purpose and movement - We had no map of the environment, no visual means of describing the battle at hand and hence no understanding of our context. Without maps, I didn’t seem to have any effective mechanism of learning from one encounter to the next or even a mechanism of effective communication - What is it that made maps useful? The first, and most obvious thing, is that they are visual. - The second thing to note with a map is it is context specific i.e. the battle at hand. - ...six absolute basic elements for any map which are visual representation, context specific, position of components relative to some form of anchor and movement of those components. - If I can’t separate out what is context specific, then how do I determine what is Doctrine i.e. universally applicable from that which is leadership i.e. context specific? - First, the process of **strategy is not a linear process but an iterative cycle**. The climate may affect your purpose, the environment may affect your strategy and your actions may affect all. Second, acting is essential to learning. Lastly your purpose isn’t fixed, it changes as your landscape changes and as you act. There is no “core”, it’s all transitional. - In order to understand the process of air combat, John Boyd developed the \[OODA loop. This is a cycle of observe the environment, orient around it, decide and then act ### Chapter-2 :[ Finding a path](https://medium.com/wardleymaps/finding-a-path-cdb1249078c0?ref=kislayverma.com) - All firms are in a constant state of flux and the ecosystem it lives within never stands still. - Why do things change? In any industrial ecosystem, novel and new things constantly appear as a consequence of the desire for companies and individuals to gain an advantage over others. Those things that are useful will be copied. They will spread until the once novel and new becomes commonplace. - Throughout our history, it has always been standardisation of components that has enabled creations of greater complexity. - The desire to differentiate creates the novel, the desire to keep up with others makes it commonplace. - The map has an anchor which is the user (in this case a public customer though other types of users exist) and their needs. The position of components in the map are shown relative to that user on a value chain, represented by the y-axis. - The components of the map also have a stage of evolution. - **Genesis**. This represents the unique, the very rare, the uncertain, the constantly changing and the newly discovered. - **Custom built**. This represents the very uncommon and that which we are still learning about. It is individually made and tailored for a specific environment - **Product (including rental)**. This represents the increasingly common, the manufactured through a repeatable process, the more defined, the better understood. - **Commodity (including utility)**. This represents scale and volume operations of production, the highly standardised, the defined, the fixed, the undifferentiated, the fit for a specific known purpose and repetition, repetition and more repetition. - This evolution is shown as the x-axis and all the components on the map are moving from left to right driven by supply and demand competition. - There is a flow of risk, information and money between components - The components can also represent different types of things. These types represent activities, practices, data and knowledge. - **Step 1 — User Needs** - Critical to mapping is the anchor and hence you must first focus on the user need. This requires you to define the scope of what you’re looking at - a common trap is not to think of your user’s needs but instead to start to describe your own needs i.e. your desire to make a profit, to sell a product or be successful - These capabilities are your highest level components and the manifestation of your user needs - **Step 2 — Value Chain** - ...a value chain can be simply determined by first asking the question of “what is the user need” and then by asking further questions of “what components do we need in order to build this capability?” - Gather a group of people familiar with the business and huddle in some room with lots of post-it notes and a huge whiteboard - On the post-it notes write down the user needs and the top level capabilities required to meet them. - Then for each capability, using more post-it notes, the group should start to write down any subcomponents that these top-level components will use. This can include any activity, data, practice or set of knowledge. - For each subcomponent further subcomponents should then be identified until a point is reached that the subcomponents are now outside of the scope of what you’re mapping - The top-level components (i.e. your capabilities, what you produce, what is most visible to the user) should be placed near the top of the value chain. Subcomponents should be placed underneath with lines drawn between components to show how they are related e.g. this component needs that component - **Step 3 — Map** - Value chains on their own are reasonably useless for understanding strategic play in an environment. This is because they lack any form of context on how it is changing i.e. they lack movement - The largest problem with creating an understanding of the context in which something operates is that this process of change and how things evolve cannot be measured over time - whilst evolution cannot be measured over time, the different stages of evolution can be described - add a horizontal line for evolution. Mark on sections for genesis, custom built, product and commodity - this step is often the main cause of arguments in the group. You will regularly come across components that parts of the group feel passionate about. They will declare it as unique despite the fact that all your competitors will have this. There is also the danger that you will describe the component by how you treat it rather than how it should be treated - There are many causes for this, some of which are due to inertia and the component being a pet project and in other cases it is because the component is actually multiple subcomponents - You can’t outsource mapping to someone else any more than you can outsource learning to play chess to a consultancy. ### Chapter 3 -[ Exploring the map](https://medium.com/wardleymaps/exploring-the-map-ad0266fad59b?ref=kislayverma.com) - Climatic patterns are those things which change the map regardless of your actions. - This can include common economic patterns or competitor actions. - **Climatic pattern: Everything evolves** - **Climatic pattern: Characteristics change** - **Climatic pattern: No one size fits all** - With any business you need to encourage coherence, co-ordination, efficiency and stability when dealing with the industrialised domain. However, the exploration and discovery of new capabilities in the uncharted domain requires you to abandon these erstwhile virtues for experimentation. Any structure whether a company or a team needs to manage both of these polar opposites. - Any significant system will have components at different stages of evolution. At any one moment in time, there is no single method that will fit all. - Invariably there are endless attempts to create a new magic one size fits all method by trying to make a single approach all encompassing or marrying together different stages e.g. lean six sigma or agile lean or prince agile - **Climatic pattern: Efficiency enables innovation** - The story of \[\[Evolution\]\] is complicated by the issue that components not only evolve but enable new higher order systems to appear. - As a component evolves to a more standard, good enough commodity then to a consumer any improvement becomes increasingly hidden. - **Climatic pattern: Higher order systems create new sources of worth** - the story of evolution doesn’t simply stop at efficiency and the consequential enablement in building higher order systems. It also has an impact on value - An idea is something with social value and it is the implementation of that idea as a new act which can create economic value when that act is useful. This process of transformation from social to economic value is known as **commodification** - At the same time that the differential benefit of a component declines, it also becomes more of a necessity and a cost of doing business. This is the process of **commoditization** - This creates a situation where the unit value of something maybe declining but the total revenue generated is increasing due to volume. - **The uncharted domainx** is associated with high production costs, high levels of uncertainty but potentially very high future opportunity. - **The transitional domain** is associated with reducing uncertainty, declining production costs, increasing volumes and highest profitability. However, whilst the environment has become more predictable, the future opportunity is also in decline - **The industrialised domain** is associated with high certainty, high levels of predictability, high volumes, low production costs and low unit margin (Red Ocean) - **Climatic pattern: No choice on evolution** - There exists a secondary impact of the Red Queen phenomenon which is it limits one organisation (or in biology one organism) from taking over the entire environment in a runaway process - **Climatic pattern: Past success breeds inertia** ### Chapter 4 -[ Doctrine](https://medium.com/wardleymaps/doctrine-8bb0015688e5?ref=kislayverma.com) - Doctrine are the basic universal principles that are applicable to all industries regardless of the landscape and its context. - **Doctrine: Focus on user need** - When you look at a map, each component represents a store of capital (whether physical, financial or otherwise). The lines between components represent capital flows from one component to another. - Discussion and data collection are a key part of determining user needs and so talk with them and talk with experts in the field. - There are two important areas where the users and the experts are usually wrong in describing their own needs. - The first area is when a component is moving between stages of evolution - The second area to note is that of the uncharted domain. These needs are both rare and highly uncertain and this means you’re going to have to gamble - **Doctrine: Use a common language** - Instead of using multiple different ways of explaining the same thing between different functions of the company then try to use one e.g. a map. - *\[My Note\] - This is very similar to the concept of “Ubiquitous Language” from Domain Driven design in a software engineering concept* - **Doctrine: Be transparent** - **Doctrine: Challenge assumptions** - **Doctrine: Remove duplication and bias** - You should not only share maps, you should collate them in an effort to remove duplication and bias i.e. rebuilding the same thing or custom building that which is already a commodity. - the same component being on different maps is fine except when we’re saying it’s a different instance of that component - I find useful in helping to highlight this problem is to create a profile diagram. I simply collate maps together, identifying commonly described components and then place them onto the profile. This gives me an idea of both duplication and bias - **Doctrine: Use appropriate methods** - The issue with outsourcing isn’t that the concept is wrong but instead that we have a tendency to outsource entire systems for which we do not understand the landscape. - The problem was not that a highly structured process with detailed specification was correctly applied to industrialised components but that the same technique was also incorrectly applied to components that were by their very nature uncertain and changing. - it’s important to challenge any bias your company may have in your maps - there are a wide range of excuses that are deployed for not breaking up entire systems into components and then applying more appropriate methods. - “we need better experts and specification” - “it’s too complex, splitting into parts will make it unmanageable” - “It will cause chaos” - The truth is usually more of a desire to have “one throat to choke” - “You’ll end up with hundreds of experimental startups” - “complexity in managing interfaces” - **Doctrine: Think small** - In order to apply appropriate methods then you need to think small. You can’t treat the entire system as one thing but you need to break it into components. - teams should be given autonomy in their space and this can be achieved by the team providing well defined interfaces for others to consume along with defined boundaries often described through some form of fitness function i.e. the team has a goal around a specific area with defined metrics for delivery - **Doctrine: Think aptitude and attitude** - **Pioneers** are brilliant people. They are able to explore the never before discovered concepts, the uncharted land - **Settlers** are brilliant people. They can turn the half-baked thing into something useful for a larger audience. They build trust. - **Town Planners** are brilliant people. They are able to take something and industrialise it taking advantage of economies of scale. - you need to populate the cells with different types of people — pioneers, settlers and town planners. - It’s really important to understand that pioneers build and operate the novel. - Don’t fall into the trap that Pioneers build new stuff and hand it off to someone else to run or operate. - **Doctrine: Design for constant evolution** - We need to somehow mimic that constant state of evolution in the outside world but within a company. The solution is to introduce a mechanism of theft which means new teams need to form and steal the work of earlier teams i.e. the settlers steal from the pioneers and productise the work. This forces the pioneers to move on. - if a cell sees something they can take tactical advantage of in their space (remember they have an overview of the entire business through the map) then they should exploit it. - Maps are a useful way to kick-start this process. They also give purpose to each cell as they know how their work fits into the overall picture. - The cells can grow in size but ultimately you should aim to subdivide into smaller cells and maps can help achieve this. - You will however increasingly have to structure the monitoring and communication between cells using a hierarchy and yes, that means you need a hierarchy on top of a cell based structure - the structure causes three separate cultures to flourish. This is somewhat counter to general thinking because the culture results from the structure and not the other way around. It also means **you don’t have a single company culture but multiple** - Doctrine are a set of beliefs over which you have choice. They are something which you apply to an organisation unlike climatic patterns which will apply to you regardless of your choice. ### Chapter 5 :[ The play and a decision to act](https://medium.com/wardleymaps/the-play-and-a-decision-to-act-8eb796b1dff1?ref=kislayverma.com) - There exists two different forms of why in business — the why of purpose (i.e. win the game) and the why of movement (i.e. move this piece over that) - One significant problem around making a choice usually stems from past success and the comfort it brings. - **Context specific play: Accelerators, decelerators and constraints** - the evolution of a component can be accelerated by an open approach, whether open source or open data. - the evolution of a component can be slowed down through the use of fear, uncertainty and doubt when crossing an inertia barrier or through the use of patents to ring-fence a technology. - the evolution of a component can be affected by constraints in underlying components - the very act of open sourcing, if a strong enough community could be created would drive a once magical wonder to becoming a commodity. Open source seemed to accelerate competition for whatever activity it was applied to. - **Context specific play: Innovate, Leverage and Commoditise** - Take an existing product that is relatively well defined and commonplace and turn it into an industrialised utility - Then encourage and enable other companies to innovate by building on top of your utility - These companies building on top of your utility are your “outside” pioneers or what we commonly call an “ecosystem”. - Your “outside” ecosystem is in fact your future sensing engine. - This translates to an increasing appearance of being highly efficient as we industrialise components to commodity forms with economies of scale but also highly customer focused due to leveraging meta data to find patterns others want. Finally, others will come to view us as highly innovative through the innovation of others. - The decision to act can impact the very purpose of your company — the strategy cycle is not only iterative, it’s a cycle - Gameplay is context specific. You need to understand the landscape before you use it. The purpose of gameplay is once you determine the possible “wheres” that you could attack (which requires you to understand landscape and anticipate change from common economic patterns) then you look at what actions you can take to create the most advantageous situation. ### Chapter 6 :[ Getting Started](https://medium.com/wardleymaps/getting-started-yourself-e1a359b785a2?ref=kislayverma.com) - understanding your landscape, the context that you’re competing in and having a modicum of situational awareness is not a luxury for strategy, it is at the very core of it. Inspiring vision statements, well trained forces, a strong culture and good technology will not save you if you fail to understand the landscape, the position of forces and their size and capabilities. - there are some basic practices that an MMORPG will teach you - The importance of situational awareness - The importance of aptitude - The importance of collaboration - The importance of preparation - **business strategy is normally a tyranny of action — how, what and when — as opposed to awareness — where and why**. - abundant communication mechanisms rather than efficient communication can itself become a problem without good situational awareness as new players constantly ask “where should we go” as they run around in a daze - **There also tends to be an element of political conflict between the business units and the shared services** and in the worst cases the shared services function can be viewed as a hindrance. - we need to separate out the delivery of shared services from the identification of what is common. - the best way to achieve this is not to remove budget from the business units (often a political bone of contention) but instead to introduce a co-ordination function. The role of the co-ordination function is to encourage compliance to policy (doctrine) often via a spend control mechanism and to enable sharing between the business units - If it’s important enough for you to create a shared and common service, then there either exists an outside market opportunity or you’re just rebuilding what already exists in the market. - With your shared services group, then you should aim to populate it with small cells of town planners providing industrialised components. Your business units will tend to become dominated by cells of pioneers and settlers providing custom to product and rental services. - start with a small co-ordination team of highly skilled people helping other business units create, share maps and learn from them - You will probably find that some business units start to offer their own home grown capabilities as common components to other business units. Don’t discourage these emergent behaviours - You can always migrate those components to a shared services group at a later date. - Recommended books: - Sun Tzu, the art of Warfare (Robert Ames translation) - Science, Strategy and War by Frans P.B. Osinga - Atlas of Military Strategy 1618–1878 by David Chandler. - The Simplicity Cycle by Dan Ward - Accidental Empires by Robert X. Cringely - Hierarchy Theory, The Challenge of Complex Systems by Howard H. Pattee - The Evolution of Technology by George Basalla - Thinking in Promises by Mark Burgess - Diffusion of Innovations, Everett Rogers. - Customer driven IT by David Moschella - Digitizing Government by Alan Brown, Jerry Fishenden and Mark Thompson - Learn or Die by Edward D.Hess - The Oxford Handbook of Innovation by Jan Fagerberg, David Mowery and Richard Nelson - The Starfish and the Spider, Ori Brafman and Rod Beckstrom - Does IT matter? by Nicholas Carr - Technological revolutions and financial capital, Carlota Perez - The Entrepreneurial State by Marriana Mazzucato - Topographical Intelligence and the American Civil War, Daniel D. Nettesheim. - The Intelligent Investor by Benjamin Graham - Cybernetics by Norbert Wiener - Systems Thinking by Jamshid Gharajedahi - The Age of Discontinuity by Peter F. Drucker - The Red Queen, William P. Barnett **Read Next :** Review and highlights from ["Thinking in Systems"](https://kislayverma.com/book-review-thinking-in-systems-a-primer/) , a fantastic primer to systems and systems thinking. ### Working around the CAP Theorem with Eric Brewer URL: https://kislayverma.com/working-around-the-cap-theorem/ Last updated: 2026-07-22T12:47:01.000Z This is my breakdown and summary of this fantastic [2012 article by Eric Brewer](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/?ref=kislayverma.com) on InfoQ. Although it is 8 years old, it shines a powerful light on many of the issues we are still struggling with in context of micro-service architectures like cross-service transactions, cascading failures and user experience management in distributed systems. --- I have written about [design considerations in distributed systems](https://kislayverma.com/design-review-checklist-for-distributed-systems/) before, and this article does a far better job of highlighting the consistency versus availability concerns that I could possibly do. Especially notable is how as an industry we have moved towards consistency and availability trade-off at various levels of system architecture instead of perceiving an up-front, all-in choice of consistency and availability. This is a direct result of improved practical understanding of working with the constraints of CAP Theorem. We now think more in terms of degrees of CAP rather than binary C-vs-A. Now we focus more on recovery from partition rather picking consistency or availability up front. Hence the emphasis is on being both consistent AND available till the time we must choose one. A network partition is not something the complete distributed System agrees upon all at once. One node may find itself with another node while the rest of the network sees no such issue. Hence the question of whether a partition has occurred is very local. Hence we need more fine grained strategy to identify and deal with partitions. The question of partition is a local decision based on latency of communication. If the latency is higher than acceptable, then we are in partition for all practical purposes. This also happens only intermittently, so not all out system choices should be made around this low probability event. Older architectures would make the C-vs-A choice at this point. However, we now understand that we also have the option of going into "partition mode" where we run the the system in such a way that allows at least limited availability along with the possibility of resolving differences between both sides of the partition after the partition is over. - Allow some operations and disallow others. Which operations to allow is determined by the invariants of the system, whether they MUST hold at all times, and how can we reconcile the differences if they are not mandatory to be enforced at all times. e.g. Allow credit (harmless) but not debit (dangerous). - We can store metadata about events on both sides before we execute those events so that we can serialize the intent once the partition is over. This is especially important when the outcomes are externalized. i.e. in the realm of other system (like debiting a credit card). This is very similar to event sourcing. - We can use data structures like CRDTs to reconcile system state after partition. - The best situation is to use commutative operations - operations that can be squashed into a single commit log ordered along their timelines to generate the final, resolved system state. e.g. the Saga pattern in distributed transactions. --- The following are the highlights of the article which provide more detail into the arguments and insights. These are just to give the essence, the entire article is an absolute must read. --- - The CAP Theorem asserts that any net­worked shared-data system can have only two of three desirable properties - Consistency, Availability, and Partition Tolerance. - The theorem first appeared in fall 1998\. It was published in 1993 and in the keynote address at the 2000 Symposium on Principles of Distributed Computing, which led to its proof. - The easiest way to understand CAP is to think of two nodes on opposite sides of a network partition. - Allowing at least one node to update state will cause the nodes to become inconsistent, thus forfeiting C. - Likewise, if the choice is to preserve consistency, one side of the partition must act as if it is unavailable, thus forfeiting A. - Only when nodes communicate is it possible to preserve both Consistency and Availability, thereby forfeiting Partition Tolerance. - For wide-area systems, designers cannot forfeit P and therefore have a difficult choice between C and A. - The "2 of 3" formulation was always misleading because it tended to oversimplify the tensions among properties. - First, because partitions are rare, there is little reason to forfeit C or A when the system is not partitioned - The choice between C and A can occur many times within the same system at very fine granularity; not only can subsystems make different choices, but the choice can change according to the operation or even the specific data or user involved - All three properties are more continuous than binary. - The modern CAP goal should be to maximize combinations of consistency and availability that make sense for the specific application. Such an approach incorporates plans for operation during a partition and for recovery afterward. - BASE : Basically Available, Soft state, Eventually consistent. - ACID : Atomicity, Consistency, Isolation, Durability - ACID: - In ACID, the C means that a transaction pre-serves all the database rules, such as unique keys. In contrast, the C in CAP refers only to singlecopy consistency, a strict subset of ACID consistency. - if a system requires ACID isolation, it can operate on at most one side during a partition because Serializability requires communication in general and thus fails across partitions. - Operationally, the essence of CAP takes place during a timeout, a period when the program must make a fundamental decision-the partition decision: - cancel the operation and thus decrease availability, or - proceed with the operation and thus risk inconsistency. - Retrying communication to achieve consistency...just delays the decision...retrying communication indefinitely is in essence choosing C over A. - a partition is a time bound on communication. Failing to achieve consistency within the time bound implies a partition and thus a choice between C and A for this operation. - there is no global notion of a partition, since some nodes might detect a partition, and others might not. - Nodes can detect a partition and enter a "partition mode". - designers can set time bounds intentionally according to target response times; systems with tighter bounds will likely enter partition mode more often and at times when the network is merely slow and not actually partitioned - Aspects of the CAP theorem are often misunderstood, particularly the scope of availability and consistency - Scope of consistency reflects the idea that, within some boundary, state is consistent, but outside that boundary all bets are off. - Independent, self-consistent subsets can make forward progress while partitioned, although it is not possible to ensure global invariants. - Conversely, if the relevant state is split across a partition or global invariants are necessary, then at best only one side can make progress and at worst no progress is possible - real systems lose both C and A under some sets of faults, so all three properties are a matter of degree - given the high latency across the wide area, it is relatively common to forfeit perfect consistency across the wide area for better performance. - Another aspect of CAP confusion is the hidden cost of forfeiting consistency, which is the need to know the system’s invariants. The subtle beauty of a consistent system is that the invariants tend to hold even when the designer does not know what they are. - Conversely, when designers choose A, which requires restoring invariants after a partition, they must be explicit about all the invariants, which is both challenging and prone to error. - **Managing Partitions** - Normal operation is a sequence of atomic operations, and thus partitions always start between operations. Once the system times out, it detects a partition, and the detecting side enters partition mode. - Once the system enters partition mode, two strategies are possible. The first is to limit some operations, thereby reducing availability. The second is to record extra information about the operations that will be helpful during partition recovery. - **Which operations should proceed?** - Given a set of invariants, the designer must decide whether or not to maintain a particular invariant during partition mode or risk violating it with the intent of restoring it during recovery. - For an invariant that must be maintained during a partition, however, the designer must prohibit or modify operations that might violate it. (In general, there is no way to tell if the operation will actually violate the invariant, since the state of the other side is not knowable.) - Externalized events, such as charging a credit card, often work this way. In this case, the strategy is to record the intent and execute it after the recovery. - Partition mode gives rise to a fundamental user-interface challenge, which is to communicate that tasks are in progress but not complete. - **Partition Recovery** - The designer must solve two hard problems during recovery: - the state on both sides must become consistent, and - there must be compensation for the mistakes made during partition mode. - It is generally easier to fix the current state by starting from the state at the time of the partition and rolling forward both sets of operations in some manner, maintaining consistent state along the way. - Most systems cannot always merge conflicts. - Conversely, some systems can always merge conflicts by choosing certain operations to be admissible during partition - Using commutative operations is the closest approach to a general framework for automatic state convergence. The system concatenates logs, sorts them into some order, and then executes them. - Unfortunately, using only commutative operations is harder than it appears; for example, addition is commutative, but addition with a bounds check is not (a zero balance, for example). - commutative replicated data types (CRDTs) - a class of data structures that provably converge after a partition - **Compensating for mistakes** - Typically, the system discovers the (invariant) violation during recovery and must implement any fix at that time - There are various ways to fix the invariants - trivial ways such as "last writer wins" (which ignores some updates) - smarter approaches that merge operation - human escalation - Recovering from externalized mistakes typically requires some history about externalized outputs. - Long-running transactions face a variation of the partition decision: is it better to hold locks for a long time to ensure consistency, or release them early and expose uncommitted data to other transactions but allow higher concurrency? - Serializing this transaction in the normal way locks all records and prevents concurrency. - Compensating transactions take a different approach by breaking the large transaction into a saga, which consists of multiple sub-transactions, each of which commits along the way - ATM design : Consistency or Availability? - Strong consistency would appear to be the logical choice, but in practice, A trumps C. The reason is straightforward enough: higher availability means higher revenue - The key invariant is that the balance should be zero or higher. Because only withdraw can violate the invariant, it will need special treatment, but the other two operations can always execute. - Under partition, modern ATMs limit the net withdrawal to at most k, where k might be $200. - When the partition ends...Restoring state is easy because the operations are commutative, but compensation can take several forms (overdraft fee, legal action) --- **Read Next** \- [Understanding distributed system as data pipelines](https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/) ### Observing is not Debugging (and other misnomers) URL: https://kislayverma.com/observing-is-not-debugging-and-other-misnomers/ Last updated: 2026-07-22T12:47:02.000Z I got a lot of feedback and support on my article about [thinking in terms of events instead of messages](https://kislayverma.com/publish-events-not-logs/) when thinking about logs. Charity Majors herself [gave a shout out](https://twitter.com/mipsytipsy/status/1310776966648225793?ref=kislayverma.com) \- so yeah I had a good week :) ![](https://kislayverma.com/content/images/2020/10/Screenshot-2020-10-04-at-10.31.21-AM.png) I am no expert on Observability or tech operations so I try not to write much about it. However, a lot of people raised some concerns enough times that I felt some more clarification was needed. In this article I will try to address some of these confusions. ### Logs are for debugging Several people pointed out that logs are critical for debugging issues in code. When some piece of code is malfunctioning, it is important to be able to trace the execution of this code step by step to determine exactly what is going on in there. Logs are very useful for this, and therefore logs cannot only be about events at boundaries as I had expressed. I view logs with two different perspectives as they are used today. One type of logs are used for observability, and the other are used for debugging. I have no problem with the latter, and I agree that it is important to have these. What I disagree with is the conflation of the two. Observability is a study of the system in motion . We observe the system “from the outside” as it goes about its business. If we have good mechanisms for observability in place, we can tell which parts of the system are working well and which are facing problems. This worldview has certain direct implications. It presupposes that we can identify “parts” of a system (hence my insistence on boundaries), and that we know what it looks like when these parts are not working well (hence the insistence on observing events at boundaries instead of “everything”). We don’t observe to debug - we observe to understand and identify system behaviour. Observing the system in motion also means that we are observing a stream of things as they happen. The only way to make sense of streams is to keep a history of them and work with aggregates. This means, almost by definition, that we need event processing systems to understand what is going on. It is unrealistic to expect humans to operate at this scale by watching logs. So observability based constructs should be aimed towards capturing as much aggregate information as possible and processing it to determine outliers. Debugging on the other hand, is about the system at rest . When we know the whereabouts of a problem, we pause the entire system (that’s exactly the job of a breakpoint in a debugger) and try to isolate the exact problem with specific inputs and scenarios. This is a one-off deep dive, and human expertise is inevitably required here since the system cannot infer the why of its own internals. Logs can be helpful here, but there are plenty of other ways of doing it once we know exactly which parts of the system are malfunctioning. Unit tests are meant for exactly this behaviour - isolate (mock) the externalities and test the system to verify what it does. Logging is one way of doing this, and arguably not the best way. I object to putting ad-hoc strings in Elasticsearch or files as an observability mechanism. It is chaotic at best, and misleading on average. The lack of structure in logging directly gives rise to the need for the other “pillars” of observability - logs are just not enough in their current form. By all means use logs to trace specific parts of a program, but think of this as a separate activity than identifying system boundaries and behaviours. Conflating fine grained debugging with coarse grained behaviours is fundamentally inefficient. ### Logs and Metrics combine to give full operational coverage “Logs and Metrics” is a way of thinking which implicitly assumes the existence of two things - a stream of time stamped messages (in a file or whatever) which humans can read called “Logs” and a database (statsd, prometheus etc) of “named magnitudes” called Metrics. It then goes on to assume that the business of Observability is to correlate these two. Many people will add a third dimension of “traces” to this mix (no one did that in response to my article, which IMO indicates that distributed tracing is a lot less adopted than the blog-osphere indicates). E.g. Logs will have something like : Timestamp 1 : Calling service A with payload XXXX Timestamp 2 : Received response from service A : YYYY Metrics will have svc-A-call-count : 1 svc-A-resp-time : ZZZZ ms We can clearly see that the second piece of information is entirely derivable from the first, if only the first *meant* something. Logs as we see them today are not *information*, they are just *data*. So it takes a human to collate these two things and figure out what is going on. The need for metrics despite having logs actually points to the fact that we are doing something very wrong The event logging paradigm is targeted at solving this problem. Events mean something - they are information that can be automatically processed into higher order information about system health. E.g. Event 1: {eventTime : 1234567, eventType: “request-sent”, target: “service A”, payload: “XXXX“, requestId : “E1”} Event 2: {eventTime : 2345678, eventType: “response-sent”, target: “service A”, payload: “YYYY“, requestId : “E1”} It is not hard to write an automatic mechanism for collating these two into a higher understanding which can be called a *metric* if we want. There is no reason for the events to be human readable (as it would be with logs) because machines can understand them and process them into the real thing that humans want to know - the system is working well (or not). The fact is that some things happen in the real world system, and by “logging” them as arbitrary strings, we lose their semantics except to the trained human eye. We then try to cover this up by introducing more things like metrics into our system that again takes a human to understand. This is clearly a poor way to use technology. Log meaningful events, and use event processing systems to derive information from them. Humans should be overseers, not operators of Observability. ### Logging is about the implementation A lot of people agreed with the idea, but objected to introducing yet another mechanism into their stack. They seemed to believe that they would require a message bus like Kafka for events while at the same time putting their “logs” in files. This was a completely unexpected response for me, and it took some time for me to parse. Just to be clear, “Logging” or “Events” to me is not about the method of delivery (e.g. things sitting on a \*.log file versus going over Kafka/RabbitMQ). The internal implementation does not matter as much as the mental model behind the act of logging. We don’t have to introduce another component - We can put events in files or logs in Kafka. It does not matter as long as we can distinguish between the two and use them appropriately. I feel that this is yet another confusion stemming from the idea of logs as “messages to myself on call” being different from events (something informationally meaningful) and hence the urge to treat them differently. **Read Next** \- [Data change stream are not domain events](https://kislayverma.com/domain-events-versus-change-data-capture/) ### The Revolution will be Unsupervised URL: https://kislayverma.com/the-revolution-will-be-unsupervised/ Last updated: 2026-07-22T12:47:02.000Z This is a guest post by [Bharath Reddy](https://www.bharathkreddy.com/?ref=kislayverma.com) . Bharath works for a large bank helping them with data analytics. He [teaches Machine Learning and AI](https://www.bharathkreddy.com/ml-bootcamp?ref=kislayverma.com) to people with no prior coding or mathematics experience. When not working or teaching - he can be found sketching or spending time with my dogs. --- To understand what I mean by an unsupervised revolution let me first tell you a brief history of AI in 3 acts. ### ACT 1 #### Early years 1956 - 1974 The field of AI research was founded at a workshop held on the campus of Dartmouth College during the summer of 1956.Those who attended would become the leaders of AI research for decades. Many of them predicted that a machine as intelligent as a human being would exist in no more than a generation and they were given millions of dollars to make this vision come true. [\[1\]](https://en.wikipedia.org/wiki/History%5Fof%5Fartificial%5Fintelligence?ref=kislayverma.com) #### First Winter 1974 -1980 Blinded by their optimism, researchers focused on so-called strong AI or general artificial intelligence (AGI) projects, attempting to build AI agents capable of problem solving, knowledge representation, learning and planning, natural language processing, perception, and motor control. This optimism helped attract significant funding into the nascent field from major players such as the Department of Defense, but the problems these researchers tackled were too ambitious and ultimately doomed to fail. Eventually, it became obvious that they had grossly underestimated the difficulty of AGI. This led to many agencies stopping the funding to this field and the difficult years that followed would later be known as “AI Winter”. #### The problems Capabilities of AI programs were limited. Even the most impressive could only handle trivial versions of the problems they were supposed to solve; all the programs were, in some sense, "toys". AI researchers had begun to run into several fundamental limits around memory and computing power that could not be overcome in the 1970s. Most of the statistical methods which existed then, could not be applied algorithmically due to these limitations. ### ACT 2 #### Boom 1980–1987 In the 1980s a form of AI program called "expert system " was adopted by corporations around the world and knowledge became the focus of mainstream AI research. In those same years, the Japanese government aggressively funded AI with its fifth generation computer project. An expert system is a program that answers questions or solves problems about a specific domain of knowledge, using logical rules that are derived from the knowledge of experts. The power of expert systems came from the expert knowledge they contained. They were part of a new direction in AI research that had been gaining ground throughout the 70s. "AI researchers were beginning to suspect— that intelligence might very well be based on the ability to use large amounts of diverse knowledge in different ways. Chess playing programs like Deep Thought were developed around this time using this prevalent paradigm. #### The money returns In 1981 Japanese government set aside $850 million for research on a large computational project, several countries responded with programs of their own. In 1982 physicist John Hopfiled was able to prove a form of neural networks (hopfield nets) could learn and process information in a completely new way. Around the same time Georfrey Hinton popularized a method of training neural networks called “back propagation”. These two discoveries led the revival of this field. #### Second AI Winter 1987-1993 Desktop computers from apple and IBM had been steadily becoming cheaper and faster. They became more powerful than more expensive LISP machines. Similarly successful expert systems were proving to be too expensive to maintain. DARPA which has been a big funding source had a change of leadership, which deemed AI was not the “next big thing”. Simultaneously the spending by Japanese government did not see good returns on their projects. All of this led to the second AI winter. ### Act 3 #### Narrow AI 1993–2011 Increase in computational power as a result of Moore’s law was effectively used with some commercial success by focusing on specific isolated problems, and pursuing that with highest standards of accountability. These successes were not due to some revolutionary new paradigm, but mostly on the application of engineering skill and on the tremendous increase in the speed and capacity of computers by the 90s. Investment and interest in AI again boomed in the first decades of the 21st century, when machine learning was successfully applied to many problems in academia and industry due to new methods, the application of powerful computer hardware, and the collection of immense data sets. Machine learning is a sub field of artificial intelligence (AI) in which computers learn from data—usually to improve their performance on a narrowly defined task—without being explicitly programmed. - 1959 Arthur Samuel. #### DEEP LEARNING BECOMES FEASIBLE 2011 Deep learning is a branch of machine learning that models high level abstractions in data by using a deep graph with many processing layers. A couple of neurons can now approximate any mathematical function. State-of-the-art deep neural network architectures can sometimes even rival human accuracy in fields like computer vision, specifically on things like the MNIST database, and traffic sign recognition. AI has re-emerged with a vengeance over the past two decades—first as an academic area of interest and now as a full-blown field attracting the brightest minds at both universities and corporations. 3 main drivers for this 1. Focus on specific tasks rather than general AI. 2. Tremendous increase in compute power, the most recent RTX card has 10 Mil times compute power than the one MIT labs had back when hopfiled came out with his network. 3. Data availability: Over the last 10 years we have had a million fold increase in the number of datasets which have been curated and made public. AI is now viewed as a breakthrough technology, akin to the advent of computers, that will have a significant impact on every single industry over the next decade. Of course, these successes in applying AI to narrowly defined problems is just a starting point, and the hope is that by combining several weak AI systems, we have a good shot at developing strong AI. This strong AI will be capable of human-level performance at many broadly defined tasks. ### The Next Revolution If you have followed the history of AI, you might have noticed a pattern. - Solution to a big challenge was attempted (AGI) - but failed, - AGI was broken to narrow set of problems (weak AI) - This have been solved mostly using large amounts of data. Most of the successful commercial applications to date—in areas such as natural Language processing , computer vision, speech recognition, and translation, have involved supervised learning, taking advantage of labeled datasets. However, most of the world’s data is unlabelled. The field of machine learning has two major branches—supervised learning and unsupervised learning and a few others but those are not of interest for our discussion. In **supervised learning**, the AI has access to labels, which it can use to improve its performance on some task. Supervised learning excels at optimizing performance in well-defined tasks with plenty of labels. Most of the problems that have been tackled by Machine learning today are by using supervised learning, in this method AI trains on data and measures its performance by comparing its prediction to what is true data. This is why labels are so powerful, they guide the AI agent by providing it a measure of error which the AI agent minimizes. Without these Labels AI has no measure of how successful it is. Increasingly we have tackled problems of increasing complexity by throwing more data at AI/ML models – this gives more features and more labels and we have ever faster computers available. We have already reached a stage where all the state of art algorithms are trained on datasets big enough to fill a couple of data centers and trained on millions of Compute units. ### So What are the problems with this ? #### Supervised AI advances are dependent on resources now. Supervised AI trains on some data it is fed however what we care about most is how well the AI generalizes it training to never seen data before. To ensure AI generalizes well, we have to feed the AI , increasingly large amounts of labeled data – and this is how increasingly AI has advanced – by gathering larger and larger “labeled “ datasets. Increasingly we have tackled problems of increasing complexity by throwing more data at AI/ML models – this gives more features and more labels and we have ever faster computers available. We have already reached a stage where all the state of art algorithms are trained on datasets big enough to fill a couple of data centers and are trained on millions of Compute units. Today, most of the Supervised AI breakthroughs depend on ability to get large labeled datasets and vast compute power. Both of which, rich organizations have in plenty and hence we see most of the breakthroughs coming from places like - Google, nvidea, MIT ,Facebook, Tesla etc. #### Supervised AI generalizes only on data it has seen. Lets say at some point in time we reach a stage where everyone has access to all the data – still the learning of Supervised AI would be limited to what it has seen. Highly anthropomorphic learning I would say. Regardless of how highly trained a supervised AI is – it would not do well in conditions it has never seen before. This is akin to training someone in classical physics all their life and expecting them to understand quantum nature of things. Or training someone to swim and expecting them to execute a good space walk. Supervised learning excels at optimizing performance in well-defined tasks with plenty of labels and would trounce unsupervised learning at narrowly defined tasks for which we have well-defined patters that do not change much over time. However Unsupervised learning makes previously intractable problems more solvable and is much more nimble at finding hidden patterns both in the historical data that is available for training and in future data. Moreover, we now have an AI approach for the huge troves of unlabeled data that exist in the world. Even though unsupervised learning is less adept than supervised learning at solving specific, narrowly defined problems, it is better at tackling more open-ended problems of the strong AI type and at generalizing this knowledge problems where patterns are unknown or constantly changing, for which we do not have enough understanding or large labeled datasets – our hope lies with unsupervised learning. #### No new paradigm The backbone of current supervised learning is to some extent – a single algorithm – Gradient Dissent. Now this algorithm can do only so much and has its own limitations. Finding a way around these limitations was by adopting increasingly novel and complex architectures. But in essence this is like having a knife and we keep changing the shape of it and sharpening it – but a true advancement would be someone discovering chemical properties and inventing a gun. Refinements of a knife is akin to progressing in same generation, discovery of chemical properties and invention of a gun – Leap to next generation. Refinements to accuracy of a gun is progressing in same generation, discovery of nuclear power and exploiting it is akin to leap to next generation. By same analogy we need to think beyond supervised AI for the leap to next generation. #### Curse of Dimensionality The fundamental way in which data is represented in a mathematical form which computers can work on is by projecting each feature of data as a dimension. For ex datasets with your height , weight, age , Blood pressure and cholesterol level would be represented with x axis as height, y axis as weight, z axis as age, and then our comprehension fails us – we cannot fathom things beyond 3 dimensions, however a machine adds more axises and each feature is represented as an axis and any dataset is represented as a point in n-dimensional graph. Now it so happens that as the number of features grows , our dimensions grow as well and as these grow, the distance between each points tends to infinite and mathematically it can be proven that the space between all the points also tends to infinite and the entire data looses its notion of similarity and dissimilarity of any two observations. This is called curse of dimensionality. Our existing supervised algorithms break down at this point. For more physics oriented amongst you think of think as singularity where all equations break down. Unsupervised Learning is well suited to help manage this by finding most salient features in original dataset and reduce the number of dimensions to a more manageable number while losing very little information in the process. ### How does Unsupervised Learning work ? Unsupervised learning works by learning the underlying structure of the data it has trained on. It does this by trying to represent the data with a set of parameters that is significantly different than the number of examples available in the dataset. By performing this representation learning, unsupervised learning is able to identify distinct patterns in the dataset and capture the essence of data. Unsupervised learning makes previously intractable problems more solvable and is much more nimble at finding hidden patterns. Though not as adept as supervised learning at narrowly defined tasks, unsupervised learning is better at tackling more open-ended problems of the strong AI type and at generalizing this knowledge. Take a toy example - email spam filter problem, we have a dataset of emails with all the text within each email labelled spam or not . In unsupervised learning, labels are not available. Therefore, the task of the AI is not well-defined, and performance cannot be so clearly measured. Now, the AI will attempt to understand the underlying structure of emails, separating the database of emails into different groups such that emails within a group are similar to each other but different from emails in other groups.This unsupervised learning problem is less clearly defined than the supervised learning problem and harder for the AI agent to solve. But, the solution is more powerful. The unsupervised AI may find groups which can be categorized as “work” , “friends”, ”promotions”,”spam” etc. AI agent may find interesting patterns above and beyond what we were initially looking for. This is how the human brain works, we are taught a few things like how a hot object burns (labels) but our understanding of the world is not just a sum total of all our memory of labels. Imagine having to take a decision after running through all your memory of what object is hot and what is not!. Our brains have developed some heuristics as survival / evolutionary traits. We have also developed a notion of abstraction – a higher form of knowledge if you will. This means in general hot things would burn, this might be gas stove, embers , running engines etc. And some events cause things to get hot like, being in the sun for a long time, being burnt, being in contact with other hot objects etc. In essence our brain is doing exactly what an unsupervised learning agent does - finding patterns and associating experiences to each pattern and solidifying these into notions about those things. For the same reasons I am of opinion that - unsupervised learning is the next frontier in AI and may hold the key to AGI. The next AI revolution may well be unsupervised! **Read Next** \- [ML will be an indispensable part of this decade's full stack team.](https://kislayverma.com/the-full-stack-team-of-this-decade/) ### Publish events, not logs URL: https://kislayverma.com/publish-events-not-logs/ Last updated: 2026-07-22T12:47:03.000Z Every programmer is familiar with logs - messages we put in our code to figure out where the code execution is and what is going on in the system. Logs are deemed to be one of the three pillars of observability, and are heavily used by developers to understand system behaviour in production environments. While much has been said about structured logging instead of the typical message logging (e.g. “Calling service A with id B...” or “Error occurred while doing X : could not find Y”), I want to push the idea further to answer the question - what to log? I propose we should approach logging from the perspective of [publishing events occurring between systems](https://kislayverma.com/programming/using-events-to-build-evolutionary-architectures/) and subsystems. I propose that there are no such things as “logs”, there are just events that happen between two interacting components, and analyzing the events gives us the health of a distributed system. ## The inevitable march to structured logging Let’s consider the evolution of logging in most systems, especially if a company is evolving alongside into microservices or some other form of distributed architecture. Developers start logging messages into log files, and referring to these log files at runtime to see what the system is doing. Debug info, errors, exceptions all reach these files. Initially the system may be running on very few servers so that developers are able to access each of these log files individually. Eventually though, one of two things happen. Either a monolith has to be run on lots of servers to keep up with scale requirements, or smaller components start to break away into independent systems and the state of the system starts to get distributed. Both of these situations over time demand that logs from all different servers be pulled into a single log aggregation system. So that we can search for them in one place and debug problems effectively. The ELK stack is a popular choice for such aggregation systems. ![](https://kislayverma.com/content/images/2020/09/logging-without-boundaries-1.jpg) But other problems now start popping up. How do we tie together work that was done across multiple systems for a single user action. How do we track interactions done over asynchronous channels like Kafka? What if the software is running across two data centers - how do we identify what is happening where? The most common solution for all these things is to add more metadata to each log message (often via a standard logging library). So we now have datacenter id, correlation id,traces/spans and other such data points getting appended to the log message. Developers are still thinking in terms in terms of strings like this. ``` 2020-09-23 17:08:13+0530 INFO [thread=main] [uId=user54672] [reqId=avsk4ghaioernkva] [cId=5] o.e.j.s.Server - Started @7217ms 2020-09-23 17:13:12+0530 INFO [tid=Timer-0] [uId=user54672] [reqId=avsk4ghaioernkva] [cId=5] c.c.g.t.LocationServiceTimerTask - Updating cities and countries ``` In fact, the log message has effectively become this. ``` { “thread” : “main”, “uId” : “user54672”, “reqId” : “avsk4ghaioernkva”, “cId” : 5, “class” : “c.c.g.t.LocationServiceTimerTask”, “Message” : “Updating cities and countries” } ``` This march from unstructured logging to some sort of structured logging is inevitable in any company that survives long enough to tell the tale. I don’t get why we don’t just accept this and do logs this way from day one. Anyhow... ## What are we looking for? Let’s dial back a little bit and think about why we are logging things anyway. We usually log things to get visibility into the state of our running code. Typical interesting things 1. Errors occurring 2. State of the system when some error occurs 3. Rate of successes/failures or various activities 4. Operations happening on different entities. This small set of sample requirements is so wide ranging that it makes many people say “log everything.” This is neither necessary nor feasible. It is not necessary because most of the things happening inside a system are trivial and boring (think most logs). We could ignore them without any risk. So logging everything would only be cluttering the systems with log statements everywhere for no reason. Logging everything is also not feasible because logging is not free of cost. The system suffers a performance penalty for every log emitted, the wallet of the CTO pays the penalty of storing the logs for search later, and the log aggregation system pays the scalability penalty if every system starts logging everything. It is not possible for the log management system to scale out to handle indiscriminate logging by all other systems. In any case, “log everything” is not much of a guideline. What is “everything”? Every single line of code? Obviously not. Ideally we want to log what matters, but how do we know what matters? “Log everything” doesn't help there, but system boundaries do. ## Boundaries are the happening places No system without some boundaries endures the real world. As part of the design process, developers divide the larger system into smaller logical or physical modules. The boundaries encapsulate the internal details of how things are done from other parts of the system. One way of looking at this is that from an observability perspective, most of the interesting activities in a system happen at its boundaries, while most of the boring things happen inside boundaries. The modules themselves might be very complicated, intricate pieces, but when observing the system, its health and state can be observed at its edges. Boundaries are of many types. They may be REST APIs and message queues, they can be interfaces and abstract classes in the code, they can be the APIs of other libraries being used in a system. The question of why something is considered a boundary instead of something is tied inextricably with the opinions of the designers of the system, and changes dynamically as a system evolveds. So to observe a system effectively over time, not only do we need to identify the seams within it, we also need to be able to systemically keep up with the changes in them. **Where are the boundaries?** Let’s say we have an invoice service, which calculates the value of an order and saves it in a database. Let’s say the invoice generation code looks somewhat like this. ``` public class InvoiceService { public Invoice generateInvoice(Order order) { if (order.ItemCount() > 0) { double amount = 0; long amountCalcStartTime = new Date().getTime(); for (Item item : order.getItems()) { amount += item.getValue(); int taxAmount = computeTax(); //Some complicated tax logic Amount += taxAmount; } long amountCalcEndTime = new Date().getTime(); LOG.info("Amount calculation took {} ms", amountCalcEndTime - amountCalcStartTime); order.setValue(amount); Invoice invoice = convertToInvoice(order); long dbWriteStartTime = new Date().getTime(); InvoiceDao dao = new InvoiceDao(); Invoice persistedInvoice = dao.saveInvoice(invoice); long dbWriteEndTime = new Date().getTime(); LOG.info("Persisting invoice to DB took {} ms", dbWriteEndTime - dbWriteStartTime); return persistedInvoice; } LOG.error("Persisting invoice to DB took {} ms", dbWriteEndTime - dbWriteStartTime); throw new IllegalArgumentException("no items in this order"); } } ``` While this is not bad code, looking at it doesnt tell me what I should be logging. Even more importantly, how can I be sure that any logs I add now will seem important enough to future developers that they retain them through refactoring/code changes. In short, there are system boundaries that aren’t very obvious. For want of boundaries, we cannot tell what is important except by relying on tribal knowledge. Let’s refactor this code into something like this. ``` public class InvoiceServiceNew { public Invoice generateInvoice(Order order) { AbstractAmountCalculator calculator = new ItemAndTaxBasedCalculator(); order.setAmount(calculator.calculate(order)); Invoice invoice = convertToInvoice(order); long dbWriteStartTime = new Date().getTime(); InvoiceDao dao = new InvoiceDao(); Invoice persistedInvoice = dao.saveInvoice(invoice); long dbWriteEndTime = new Date().getTime(); LOG.info("Persisting invoice to DB took {} ms", dbWriteEndTime - dbWriteStartTime); return persistedInvoice; } } public abstract class AbstractAmountCalculator { public Double calculate(Order order) { try { long amountCalcStartTime = new Date().getTime(); Double amount = doCalculate(order); long amountCalcEndTime = new Date().getTime(); LOG.info("Amount calculation took {} ms", amountCalcEndTime - amountCalcStartTime); LOG.info("Invoice amount for order id {} is {}", order.getId() - amount); return amount; } catch (Exception e) { Log.error(e.getMessage()); throw e; } } protected abstract Double doCalculate(Order order); } public class ItemAndTaxBasedCalculator extends AbstractAmountCalculator { @Override protected Double doCalculate(Order order) { if (order.ItemCount() > 0) { for (Item item : order.getItems()) { amount += item.getValue(); int taxAmount = computeTax(item); // Some complicated tax logic LOG.info("Tax for item {} is {}", item.getId() - taxAmount); amount+= taxAmount; } return amount; } throw new IllegalArgumentException("no items in this order"); } } ``` Now there is a very clear boundary between the service and the calculator. Not only is this boundary likely to endure over time since it is more formalized, it is also hardened to take care of logging what happens at the boundary. The logging is also likely to endure better because it is baked into the scaffolding of the system. We can do further refactoring of a similar type with tax calculations and with the database persistence layer ![](https://kislayverma.com/content/images/2020/09/logging-with-boundaries.jpg) While this is a simplified example, the principle of defining boundaries and logging interactions cross them is a valuable one. Embedding the logging within the seams of the system is even better since that way it will not go away unless the boundary itself changes. In this way we can use “what to log” and “where to log” as nudges towards good design rather than ambiguous directives to log “the important things”. ## Structured logs are just half assed event Whether we think of them that way or not, log messages are events that have been poorly thought through. If we look at our system now, we can see structured logs with metadata being emitted from system boundaries. This is literally the [definition of domain events](https://kislayverma.com/domain-events-versus-change-data-capture/)! Domain events are emitted from a system domain when something of potential note to the outside world happens inside the domain. That is exactly what we are doing here. So we should wholeheartedly acknowledge the idea that there are no “logs” - there are just events which define what happened inside a boundary alongside all relevant metadata. Here’s a sample of our world view in this new age. ``` { “eventName” : “INVOICE_AMOUNT_CALACULATION” “Type” : SUCCESS, “eventTime” : 12344847339485, “Metadata” : { “thread” : “main”, “uId” : “user54672”, “reqId” : “avsk4ghaioernkva”, “cId” : 5, “class” : “c.c.g.t.ItemAndTaxBasedCalculator”, “Message” : “Invoice amount is 1000” “OrdeId” : 1, “InvoiceAmount” : 1000 } } ``` This paradigm shift matters because treating logs as events makes them first class citizens in the design of the system. We are no longer logging requests, responses, errors, stacktraces etc willy-nilly - we have a formal vocabulary for defining what is important in our system, and we are logging all events at the intersections of our technical domains. Not only this, now logging can be a means of informing our system design. If there are some things that we feel like we should be logging, but we cannot find a boundary along which this logging should happen, or we cannot structure the log cleanly, we may be looking at a poorly defined or missing system interface. ![](https://kislayverma.com/content/images/2020/09/logging-events.jpg) ## Single mode of observability If events represent all interesting activity happening at the system's boundaries, then we ought to be able to observe the important bits of our system’s runtime state through them. And if this is true, it follows logically that we don’t need multi-modes of observability. Metric, traces, etc are all derived attributes of the combined event stream of the system, and what we actually need is a powerful means of querying these event streams/stores. Standard database systems, stream processing systems, complex event processing can all be leveraged on this single source of truth to generate a rich variety of insight into the live system. This reduces the need to maintain multiple sources of truth for the same information - as we do when we treat logs, metrics, and traces separate from each other. System events right with business metadata can serve the same purpose very effectively. I hope I have conveyed why I believe that logging as understood commonly is an ad hoc activity, and cannot handle the unknown-unknowns of a production system. We need to switch to an event perspective to leverage it more effectively for system design and reliability. **Read next** : More articles about [distributed systems.](https://kislayverma.com/content/files/2026/07/distributed-systems-2.html) ### For the Layman (Ep. 3) - What is programming? URL: https://kislayverma.com/for-the-layman-ep-3-what-is-programming/ Last updated: 2026-07-22T12:47:04.000Z Hello Everyone! Welcome to this episode of the “ [For the Layman](https://kislayverma.com/category/for-the-layman/) ” series. Today we are going to talk about the root of all good or evil - What is Programming? --- Tl;dr - Computers are extremely fast and completely stupid. To make use of a computer’s speed, a human has to provide all of the intelligence. Providing this intelligence is the exact art and subtle science of programming. --- I have a restaurateur friend (let’s call him K for now) who has done well for himself in life. He lives in a nice house and is able to afford the services of a full time butler/gopher of sorts. This guy takes care of his laundry, odd jobs and such. One of these evenings, a few of us were all just relaxing after one hell of a party, and we felt like having some Maggi. Suresh the Butler was summoned and instructed to make a lot of Maggi. Now I should tell you that Suresh wasn’t the sharpest fellow you might ever meet. He might actually be closer to the bluntest fellow you might meet. What followed was the funniest conversation I have ever heard with Suresh getting into grisly details like “What is yellow packet”, “What exactly is ‘all of us’”, “How to open the cupboard”, “how exactly is the packet to be opened” etc etc and the mix of rage and despair creeping over K's face. At the end of all that, though, we got some pretty good Maggi. ![](https://kislayverma.com/content/images/2020/09/suresh-and-k.jpg) That conversation has stuck with me as a pretty accurate description of what programming is. We have Suresh The Computer that can do pretty much anything we want but must be told in a maddeningly precise manner about how to do it. And we have P The Programmer who can create the next Facebook or Google if only he can explain to Suresh what “Yellow” really, really means. **Programming is the process of telling a computer exactly what activities it must do so that we can achieve our desired goal**. To understand programming, we first need to understand a little bit about computers. --- The prevalent image of a computer is a box attached to a screen and a keyboard and a mouse and a printer and so on. The fact is that all of these things are accessories to let humans use a computer ([Cloud computers](https://kislayverma.com/for-the-layman-ep-2-what-is-the-cloud/), for example, have no monitor or keyboard or mouse). The actual computer, the real heart of it all, is just the Intel/AMD/whatever else **microprocessor chip** sitting on the motherboard. That is where all the “computing” happens. The other interesting bit (pun intended) about computers is that they are **digital**. This means that they can be either on or off. There are no half states. This is different from **analog machines** like the volume knob on a speaker or a throttle on a motorcycle which has many levels through which it can be gradually rotated. Computers are not like that - they are either computing, or they are not. In other words, their output is **Binary - zero or one**. ![](https://kislayverma.com/content/images/2020/09/analog-vs-digital.jpg) The simplest digital component is literally a switch. On and off. All electronic devices, computers (i.e. the microprocessor) are made up of speciall types of switches called **Logic Gates**. A [logic gate](https://en.wikipedia.org/wiki/Logic%5Fgate?ref=kislayverma.com) is a little circuit which has ‘n’ inputs, one output, and the ability to decide what should be the output depending on its inputs. Already beginning to sound like programming, isn’t it? There are some fundamental types of gates like **AND** (output is ‘ON’ if all inputs are ‘ON’), **OR** (output is ‘ON’ if any input is ‘ON’), **NOR** (output is ‘ON’ if all inputs are off) and so on. **Integrated circuits (or ICs)** combine millions of gates into intricate patterns using principles called [De Morgan’s Laws](https://en.wikipedia.org/wiki/De%5FMorgan%27s%5Flaws?ref=kislayverma.com) to create a vast variety of input and output combinations. Microprocessors compose many ICs to provide even richer combinations. These are incredibly complicated arrangements of logic gates and it takes a fat book to describe what will happen if you change one input in this way or that. Understanding it takes a superhuman electrician, also known as an **Electronics Engineer/Hardware Engineer**. ![](https://kislayverma.com/content/images/2020/09/gate-makes-ic-makes-uprocessor.jpg) Despite all this complexity, this hardware does not have any intent/purpose. It is a tool, and the software engineer is now called upon to make this equipment do something useful. The job of the programmer is to analyze a problem, break it down into a specific set of steps, each of which involves flipping one or more switches (gates) so that we get the desired output. E.g. To open a website in your browser, Step 1 : Turn input 100346 to zero. Step 2 : Turn inputs 3454562348 and 846453 to one simultaneously, and immediately change input 4733 to off. Step 3 : …… This is obviously a made up example, but the reality is not incredibly different. **Programming is the process of writing a set of instructions that can manipulate the microprocessor inputs in a certain way to achieve a specific final output**. ![](https://kislayverma.com/content/images/2020/09/old-school-programming.jpg) Early programmers worked at this level of complexity and made tremendous advancements in the field of computing. [Football field sized computers](https://en.wikipedia.org/wiki/ENIAC?ref=kislayverma.com#:~:text=ENIAC%20%28%2F%CB%88%C9%9Bni,of%20numerical%20problems%22%20through%20reprogramming.) and [punched cards](https://en.wikipedia.org/wiki/Punched%5Fcard?ref=kislayverma.com) are relics of this era. But as you might imagine, it takes a special kind of genius to do anything in this way. Much like multiplying large Roman numbers, the task was complicated, error prone, and difficult to explain to other humans. This severely limited what we could practically do with computers. Programmers needed a better way to talk to computers, and this led to the development of **programming languages**. Assembly language is among the lowest level programming languages there is today. While it is still very close to how the hardware is built, it allows programmers to say things like `***mov eax, 3*** `(Put the value 3 in the location named `eax` in the circuit) or ***`add eax, ebx, ecx`*** (Add the values at locations named `eax` and `ebx` in the circuit and put the result at the location named `ecx`) instead of talking about every single input and output in the hardware - i.e it has a higher level of abstraction. It has the concept of data and supports a set of operations that can be performed on this data - a huge improvement from on/off based programming. The story of programming languages is the story of increasing levels of **abstraction** in programming. Abstraction is the process hiding the complexity of something behind something simple. Modern programming languages offer very high level statements that, impossible though it sounds, still map to the on/off of the computer hardware. Something trivial looking *`System.out.println(“Hello World!”)`* (A statement in Java language to print ‘Hello world!’ on the screen) goes through hundreds of transformations to generate possibly thousands of little on/off combinations that cause the pixels on the screen to glow in that specific way. Programming languages, just like real world languages, have different flavours and do different things well. Some are formal (Java), some are flexible (Perl), some are easy to learn (Python), and so on. Programmers choose languages depending on their preference of style, and on the problem they need to solve. A program, once written in any programming language, is typically passed through a **compiler** (sometimes an **interpreter**, but never mind that for now) to generate the exact on-off instructions which the computer can execute to find the smallest number in any given set of numbers. Compilers are typically written by the inventors of a programming language and are the foundation of all high level programming languages. Without compilers, we would have to write the same program separately for each type of machine. Compilers take care of this problem in one go by being machine specific themselves so that our programs don’t have to be. They allow programmers to rise above the minutiae of hardware to a more human-friendly level of expression. ![](https://kislayverma.com/content/images/2020/09/prog-with-hl-lang.jpg) Now let’s look at the actual purpose of programming. Programs are written to solve problems that would take humans too long to solve. How does this work? The first phase is to figure out a step-by-step way to solve the problem, regardless of the programming aspect. The “step-by-step” is really REALY important because any hand-waving here will just not work because we eventually need to translate these steps into the language of switches - on and off. The solution, described in such a manner, is called an **Algorithm**. Algorithms are not just the realm of artificial intelligence and social media feeds and so on. “Algorithm” is just a fancy way of saying “solution to the problem”. --- Let’s imagine that we have a bunch of numbers and we want to find the smallest number among them. How can we do this? One way is to pick one of the numbers randomly, compare it with all the other numbers, and see if it is the smallest number. If not, we repeat this process till we find the smallest number. Note how we did not say “find the smallest number and show it as the output” because it is not clear “how” that can be done. There’s a reason programmers are pedantic about things - that’s the only kind of communication a computer understands. The alogrithm described above will work, but intuitively seems very inefficient. If we lots of input of numbers, randomly searching for the smallest numbers might take very long. In programmerese, we say this algorithm has high **time complexity** (takes too damn long to run). Here’s a better solution. We go over all the input number one by one and keep the smallest number we have seen so far in our hand. If we see a smaller one, we pick it up and drop the one in our hand. At the end of just one cycle over all numbers, we will have the smallest number. This algorithm is much faster (lower time complexity) but needs space in our hand so that we can hold on the smallest number we have seen so far. In the computer world, this space is represented by memory (RAM) and prgrammers refer to the amount of memory required by an algorithm as its **space complexity**. Space and time complexity can often be traded-off depending on what computing resources we have in abundance and what is scarce. Here is a rough Java program showing one pass over all input numbers to find the smallest number as described by the second algorithm. Java is a high level language offering a lot of abstraction from the underlying hardware. So you can see how the program reads reasonably close to normal English. ``` int minimumNumber = Integer.MAX_VALUE; //The largest integer value supported by the computer inputNumberList.forEach(currentNumber -> { If (currentNumber < minimumNumber) { minimumNumber = currentNumber; } } System.out.println(“The minimum number is “ + minimumNumber); ``` Of course, if you miss any punctuation mark or bracket, the program will not work at all. Computers are EXTREMELY fussy about **syntax** (the precise way of writing a program in any language), and all language have a different syntax. --- I hope this has given you some insight into what programs and programming are. If you find that I used any technical term which would not be accessible to non-programmers, please let me know and I will try to re-express it in simpler terms. **Read Next** : More articles on [programming](https://kislayverma.com/category/programming/). ### Book Review : The Great Mental Models (General Thinking Concepts) URL: https://kislayverma.com/book-review-the-great-mental-models-general-thinking-concepts/ Last updated: 2020-09-13T06:56:19.000Z > The old saying goes, "To the man with a hammer, everything looks like a nail." But anyone who has done any kind of project knows a hammer often isn't enough. > > The more tools you have at your disposal, the more likely you'll use the right tool for the job — and get it done right. > > The same is true when it comes to your thinking. The quality of your outcomes depends on the mental models in your head. And most people are going through life with little more than a hammer. > > Until now. > > The Great Mental Models: General Thinking Concepts is the first book in The Great Mental Models series designed to upgrade your thinking with the best, most useful and powerful tools so you always have the right one on hand. > > This volume details nine of the most versatile, all-purpose mental models you can use right away to improve your decision making, productivity, and how clearly you see the world. You will discover what forces govern the universe and how to focus your efforts so you can harness them to your advantage, rather than fight with them or worse yet— ignore them. > > [Goodreads blurb](https://www.goodreads.com/book/show/44245196-the-great-mental-models?ref=kislayverma.com) This [first volume in the Great Mental Models series](https://www.amazon.in/gp/product/B07P79P8ST?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B07P79P8ST&ref=kislayverma.com) was, in all honesty, a complete letdown for me. I have been an avid reader of [Shane Parrish](https://twitter.com/ShaneAParrish?ref=kislayverma.com) and his [FS blog](https://fs.blog/?ref=kislayverma.com) over the last few months. It is a fantastic place to learn things, and to learn about how to learn things. The blog brings an extremely rich and nuanced perspective to everything it covers. This book, on the other hand, felt like those abridged versions of great classics that are published for young children. While it covers some very extremely important and versatile modes of thinking, the material is so shallow as to be only suitable for those who have never heard these words before. The deep analysis and application of the simple concepts to deep problems which reading FS blog such a delight are entirely missing from the book. The following mental models are covered: 1. The map is not the territory 2. Circle of competence 3. First principles thinking 4. Thought experiment 5. Second-order thinking 6. Probabilistic thinking 7. Inversion 8. Occam’s razor 9. Hanlon’s razor If you google these words, you will get articles with deeper insights into each of these than the book provides. I was hoping that there would at least be some examples that would demonstrate the fundamental nature of these models and how they can be applied to seemingly unrelated issues. But I did not find those. All this said, there is nothing wrong with the book. It is written simply and well, and if you have not heard of the models mentioned above (or have heard the names but never thought of them explicitly as mental models), then this is a good place to start. At the very least it gives some great place to start googling about widely-applicable thought patterns. And unlike some other books (like Alain De Botton’s Consolations of Philosophy), it does not trivialize the concepts it covers. It sets them up nicely and encourages the reader to explore further Reading books like this makes me wonder where the problem actually lies when it comes to critical thinking. My feeling is that a lot of people know about these models but fail to identify their applicability in real scenarios and so the whole point of knowing them is lost. Perhaps a better book would be an exercise book of sorts with real life problems that you could try to apply the models on and debate your solution with a community? That might help some more muscle memory in applying these concepts properly. The problem is not the lack of knowledge, I think, but clear headed application of existing knowledge. I have included my highlights of the book below. This should give you a taste of what the book talks about and reads like. 1. A mental model is simply a representation of how something works. We cannot keep all of the details of the world in our brains, so we use models to simplify the complex into understandable and organizable chunks. 2. In life and business, the person with the fewest blind spots wins. Removing blind spots means we see, interact with, and move closer to understanding reality. 3. The fundamentals of knowledge are available to everyone. There is no discipline that is off limits—the core ideas from all fields of study contain principles that reveal how the universe works, and are therefore essential to navigating it. 4. Being able to draw on a repertoire of mental models can help us minimize risk by understanding the forces that are at play. 5. To see a problem for what it is, we must first break it down into its substantive parts so the interconnections can reveal themselves. 6. Most problems are multidimensional, and thus having more lenses often offers significant help with the problems we are facing. 7. Understanding must constantly be tested against reality and updated accordingly. This isn’t a box we can tick, a task with a definite beginning and end, but a continuous process. 8. Our failures to update from interacting with reality spring primarily from three things: not having the right perspective or vantage point, ego-induced denial, and distance from the consequences of our decisions. 9. Many of us tend to have too much invested in our opinions of ourselves to see the world’s feedback—the feedback we need to update our beliefs about reality. 10. We’re so afraid about what others will say about us that we fail to put our ideas out there and subject them to criticism. 11. If we do put our ideas out there and they are criticized, our ego steps in to protect us. We become invested in defending instead of upgrading our ideas. 12. In the real world you will either understand and adapt to find success or you will fail. 13. The world does not act on us as much as it reveals itself to us and we respond 14. Better models mean better thinking. 15. The degree to which our models accurately explain reality is the degree to which they improve our thinking. 16. What successful people do is file away a massive, but finite, amount of fundamental, established, essentially unchanging knowledge that can be used in evaluating the infinite number of unique scenarios which show up in the real world. 17. If a model works, we must invest the time and energy into understanding why it worked 18. The Map is not the Territory - the description of the thing is not the thing itself. The model is not reality. The abstraction is not the abstracted. 19. A map may have a structure similar or dissimilar to the structure of the territory. 20. An ideal map would contain the map of the map, the map of the map of the map, etc., endlessly. We may call this characteristic self-reflexiveness. 21. We run into problems when our knowledge becomes of the map, rather than the actual underlying territory it describes. 22. Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful. 23. In a true science, as opposed to a pseudo-science, the following statement can be easily made: “If x happens, it would show demonstrably that theory y is not true.” 24. A theory is part of empirical science if and only if it conflicts with possible experiences and is therefore in principle falsifiable by experience. 25. First principles thinking identifies the elements that are, in the context of any given situation, non-reducible. 26. everything that is not a law of nature is just a shared belief. 27. First principles thinking helps us avoid the problem of relying on someone else’s tactics without understanding the rationale behind them. 28. As to methods, there may be a million and then some, but principles are few. The man who grasps principles can successfully select his own methods. The man who tries methods, ignoring principles, is sure to have trouble. 29. Thought experiments are more than daydreaming. They require the same rigor as a traditional experiment in order to be useful. 30. One of the real powers of the thought experiment is that there is no limit to the number of times you can change a variable to see if it influences the outcome. 31. Experimenting to discover the full spectrum of possible outcomes gives you a better appreciation for what you can influence and what you can reasonably expect to happen. 32. The rigor of the scientific method is indispensable if we want to draw conclusions that are actually useful. 33. Thought experiments tell you about the limits of what you know and the limits of what you should attempt. In order to improve our decision-making 34. Very often, the second level of effects is not considered until it’s too late. This concept is often referred to as the “Law of Unintended Consequences” for this very reason. 35. the UC Santa Barbara ecologist and economist Garrett Hardin proposed his First Law of Ecology: “You can never merely do one thing.” 36. High degrees of connections make second-order thinking all the more critical, because denser webs of relationships make it easier for actions to have far-reaching consequences. 37. Being aware of second-order consequences and using them to guide your decision-making may mean the short term is less spectacular, but the payoffs for the long term can be enormous. 38. Second-order thinking needs to evaluate the most likely effects and their most likely consequences, checking our understanding of what the typical results of our actions will be. If we worried about all possible effects of effects of our actions, we would likely never do anything, 39. The theory of probability is the only mathematical tool available to help map the unknown and the uncontrollable. 40. Probabilistic thinking is essentially trying to estimate, using some tools of math and logic, the likelihood of any specific outcome coming to 41. Probabilistic thinking is essentially trying to estimate, using some tools of math and logic, the likelihood of any specific outcome coming to pass. 42. Our lack of perfect information about the world gives rise to all of probability theory, 43. The core of Bayesian thinking (or Bayesian updating, as it can be called) is this: given that we have limited but useful information about the world, and are constantly encountering new information, we should probably take into account what we already know when we learn something new. 44. For each bit of prior knowledge, you are not putting it in a binary structure, saying it is true or not. You’re assigning it a probability of being true. Therefore, you can’t let your priors get in the way of processing new knowledge. In Bayesian terms, this is called the likelihood ratio or the Bayes factor. 45. In a bell curve the extremes are predictable. There can only be so much deviation from the mean. In a fat-tailed curve there is no real cap on extreme events. 46. We can think about three categories of objects: Ones that are harmed by volatility and unpredictability, ones that are neutral to volatility and unpredictability, and finally, ones that benefit from it. 47. We notice two things happening at the same time (correlation) and mistakenly conclude that one causes the other (causation). 48. in any situation where change is desired, successful management of that change requires applied inversion. 49. Florence Nightingale is often remembered as the founder of modern nursing, but she was also an excellent statistician and was the first woman elected to the Royal Statistical Society in 1858. 50. Simpler explanations are more likely to be true than complicated ones. This is the essence of Occam’s Razor, a classic principle of logic and problem-solving 51. Occam’s Razor is a great tool for avoiding unnecessary complexity by helping you identify and commit to the simplest explanation possible. 52. Writing about the truth or untruth of miracles, Hume stated that we should default to skepticism about them. 53. With limited time and resources, it is not possible to track down every theory with a plausible explanation of a complex, uncertain event. Without the filter of Occam’s Razor, we are stuck chasing down dead ends. 54. Sometimes unnecessary complexity just papers over the systemic flaws that will eventually choke us. 55. irreducible complexity, like simplicity, is a part of our reality. Therefore, we can’t use this Razor to create artificial simplicity. If 56. Hanlon’s Razor states that we should not attribute to malice that which is more easily explained by stupidity. 57. The explanation most likely to be right is the one that contains the least amount of intent. 58. we’re deeply affected by vivid, available evidence, to such a degree that we’re willing to make judgments that violate simple logic. We over-conclude based on the available information. We have no trouble packaging in unrelated factors if they happen to occur in proximity to what we already believe. 59. Hanlon’s Razor, when practiced diligently as a counter to confirmation bias, empowers us, and gives us far more realistic and effective options for remedying bad situations. **Read Next** : Learn some popular [technology concepts from first principles](https://kislayverma.com/category/for-the-layman/) ### More than testing - writing unit tests for better design URL: https://kislayverma.com/more-than-testing-writing-unit-tests-for-better-design/ Last updated: 2026-07-22T12:47:05.000Z For the first nearly ten years of my programming career, I hardly wrote any unit tests. I wrote a lot of code, and tested almost all of it by running it and testing for end-to-end behaviour, mostly manually. However , as I started inheriting larger and larger codebases and projects across multiple teams in my role as an architect, I began to see why everyone went on about unit testing. There usually isn’t a faster way to know that the code you are writing is safe and works as expected. This is a huge safety net during refactoring complicated code. It is certainly cheaper than having to start a service with its datastore and other attending paraphernalia just to see if a small change is fine. Good unit test coverage also serves as a good and fast way of ensuring that you are shipping the right thing in the land of CI-CD etc. Running integration tests which require elaborate set-up if you want to deploy very often can be very expensive on the infra. However, after writing unit tests regularly for a few years now, I have realized that there is a much deeper value in them. Unit tests (with mocking etc) don’t just test code, they make sure that the code is *testable* in the first place. The act of writing tests helps improve the design of the code during the process of writing. Ever since I realized this, I have become a lot more rigorous in writing tests - not because I care a lot for the testing itself, but because writing them makes my code better designed, more modular, and reveals hidden boundaries in it. ## To test is to draw boundaries Let’s consider what we are doing when we write tests. We provide some input to a piece of code (bunch of classes, class, method), and we verify that the output is what we expected it to be. This idea of input and output implicitly defines boundaries - a boundary from outside which we feed in the input, and a boundary outside which we receive and inspect the output. There may be many code paths inside this boundary, and we want to exercise all the paths in our testing. ![](https://kislayverma.com/content/images/2020/09/testing-boundary.jpg) The definition of expected output depends on the boundary we choose. A REST style *create* API can be tested in several ways. If we choose to include the database inside the boundary, the expected output is an entry in the database (for a valid input). If we draw the boundary just before the persistence layer, expected output may be a valid entity for insertion or a lack of exception or something like that. So the tests are necessarily impacted by the choice of boundary. And good testing strategies exploit this to discover/create boundaries in the code and make them more concrete. This improves cohesion in the code (we identify things which can be collectively hidden behind a boundary) and decreases coupling (we separate things that should not get bundled behind a boundary) ## But boundaries are often hidden ``` public boolean isUserActive(String userId) { RestClientForUserService client = new RestClientForUserService(); User user = client.get(userId); Return user.getIsActive(); } ``` To test the above code, I would consider the method boundary as the boundary for both input and output and hence we should be able to test the behaviour of the method for both active and inactive users. However, doing this is difficult, because embedded inside the method is another boundary, i.e. the interface with user service. ![](https://kislayverma.com/content/images/2020/09/revealing-code-boundary.jpg) So to be able to write a test for this method, we need to be able to control what User service does. This can be done by actually setting up the user service to run this test. But this creates a hard dependency for testing - not a good idea. So we need to make this interface with user service externally accessible so that we can create the different conditions required to test all aspects of this method. We can do this by passing the client as an argument - this makes the hidden boundary more clear. ``` Public boolean isUserActive(RestClientForUserService client, String userId) { User user = client.get(userId); return user.getIsActive(); } ``` Now we can mock this client in a test to simulate behaviour of user service. This is, in a sense, dependency injection, and can be done at class level. This way of making explicit the dependency actually shows us one more thing - this method doesn’t need to know about User Service per se. What it needs is a source of User objects. So we can further decouple it from user service by introducing an interface and a service backed implementation for it. ``` public interface IUserDataLoader { User getUser(userId); } public class UserServiceClient implements IUserDataLoader { public User getUser(userId) { RestClientForUserService client = new RestClientForUserService(); return client.get(userId); } } public boolean isUserActive(IUserDataLoader userInterface, String userId) { User user = userInterface.get(userId); return user.getIsActive(); } ``` This whole redesign has emerged just by trying to write a unit test. The boundaries of our code are much more clear, and we see a strong separation of concerns. This level of refactoring may be useful in some scenarios and overkill in others, but what counts is the thought process behind the thing. Testing makes the process of decoupling much more explicit at the time of writing the code. I spoke about dependency injection earlier. Here’s a typical example from Spring land. ``` @Component public class MyClass { @Autowired RestClientForUserService client; public boolean isUserActive(String userId) { RestClientForUserService client = new RestClientForUserService(); User user = client.get(userId); return user.getIsActive(); } } ``` I used to love this way of injecting dependencies - no pesky constructors or getter/setter boilerplate. But as I started testing this, I kept getting stuck because the boundary represented by the User service client was not accessible. I don’t like the idea of setting up entire application contexts in tests - I prefer a little less magic and simple tests run faster too. So over time I have come to like the constructor injection method. While writing tests, I now have control over all dependencies (which essentially represent boundaries) and I can manipulate them (via mocking or dummy implementations) to test all behaviour of MyClass. ``` @Component public class MyClass { RestClientForUserService client; @Autowired public MyClass(RestClientForUserService client) { this.client = client; } public boolean isUserActive(String userId) { User user = client.get(userId); return user.getIsActive(); } } ``` --- Let’s take a slightly larger example composed of multiple steps. ``` Public class Account { String userId; Date createdDate; Date balance; } // Return true if account created, false otherwise Public boolean createUserAccount(Account account) { RestClientForUserService client = new RestClientForUserService(); User user = client.get(userId); If (user == null || !user.isActive()) { return false; } AccountDao dao = new AccountDao(); try { dao.create(account); } catch (Exception e) { log.error(“Failed creating account”, e); return false; } return true; } ``` If we want to test this method properly, we need to control every place where code branches off. In this example, branching is occurring at the user object and when trying to write the users to the database (UserDao is the class that inserts the object to the database, in JPA world this would be UserRepository). Let’s try a different approach than dependency injection here by splitting the original method into multiple methods and then hook into each of them via mocking. ``` Public class Account { String userId; Date createdDate; Date balance; } // Return true if account created, false otherwise Public boolean createUserAccount(Account account) { User user = getUser(userId); If (user == null || !user.isActive()) { return false; } try { persistAccount(account); } catch (Exception e) { log.error(“Failed creating account”, e); return false; } return true; } @VisibleForTesting protected User getUser(String userId) { RestClientForUserService client = new RestClientForUserService(); return client.get(userId); } @VisiblForTesting protected void persistAccount(Account account) { AccountDao dao = new AccountDao(); dao.create(account); } ``` This code can now be tested by mocking the two new methods - something like this. ``` Mockito.doReturn(new User(active = false)).when(classUnderTest).getUser(Mockito.any()); ``` If we want, we can still use dependency injection to decouple the new protected methods from RestClientForUserService and UserDao even further. The @VisibleForTesting indicates that the method has been made protected or public so that it can be tested and should not be used by mainline code. It is, however, purely indicative. This is why I am not as big a fan of this approach as compared to the simpler DI based approach - it sometimes forces us to expose methods which would otherwise have been private. In both these cases, our attempts at testing a method have done something awesome. They have turned the method from imperative to declarative. In both cases, the updated code simply states what is to be done while delegating the “how” to other components. It is much more decoupled that the original version. --- ## Seams of Code In his wonderful book "[Working Effectively with Legacy Code](https://www.amazon.in/gp/product/0131177052?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=0131177052&ref=kislayverma.com)", Michael Feathers introduced us to the notion of "seams in code". These are the places in our code where two components interact and thus where testing should be done. What we have done so far in this article is create these seams so that components can be identified and there are clear hooks to test the interactions between. But as I have [written before](https://kislayverma.com/working-in-the-seams-of-code-with-resource-locator/) , this idea can be pushed further to say that seams are the places where software grows and evolves. If we have clearly drawn lines in the codebases where we can, through tests and other automated means verify the behaviour of both sides, it means that places offer the ideal starting point for software evolution. Both sides of a seam in the code can evolve in a decoupled manner so long as the across-the-seam contract is honoured. This makes the small refactoring we have done above much more powerful than it might seem at first glance. It is not just about testing or mocking or clean code - it is about laying the groundwork for all future evolution of the codebase. --- This is why I now write unit tests - to make the design of my code better. Read Next - [Overcoming IO overhead in microservices](https://kislayverma.com/overcoming-io-overhead-in-micro-services/) ### A case against "Platform Teams" URL: https://kislayverma.com/a-case-against-platform-teams/ Last updated: 2026-07-22T12:47:06.000Z Most technology companies above a certain size start thinking about creating an internal platform team to build/manage systems that are used by multiple teams/products. This is a very high leverage team since they can beneficially impact many products at once and super-charge the organization. However, today I want to put some things that do not work well with internal platform teams, at least in the versions that I have encountered. Tl;dr - It is better to operate multiple platform teams specializing in their own techno-business domains than to operate a single platform team. [Platform Thinking](https://kislayverma.com/category/platform-thinking/) is not about reuse, it is about facilitating evolution, and fixating on reuse destroys that opportunity. Instil platform thinking in all teams and allow self-reliant domains and platforms to emerge organically instead of forcing the issue up front. **What does this team do?** Having a separate platforms team often means that all horizontal concerns start getting pushed to them. Eventually, this team cannot identify its customers and simply ends up supporting many useful but disjoint systems. It is difficult to set goals and a north star for such a company because they are decoupled from the end user “by definition”. The only intent of such a team is to maximize the reuse of systems within an organization without any regard to the purposes of those systems and the expertise needed to run those systems. In worse situations, other teams have no qualms building reusable components. But when it comes to supporting the reuse in production, respecting SLO and SLAs with other teams, people immediately start looking for a platform team to dump the operations on. The result is an operations laden central team forever doing firefighting because of its poor understanding of what it owns. As the number of shared use-cases in a company grows, the platform team becomes the repository of all kinds of unconnected components whose business purpose they are disconnected from. Eventually we end up with a “team” whose team members have no idea what the others are working on as each member ends up specializing in some part or the other. We get technical specialists but not subject matter experts in the topics that most impact the business. If business starts to get into trouble, the platform team and its work is often among the first to go on the chopping block. This is because it is very difficult to explain to business how this collection of quasi-experts will help them make money. **The developer gold rush** There is something fundamentally unsound about the popular two-dimensional view of an organization’s technical stack. It biases us to the idea that things at the bottom are more foundational in nature, or more complex. It is certainly true that lower layers support more “scale” than upper layers. All of these things cause developers to rush to join the internal platform team as soon as it is incubated. The work is seen as more prestigious or more central, in some way, to the organization. The fact is that at best it is more technical in nature so developers don’t have to deal with the messiness of the real world, customer facing products. This represents an organizational challenge in multiple ways. One is just managing who gets to do what and which roles are considered cool (why aren’t non-platform roles considered “business features“ and not “awesome engineering”). The other part is managing expectations in the team that eventually does get formed, Creating the team is easy, but keeping them business/customer focussed is harder than expected. Some many platform teams are mired in a strange mix of technical arrogance and customer detachment, making them far less powerful than they can be. **Other teams aren’t platform teams?** If there is a platform team, then does it mean that other teams aren’t platform teams? With a catch-all team in place, other teams start abandoning platform thinking and start thinking in product silos. If platformization has value, then it should be the mindset and strategy for all teams and not the domain on one single team. Since all business problems are made up of [domain and organization context](https://kislayverma.com/platforms-and-dogfood-everywhere/), it stands to reason that all teams should be producing and operating platformized components. These platforms may be produced on top of others which are owned by other teams. The purpose of a platform or a generic product is not just reuse, but also to model a domain boundary where organizational expertise is centralized. The platform team cannot keep pulling in horizontal components wherever they arise because then they would have to be experts in all aspects of the business. The typical definition of a platform team which owns a bottom tier of reusable things is just not compatible with the way organizations work. And that leads us to... **Lines of ownership** ![](https://kislayverma.com/content/images/2020/08/a-typical-tech-stack.jpg) The tech stack of a company can be visualized with higher order system abstractions at the top and lower ones at the bottom. Which means that the skill set of operating the architecture can vary substantially as you go down the stack. This gives credence to the idea that the lower layers are somehow “different” from the upper layer and should be managed differently. When we look at the technical stack, how should the lines of ownership run? Should they run horizontally, along components of similar level of abstraction, or should they run vertically, grouping systems generating end-to-end business value? If you are running an autonomous, cross functional team (and many many companies claim to do so), does the depth of the stack matter? The very premise of this team is to be able to work at all levels of the stack and in loose alignment with other teams at each level. In such a setup, the idea of a platform team with fixed horizontal charter is meaningless. Every team creates the layers of the platform that it requires, and shares them with others as needed. This keeps a low-overhead alignment process running because a component isn’t created only for reuse, it is created first for use and then gets reused as the need arises. This kind of ownership also solves the problem of orphaned components which everyone critically depends upon but no one team maintains. Open sourcing is a great idea for writing code but a terrible idea for operations. Software MUST have an operational owner - one team that is responsible for ensuring that it is running as it is supposed to and meeting the benchmarks it is supposed to. Autonomous teams operate what they build, and if we can teach them to build platforms, then we don’t need to explicitly create platform teams. --- But with that said, One of the big arguments against autonomous teams is that they often end up doing a bunch of repeated work. This is obviously suboptimal, so how can we minimize this. **Solving the reuse problem** This problem can be minimized by having *all* problems solved in a platformized manner such that once invented, all teams can benefit from the systems that emerge. There might be short term problems in aligning around launch dates and such, but in the longer run, all use cases of a certain capability or entity start getting centralized in one place. However, this does not mean that the team which built this platform doesn't own it anymore. Reuse does not mean separating ownership from creation. A well designed platform is designed to keep the platform team out of the client’s decision cycle (external programmability), so if a notification platform is built by the marketing team and adopted by many others, the marketing team can continue to own and operate it - nothing wrong with that. Once a reusable thing finds more than 3 customer (Atwood's law), then it ***may*** be time to consider this a horizontal responsibility and move it to a separate team. I don't say move it to a general purpose platform team, but to a specific team that can specialize in the domain modelled by the component and can commit to evolving it to the benefit of all its customers. E.g. A notifications system can kick-start a new business domain altogether - a little Twilio in its own right. Or it may be only a shared technical capability like Managed Elasticsearch which can be handled by a team of storage/elasticsearch experts in the storage platform team/ES platform team. ![](https://kislayverma.com/content/images/2020/08/a-new-domain-is-born.jpg) The above example shows that new teams and domains can be spawned off from every team’s work at either the same level of abstraction if the team originally owns all levels of abstraction in its work (aka a full-stack team). A new domain is a whole new reusable component in a sense. We grow the organization vertically by [layering components](https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/) , and the layers are built creating reusable components atop each other. We also grow the organization horizontally by adding whole new problem spaces that need solving, and each of these will in turn new opportunities for deepening. This organic process goes against the idea of a fixed, co-located platform charter. The idea of a "platform team" confuses the idea of ownership and reuse just as it confuses the idea of “depth” of the platform in the stack with domain knowledge. I think it is far better to think in terms of teams in specific domains each developing their own platforms, and moving cross teams and merging as either redundancy or reuse is discovered. To my mind, the only platform teams on day one are the infra team (managing the hardware stuff) and perhaps the authentication/authorization team, and even these I think of business/tech capabilities rather than reuse/depth-in-the-stack driven teams. All other platforms should emerge from inside product teams. Think cellular division rather than divine hand. Read Next : [We need two pizza problems before two-pizza teams](https://kislayverma.com/independence-autonomy-and-too-many-small-teams/) ### For the Layman (Ep. 2) - What is the Cloud? URL: https://kislayverma.com/for-the-layman-ep-2-what-is-the-cloud/ Last updated: 2026-07-22T12:47:07.000Z Hello everyone! In this episode of the “ [For the layman](https://kislayverma.com/category/for-the-layman/) ” series, I am going to discuss what the “cloud” is. We’ve all heard of iCloud, AWS, cloud computing etc, and understand the power of their ubiquitous presence. Let’s understand the history and the mechanics of the Cloud a little more. --- **tl;dr -** The Cloud is Uber for computers. It is made up of hundreds of thousands of computers (aka servers) that users can rent out to run their software in exchange for a regular fee - mostly based on usage. Cloud companies take care of running the hardware with all kinds of best practices so that software developers and software development companies don’t have to and can focus on their core work. --- Have you ever seen one of those high intensity surgery scenes in movies where the doctor keeps shouting at the nurses “Give me a clamp” or “Give me 5cc of ”? For the longest time, software development used to be like (only less high stakes for the most part). Programmers would write awesome [applications that could eat the world](https://a16z.com/2011/08/20/why-software-is-eating-the-world/?ref=kislayverma.com) . But running these awesome apps required big computers. So while the programmer finished writing the last line of code or polishing the last bit of documentation (HAHAHAHA), she would start screaming, “Give me a big computer”. ![](https://kislayverma.com/content/images/2020/08/give-me-a-big-computer.jpg) For many years, this meant going out and buying the computer, plugging it in, installing a bazillion things on it, and forever making sure no one pulled out the plug by mistake. I’m not even getting into the bureaucracy of procuring said computer. As companies started churning out more and more software, this got to be a real problem. With the rise of the cloud, today the application developer could herself set up a new computer with no more than a few clicks of the mouse. No fuss at all. **The cloud has commoditized hardware and the operations around it.** To understand the rise of cloud computing, let’s look a little bit at the history of computers and their use. --- The first programmable computer was (arguably) the one developed by Alan Turing’s team in Bletchley Park as part of the allied effort at intelligence gathering in WW2\. “**Programmable Computer**” means that this was a machine that could be arranged to do any calculation. Configured one way, it could crack the Nazi ENIGMA code. Set up another way, it could compute the [value of Pi](https://www.wired.com/2016/03/six-things-probably-didnt-know-pi/?ref=kislayverma.com) to the umpy-umpth decimal. This is remarkable by itself because most machines now or since do only one thing. A screwdriver, a bulldozer, an airplane, all of them do exactly one thing. Even the “swiss knife” is a collection of many machines, each for one purpose. So this “programmable computer” was a breakthrough in its own right because you could get it to do anything by programming it in different ways. Anyhow, with a dazzling array of innovations in hardware and computer science, it came to be that computers became commonplace. They could be found in people’s homes, and what we know today as Silicon Valley was beginning to take shape. However, access to computers at the scale at which this new computing revolution wanted it was still not easy. You could easily buy a computer for your home or office, but buying 50 computers was still a pain. Marc Randolph recounts the story from the launch day of Netflix in his book [That will never work](https://www.amazon.in/gp/product/B07QRVWBX2?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B07QRVWBX2&ref=kislayverma.com) about how his team were constantly rushing out to buy more computers from a neighbourhood shop because of the unexpectedly large number of people visiting the website. Since this buy-on-demand model did not work very well, software companies started buying and keeping computers in their office (this is still done widely and is called **on-premise/on-prem**). IT teams would set up the computers, power system, air conditioning, cable layouts, failsafes etc in some designated area in the office. This meant that the company always had a buffer stock of computing power at hand. The cost of this was the capital tied up in all the servers whether being used or not. ![](https://kislayverma.com/content/images/2020/08/on-prem-deployments.jpg) And you know what the sneaky software developers did (and have been doing since)? They found out that getting computers wouldn’t be so much of a pain anymore, so they started churning out even more software. Sooo much more of it. Easy supply further fuelled the hunger for computational resources. The other problem for companies was that off-the-shelf computers came in certain specific sizes. Roughly speaking, **a computer is made up of storage (Hard disk), computing power (CPU cores), memory (RAM) and the network (LAN/Internet)**. But in the new digital age, all kinds of programs were being written which demanded each of these in different combinations. Some wanted a lot of network but didn't do much computing (Netflix, Zoom), some stored a lot of data but didn't have to be fast (data backup systems), and so on. Most companies couldn’t afford to keep all combinations of such machines at hand. It was too expensive to buy and they would have had to keep a team of engineers for this alone. So some smart folks at IBM, the premier computing company of its time, came up with the idea of **Virtualization**. Virtualization allowed us to hide multiple computers and all their resources behind a curtain and think in terms of their aggregate resources. E.g. Ten 4 core computers with 8 GB RAM and 256 GB storage each became a collection of 40 cores, 80 GB RAM, and 2560GB of storage which could be parcelled out into “**virtual machines**” as per the programmer’s requirement. e.g. All of the above mentioned resources could be combined to make 2 huge virtual machines (aka VMs) of 20 cores, 40 GB RAM, and 1280GB storage. Wide adoption of virtualization has since hidden the actual computers (aka **bare metal**) from the programmers view. ![](https://kislayverma.com/content/images/2020/08/virtualization.jpg) While virtualization gave organization’s huge leeway in terms of deploying computational resources efficiently, it also introduced a huge “**Virtualization Layer**” to the already complex stack of hundreds, sometimes thousands of computers they were running. Some companies were reaching the stage where they couldn’t house all their servers in their offices and had to open up **data-centers** (separate facilities for storing and operating all the servers). This was obviously a huge capital investment and distraction for these companies. They didn’t want to be in the business of running hardware, but they had no choice. Around the dot com crash of the 90s, some companies (especially Amazon) had a lot of computers and expertise in operating them, and were looking to make some money. They realized that with the internet potentially connecting all computers of the world to each other, they could not only rent out their servers to people who needed to use them, but also sell their expertise in operating them reliably as a service with the hardware. This was the origin of the **cloud** (or **cloud computing** or **cloud services**) as we know it today. The users of the cloud (the renters of the servers) essentially reach out into the Cloud to lay their hands on mostly Virtual Machines as per their computing needs without worrying about where they kept, which brand they are, and whether they are connected to the power supply or not. Cloud companies like AWS, Azure, GCP, Digital Ocean (where this blog is hosted) take care of all that so we can just focus on running the cool Facemash applications we have written. Users pay an hourly, or monthly, or some other sort of recurring charge based on their usage of these machines. No more scrambling around to buy computers, connect them etc etc. With heavy automation, today even individual developers can manage more computing power than entire teams not too long ago. ![](https://kislayverma.com/content/images/2020/08/cloud-providers-and-users.jpg) In the last decade, this process of cloudification has gone beyond hardware and even into the realm of software. Most cloud providers have “**managed services**” under which they provide well set up and maintained installations of the most commonly used software already running. We don’t even have to set up this software now - we just rent a running version of it for our own purposes -as if it was a physical thing. This is a huge blessing for startups since some of the most commonly used types of software like databases, firewalls, caches, message queues are available at only a few clicks notice and don’t have to be installed and operated. The barrier to entry in the software world has never been lower with more and more of the grunt work being moved off to the cloud. Think of cloud hardware and software in terms of the “[distributed car](https://kislayverma.com/for-the-layman-ep-1-what-is-a-distributed-system/)” I had discussed in a previous article. The engine, the wheels, the fuel injection system etc are all in the cloud now, and you can have as many of them as you can pay for. As the car maker, you can now focus on your unique value addition - the branding, the interiors and leather seats, the new electric battery that only you hold the patent to, and so on. Today, the cloud is part of the internet’s infrastructure like highways or schools for the real world. Most people want highways and schools as a means to an end. They don’t want to have to build them themselves. The cloud plays the same role for anyone that wants to build software. The result - omnipresent software and huge cloud provider bills :) **Read Next** \- More articles on [distributed systems](https://kislayverma.com/content/files/2026/07/distributed-systems-3.html). ### The Golden Rule of Platforms URL: https://kislayverma.com/the-golden-rule-of-platforms/ Last updated: 2026-07-22T12:47:09.000Z The Golden rule of platforms ([™ Steve Yegge](https://kislayverma.com/distilled-steve-yegge-s-platform-rant/)) is simple and well known - ***Eat your own dogfood*.** In the context of building technical platforms, it means that the first user of your platform is you. While this seems a fairly simple thing, certainly too simple to be THE golden rule, this guideline has very deep repercussions around why a platform is built, how it is built, and the impact it has on an organization. The Golden Rule plays the essential counterfoil to the other cardinal rule of building platforms - **a platform must be externally programmable**. External programmability ensures that anyone can build cool new things on top of a platform without having to buy into your business opinions. “Eat your own dogfood” turns external programmability inwards by dictating that even internal teams, even the platform owner itself, must use the platform just as if it was provided by an external party. **No backdoors. No special access. No admins. Nothing that you wouldn’t allow a complete stranger to do.** External programmability is the knife that separates platforms from products. The Golden Rule is this knife applied to the platform owner’s business. ![](https://kislayverma.com/content/images/2020/07/knife-of-platformization.jpg) Here are some ramifications of building a technical platform based on dogfooding. ### A Good Platform is a Great Product I have written before about [why an organization should adopt platform architecture and strategy](https://kislayverma.com/why-you-should-build-a-platform/). Eating your own dogfood means that the reasons for building a platform are also the reasons for using said platform. A platform is typically sold as a set of tools that take care of “solved problems” and allows the buyer to focus on solving new problems. For a team to grow and solve business problems at a deeper and deeper level, it is imperative that they [not be repeating themselves](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/). Explicitly adopting platform architecture forces us to think about solved problems and unsolved problems separately. A good platform lays down the building blocks and [clear rules around using them](https://kislayverma.com/platform-nuts-bolts-enforcing-constraints-in-platform-architectures/). As a follower of those same rules, the platform owner gets a first hand understanding of the good, the bad, and the ugly. Dogfooding tells us about the “maker experience” first hand. ### Externalizable from day one If we are to eat our own dogfood, then ALL the rules and mechanisms for using the platform must be in place just as they would have to be if we were to let others (other teams in the company, people outside the company) use the platform. This means that in practice, a platform done right is externalizable and ready for consumption by anyone from its first day. A good effect of this is that the organization's boundary starts becoming fuzzy from a technical perspective, [switching from access control mode to cooperation mode](https://kislayverma.com/control-and-chaos-in-platform-systems/) . *Anyone* who wants to co-create alongside internal teams on this platform is now free to do so. We may never expose the platform externally, but it forces the business and other technical teams to think in a way that does not presuppose exclusive control and encourage co-creation of value with other partners (via close knit technology integrations). I can assure you that given such capabilities, business folks can get very creative very fast! ![](https://kislayverma.com/content/images/2020/08/platforms-for-cooperation-1.jpg) **Layered Organization Structure** *Eating our own dogfood* means that we have to think about making dog food separately from eating it. [Every business problem contains two parts](https://kislayverma.com/platforms-and-dogfood-everywhere/) . One is the part which is shared by all problems of that domain regardless of the organization (what I call **Domain Context**). The other part represents the organization’s specific ways of solving the problem - the **Organization Context**. Eating your own dogfood forces the split between these two in the organization's technical architecture. I outlined a [case study](https://kislayverma.com/platforms-and-dogfood-everywhere/) of such a split in an earlier post on building a notification system. Hierarchical, loosely coupled layers are an emergent property of platform architecture. And since an organization cannot help but [ship its communication structures](https://en.wikipedia.org/wiki/Conway%27s%5Flaw?ref=kislayverma.com) , we can leverage this phenomenon in reverse to let the technical architecture lead us to an organization structure which is best suited to exploit this loose coupling and agility. Platforms architecture facilitates fast, unmediated (by humans, all mediation is over well defined programmatic interfaces) cooperation between teams that focus on achieving their specific business goals using the various platforms surrounding them - including the one they made themselves. ![](https://kislayverma.com/content/images/2020/08/platforms-and-business-process-2.jpg) You can visualize this as a set of value generating, opinionated business processes sitting on top of a set of platforms. As these processes grow larger and more complex, they themselves start splitting into higher order platforms and even higher order business solutions built on top of them. ![](https://kislayverma.com/content/images/2020/08/higher-order-platforms-and-processes-1.jpg) --- This is a brief look at the impact of The Golden Rule of Platform on technical architecture organizational attitudes. It is worth it to consciously apply platform thinking and this rule to every business or technical problem your are solving to what kind of structures emerge as a result and how we can benefit from them. **Read Next** \- [Using API Gateways to build hierarchical architecture](https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/) ### Book Review : Thinking in Systems - A Primer URL: https://kislayverma.com/book-review-thinking-in-systems-a-primer/ Last updated: 2026-07-22T12:47:10.000Z > “Meadows’ *Thinking in Systems*, is a concise and crucial book offering insight for problem solving on scales ranging from the personal to the global. Edited by the Sustainability Institute’s Diana Wright, this essential primer brings systems thinking out of the realm of computers and equations and into the tangible world, showing readers how to develop the systems-thinking skills that thought leaders across the globe consider critical for 21st-century life. > > In a world growing ever more complicated, crowded, and interdependent, *Thinking in Systems* helps readers avoid confusion and helplessness, the first step toward finding proactive and effective solutions.” > > [Goodreads](https://www.goodreads.com/book/show/3828902-thinking-in-systems?ref=kislayverma.com) *Thinking in Systems: A Primer* by Donella Meadows is an absolutely wonderful introduction to the world of systems and systems thinking. It lays out the conceptual landscape of the entire discipline in language that is accessible to everyone, including non-technical people. It is a basic book which covers only the most basic principles of systems thinking and modelling. The initial chapter lays out the world of systems, the second chapter explains some basic terminology of systems like stock, flows and feedback. This is used in the subsequent chapter to showcase some typical system structures in what Domella calls the “Systems Zoo”. It is fascinating to explore what remarkable behaviours can emerge even with very simple arrangements of water, taps, and bathtubs! I have written before about [modelling complex technical systems like meshes of water hoses](https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/) , but this book is the master class in that way of thinking about absolutely everything in the world. The next section covers the properties of complex systems in some details and explains how the principles for feedback loops and stocks lead to a rich diversity of nonlinear behaviour and how the human mind is singularly ill-equipped to comprehend all these forces acting on each other. The resultant “bounded rationality” of actors in systems is a major cause of all the confusion we see in the world around us. Next come some particularly perverse but unfortunately common system behaviours the author called “System Traps”. These are followed by a list of leverage points which can be acted upon to alter system behaviour effectively, but we have to understand them right. The book closes with some heartfelt advice on being an effective systems thinker (or any kind of thinker at all) More than teaching the technicalities, the author places a far greater importance on exposing the rich, complicated, and unexpected nature of real world systems through day-to-day examples. She repeatedly cautions against the idea of “control” in a world which defies control by its very nature. In many ways, this is a book about people and their attitudes towards the world. If you have ever heard the phrase “all technical problems are people problems” - this book will make it clear exactly why that is. It warns against false causalities, simplistic explanations, and impulsive action just as it exhorts empathy and humility in our intellectual endeavours. A lot of the theory about teams and organizations has emerged from the principles of systems thinking, and this book explains those principles very effectively. If you want to understand the world we live in and the systems that exist all around us, [Thinking in System : A Primer](https://www.amazon.in/gp/product/1603580557?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1603580557&ref=kislayverma.com) is an absolutely essential starting point. The book contains nice summaries throughout chapters which sum up sections, and there is a comprehensive redux at the end. The following are some excerpts from the book that I personally felt were most impactful. Hope they convince you to get your own copy ASAP! --- --- ### System and System Thinking Overview 1. A system is an interconnected set of elements that is coherently organized in a way that achieves something. 2. Systems must consist of three kinds of things: elements, interconnections, and a function or purpose. 3. The system, to a large extent, causes its own behavior! An outside event may unleash that behavior, but the same outside event applied to a different system is likely to produce a different result. 4. Once we see the relationship between structure and behavior, we can begin to understand how systems work, what makes them produce poor results, and how to shift them into better behavior patterns. 5. Systems happen all at once. They are connected not just in one direction, but in many directions simultaneously. 6. A system is more than the sum of its parts. It may exhibit adaptive, dynamic, goal-seeking, self-preserving, and sometimes evolutionary behavior. 7. There is an integrity or wholeness about a system and an active set of mechanisms to maintain that integrity. 8. Systems can be self-organizing, and often are self-repairing over at least some range of disruptions. They are resilient, and many of them are evolutionary. 9. Many of the interconnections in systems operate through the flow of information. Information holds systems together and plays a great role in determining how they operate. 10. A system’s function or purpose is not necessarily spoken, written, or expressed explicitly, except through the operation of the system. Purposes are deduced from behavior, not from rhetoric or stated goals. 11. An important function of almost every system is to ensure its own perpetuation. 12. One of the most frustrating aspects of systems is that the purposes of subunits may add up to an overall behavior that no one wants. 13. Keeping sub-purposes and overall system purposes in harmony is an essential function of successful systems. 14. Change in purpose changes a system profoundly, even if every element and interconnection remains the same. ### Systems Terminology and Modelling #### Stocks and Flows 1. Stocks are the elements of the system that you can see, feel, count, or measure at any given time. 2. Stocks change over time through the actions of a flow. 3. All models, system diagrams, and descriptions, whether mental models or mathematical, are simplifications of the real world. 4. A stock takes time to change, because flows take time to flow - Results are lagging indicators of processes 5. Stocks act as delays or buffers or shock absorbers in systems. 6. Changes in stocks set the pace of the dynamics of systems. 7. Time lags that come from slowly changing stocks can cause problems in systems, but they also can be sources of stability. 8. The presence of stocks allows inflows and outflows to be independent of each other and temporarily out of balance with each other. 9. Stocks allow inflows and outflows to be decoupled and to be independent and temporarily out of balance with each other. 10. Systems thinkers see the world as a collection of stocks along with the mechanisms for regulating the levels in the stocks by manipulating flows. #### Feedback Loops 1. A feedback loop is formed when changes in a stock affect the flows into or out of that same stock. 2. If you see a behavior that persists over time, there is likely a mechanism creating that consistent behavior. That mechanism operates through a feedback loop. 3. Balancing feedback loops are equilibrating or goal-seeking structures in systems and are both sources of stability and sources of resistance to change. 4. Reinforcing loops are found wherever a system element has the ability to reproduce itself or to grow as a constant fraction of itself. 5. Reinforcing feedback loops are self-enhancing, leading to exponential growth or to runaway collapses over time. They are found whenever a stock has the capacity to reinforce or reproduce itself. 6. The time it takes for an exponentially growing stock to double in size, the “doubling time,” equals approximately 70 divided by the growth rate (expressed as a percentage). 7. The information delivered by a feedback loop—even nonphysical feedback—can only affect future behavior; it can’t deliver a signal fast enough to correct behavior that drove the current feedback. Even non-physical information takes time to feedback into the system. 8. Complex behaviors of systems often arise as the relative strengths of feedback loops shift, causing first one loop and then another to dominate behavior. A stock governed by linked reinforcing and balancing loops will grow exponentially if the reinforcing loop dominates the balancing one. It will die off if the balancing loop dominates the reinforcing one. It will level off if the two loops are of equal strength. 9. In physical, exponentially growing systems, there must be at least one reinforcing loop driving the growth and at least one balancing loop constraining the growth, because no physical system can grow forever in a finite environment. 10. Systems with similar feedback structures produce similar dynamic behaviors. #### Delays in Systems 1. Delays are pervasive in systems, and they are strong determinants of behavior. Changing the length of a delay may (or may not, depending on the type of delay and the relative lengths of other delays) make a large change in the behavior of a system. 2. A delay in a balancing feedback loop makes a system likely to oscillate. 3. The higher and faster you grow, the farther and faster you fall, when you’re building up a capital stock dependent on a nonrenewable resource. ### Some examples of Systems 1. Stock with Two Competing Balancing Loops—a Thermostat 2. Stock with One Reinforcing Loop and One Balancing Loop—Population and Industrial Economy 3. A System with Delays—Business Inventory 4. A Renewable Stock Constrained by a Nonrenewable Stock—an Oil Economy 5. Renewable Stock Constrained by a Renewable Stock—a Fishing Economy 6. Nonrenewable resources are stock-limited. The entire stock is available at once, and can be extracted at any rate (limited mainly by extraction capital). But since the stock is not renewed, the faster the extraction rate, the shorter the lifetime of the resource. 7. Renewable resources are flow-limited. They can support extraction or harvest indefinitely, but only at a finite flow rate equal to their regeneration rate. If they are extracted faster than they regenerate, they may eventually be driven below a critical threshold and become, for all practical purposes, nonrenewable. ### Properties of Complex Systems #### Resilience 1. Resilience is a measure of a system’s ability to survive and persist within a variable environment. The opposite of resilience is brittleness or rigidity. 2. There are always limits to resilience. 3. Resilience is not the same thing as being static or constant over time. Resilient systems can be very dynamic. 4. Systems need to be managed not only for productivity or stability, they also need to be managed for resilience—the ability to recover from perturbation, the ability to restore or repair themselves. #### Self Organization 1. Capacity of a system to make its own structure more complex is called self-organization 2. Self-organization produces heterogeneity and unpredictability. 3. It is likely to come up with whole new structures, whole new ways of doing things. 4. It requires freedom and experimentation, and a certain amount of disorder. 5. Conditions that encourage self-organization often can be scary for individuals and threatening to power structures. 6. If subsystems can largely take care of themselves, regulate themselves, maintain themselves, and yet serve the needs of the larger system, while the larger system coordinates and enhances the functioning of the subsystems, a stable, resilient, and efficient structure results. #### Evolution 1. Complex systems can evolve from simple systems only if there are stable intermediate forms. 2. In hierarchical systems relationships within each subsystem are denser and stronger than relationships between subsystems. Everything is still connected to everything. If these differential information links within and between each level of the hierarchy are designed right, feedback delays are minimized. 3. Hierarchical systems evolve from the bottom up. 4. The purpose of the upper layers of the hierarchy is to serve the purposes of the lower layers. This is something, unfortunately, that both the higher and the lower levels of a greatly articulated hierarchy easily can forget. Therefore, many systems are not meeting our goals because of malfunctioning hierarchies. ### Nonlinear Relationships 1. Systems fool us by presenting themselves—or we fool ourselves by seeing the world—as a series of events. 2. It’s endlessly engrossing to take in the world as a series of events, and constantly surprising, because that way of seeing the world has almost no predictive or explanatory value. 3. System structure is the source of system behavior. System behavior reveals itself as a series of events over time. 4. A linear relationship between two elements in a system can be drawn on a graph with a straight line. 5. A nonlinear relationship is one in which the cause does not produce a proportional effect. The relationship between cause and effect can only be drawn with curves or wiggles, not with a straight line. ### System Boundaries 1. The greatest complexities arise exactly at boundaries. 2. There are no separate systems. The world is a continuum. Where to draw a boundary around a system depends on the purpose of the discussion—the questions we want to ask. 3. The right boundary for thinking about a problem rarely coincides with the boundary of an academic discipline, or with a political boundary. 4. Boundaries are of our own making, and that they can and should be reconsidered for each new discussion, problem, or purpose. ### Limiting Factors 1. At any given time, the input that is most important to a system is the one that is most limiting. 2. Insight comes not only from recognizing which factor is limiting, but from seeing that growth itself depletes or enhances limits and therefore changes what is limiting. 3. To shift attention from the abundant factors to the next potential limiting factor is to gain real understanding of, and control over, the growth process. 4. For any physical entity in a finite environment, perpetual growth is impossible. Ultimately, the choice is not to grow forever but to decide what limits to live within. ### Bounded Rationality 1. When there are long delays in feedback loops, some sort of foresight is essential. To act only when a problem becomes obvious is to miss an important opportunity to solve the problem. 2. Bounded rationality means that people make quite reasonable decisions based on the information they have. But they don’t have perfect information, especially about more distant parts of the system. ### System traps 1. **Policy Resistance** 1. The most effective way of dealing with policy resistance is to find a way of aligning the various goals of the subsystems, usually by providing an overarching goal that allows all actors to break out of their bounded rationality. 2. Harmonization of goals in a system is not always possible. It can be found only by letting go of more narrow goals and considering the long-term welfare of the entire system. 2. **Tragedy of the commons** : Tragedy of the commons comes about when there is escalation, or just simple growth, in a commonly shared, erodable environment. 3. **Drift to Low Performance** : Allowing performance standards to be influenced by past performance, especially if there is a negative bias in perceiving past performance, sets up a reinforcing feedback loop of eroding goals that sets a system drifting toward low performance. 4. **Escalation** : When the state of one stock is determined by trying to surpass the state of another stock—and vice versa—then there is a reinforcing feedback loop carrying the system into an arms race, a wealth race, a smear campaign, escalating loudness, escalating violence. The escalation is exponential and can lead to extremes surprisingly quickly. 5. **Success to the successful** 1. This system trap is found whenever the winners of a competition receive, as part of the reward, the means to compete even more effectively in the future. 2. If the winners of a competition are systematically rewarded with the means to win again, a reinforcing feedback loop is created by which, if it is allowed to proceed uninhibited, the winners eventually take all, while the losers are eliminated. 6. **Shifting responsibility to the intervenor** : The trap is formed if the intervention, whether by active destruction or simple neglect, undermines the original capacity of the system to maintain itself. 7. **Rule Beating** 1. Rule beating is usually a response of the lower levels in a hierarchy to overrigid, deleterious, unworkable, or ill-defined rules from above. 2. Rules to govern a system can lead to rule beating—perverse behavior that gives the appearance of obeying the rules or achieving the goals, but that actually distorts the system. 3. The way out is to design rules to release creativity not in the direction of beating the rules, but in the direction of achieving the purpose of the rules. 8. **Chasing the wrong goal** 1. Systems have a terrible tendency to produce exactly and only what you ask them to produce. 2. ... confuse effort with result, one of the most common mistakes in designing systems around the wrong goal. 3. System behavior is particularly sensitive to the goals of feedback loops. If the goals—the indicators of satisfaction of the rules—are defined inaccurately or incompletely, the system may obediently work to produce a result that is not really intended or wanted. ### Identifying and exploiting leverage points 1. One of the big mistakes we make is to strip away “emergency” response mechanisms because they aren’t often used and they appear to be costly.. In the long term, we drastically narrow the range of conditions over which the system can survive. 2. Democracy works better without the brainwashing power of centralized mass communications. Traditional controls on fishing were sufficient until sonar spotting and drift nets and other technologies made it possible for a few actors to catch the last fish. The power of big industry calls for the power of big government to hold it in check; a global economy makes global regulations necessary. 3. The ability to self-organize is the strongest form of system resilience. A system that can evolve can survive almost any change, by changing itself. 4. Self-organization is basically a matter of an evolutionary raw material—a highly variable stock of information from which to select possible patterns—and a means for experimentation, for selecting and testing new patterns. 5. Encouraging variability and experimentation and diversity means “losing control.” 6. Changing the players in the system is a low-level intervention, as long as the players fit into the same old system. The exception to that rule is at the top, where a single player can have the power to change the system’s goal. 7. The shared idea in the minds of society, the great big unstated assumptions, constitute that society’s paradigm, or deepest set of beliefs about how the world works. These beliefs are unstated because it is unnecessary to state them—everyone already knows them. 8. Paradigms are the sources of systems. 9. Paradigms are shared opinions, not facts. There are no ‘true” paradigms. 10. Systems modelers say that we change paradigms by building a model of the system, which takes us outside the system and forces us to see it whole. 11. If no paradigm is right, you can choose whatever one will help to achieve your purpose. 12. Magical leverage points are not easily accessible, even if we know where they are and which direction to push on them. ### Advice on working with systems effectively 1. **Get the Beat of the System** : Before you disturb the system in any way, watch how it behaves. 2. **Expose Your Mental Models to the Light of Day** 3. **Mental flexibility** —the willingness to redraw boundaries, to notice that a system has shifted into a new mode, to see how to redesign structure—is a necessity when you live in a world of flexible systems. 4. **Hono** **u** **r, Respect, and Distribute Information** 5. **Use Language with Care and Enrich It with Systems Concepts** : The first step in respecting language is keeping it as concrete, meaningful, and truthful as possible—part of the job of keeping information streams clear. The second step is to enlarge language to make it consistent with our enlarged understanding of systems. 6. **Pay Attention to What Is Important, Not Just What Is Quantifiable** \- Pretending that something doesn’t exist if it’s hard to quantify leads to faulty models. 7. **Make Feedback Policies for Feedback Systems** 8. **Go for the Good of the Whole** : Don’t maximize parts of systems or subsystems while ignoring the whole. 9. **Listen to the Wisdom of the System** 1. Aid and encourage the forces and structures that help the system run itself. 2. Don’t be an unthinking intervenor and destroy the system’s own self-maintenance capacities. 3. Locate Responsibility in the System That’s a guideline both for analysis and design. In analysis, it means looking for the ways the system creates its own behavior. 4. “Intrinsic responsibility” means that the system is designed to send feedback about the consequences of decision making directly and quickly and compellingly to the decision makers. 10. **Stay Humble—Stay a Learner** 1. We can celebrate and encourage self-organization, disorder, variety, and diversity. 2. You need to be watching both the short and the long term—the whole system. 11. **Defy the Disciplines** \- In spite of what you majored in, or what the textbooks say, or what you think you’re an expert at, follow a system wherever it leads. 12. **Expand the Boundary of Caring** \- Most people already know about the interconnections that make moral and practical rules turn out to be the same rules. They just have to bring themselves to believe that which they know. 13. **Don’t Erode the Goal of Goodness** Read Next : More articles about [organizing teams](https://kislayverma.com/content/files/2026/07/organizing-teams.html). ### Layering domains and microservices using API Gateways URL: https://kislayverma.com/layering-domains-and-microservices-using-api-gateways/ Last updated: 2020-08-02T13:25:35.000Z TL;DR - Hierarchically arranged API gateways can be used to build “macro” bounded contexts. This can make microservices “less”of a problem by reducing them to an implementation detail within that macro context. This results in a simplified architecture composed of stable interactions between hierarchical domain boundaries. --- Microservice architectures have many advantages as well disadvantages, and the internet is rife with debate about the size of a "micro"-service. A less talked about problem is the “conceptual sprawl” that individual microservices modelling a rich problem space create. There can be dozens or hundreds of well-defined bounded contexts each represented by a microservice. However, discovering the right combination of these services to fulfil a new business requirement becomes an increasing challenge. Each individual part (service) may be well designed, but we start losing our way in the exponential number of interactions between them. ![](https://kislayverma.com/content/images/2020/08/too-many-services.jpg) ### Case Study : Backend for Frontend A “backend for frontend” is a service supporting a frontend by combining data from multiple underlying services. This is the node server backing a react app, the controller layer behind thymeleaf/JSP pages and so on. Let’s walk through how new UIs are often built. The first version almost always directly calls existing APIs across many services (likely because there aren’t many such calls). As more and more services need to be called and their data merged into each other in more complicated ways, a debate arises as to whether all this should be done in the frontend or the backend and which team should do this. The typical response these days is to spin off a “service” which does whatever the frontend team wants it to do. ![](https://kislayverma.com/content/images/2020/08/custom-backend-for-frontend-1.jpg) While we can think of this new service as a system that supports a particular UI, what it is actually doing is providing an abstraction that allows the frontend to have only a limited exposure to the full complexity of the backend. ### The Cognitive Overhead Problem The problem described above happens because a single frontend team cannot possibly deal with a microservice-based backend fragmented into tiny “domain” capsules exposed all at once. The cognitive overhead in combining these tiny pieces into larger things is tremendous. One has to understand all of them as they relate to each other and the ways they communicate with each other (e.g. are they eventually consistent with each other? Do they have independent state machines and how do they relate to each other?). The other problem is that the underlying architecture is constantly shifting, with more microservices coming up, services getting deprecated and changing contracts etc. All this makes our “abstraction service” brittle. The microservice owners have to be on the lookout for all consumers lest they break them with any change. Service users are always tinkering with their service to keep it in line with the underlying services. Migrations become a permanent fixture on the sprint board. ![](https://kislayverma.com/content/images/2020/08/custom-means-no-reuse-1.jpg) Given this scenario, a specialized solution in the form of one service for one UI is actually a good solution - it is the only way to move forward! Anyone looking to combine information across multiple domains as defined by each microservice has to create a solution of their own because that is all they can do. But in the larger scheme of things, this is clearly not a good place to be in. How can we reduce this complexity in combining things so that we are better able to leverage the existing systems? ### Organization in Complex Systems Here’s a quick, shallow recap of organization in complex systems. Complex systems are made of hierarchies of independent subsystems. The hierarchical arrangement emerges as an evolutionary response to changing environment and leads to more and more sophisticated features. Each level is a response to needs faced by one or more lower level systems and it abstracts their details while still facilitating their functioning. The overall system is therefore partially resilient to failure of its somewhat independent subsystems. Collectively, this gives to an ever rising dynamic complexity which is highly resilient and adaptive. ![](https://kislayverma.com/content/images/2020/08/organization-in-complex-systems-1.jpg) If this sounds remarkably similar to distributed system design (buzzwords et al) because it is. Large distributed systems are rich complex systems with all constituent parts interacting with each other and triggering concurrent changes in each other. So let’s try to bring some systems thinking to bear on the problem of custom backends-for-frontends. The custom backend-for-frontend we just saw is the microservice architecture’s highly specialized evolutionary response to the pressure of the external world, viz. the functional requirements of the frontend and the limited time in which to meet those requirements. Evolutionary timescales are generally long because the system tries out many combinations before “discovering” what works best in the new environment. However, by putting a timeline on “survival” we have hamstrung the usual mechanics resulting in a system response much like growing corns to protect against shoes that pinch instead of allowing better shoes to “emerge”. ![](https://kislayverma.com/content/images/2020/08/coevolving-systems-and-environment-1.jpg) The evolutionary pressure is also intense because there is no sense of hierarchy in our microservice architecture. The entire complexity of the domain is laid bare all at once to everyone. While upstream-downstream relationships can be inferred between domains looking at data and call flow, there is no concept of hierarchy of ideas that can simplify comprehension for new users. So while domains are the building blocks of our systems, we have not yet managed to layer these building blocks in layers of increasing complexity in representation and function. ### Domain Modelling is Hierarchical Bounded contexts are the philosophical building blocks of microservice architectures. If we want to layer our architecture, we need to layer our concepts. And as you might imagine, this is not difficult at all! We have the entire organization’s structure to be inspired, and since domain driven systems tie in very closely with how organizations are organized, there is plenty of opportunity to copy-paste. Our organization’s structure clearly tells us that a “domain” can mean very different things at different levels of abstractions. As soon as we say “abstraction”, we know that we are in a hierarchical world. If you have ever seen a junior developer try to explain a production outage to a senior manager, you know what I am talking about. The minutiae of system implementation don’t matter to the senior manager because at his level of operation, “outage due to timeout in calling payment authentication service from checkout validator service” is interpreted as “outage in checkout due to payment system”. He doesn’t care about “timeout”, “authentication”, “validator” or “service” - he cares about “checkout”, “outage”, and “payment”. The CEO doesn’t even care about “checkout” and “payment”, he probably just hears “tech” and “outage”. This gives us a direct line to solving our problem - let’s bundle our bounded contexts in a way that all contexts in one group can be represented by one word. This is the reverse process of how we break down the system into microservices; a microservice-to-monolith migration, if you will. ### Representing domain hierarchies So we want to bundle multiple closely allied domains (physically represented by their corresponding microservices) into a single umbrella.and then those umbrellas into a larger umbrella and so on. Note that the bundling process is subjective because people can disagree with what “closely allied” means. An instinct for naming is a good guide - the collective name should not sound dissonant from the individuals. ![](https://kislayverma.com/content/images/2020/08/domain-boundaries-2.jpg) In the real world, a higher level domain can be represented by a separate team (Payment Coordination Team) or a separate manager at a certain level (VP, Director, Architect etc). How do we do this in the technical world? If we want to build an interface that can span across multiple internal systems and combine them into a cohesive experience wrt system vocabulary and capabilities, then we need not look farther than an API Gateway. ### API Gateways API gateways are used as single points of ingress and egress of data from the system. They are typically the points where administrative functions like authentication, rate limiting etc are applied. Companies often employ “public” API gateways to expose a limited subset of their tech stack to external users (external meaning any software not running on the company’s servers, including mobile apps). ![](https://kislayverma.com/content/images/2020/08/api-gateway-2.jpg) The traditional role of API gateways has been that of gatekeepers of all traffic coming into the company. They are meant to be lightweight with no business logic (though they may contain minimal schema transformations), and they often expose an aggregate of the capabilities of multiple internal components as a single unit. E.g. Signing up for a website via its API gateway might create an account in the “User Service” and create a newsletter subscription in the “Newsletter Service” in one go. ### API Gateway as domain boundaries The backend-for-frontend we saw earlier is essentially an API gateway designed from the wrong side, i.e., by consumers of the API rather than publishers of APIs. The consumer is compelled to do so because no standard conceptual hierarchies exist for them to leverage. We can apply this API gateway’s aggregation capability in an internal context to develop coarse-grained system boundaries between our microservices to model hierarchical domains. We can hide the microsevices we had previously put under the same umbrella behind an API gateway which now represents the new domain boundary. No one is allowed to call these services directly. Any capabilities that need to be exposed must be exposed via the API gateway. ![](https://kislayverma.com/content/images/2020/08/use-api-gateway-for-building-domain-boundaries-1.jpg) Let’s take the example of an order being created in a typical order management system. An order entity has to be created in “Order service”, tracking its fulfilment has to be started in “Order tracking service”, an invoice has to be created in “Invoicing Service”. A request to fulfil this is then sent to “Fulfilment Request Service” which invokes the actual fulfilment mechanism implemented inside “Fulfilment Orchestration Service”. The “checkout” system creating the order has to understand all of these things to understand how the order should be placed. ![](https://kislayverma.com/content/images/2020/08/creating-order-without-composite-domains.jpg) Let’s define two aggregate domains : Order and Warehousing. The order domain abstract the Order Service, order tracking service, and invoicing service. The warehousing domain abstracts the fulfilment request service and fulfillment orchestration service, packing service. It doesn’t matter exactly what these services do. What matters is that the two API gateways dropped in to represent the domain boundaries abstract the system creating the order (perhaps checkout service) from the multiple operations required to create an order completely. Instead, that system only sees a single createOrder(Order) interface. ![](https://kislayverma.com/content/images/2020/08/creating-order-with-composite-domains.jpg) This is a massive reduction in the amount of domain language and complexity that new upstream systems have to deal with. The warehousing domain has, in fact, vanished from the view of the checkout system completely. This is the concept of layered domains at play. ### Architecture informs the Organization As you keep applying this principle to build progressively larger domains which contain smaller domains which in turn contain even smaller domains, the picture that emerges is one of how your technology capabilities are organized. This is the true representation of the technology landscape in the real world, and hopefully this is how the technology organization is set up too. If this is not so, big red flags should be waved immediately because now you are fighting Conway’s law(“An organization ships its org chart”). If the org chart and domain models are not aligned, then somewhere, some teams are definitely struggling with massive communication overheads and friction with other teams. So modelling domains is an exercise in organization management just as much as it is an exercise in technical decoupling.The emergent architecture of our system truly has the power to inform the structure of the organization, just as the organization structure was used to identify domain boundaries in the first place. What a beautiful, co-evolving world :) ### Drawbacks The biggest objection I have heard when proposing this approach is that it adds one extra network hop when crossing boundaries. This is a legitimate problem for latency sensitive applications. However, I usually dismiss this problem in favour of the far larger problem of not being able to make sense of a large scale microservice architecture. If you have many well-defined (assumption) microservices then communication across them is anyway inevitable. Paying the one-extra-hop tax is far better than paying the I-don’t-know-which-service-to-call tax. ### Conclusion I hope this article has given you food for thought about how evolutionary and systems thinking can be applied to manage the conceptual sprawl created by microservice architecture. API gateways, or home grown “aggregator services”, or any other approach which can physically represent domain boundaries (crazy idea - separate VPNs for each domain!) can be powerful tools in defining the external as well as internal boundaries of our systems. ### Independence, autonomy, and too many small teams URL: https://kislayverma.com/independence-autonomy-and-too-many-small-teams/ Last updated: 2026-07-22T12:47:11.000Z "The two pizza team" paradigm has become really popular in the context of organizing software teams. The idea is to have small, self-reliant teams working independently to solve problems. The "two-pizza" refers to the guidance to keep the team small enough that we need not order more than two pizzas to feed them. ![](https://kislayverma.com/content/images/2020/07/dev-prod-with-team-size.jpg) These teams are all around us today, but we are not seeing the kind of productivity or shipping velocity that we expected to see. Most developers, especially in larger organizations, complain about meetings, the burden of communication, the slow pace of change etc. Startups often sell themselves to potential candidates by touting their agility. But why should larger companies be slow if everyone is using the same "empowered team"/"autonomous team" model for organizing themselves? I argue that **we have lost the original intent of the "two pizza teams"** (aka autonomous teams). The way we are organizing and scaling teams as companies grow larger is exactly opposite to the spirit of the idea, and this is making the already difficult problem of keeping teams motivated and agile even more difficult. ### Why two pizza teams The concept of the "two pizza"/"autonomous" team arose from attempts to minimize communication between multiple teams and empowering a single team to be in complete control of delivering customer value. Before this idea gained popularity, the most common way of organizing teams was in terms of Backend, UI, DBA, Ops etc.. Building any feature required a lot of communication, collaboration, and alignment between all these groups. The notion of a small team which had all the skills required to solve a single specific problem was intended to solve this problem. A single team would be given the entirety of the problem statement and the tools it needed to solve it. This would ideally remove the problem of aligning multiple teams around shared roadmaps and delivery schedules and allow the team to own the problem solving process end-to-end. There are two core concepts underlying the two-pizza team - **mission** and **independence**. Ideally, end-to-end means all the way to the customer or the business. Hence, **every autonomous team is expected to generate direct business value all by itself**, without a lot of overlap with other teams. Additionally, **the team should be able to meets its goals independently** (i.e. without reliance on or interference from other teams) This reduces the cross team communication overhead because its members have all the skills needed to solve the problem. ### What went wrong Somewhere along the line, we forgot about "reducing communication" just started fixating on assigning independent teams to problem statements that were essentially tiny slices of business problems. As the problem space gets more finely sliced in hopes of achieving scale at each step, so does the number of teams. e.g. What might have been a "Data Delivery Team" charged with delivering fresh data to customers unfortunately becomes "Data ingestion Team", "Data processing Team" and "Data Release Team" (real world example). This causes problems with both the core tenets of the autonomous team philosophy. ![](https://kislayverma.com/content/images/2020/07/narrow-missions.jpg) The **mission is diluted because most of the teams are now working on problems which are subsets of the original problem and as such not valuable in themselves**. The success of a team can no longer guarantee the improvement of at least one business metric. The team's work is not tied to a business objective any longer, so it gets tied to tightly defined execution scopes."Data ingestion team" is supposed to pull in data, that's it. While the team is still allowed to do this however it wants, succeeding in its objective will not mean that fresh data is being delivered to the customer. Autonomy is diluted as a direct fallout. **Where a single team could have come together to solve problems of data delivery, now multiple teams with different managers and different roadmaps must come together to deliver anything to the customer**. This creates exactly the kind of coordination overhead that these small teams were created to solve in the first place. --- **Coordination, also known as alignment, communication, shared roadmap, Gantt chart and many other positive sounding names, is the arch-enemy of the autonomous team**. We have lots of small teams with lots of independence and no direct impact that keeping everyone pointed in one direction has become a nightmare. Program Managers paper over the ever-increasing complexity involved in coordinating priorities, efforts, and timelines, with developers and managers increasingly wash their hands of the whole business of communicating with other teams. We took a good idea too far. The current "small" team is certainly small, but we forgot the part about reducing communication. Instead we are now offered "small, highly collaborative" teams as some sort of new ideal, as if high collaboration is something to be desired during execution. This is a huge blind spot for organizations. The implicit goodness of small teams (devoid of any of the original context) is now internalized to such an extent that deep collaboration between small teams is considered a good thing in the organization. Team productivity and motivation keeps falling even as their sizes increase. And somehow we still believe this way of organizing work among teams is efficient, and something else must be causing the diminishing productivity. --- --- ### Collaboration is not good Collaboration is a great idea during ideation and brainstorming. It helps in getting a lot of input from lots of different people and can help us discover new aspects which we may not consider ourselves. It can expose other approaches to solving the problem, or even some versions of this problem that have already been solved by others. **Broad stroked collaboration between people and systems is how we build something that is greater than the sum of its parts.** However, **once we reach the execution phase, collaboration is extremely costly.** Collaboration means that two people or teams cannot just focus on doing their jobs, they also have to worry about when the other person is going to finish theirs and how they are going to synchronize with each other to hit the finish line. ![](https://kislayverma.com/content/images/2020/07/org-prod-with-team-count.jpg) I can quote any number of engineering principles to refute the deification of collaborative teams which seems to be the norm today. In general, it is assumed that multithreading improves the performance of a system. This is completely true if each thread can execute completely independently on its own core without talking to any other thread or sharing any data. However, the moment we end with some shared state or more threads than CPUs, the synchronization overhead can quickly outstrip all the performance gains from using the technique. [Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s%5Flaw?ref=kislayverma.com) puts a finite upper limit on the increased output when an extra worker is added into the mix. The well-known problem is that every extra worker imposes non-parallelizable, exponentially increasing communication overhead. At some point, adding another worker makes things worse rather than better. "*9 developers cannot finish a 9 month project in one month*" is a well-known idea in software management. And yet we see managers trying to grow and manage their teams in isolation. Why is that? My hypothesis is that we see individual teams delivering quickly and we interpret this to mean that adding more members to these teams will further scale the output. In a world where we examine teams by isolating them from the overall system context, it is very easy to confuse independent with autonomous. ### Independence is not Autonomy In a highly collaborative setup, the correct way of looking at the organization is like a factory floor. [**Theory of constraints**](https://en.wikipedia.org/wiki/Theory%5Fof%5Fconstraints?ref=kislayverma.com#:~:text=The%20theory%20of%20constraints%20%28TOC%29%20is%20an%20overall%20management%20philosophy,Critical%20Chain%2C%20published%20in%201997.) tells us that the only way to increase output in this situation is to broaden the bottlenecks and widen the entire "flow of work" from conception to final delivery to the customer. ![](https://kislayverma.com/content/images/2020/07/sequential-teams.jpg) But when we look at each step individually and out of context of the rest of the "pipeline", "delivery" takes on a very different meaning. It just means "out of my door". Any attempts to improve output with such a perspective will only create local maxima which means nothing for the final output - it might even be detrimental. This is where we end up with too many small teams which are only looking at their little slice of work. That is what they are incentivised to do. If the outputs of our teams are arranged sequentially, they are not autonomous in delivery, no matter how independent they might be in their day to day work. Anyone who has managed a supply chain can tell, independent teams organized in a sequential manner MANDATE a lot of communication and feedback. The lines of communication have not been removed by the creating small teams - they have been multiplied manifold. What's worse, communication is happening at team boundaries where it is actually the weakest (the [communication overhead in microservices](https://kislayverma.com/overcoming-io-overhead-in-micro-services/) springs to mind readily). ![](https://kislayverma.com/content/images/2020/07/too-many-small-teams.jpg) Without an autonomous mission, independent teams are forever shackled by the communication overhead. ### Autonomy is a customer facing construct What happens if we organize our independent teams by problem statement rather than function? A team that is authorized to solve a customer problem using any means at its disposal in essence owns the entire "pipeline" of tasks that need to be done. There is no cross-team collaboration and communication required for this team to do its job. This team not only operates independently, it also delivers value to the customer independently. **A team is autonomous when it "delivers value to the customer" independently**. In his novel "[Goal](https://www.amazon.in/gp/product/8185984565?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=8185984565&ref=kislayverma.com)", Eliyahu M. Goldratt highlights this beautifully by having his protagonist (who is plant manager) define his success to mean increased sales. Sales is usually not the problem of the manufacturing team, but the novel highlights how defining a customer facing "global" goal transforms the poorly performing plant. Scaling this team will directly add more business value because the new resources will be (should be?) deployed in the most optimum way within the team's pipeline. This is no different from the previous situation of task owners creating local maxima, but because the team owns the final customer problem, even a local maximum is a win for the customer. This is what the original "two-pizza" team was intended to do. Not merely to execute independently, but to deliver customer value independently. **The core idea behind autonomy is not arranging teams to maximize outputs of steps in the value chain, but defining value chains in such a way that a single, tight-knit team can be unleashed upon it**. Every team has to define its output in terms of customer success because that's how the organizational boundary is drawn. ### Scaling an autonomous team The problem of scaling teams applies recursively, no matter the paradigm we choose to apply. Let's say we have autonomous teams and all that, now how do we scale their output to achieve ever larger objectives? This is a central problem faced by large organizations. In the early days of a company, domain experts emerge and lead small teams that solve specific customer problems. Since the size of the company is small, having direct customer impact is typically not difficult. However, as the organization grows large in scope, size and ambition, each little team is now expected to deliver more and more. This typically means growing team sizes by adding more members. Then each team starts internally splitting into separate sub-teams, and we are back to independent teams rather than autonomous teams. How can we scale our teams to deliver on more and more ambitious goals while retaining their autonomous nature? I will offer some opinions on this in the next post. Read Next : How to use [Agile for more than just great execution](https://kislayverma.com/agile-for-innovation-going-beyond-execution-excellence/) ### Book Review : Atomic Habits URL: https://kislayverma.com/book-review-atomic-habits/ Last updated: 2026-07-22T12:47:11.000Z [![](https://kislayverma.com/content/images/2020/07/atomic-habits-1.jpg)](https://www.amazon.in/gp/product/1847941834?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1847941834&ref=kislayverma.com) Buy on [Amazon](https://www.amazon.in/gp/product/1847941834?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1847941834&ref=kislayverma.com) > No matter your goals, *Atomic Habits* offers a proven framework for improving--every day. James Clear, one of the world's leading experts on habit formation, reveals practical strategies that will teach you exactly how to form good habits, break bad ones, and master the tiny behaviors that lead to remarkable results. > > [Goodreads](https://www.goodreads.com/book/show/40121378-atomic-habits?ac=1&from%5Fsearch=true&qid=rMzwmpSitp&rank=1&ref=kislayverma.com) *Atomic Habits* came highly recommended from many people I know and from the Twitterverse. Reading the book, though, was a bit of a mixed experience. I really like the accessible, informal tone of the book. It felt like a long blog post written by a friend rather than reading a \~250 page long book. There isn't a lot of "scholarly" pretension anywhere in the book, though James undoubtedly knows what he is talking about very deeply from an academic perspective too. He builds out a nice 4 point framework for building habits and explains some good practices built around behavioural trigger that can help us reshape our lives. The most powerful takeaways for me were: - Think in terms of processes and journeys rather than fixed, boolean goals. - Tiny changes add up over time. Neither of these are new ideas by any means - the [agile process](https://kislayverma.com/agile-for-innovation-going-beyond-execution-excellence/) is essentially the former and compound interest the latter - but somehow neither is very intuitive to us and they are where I think most people fail over the long term. James reiterates these points many, many times in the book, and cautions against the big-bang success narratives riddled with survivorship bias. *Atomic Habits* is a great book if you are looking for something prescriptive which will lay out a bunch of do's and dont's for creating new habits and breaking old ones. It is full of directly actionable advice. My problem with the book is actually what I mentioned above - there are no *new* ideas in this book. I was looking to learn more deeply about how habits work with respect to the human mind and psychology. I realized that this was the wrong expectation only after I a long way into the book. Even so, I came away from the book without the feeling of having "learnt" anything. There are no really new or groundbreaking ideas here. The book is essentially a "practitioner's guide" of the many years James has spent learning and talking about the subject. [*Thinking Fast, and Slow*](https://www.amazon.in/gp/product/B005MJFA2W?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B005MJFA2W&ref=kislayverma.com) is a better book for those who want a deeper study of the mind. --- --- Here are my highlights from reading "Atomic Habits". This is not a summary of the book. James does that himself by giving very succinct summaries at the end of each chapter. These are just some passages from the book that stood out for me. 1. In the long run, the quality of our lives often depends on the quality of our habits. 2. It is so easy to overestimate the importance of one defining moment and underestimate the value of making small improvements on a daily basis. 3. Habits are the compound interest of self-improvement. 4. Be far more concerned with your current trajectory than with your current results. 5. Your outcomes are a lagging measure of your habits. Your net worth is a lagging measure of your financial habits. Your weight is a lagging measure of your eating habits. Your knowledge is a lagging measure of your learning habits. Your clutter is a lagging measure of your cleaning habits. You get what you repeat. 6. Habits often appear to make no difference until you cross a critical threshold and unlock a new level of performance. 7. Mastery requires patience. 8. Goals are about the results you want to achieve. Systems are about the processes that lead to those results. 9. If you want better results, then forget about setting goals. Focus on your system instead. 10. Goals are good for setting a direction, but systems are best for making progress. 11. Goal setting suffers from a serious case of survivorship bias. We concentrate on the people who end up winning—the survivors—and mistakenly assume that ambitious goals led to their success while overlooking all of the people who had the same objective but didn’t succeed. 12. We think we need to change our results, but the results are not the problem. What we really need to change are the systems that cause those results. 13. Fix the inputs and the outputs will fix themselves. 14. The problem with a goals-first mentality is that you’re continually putting happiness off until the next milestone. 15. When you fall in love with the process rather than the product, you don’t have to wait to give yourself permission to be happy. You can be satisfied anytime your system is running. 16. The purpose of setting goals is to win the game. The purpose of building systems is to continue playing the game. True long-term thinking is goal-less thinking. It’s not about any single accomplishment. It is about the cycle of endless refinement and continuous improvement. 17. You do not rise to the level of your goals. You fall to the level of your systems. 18. An atomic habit is a little habit that is part of a larger system. 19. Behind every system of actions are a system of beliefs. 20. There are a set of beliefs and assumptions that shape the system, an identity behind the habits. Behaviour that is incongruent with the self will not last. 21. The ultimate form of intrinsic motivation is when a habit becomes part of your identity. It’s one thing to say I’m the type of person who wants this. It’s something very different to say I’m the type of person who is this. 22. True behaviour change is identity change. 23. The real reason you fail to stick with habits is that your self-image gets in the way. 24. Your habits are how you embody your identity. 25. The process of building habits is actually the process of becoming yourself. 26. Each habit not only gets results but also teaches you something far more important: to trust yourself. You start to believe you can actually accomplish these things. When the votes mount up and the evidence begins to change, the story you tell yourself begins to change as well. 27. Decide the type of person you want to be. Prove it to yourself with small wins. 28. Work backward from the results you want to the type of person who could get those results. 29. Your identity is not set in stone. You have a choice in every moment. 30. “Habits are, simply, reliable solutions to recurring problems in our environment.” 31. One of our greatest challenges in changing habits is maintaining awareness of what we are actually doing. 32. People who make a specific plan for when and where they will perform a new habit. 33. Being specific about what you want and how you will achieve it helps you say no to things that derail progress, distract your attention, and pull you off course. 34. [The Diderot Effect](https://en.wikipedia.org/wiki/Diderot%5Feffect?ref=kislayverma.com#:~:text=The%20Diderot%20effect%20is%20a,be%20complementary%20to%20one%20another.) states that obtaining a new possession often creates a spiral of consumption that leads to additional purchases. 35. Environment is the invisible hand that shapes human behavior. 36. Make sure the best choice is the most obvious one. 37. If you’re overweight, a smoker, or an addict, you’ve been told your entire life that it is because you lack self-control—maybe even that you’re a bad person. 38. Bad habits are autocatalytic: the process feeds itself. 39. Self-control is a short-term strategy, not a long-term one. You may be able to resist temptation once or twice, but it’s unlikely you can muster the willpower to override your desires every time. 40. It is the anticipation of a reward—not the fulfillment of it—that gets us to take action. 41. Temptation bundling works by linking an action you want to do with an action you need to do. 42. Temptation bundling is one way to apply a psychology theory known as [Premack’s Principle](https://en.wikipedia.org/wiki/Premack%27s%5Fprinciple?ref=kislayverma.com). Named after the work of professor David Premack, the principle states that “more probable behaviors will reinforce less probable behaviors.” 43. Join a culture where (1) your desired behavior is the normal behavior and (2) you already have something in common with the group. 44. Nothing sustains motivation better than belonging to the tribe. 45. Habits are all about associations. 46. The cause of your habits is actually the prediction that precedes them. These predictions lead to feelings, which is how we typically describe a craving—a feeling, a desire, an urge. 47. Habit formation is the process by which a behavior becomes progressively more automatic through repetition. The more you repeat an activity, the more the structure of your brain changes to become efficient at that activity. 48. A commitment device is a choice you make in the present that controls your actions in the future.2 It is a way to lock in future behavior, bind you to good habits, and restrict you from bad ones. 49. Commitment devices are useful because they enable you to take advantage of good intentions before you can fall victim to temptation. 50. The best way to break a bad habit is to make it impractical to do. Increase the friction until you don’t even have the option to act. 51. Technology can transform actions that were once hard, annoying, and complicated into behaviors that are easy, painless, and simple. 52. Mathematician and philosopher [Alfred North Whitehead](https://en.wikipedia.org/wiki/Alfred%5FNorth%5FWhitehead?ref=kislayverma.com) wrote, “Civilization advances by extending the number of operations we can perform without thinking about them.” 53. We are more likely to repeat a behavior when the experience is satisfying. 54. The first three laws of behavior change—make it obvious, make it attractive, and make it easy—increase the odds that a behavior will be performed this time. The fourth law of behavior change—make it satisfying—increases the odds that a behavior will be repeated next time. It completes the habit loop. 55. You value the present more than the future. 56. A reward that is certain right now is typically worth more than one that is merely possible in the future. 57. The costs of your good habits are in the present. The costs of your bad habits are in the future. 58. Let’s update the Cardinal Rule of Behavior Change: What is immediately rewarded is repeated. What is immediately punished is avoided. 59. In a perfect world, the reward for a good habit is the habit itself. In the real world, good habits tend to feel worthwhile only after they have provided you with something. 60. Use reinforcement, which refers to the process of using an immediate reward to increase the rate of a behavior. 61. Incentives can start a habit. Identity sustains a habit. 62. Making progress is satisfying, and visual measures—like moving paper clips or hairpins or marbles—provide clear evidence of your progress. As a result, they reinforce your behavior and add a little bit of immediate satisfaction to any activity. 63. Habit tracking also helps keep your eye on the ball: you’re focused on the process rather than the result. 64. The problem is not slipping up; the problem is thinking that if you can’t do something perfectly, then you shouldn’t do it at all. 65. The dark side of tracking a particular behavior is that we become driven by the number rather than the purpose behind it. 66. Named after the economist Charles Goodhart, the [principle](https://en.wikipedia.org/wiki/Goodhart%27s%5Flaw?ref=kislayverma.com) states, “When a measure becomes a target, it ceases to be a good measure.” 67. Measurement is only useful when it guides you and adds context to a larger picture, not when it consumes you. 68. The more immediate the pain, the less likely the behavior. 69. People are born with different abilities. 70. Our environment determines the suitability of our genes and the utility of our natural talents. 71. Genes can predispose, but they don’t predetermine. 72. The takeaway is that you should build habits that work for your personality. 73. Choose the habit that best suits you, not the one that is most popular. 74. A good player works hard to win the game everyone else is playing. A great player creates a new game that favors their strengths and avoids their weaknesses. 75. Work hard on the things that come easy. 76. [The Goldilocks principle](https://en.wikipedia.org/wiki/Goldilocks%5Fprinciple?ref=kislayverma.com#:~:text=In%20cognitive%20science%20and%20developmental,current%20representation%20of%20the%20world.) states that humans experience peak motivation when working on tasks that are right on the edge of their current abilities. Not too hard. Not too easy. Just right. 77. [A flow state](https://en.wikipedia.org/wiki/Flow%5F%28psychology%29?ref=kislayverma.com#:~:text=In%20positive%20psychology%2C%20a%20flow,the%20process%20of%20the%20activity.) is the experience of being “in the zone” and fully immersed in an activity. Scientists have tried to quantify this feeling. They found that to achieve a state of flow, a task must be roughly 4 percent beyond your current ability. 78. At some point it comes down to who can handle the boredom of training every day, doing the same lifts over and over and over. 79. Mastery requires practice. But the more you practice something, the more boring and routine it becomes. Once the beginner gains have been made and we learn what to expect, our interest starts to fade. 80. The greatest threat to success is not failure but boredom. We get bored with habits because they stop delighting us. 81. Professionals stick to the schedule; amateurs let life get in the way. 82. Habits are necessary, but not sufficient for mastery. What you need is a combination of automatic habits and deliberate practice. 83. The more sacred an idea is to us—that is, the more deeply it is tied to our identity—the more strongly we will defend it against criticism. 84. When you cling too tightly to one identity, you become brittle. Lose that one thing and you lose yourself. 85. Success is not a goal to reach or a finish line to cross. It is a system to improve, an endless process to refine. Read next : More [book reviews](https://kislayverma.com/category/books/). ### For the Layman (Ep. 1) - What is a Distributed System? URL: https://kislayverma.com/for-the-layman-ep-1-what-is-a-distributed-system/ Last updated: 2026-07-22T12:47:12.000Z Hello Folks! I am starting a new series of articles called “[For the Layman](https://www.kislayverma.com/category/for-the-layman/?ref=kislayverma.com)” to cover some frequently encountered software engineering concepts to non-developers or beginners. The articles with try to explain these concepts in simple terms with as little jargon as I can manage. In this first episode of the series, let’s understand "[distributed systems](https://kislayverma.com/content/files/2026/07/distributed-systems-4.html)". --- As the name suggests, a distributed system is a "**system**" whose components are"**distributed**". Let's look at both those words individually. --- ### What is a system **A "system" is a set of parts working together to deliver a certain functionality.** A clock is a system of springs and gears that tells time reliably. A car is built up of many parts which allow us to be driven from one place to another. Let's call each part a "component". ![](https://kislayverma.com/content/images/2020/07/car-as-a-system.jpg) ### Monoliths : "Not" distributed systems Most mechanical systems are not, and cannot be "distributed". Components in most hardware systems assume "local" availability of their partnering components. Piston rods expect to be welded to crank shafts, keyboards expect to be connected to processors etc etc. A software system built along these is sometimes called a “**monolith**”. ![](https://kislayverma.com/content/images/2020/07/car-system.jpg) #### Vertical Scaling A side effect of having to put all components next to each other is that to build a more powerful system, we need to fit more pieces on the same chassis (so to say). If we want to add more engines so our car can go faster, we need to add a larger engine with more cylinders. This in turn requires a large car, and so on. ![](https://kislayverma.com/content/images/2020/07/vertically-scaled-car.jpg) You can also think of trying to add more power to your laptop by adding more processors or more memory. It can be done but makes the laptop larger and larger - eventually we end up with a desktop rather than a laptop. **This process of adding more and more power to a single physical system is called "Vertical Scaling"**. It is an important strategy ([Moore's law](https://en.wikipedia.org/wiki/Moore%27s%5Flaw?ref=kislayverma.com) is essentially about vertical scaling), but comes with hard physical limits that are very difficult to surmount. #### Location Coupling Mechanical systems also assume a certain "guarantee" in terms of collaborating with each other. No unreliability is expected between the turning of a gear and the turning of the clock hand (there may be errors of precision, but that is not the question here). A clock is designed with the assumption that certain things will cause certain other things to happen, and if they don't then the clock is considered broken. **The functioning of the complete system depends strongly on all of the components always being physically present in a certain location at a certain time.** This type of dependency pattern is called a form of "**tight coupling**" between components. #### Globally Consistent State This hard dependency on all parts strictly working together has an interesting implication. **It means that if we know what state (e.g. position, location) one component is in, we necessarily also know the states of all other components**. If component A is not in the position that component B expects it to be, then we have a problem. As a result, our knowledge of the system at any point of time is complete and consistent. This is called "**Globally Consistent state**" or just **"Consistent State"**. --- As you may be able to see by now, **it is difficult to build very large systems when everything MUST be co-located and MUST work all the time**. A large system built using these principles is brittle - any small failure can cause a complete outage. It is also not scalable - as the size grows, not only do we have to keep fitting everything next to each other (imagine an engine with thousands of cylinders all of which must be next to each other and must coordinate completely), but we also have to have complete knowledge of all component at all times to be able to understand if the system is working properly. The cognitive load such a system creates is tremendous and increases exponentially with every new component. Modern software architecture is as much about handling unprecedented scale as it is about solving business problems. Whether it is the billions of people using Facebook or the surges of online shoppers on Singles Day, software systems today are expected to deliver tremendous performance and continue to function even when parts fail. To fulfil these requirements, distributed systems have emerged as an alternative paradigm for constructing systems built out of many components. --- --- ### What is a distributed system *Server* : A computer connected to the internet or some other network. **Distributed systems are made up of "independent" components which are not necessarily located next to each other.** This seemingly simple definition of distributed systems has huge ramifications and gives these systems their unique strengths and weaknesses. Let's cover some of these in detail. #### Location Transparency Components in a distributed communicate with each other via methods/protocols that don't require the calling component to know where the called component is located. So the engine can potentially be located at home even when you drive your car. Somehow when the accelerator is pressed, the engine generates more power which is somehow transferred to the wheels. Another analogy is the remote working style we now see everywhere. Team members are not physically located together, but still cooperate by performing their respective jobs in benefit of a shared objective. ![](https://kislayverma.com/content/images/2020/07/distributed-car.jpg) This is called "**Location Transparency**" which is a form of "**Loose Coupling**" (all developers get a dreamy eyed look when they hear this word - try it!) The advantages of location transparency are obvious. **If software components need not be co-located, then we can move them to different physical machines, each of which can then be vertically scaled.** i.e. We can buy powerful machines for each component separately instead of fitting all components on one machine. This directly leads to a more powerful system. Note that we are not "mandating" that components must be located on separate servers, just that it shouldn’t matter where they are located as long as there is a way to locate them. **How do these distributed components find each other**? There are many ways to do this. One of the most popular mechanisms is [DNS (Domain Name System)](https://en.wikipedia.org/wiki/Domain%5FName%5FSystem?ref=kislayverma.com) which maps names to IP Addresses (unique identities of servers all over the internet). The backbone of all networking is the ability to locate one specific machine given its IP address or domain name. This seemingly small (but actually extremely complicated) technique has allowed software to eat the world. #### Partial Failure Mode A fallout of the distributed nature is that our failure mode is not all-or-nothing anymore. The "card reader" component may have failed but the account management component may be running. This means that some functionality related to account management may still be accessible even though our card swiping users are frustrated. #### Horizontal Scalability Yet another corollary of location transparency is that there need not necessarily be only one more instance of a component. If it doesn't matter where the engine is located, we can now add ten or more independent engines to add that much more power. ![](https://kislayverma.com/content/images/2020/07/horizontally-scalable-car.jpg) It is even possible to add and remove engines from the car as needed. **This ability to add more instances of a component is called "horizontal scaling"**, and is the currently preferred mechanism for scaling software systems since it bypasses the physical limitations of how powerful a single server can be - we just add more low power servers to compensate. So we have *a bunch of components living on different "servers" (machines) and communicating over a "network" (internet/LAN).* Is that it? #### Eventual Consistency There is one more interesting thing to understand here - The network is unreliable and slow. ![](https://kislayverma.com/content/images/2020/07/network-is-unreliable.jpg) If your Zoom call has ever hung mid-sentence or Youtube has "buffered", you know what I'm talking about. Data is sent from one component to the other, but sometimes doesn't reach it or reaches after some noticeable time. Maybe the wire is cut, maybe the other component reads the data but crashes before it could do anything with it, maybe a lot of data was flowing over the wire and hence everything is stuck. Any which way this occurs, this results in different parts of the system not having complete information about each other. This happens all the time in the systems we encounter on the web - payment was taken but order could not be placed (payment component is not able to talk to order component), money transfer is triggered but will reflect in 24 hours (transfer component know that there is to be a transfer but the account component has not been told yet). ![](https://kislayverma.com/content/images/2020/07/inconsistent-states.jpg) Another way of putting this is to say that the independent components of a distributed system understand their own "state" (e.g. position, orientation, amount of load) absolutely but may be out of sync (to a greater or lesser extent) with other components. The "out of sync-ness" is a result of the network acting as a queue of unshared knowledge between them. This is called "**inconsistency**" and is the other side of the **Global State** we encountered in non-distributed systems. The remedy for this in distributed systems is "**Eventual Consistency**", meaning **we must implement mechanisms which will ensure that all components "eventually" agree with each other on what the overall state of the system is**. Note that this is a catchup game, and components are out of sync-by-design rather than by mistake. This gives us some buffer time in which we can perform knowledge transfer instead of making every component aware of everything instantaneously (a physical impossibility in the distributed world given that nothing can travel faster than speed of light). ### The bad parts While distributed systems can be extremely resilient to failures and very responsive under high loads, building well designed distributed systems is an extremely complicated undertaking. The first problem is user experience. There is no way to hide the eventually consistent nature of the system from the users. With instant gratification being the increasingly accepted norm, it can sometimes take a lot of clever UX to keep the system distributed and the user happy. Distributed system is necessarily much more complicated than a "monolithic" (everything in one place) design to compensate for the [fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies%5Fof%5Fdistributed%5Fcomputing?ref=kislayverma.com) . A lot of new tools have evolved to help developers build reliable distributed systems, but this is still far from an easy or solved problem. --- I hope you got a basic understanding of distributed systems from this article. If you are a beginner and found that parts of the article were still too technical to understand, let me know in the comments and I will try to break things down in simpler terms. --- The "For the Layman" series will try to explain deep software engineering concepts in very simple language. Sign up to the mailing list (bottom of this page) to receive the next episode right in your inbox. ### Adios Uber, and ideas in the afterglow URL: https://kislayverma.com/adios-uber-and-ideas-in-the-afterglow/ Last updated: 2020-07-22T07:35:53.000Z Howdy! Hope you folks are doing all right, staying at home, and wearing masks if you must go out. Last week was my last working day with Uber Engineering. It was a fun 16 month ride (I'll write more about it once I have digested it a little) and now I'm sitting back and relaxing for a while. It's a pleasant fiction, of course. I cannot relax if I have a computer and internet. Instead, I have been thinking a lot about Peter Thiel's [*Zero to One*](https://www.kislayverma.com/books/book-review-zero-to-one/?ref=kislayverma.com). The idea that I keep coming back to is about how a lot of things we do today haven't changed a lot for a very long time - houses, clothes, books, cars. After reading that, I've grown a heightened sense of "old" about the things we run into everyday, and more than earlier I have this craving to build some new-ness into my own life - a more seamless merging of technology into the day-to-day existence (I started by putting laptop and mobile chargers in every corner of every room - seamless charging is a good start!). Here are some ideas which have stuck with me for some time on this theme. I'm happy to know if these are solved problems, and even happier to know what you guys think about them. ### E-books should be more fun! ![](https://kislayverma.com/content/images/2020/07/ebook.jpg) stacked books with a tablet on top I read lots of e-books and I love the instant availability and access that this is simply not possible with physical books. However, I am beginning to feel that there is a lot more that can be done with e-book. The entire current "experience" of an ebook is identical to the physical book - bunch of words on a two dimensional piece of digital paper with perhaps the ability to highlight and take notes. Then there are separate audio-books where the same uni-dimensional experience is repeated, except with spoken word. Given all the different forms of media that a browser can deliver today, it seems subpar that we should still have to consume a book in exactly the same way. Can we put together audio, video, text, and hyperlinks to build a richer experience that is also more seamless across devices and modes of consumption (e.g. I start reading a book textually, need to drive so switch over to audio podcast style, and cut-back over to text once I am back home). Another way to think about this is in terms of what the difference is between a website and an ebook except that the book has a specific narrative while a website is more "undirected"? These are a bunch of disconnected thoughts and I'm not really what I want the ideal ebook, but it is something I'm thinking a lot about. --- --- ## Better Diagrams Like my ebook objection, I have a problem with the how software architecture diagrams have remained the same boring 2D structures which can talk about only one perspective. One thing to be said before anything else is that programmers don't really know how to do diagrams in the first place. There are no universal standards, though [UML](https://www.uml.org/?ref=kislayverma.com) comes close and the [C4 model](https://c4model.com/?ref=kislayverma.com) is gathering steam recently. But even so, is there really no other way to deliver architectural context than the ad-hoc boxes-and-lines scheme we use today? In line with the narrative approach to software architecture I recently wrote about, How would it be to be able to use our architecture diagrams to interactively navigate a 3-dimensional space from higher level views to lower level views with the temporal element of data/call flow mixed in? Would this be better or worse than the current way of building a narrative using multiple different diagrams? I don't know, but a good delivery mechanism might finally put an end to feedback like "can you depict this visually - all this text is not very intuitive". Can we meld images, explanations, text etc together to deliver that "intuitive" experience? I did a little [Twitter](https://twitter.com/kislayverma/status/1256851601056133121?ref=kislayverma.com) and [Reddit](https://www.reddit.com/r/programming/comments/gdu39e/would%5F3d%5Farchitecture%5Fdiagrams%5Fbe%5Fmore%5For%5Fless/?ref=kislayverma.com) enquiry for this, with Simon Brown (the author of the C4 model) and many others sharing great insights, but eventually not a strong leaning towards yea or nay. ## A Distributed Web ![](https://kislayverma.com/content/images/2020/07/decentralized-web.png) A distributed, semantic web is not a new idea. The idea of owning ALL my data without losing the insane power of the internet platform appeals to me tremendously. Last week I wanted to buy a song and use it (without any copyright claim) in a presentation I was putting together. I couldn't find a way to do this simple thing! The way I look at this is if I pay for something, it leaves the seller's control and comes into mine, and it should be mine to do with as I please, where I please (within legal limits). But that apparently does not work - An iTunes song will play in one place, a Kindle book is elsewhere, and my calendar is running elsewhere. I want all of my stuff in one place. In MY place. I want to be able to access it regardless of what the seller thinks of me these days, and I want other people to be able to build applications that can work with my data to give me experiences without me having to surrender the data to them. The launch of [Inrupt](https://inrupt.com/?ref=kislayverma.com) by [Sir Tim Berners Lee](https://www.w3.org/People/Berners-Lee/?ref=kislayverma.com) and all the [Solid](https://solid.mit.edu/?ref=kislayverma.com) related work being done has put ideas of democratizing this in my head. I'm thinking super-accessible self-sovereign identity, everyone in complete ownership of their data, and companies that use this data to build delightful experiences without having to resort to data thievery and criminal privacy violations. I imagine everyone with their own digital homes on the web, with all their data and little tools to work with it available anywhere. I had friends before there was a Facebook - I want the internet to enable that for me again. I didn't know the trends around the world instantly before there was Twitter, and I want that too. How can we get both? --- This is what I'm spending my idle days thinking about. I'd love to hear what you guys think about these - drop a line in the comments. If you are working on some other cool ideas, put it here and we can share the excitement! --- I write about once a week about books, software architecture, and building platforms. Sign up for the mailing list at the bottom of the page to receive more updates directly in your inbox. ### A Narrative Approach to Software Design URL: https://kislayverma.com/a-narrative-approach-to-software-design/ Last updated: 2020-07-22T07:28:54.000Z To me, the design stage is the most intellectually stimulating part of the software development cycle. After all, the design decides what the code will do, and how it will do it. The design process gives technical shape to what are till then only business requirements. We get to the deepest what's and why's at this stage - it is a lot of fun! Here's my rule of thumb about approaching software architecture - **Approach designing a system/feature like you are writing an investigative story**. That's essentially what the design exercise is - we are poking around in the problem space/requirements to figure out how we might solve it. And like any good investigator, we ought not to make assumptions early on or jump back and forth between incidents. All we have is a burglary - deciding to shadow the sinister looking neighbour up front is probably not the best of course of action. I love crafting architecture narratives because they sit "above" any technical approaches and provides a framework for applying any of them. It also like it party because it has worked very well for "most" of the problems "I have worked on" (which are mostly application engineering problems using service-based architecture), and partly because even when it doesn't provide actual technical design (e.g. in big data related problems), it helps me understand the problem domain deeply. Start from a broad lay-of-the-land, then identify specific facts and occurrences, then progressively focus on each of them to see how they tick - all the while keeping the thread of overall technical story in hand. The process should feel very organic and evolutionary - a breakdown of high level business requirement into somewhat lower level technical constructs into technical components into the lowest level of implementation details and technology choices. It is a little bit like doing [Bog Post Driven Development](https://news.ycombinator.com/item?id=9584806&ref=kislayverma.com) for engineering teams. You can slice this breakdown in many ways, but in most cases a few logical levels usually suffice. ### High Level View ![](https://kislayverma.com/content/images/2020/07/reco-high-level.jpg) This fits our system or feature in the overall big picture and defines why we need it. It also assigns our system-specific responsibilities which no other system can fulfill. This is where we think about whether we need this thing in our organization and who might be consuming it. ### Components Level Design ![](https://kislayverma.com/content/images/2020/07/reco-component-level.jpg) Now we look at our system itself and try to figure out what it is made up of. Any system of some complexity with be composed of multiple sub-parts and here we identify them, name them, and attribute behaviours to them. Every time a component is described, it feeds the narrative of the overall system by defining the unique role it plays in its functioning and how it works with the other components. This level forces decisions around domain boundaries and how different concepts of the system interact with each other. ### Low Level Design ![](https://kislayverma.com/content/images/2020/07/reco-low-level.jpg) Now we look inside each component and explain how it actually works on the inside. This is usually the point at which we will be forced to make some concrete choices about technology choices and other physical details like deployment and performance. Don't critique the oversimplifies example taken above, but focus on the fractal like approach applies a different set of design principles at each level but in the same philosophical manner of composition and decomposition. Enterprise Design Architecture principles might apply at the highest level, Reactive design principles at the component level, and SOLID principles at the low level - but the entire exercise if about explaining the how the parts and the sum of the parts functions. Of course, not all design will need all of these level. Don't fit them to this pattern forcibly. Build your own story - it can be a short one too! --- --- ### Why write architecture narratives There are several advantages to using this approach for your next system. #### Gut Feel Checks The narrative centric approach comes with gut-feeling checks around jarring changes in levels of abstraction. If your technical story feels like you suddenly skipped a few chapters and need to go back and check what happened, you have either missed laying out some important details or have jumped over them by making some assumptions. If you feel this is happening like going directly to implementation details, or talking about technologies that you might use, or naming service that you will build, it is usually a good idea to step back and focus on getting the plot line in the right order. What is it -> what does it look like -> what does it do -> how does it do it. A story that jumps all over the place is usually not very good (unless it's a time travel story - in which case it is all right). #### Build a shared vision Another advantage of explicitly building a story comes when you have to share it/convey it to others. If you are working with a team - sticking to a narrative breakdown allows the entire team to align with and contribute to the plot as it is being shaped. Not only does this put a lot more options on the table early, it will also set a shared vision around which all further decisions can be based. And once the design is ready and to be shared with managers or stakeholders or other engineers who might not yet understand the problem domain as you now do - you already have a story to sell! Not just the final ready artefact in the shape of implementable design, but also the process of how your team got there. This will help everyone understand the design better, it will also help them the problem structure and thus give better feedback. A steady, systematic breakdown of the problem from high level requirements to low level implementation details demonstrates that you put in the necessary diligence. The documentation will also be more self-explanatory because the processing backing it is self-explanatory. If you are selling a large idea to management, this can be invaluable. Also when you have to explain why some seemingly trivial feature will take more than 3 hours to implement. #### Postpone decisions as much as possible The last advantage of design-as-a-story is that you get to postpone decision. No good story start with lots of details. The best of them build characters over an extended period so that the characters have had to pass through many situation and evolve the version you actually fall in love with. Likewise we don't want to start out by saying "recommendation engine will be built using Spark and Hadoop". That can wait. Before we reach that point, we want to talk A LOT about what the recommendation engine is, who needs it, how it recommends, how it might change over time, who will build it etc etc. All of these things are important in making that final decision about exactly what we will build it with. We want to get know the persona of the engine before we pick his favourite weapon. The narrative approach is a good way to force that discipline on the design - concrete things are always decided as late as possible. ### Caveat Emptor With all that said, understand that everyone tells stories differently. This is just my way of doing things as I have learnt from experience and from others (Thanks [Shamik](https://www.linkedin.com/in/shamiksharma/?ref=kislayverma.com), [Abhinav](https://www.linkedin.com/in/abhinavyajurvedi/?ref=kislayverma.com), [Sanjay](https://www.linkedin.com/in/mailsanjayyadav/?ref=kislayverma.com), [Apoorva](https://www.linkedin.com/in/apoorva-gaurav-854057b/?ref=kislayverma.com), [Govind](https://www.linkedin.com/in/govindvenkatramankrishnan/?ref=kislayverma.com)) - it is certainly not the only way. Other people certainly approach it differently. There are data-first approaches, API-first approaches, UX-first approaches, Behaviour-driven-design and so on. You can use any of them to build great design - but remember that while people can always quibble with design, everyone like a good story :) Read next - [Highlight from "97 things every software architect should know"](https://www.kislayverma.com/post/highlights-97-things-every-software-architect-should-know?ref=kislayverma.com) ### Book Highlights: 97 things every software architect should know URL: https://kislayverma.com/highlights-97-things-every-software-architect-should-know/ Last updated: 2020-07-22T07:23:02.000Z ![](https://kislayverma.com/content/images/2020/07/97-things-every-software-architect-should-know.jpg) > "In this truly unique technical book, today's leading software architects present valuable principles on key development issues that go way beyond technology. More than four dozen architects -- including Neal Ford, Michael Nygard, and Bill de Hora -- offer advice for communicating with stakeholders, eliminating complexity, empowering developers, and many more practical lessons they've learned from years of experience." *"*[*97 things every software architect should know*](https://www.amazon.in/Things-Every-Software-Architect-Should-ebook/dp/B0026OR30S/ref=tmm%5Fkin%5Fswatch%5F0?%5Fencoding=UTF8&qid=&sr=&tag=kislayverma-21&ref=kislayverma.com)*"* is a collection of (very) short essays by some of the most effective software architects of our times, and contains practical as well as philosophical guidance for aspiring and practising software architects. Each essay is extremely focussed on a specific them. Most of them talk about concrete scenarios and how to deal with them. They are also independent and can be read one-by-one - there is no progression from one essay to the next. As someone who has been this role for some time now, it really resonated with me that a lot of the advice is not technical but rather focussed around team work and enablement. These are the parts that are most difficult to internalize for architects because we are trained to think like developers and measure ourselves in terms of "code shipped". The role of an architect is more than, as the many voices on this book repeatedly tell us. The moment I started reading "97 things every software architect should know", I realized that a lot of highlighting was going to happen :). I have included here what I feel are the highlights from the book and cover most of the messages conveyed in it. They are put here is the order of appearance in the book, and like the book, they do not follow a thematic progression. Pulling out highlights like this deprives them of some of the context they are written under, but I feel that these are essential nuggets and stand on their own. This is good place to start if you want a flavour of what you will find inside the book. I strongly recommend this book for anyone interested in playing the role of a software architect. --- --- 1. Architects are expected to know the technologies and software platforms on which their organizations run as well as the businesses that they serve. 2. Always put the customer’s long-term needs ahead of your own short-term needs and you won’t go wrong. 3. "Essential Complexity" represents the difficulty inherent in any problem. 4. "Accidental Complexity" grows from the things we feel we must build to mitigate essential complexity. 5. In large-scale software, though, removing accidental complexity while retaining the solution to the essential complexity is challenging. 6. Prefer frameworks derived from working code rather than ones cast down from ivory towers. 7. Projects are built by people, and those people are the foundation for success and failure. 8. Being clear and concise in the way you communicate your ideas is vital to the success of any software project. 9. Having the developer on your side creates a collaborative environment whereby decisions you make as an architect are validated. In turn, you get buy-in from developers by keeping them involved in the architecture process. 10. Experienced architects understand that they need to “sell” their ideas and need to communicate effectively in order to do that. 11. A better expression than ‘common sense’ is contextual sense — a knowledge of what is reasonable within a given context. 12. Sufficiently different nonfunctional properties of a subsystem create a boundary across which managing inconsistent representations is tractable. 13. To the extent that the business community fails to fulfil its responsibility to provide direction, answer questions, and make business decisions for the software development team, it is actually delegating the business decision making to software developers.The architect must provide the macro-context for this ongoing series of micro-decisions made by developers, by communicating and protecting the software architecture and business objectives, and must seek to ensure that developers do not make business decisions. 14. The long-term interests of the software development team are best served when business drives. 15. The pursuit of speculative generality often leads to solutions that are not anchored in the reality of actual development. They are based on assumptions that later turn out to be wrong, offer choices that later turn out not to be useful, and accumulate baggage that becomes difficult or impossible to remove. 16. A good architect should be able to spot a problem, call the team together, and without picking out a victim, explain what the problem is or might be and provide an elegant workaround or solution. 17. *Build as a big bang event* in project development is dead. 18. You’ll commonly see attempts to require overtime or sacrifice “less important scheduled tasks” (like unit testing) as a way to reduce delivery dates, or increase functionality while keeping the delivery dates as is. 19. Every software architect should know and understand that you can’t have it all. 20. Enough cannot be said about the importance of building a solid data model from Day One. 21. While business rules and user interfaces do evolve rapidly, the structures and relationships within the data you collect often do not. 22. Migrating data from one schema to another in situ is difficult at best, time consuming always, and error prone often. 23. The database is the final gatekeeper of your precious data. The application layer, which is by design ephemeral, cannot be its own watchdog. 24. The presence of two options is an indicator that you need to consider uncertainty in the design. 25. All architecture is design but not all design is architecture. Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change. 26. Effective architecture is one that generally reduces the significance of design decisions. 27. Issues that seemed trivial early in the project become critical after it is too late to fix them. 28. Individuals often face resistance when the rest of the team does not share their experience or knowledge. 29. Defensiveness is easy. Learning to stop it is hard. Pride in our accomplishments is easy. Recognizing our limitations without conscious effort is hard. 30. Did you give everyone’s ideas the respect and acknowledgment they deserved? 31. If it looks good, it probably is good. 32. The architect should constantly be on the lookout for decisions that will have to be made soon. 33. Effective software architects understand not only technology but also the business domain of a problem space. Without business domain knowledge, it is difficult to understand the business problem, goals, and requirements, and therefore difficult to design an effective architecture to meet the requirements of the business. 34. Programming is an act of design, not an act of construction. 35. Over time, a good solution to the right challenge will probably outlast all others. 36. Was the solution an appropriate one for the problem? Did it solve the needs of the problem? Keep these as your measure — you will be a lot happier. Be happy with all that old stuff. 37. Expanding scope is the enemy of success because the probability of failure grows faster than expected. 38. Question any requirements not explained in terms of measurable value to the customer. If it has no effect on the company’s bottom line, why is it a requirement? 39. Important requirements usually remain important as the business changes, while others change or even evaporate. 40. Stewardship, taking responsibility and care of another’s property, is the appropriate role of an architect. 41. Value stewardship over showmanship; never forget that you are playing with other people’s money. 42. It’s not ethical to worsen the lives of others, even a small bit, just to make things easy for yourself. 43. We should plan to deploy one component at a time - it forces us to create well-defined interfaces between components. 44. When we deploy software, we are exposing ourselves to the accumulated technical risk embodied in the code. By deploying one component at a time, we spread technical risk out over a longer period of time. 45. It’s rare to find a technique that simultaneously provides higher commercial value and better architectural qualities, but early deployment of individual components offers both. 46. Performance of the people building the system is often called productivity, and it is important because it directly affects the cost and schedule of the project. 47. To be an effective software architect you must understand the basic architecture and design patterns, recognize when those patterns are being used, know when to apply the patterns, and be able to communicate to other architects and developers using them. 48. Enterprise architecture patterns define the framework for the high-level architecture. Some of the more common architecture patterns include event-driven architecture (EDA), service-oriented architecture (SOA), resource-oriented architecture (ROA), and pipeline architecture. 49. Application architecture patterns specify how applications or subsystems within the scope of a larger enterprise architecture should be designed. 50. Integration patterns are important for designing and communicating concepts surrounding the sharing of information and functionality between components, applications, and subsystems. 51. Anti-patterns, a term coined by Andrew Koenig, are repeatable processes that produce ineffective results. 52. Context is king, and simplicity its humble servant. 53. While newsgroups rage with the flames of technology debates of X versus Y, it is idle amusement. The reason these debates rage is often not because of huge disparities in their technical merits, but rather because there are more subtle differences between them, and what features individuals value more tha n others when there is no guiding context to act as a trump card. 54. In architecture as in all other operative arts, the end must direct the operation. The end is to build well. Well building has three conditions: Commodity, Firmness and Delight. 55. Duplication is evil. Repetitive work slows down development. 56. It’s not the domain logic that is copied; it’s the infrastructure code that just has to be there to make it work. 57. It’s crucial that you can envision the effects your examples have. 58. As an architect, you need to be highly sensitive to any kind of repetitive patterns, since anything you write will (ironically) be repeated. 59. Repetition in code is something that developers eventually learn to filter out and ignore when reading the code, once they figure out where the interesting variabilities are found, but even if the developers get used to it, it slows them down. 60. Repetition won’t go away unless someone does something about it. That someone is you. 61. Be ready to respond to events at any time in any order, regaining your context as needed. Make asynchronous requests concurrently instead of calling methods one by one. Avoid complete chaos by modelling your application using event-driven process chains or state models. Reconcile errors through compensation, retry, or tentative operations. 62. Building loosely coupled systems is a bit of a drag, so why do we bother? Because we want our systems to be flexible so they do not break apart at the slightest change. 63. Building a system that is flexible generally means the architecture is more complex and it’s more difficult to get the proverbial “big picture.” 64. An architect strives to merge realities with vision; past success with future direction; business and management expectations with development constraints. Creating these bridges is a major part of being an architect. 65. Like Janus, a software architect needs to be a keeper of doors and passageways, spanning the old and the new, incorporating creativity with sound engineering to fulfil today’s requirements while planning to meet tomorrow’s expectations. 66. From an architect’s point of view, the hard part is to find the natural places to locate boundaries and define the appropriate interfaces needed to build a working system. This is especially difficult in large enterprise systems, often characterized by few natural boundaries and intertangled domains. 67. A bounded context is an area where a model or concept is uniquely defined. 68. The role of an architect is usually to impose constraints, but you also have the opportunity to be an enabler. 69. Make sure developers have the tools they need. 70. The work life of a developer should be hands-on and practical, but also should be actively academic. 71. Let developers make their own decisions wherever it won’t contradict the overall goal of the software design. But put constraints where they count, not only to guarantee quality, but also to further empower developers. Create standards for the sake of consistency, but also to reduce the number of troublesome, insignificant decisions that aren’t part of the essential problem developers are solving. 72. One type of documentation that ages well, doesn’t require much effort, and almost always pays off is a record of the rationale behind decisions that are made regarding the software architecture. 73. The documentation should answer the basic questions “What was that decision we made?”, and “Why did we make that decision?”. A secondary question that is often asked and should be documented is “What other solutions were considered, and why were they rejected?” 74. Best practices in software architecture state that you should document the rationale behind each decision that is made, especially when that decision involves a tradeoff. 75. At an individual level, we are all trying to grow and come to understand how to build larger and larger systems. The course of our careers will take us toward ever-increasing challenges, for which we want our past experiences to help guide us. 76. Testing your knowledge against the real world is scary, particularly when you find out that something dear is myth, incorrect, or was never true; it’s hard to be wrong. 77. Given the state of so much of our software, it is clearly important for us to take every opportunity to share the things we know, what we think we know, and what we’ve seen. 78. Stamping patterns all over a project unnecessarily is over-engineering. 79. The support and maintenance of an application should never, ever be an afterthought. 80. Sometime accepting a constraint or giving up on a property can lead to a better architecture, one that is easier and less expensive to build and run. 81. When creating your architecture, you should explicitly use principles, axioms, and analogies to guide the creation. This gives the architecture a number of benefits that are not present if you simply create by implicitly leveraging your experience, opinions, and tastes. 82. An architecture with clear principles is an architecture that frees its architect from reviewing everything and being everywhere. It gives architects greater leverage. 83. Start with a walking skeleton, keep it running, and grow it incrementally. 84. Seeing the system entirely by the structure of its underlying information — can reduce even the most complicated system down to a tangible collection of details. 85. Data sits at the core of most problems. 86. What we don’t want to do is apply a complicated solution to an easy problem. 87. Keep the simple stuff simple. 88. Your code is your currency. 89. As an architect, your primary goal should be to create a solution that is feasible, maintainable, and of course addresses the issue at hand. 90. If you design it, you should be able to code it. 91. Not everything need directly translate in monetary gains, but our investments should result in added value. 92. The ROI of each option can be determined by examining its costs and projected profits, and can be used as a base for selection from available options. 93. Consider architectural decisions as investments and take into account the associated rate of return. 94. Even if your system is bleeding edge and developed in the latest technology, it will be legacy to the next guy. Deal with it! 95. Good design will document itself in many ways. 96. Legacy tends to be a bad word in software circles, but in reality, all software systems should endure the tag. It is not a bad thing. 97. If you can only think of one solution to a problem, you’re in trouble. 98. If you find yourself in the situation where you automatically know the solution, without having done any comparison to other approaches, stop, take a step back, and ask yourself if you can think of another way to do it. 99. A good architect reduces complexity to a minimum and can design a solution whose abstractions provide solid foundations to build upon, but are pragmatic enough to weather change. 100. The great architect understands the impact of change — not just in isolated software modules, but also between people and between systems. 101. The architect’s role is not necessarily to manage change, but rather to ensure that change is manageable. 102. Shortcuts taken during the initial development phase of a project can result in significant maintenance costs later. 103. Poorly designed features can become the foundation for future features, making corrective action later even more costly. 104. Don’t give in to the temptation to make your design, or your implementation, perfect! Aim for “good enough” and stop when you’ve achieved it. 105. Show the business domain experts the respect you expect to receive; this is the last group of people you want viewing you as unapproachable. 106. Don’t allow yourself to become a disgruntled genius who spends all of his time trying to impress others by making witty, condescendin g statements about how poorly the company is run. They won’t be impressed. They’ve met that guy before and they don’t really like him. 107. Find a way to establish a good relationship with the business and don’t let your ego damage it. 108. Before anything, an architect is a developer. 109. If you don’t know what a thing should be called, you cannot know what it is. If you don’t know what it is, you cannot sit down and write the code. 110. An architect should be able to look at a whole mess of concepts and data and process and separate them into smaller pieces or “chunks.” The important thing about those problem chunks is that they are stable, allowing them to be solved by a system chunk that is finite and stable in scope. 111. If the problem is stable, then when it is solved, it is solved permanently. 112. Diligence also requires an architect to succeed at the deceptively simple task of making and keeping commitments. 113. Better is possible. It does not take genius. It takes diligence. It takes moral clarity. It takes ingenuity. And above all, it takes a willingness to try. 114. Don’t be clever. Be as dumb as you possibly can and still create the appropriate design. 115. There’s usually no huge advantage to being the first to adopt new technology, but there can be several drawbacks. 116. Your customer is not your customer. Your customer’s customer is your customer. 117. During requirement gathering, allow your customer to express only the Platonic ideal, his concept and goals, rather than dictating a solution or even using technical terms. 118. No matter how in-depth, how well researched, and how well thought-out your design, it will never come out looking the same as in your head. 119. By accepting that design is an ongoing and empirical process in a forever-changing world, we learn that the design process must be flexible and ongoing. 120. If you can control how people perceive the architectural approach you propose, it’s virtually guaranteed that you can control how they will react to your proposal. 121. Make a strong business case for your architecture. People who have the budget authority to sponsor your ideas are almost always business-driven. - Establish the value proposition. - Build metrics to quantify. - Link back to traditional business measures. - Know where to stop.F - ind the right timing. 122. Make data and schema management a seamless part of your automated build and testing process early on and include an undo button. 123. Sometimes the best solution is no solution. Many software problems need not be solved at all. They only appear as problems because we look only at the symptoms. 124. Make sure it’s tough to crack the starting lineup, and once you’ve got a winning team, go the extra mile to keep it together. 125. It is simply not possible to future-proof an architecture. 126. Your goal as an architect is to be aware of and measure the threat of acceptance problems and work toward mitigating those threats. 127. Many missed requirements and bugs in software can be traced to ambiguous, general language. 128. The architect should also look at doing user interaction testing while the product is still in beta with actual end users, and incorporate their feedback into the final product. 129. It is the architect’s responsibility to make the most common interactions not only easy but also rewarding for the end user. 130. Resist trying to design a large complete system to “meet or exceed” the known requirements and desired properties, no matter how tempting that might be. Have a grand vision, but not a grand design. 131. Design the smallest system you can, help deliver it, and let it evolve toward the grand vision. Next - More [book reviews](https://www.kislayverma.com/category/books/?ref=kislayverma.com). ### Design review checklist for Distributed Systems URL: https://kislayverma.com/design-review-checklist-for-distributed-systems/ Last updated: 2020-07-22T07:01:13.000Z Distributed system design is a hard problem, made all the worse because **the design process gives no direct feedback**. Problems stemming from faulty design often show up as scalability problems, resilience problems or data issues. However, solving those problems is often the equivalent of addressing the symptoms and not the disease - we may be able to patch up the system enough to keep it running, but the underlying design issues remain and can be triggered again under different circumstances. It takes a lot of effort, not to mention organizational wrangling, to be able to analyze design related root causes when the system is failing in production. As with an earlier post on [code reviews for distributed systems](https://www.kislayverma.com/programming/code-review-checklist-for-distributed-systems/?ref=kislayverma.com), this article is a simplistic checklist of things that I look out for when reviewing design of a distributed system functionality (anything which requires multiple systems to work together). I think of distributed design issues in terms of three buckets : **Consistency-vs-Availability**, **Domain Coupling**, and **Observability**. The first two often leak into each other because a distributed system is like a complex mesh and each design choice impacts multiple other things. Each bucket is a huge huge topic in its own right, so the following guidelines represent a minimum level of scrutiny that I believe should be imposed on any design. Depending on the use-case and context of the problem, you should go far deeper into specific aspects after these basics have been checked off. Conversely, if there's a problem with these, be extremely cautious. ### Consistency or Availability I should remind you upfront that when I say “system” in this article, I mean a collection of independent systems which collaborate in different ways to deliver the final user experience. And when we talk about consistency versus availability, we are talking about all the involved systems. [CAP Theorem](https://en.wikipedia.org/wiki/CAP%5Ftheorem?ref=kislayverma.com) tells us that we can choose any two out of consistency, availability, and partition tolerance for our system. However, network partitions are a fact of life, hence the true choice of CAP theorem is between consistency and availability - we can have an **"AP"** system (available, aka continues to work under partition) or a **"CP"** system (fails if all involved components are not alive and well). **A fundamental rule of software architecture is that all software fails**. So let's say that we need consistency across three components for a feature to work as designed. This makes the feature brittle because if any one of the components is down, the feature does not work. We now have the dreaded “*Single Point of Failure”* \- in fact we have three of them!!! The more we try to make the entire system consistent, the more we make it susceptible to failure at the slightest glitch. The more components that must stay in sync, the worse this gets. Fortunately, there is a way out. When looking at individual systems, CP and AP can be binary choices (e.g. MySQL is consistent, Cassandra is not), but when looking at a distributed system, they can start taking on shades of grey. Each component may be consistent (order, inventory and payment), but the overall system can still be designed to be eventually consistent. This gives us the necessary leeway to add some availability to our system. So the main guideline here is to design the OVERALL system for availability, such that the complete functionality can be achieved over time despite individual subsystems being intermittently unavailable. #### Use asynchronous message passing for communication This is the most powerful weapon we have for taking off the pressure of consistency in favour of availability and features as a major guideline in the [Reactive Manifesto](https://www.reactivemanifesto.org/?ref=kislayverma.com). Consider making the communication between components asynchronous (message passing over a message broker) instead of making it a request-response style API call. If [synchronous communication is the crack cocaine of Silicon Valley](https://hackernoon.com/synchronous-communication-is-the-new-cocaine-in-silicon-valley-oi22734tj?ref=kislayverma.com), synchronous API calls are the crack cocaine of the distributed system design. Consider this - if two systems don’t have to be consistent, then why should we do anything immediately (as the synchronous communication model demands). A request-response model creates a form of [temporal coupling](https://www.pluralsight.com/tech-blog/forms-of-temporal-coupling/?ref=kislayverma.com) ("serve this request right now!") between the caller and callee, which causes the former to fail if the latter is unavailable at a time, which can then cause cascading failures in its callers and so on. Asynchronous communication allows the called system to process requests at its own pace, thereby taking off the pressure of availability. While asynchronous messaging is a powerful tool, there are several things that MUST be kept in mind when adopting it. 1. **Define the minimal acceptable user experience** \- For every end-user experience, define the absolute minimal consistent experience. E.g. If a user wins an online game, must we credit bonus points, award him a new position on the leaderboard, notify all his friends, and send him a notification in an all-or-nothing manner? As should be obvious, the more we can agree to do outside the core, consistent experience, the less likely it is that we will encounter system failure. We must be ruthless about this when discussing requirements and do only the minimum needed to support that - everything else should be done asynchronously. 2. **Explicitly guarantee eventual consistency** : A single user action or client request can modify data across multiple components, and the design should guarantee that all these systems will come into consensus with some specified time - even if it is via a distributed rollback. 3. **Guarantee SLA for consistency systemically** : This point is worth calling out all by itself. We may have a plan for eventual consistency, but without the mechanism for setting and enforcing a set time frame on it, it is impossible to detect failure from slow processing. Since we cannot determine whether an event threw an error during processing or whether it got dropped in the network, an explicit hard bound in time is necessary to maintain the eventual consistency guarantee. --- --- ### Domain Coupling Good distributed system design essentially hinges on separating distinct things from each other at the right level of abstraction. This line of separation is called the domain boundary and is identified by a unique language of communication and interfaces for functionality unique to that domain. E.g. A message broker domain encapsulates messages, delivery guarantees, storage media etc. Payments domain encapsulates transactions, payment gateways etc. While this broadly maps to the concept of bounded-context in microservice language, the idea is applicable broadly and recursively. E.g. Payments can be a domain, inside which payment gateways and transactions can be sub-domains, and so on. The main idea of designing a distributed system in terms of domain is in terms of decoupling domains at the same level, while building cohesion at the level of their parent domain (if any).e.g. In the example above, we would try to keep payment gateways and transactions as decoupled from each other as possible in terms of implementation, but being part of the same domain should dictate a certain consistency in terminology and data models. 1. **Create domain boundaries** : With the [huge uptick in adoption of microservices](https://www.oreilly.com/radar/cloud-adoption-in-2020/?ref=kislayverma.com), it becomes very important to identify the domain of each component in a manner decoupled from the underlying technical implementation. It is important to identify which components belong to which domain and how external systems talk to these systems. We might use multiple services to handle a shipment (tracking service, scanning service, audit service etc), but external users should work with a cohesive “logistics” domain entity and API. A very good tool for building domain boundaries are [API gateways](https://microservices.io/patterns/apigateway.html?ref=kislayverma.com) which can abstract the internal details of a domain behind higher order APIs. 2. **Use standard domain language to communicate between systems** : Chatter between two components should be in terms of existing entities + their states + actions possible on them instead of some newly created constructs which belong to neither domain. We can use constructs from either side of the communication for this, depending on whether we want events or messages. If you find that communicating between two components requires creation of some special language, it “might” mean that there is some problem in the way these two components are separated or perhaps we are missing another component that should exist between these two. 3. **Separate multi-domain/multi-component actions into workflows** \- A very common way domain coupling occurs is when one component starts taking end-to-end control of a multi-component workflow. This means that this one domain now knows about various other domains, their behaviour, and the nature of a “workflow” outside of its own boundary. This awareness makes the component coupled to the existing workflows and hence difficult to evolve. 4. If our features invoke multiple components, we should separate this orchestration out of the core services modelling the domain into stateful orchestration components. This can be dedicated orchestrating services or some sort of BPM systems. Statefulness means that we get benefits like retries, error reports, SLA etc from one place. 5. Using choreography to build workflows is also a viable option if we cannot use explicit orchestration for everything, but we should use some way of tracking task completion SLA to reduce the brittleness for long-lived workflows. 6. **Model business processes across components** : A corollary to the above point is that we should model businesses processes end-to-end regardless of technical boundaries. Since we are already decoupling domains by offloading orchestration to workflows, it doesn’t make sense to build workflows within narrow team boundaries. We should make the workflows encompass as much of the business process as a whole as we can - this will build central repositories of business knowledge and provide deep visibility into the state of operations. 7. **Prefer events over messages** : A further component on decoupling domains is to prefer the use of pub-sub style events rather than targeted messages. While this is contingent on many other factors, it further enhances agnosticity towards other domains because the publisher doesn’t care about consumers in the pub-sub model, hence coupling between publisher and consumer domain is eliminated. ### Observability In the words of [Charity Majors](https://twitter.com/mipsytipsy?ref=kislayverma.com), ***Observability is the ability to answer new questions about a system without having to peek inside it***. I think of observability in two flavours : technical and business. We should be able to explain the technical state of the system, and we should be able to determine if it is doing what it is supposed to do from a business metrics perspective. 1. **Use event data to build metrics** : The basic unit of work inside any system is an event of some sort occurring explained in the system’s own domain language. The event can be of any type (e.g. ORDER\_CREATED, REQUEST\_RECEIVED, ERROR\_RESPONSE\_RETURNED). An idea worth pursuing is that instead of the common logs-traces-metrics approach, we consider the entire system information being modelled and emitted as events and stored outside in an analytical database for deriving intelligence. The benefit of having raw event data is that in sync with the domain modelling approach (something happened), and raw data can be used to derive new metrics at any time. This is way better than trawling through some combination of unstructured text logs, spans, traces, and arbitrary Prometheus/Statsd type metrics. 2. **Store raw data in a central place** : The only way to honour the definition of Observability given above is to store raw data from systems because the moment we start dealing only in pre-defined metrics, we are bound to them and lose the ability to answer any “new” questions. One common argument against raw data is volume, but there are ways to mitigate that (sampling etc.), while the ability to diagnose your systems in face of new failure modes is priceless. And since the overall system is distributed, having the observability data also in distributed silos will be much less powerful than bringing all of it together to derive holistic insights. ### General Guidelines 1. **Use asynchronous frameworks for implementation** \- I have written before about how we can [use asynchronous programming to scale our systems significantly](https://www.kislayverma.com/programming/overcoming-io-overhead-in-micro-services/?ref=kislayverma.com). So when invoking a remote system, we should ensure that we are doing it in an asynchronous manner so as to not block application threads. This is a bit of early implementation/design choice and if your application framework does not allow this or the base framework of the application is not designed for this then you are out of luck. If you have the option of going asynchronous, always take it - your system and your team will thank you for it when that unexpected spurt of traffic comes. 2. **Know your callers** \- By definition, no one is in charge in a distributed system. As a result, we should take as many precautions for internal systems as we might take for external systems. The least of these is to enforce rate limits if you can, but at least know your callers. This will help isolate the origin of trouble when things go wrong - identifying sources of tracking using IP addresses when the system is on fire is not fun. 3. **Know when to fail** \- I have spoken a lot about how to make our systems more available under varying conditions, but there are situations when failing is better than doing the wrong thing (Someday I will write an article about how our message passing based design resulted in way more orders than we had inventory for). Consistency is a perfectly acceptable choice in many scenarios, and we should be careful to not overcompensate against them. --- I hope you find these guidelines useful in reducing the most commonly found issues in distributed systems design. I would to hear if you have some other considerations that you find simple to apply but very effective - we can add them here! ### Code review checklist for distributed systems URL: https://kislayverma.com/code-review-checklist-for-distributed-systems/ Last updated: 2026-07-22T12:47:13.000Z ![](https://kislayverma.com/content/images/2020/07/chris-ried-ieic5Tq8YMk-unsplash.jpg) Photo by Chris Ried on Unsplash **\[Update\]** *I have made several additions to the original post based on the excellent feedback I have received. The new recommendations are marked out in italics with credits at the end of the article.* --- Microservice architecture is a widely adopted practice now in the software engineering world. Organizations that adopt this architectural style find themselves dealing with the added complexity of distributed failures (over and above the complexities of implementing business logic). The [fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies%5Fof%5Fdistributed%5Fcomputing?ref=kislayverma.com) are well documented, but subtle to detect. As a result, building large scale, reliable distributed systems architectures is a hard problem. As a corollary, code that looks fine in a non-distributed system can become a huge problem the moment we introduce the complexity of a network interaction to it. After encountering failure patterns in production code for several years and having root caused them to various bits of code, I (like many others) have come to identify some of the more commonly occurring failure patterns. These vary slightly across companies and language stacks (depending on the maturity of the internal infrastructure and tooling), but one or more of these are very often cause of production issues. Here are some code review guidelines that serve as my base checklist for reviewing code relating to inter-system communication in a distributed environment. Not all of them apply all the time, but they are all pretty basic problems, so I find it useful and comforting to mechanically go down this list, flagging missing items for further discussion. It is, in that sense, a dumb checklist that you would likely ALWAYS want to be followed. ## When invoking remote systems ### What happens when remote system fails? No matter how much care a system is designed with, it will fail at some point- that's a fact on running software in production. It may fail due to a bug, or some infrastructure issues, or due to sudden spike in traffic, or with the slow decay of neglect, but fail it will. How the callers handle this failure will determine the resilience and robustness of the overall architecture. 1. **Define a path for error handling** : There must be explicitly defined paths in the code for error handling instead of just letting your system explode in the end users face. Whether it is a well designed error page, an exception log with an error metric, or a circuit breaker with a fallback mechanism, errors must be handled explicitly. 2. **Have a plan for recovery** : Consider every single remote interaction in your code, and figure out what we need to do to recover the work which was interrupted. Does our workflow need to be stateful so that it be triggered from the point of failure? Do we publish all failed payloads to a retry queue/DB table and retry them whenever the remote system comes back up? Do we have a script to compare the databases of two system and bring them in sync somehow? An explicit and preferably systemic plan for recovery should be implemented and deployed before the actual code is deployed. ### What happens when the remote system slows down? This is even more insidious than outright failure because we do not know whether the remote system is working or not. The following things should always be checked to handle this scenario. *Some of these concerns can be addressed transparent to application code if we are using Service Mesh technologies like Istio. Even so, we should make sure that they are being taken care of, regardless of the how*. 1. **Always set timeouts on remote system calls** : This includes timeouts on remote API calls, event publishing and database calls. I find this simple flaw in so much code that it is shocking and yet-not-unexpectd at the same time. Check if finite and reasonable timeouts are being set for all remote system in invocations to avoid wasting resources in waiting should the remote system become unresponsive for some reason. 2. **Retry on timeout** : Network and systems are unreliable and retries are an absolute must for system resilience. Having retries will usually eliminate a lot of the "blips" in system-to-system interaction. *If possible, use some sort of backoff in your retries (fixed, exponential). Adding a little jitter to the retry mechanism can give some breathing room to the called system if it is under load and may lead to better success rate.* The flip side of retries is idempotency, which we will cover later in this article. 3. **Use circuit breaker** : There aren't a lot of implementations that come pre-packaged with this functionality but I have seen companies writing their own wrappers internally. If you have this choice, definitely exercise it. If you don't, consider investing in building it. Having a well-defined framework for defining fallbacks in case of error sets a good precedent 4. **Don't handle timeouts like a failure** \- timeouts are not failures but indeterminate scenarios, and should be handled in a manner which supports resolution of the indeterminacy. We should build explicit resolution mechanisms that will allow the systems to get into sync for cases where timeouts occurred. This could range from simple reconciliation scripts to stateful workflows to dead letter queues and more. 5. **Don't invoke remote systems inside transactions** \- *When the remote system slows down, you will end up holding on to your database connection for longer, and this can rapidly lead to running out of database connections and therefore outage for your own system.* **Use smart batching :** If you are working with lots of data, make batch remote calls (API calls, DB reads) instead on one-by-one to remove the network overhead. But remember that the large the batch size, the greater the overall latency and greater the unit of work which can fail. So optimize batch sizes for performance as well as failure tolerance. --- --- ### When building a system others will invoke 1. **All APIs MUST be idempotent** : This is the flip side of retrying API timeouts. Your callers can only retry if your APIs are safe to retry and do not cause unexpected side effects. By APIs I mean both synchronous APIs and any messaging interfaces - client may publish the same message twice (or the broker may deliver it twice). 2. **Define response time and throughput SLAs explicitly and code to adhere to them** : In distributed systems, it is far better to fail fast than let your callers wait. Admittedly throughput SLAs are hard to implement (distributed rate limiting being a hard problem to solve by itself), but we should be cognizant of our SLAs and provision for failing the calls proactively if we are going over it. Another import aspect of this is knowing the response times your downstream systems so that you can determine what is the fastest your system can be. 3. **Define and limit batch APIs** : If exposing batch APIs, maximum batch sizes should be explicitly defined and limited by the SLA we want to promise. This is a corollary to honouring SLAs. 4. **Think about Observability up-front** : Observability means having the ability to analyze the behaviour of a system without having to look at its insides. Think upfront about what metrics should you gather about your system and what data you should gather that will enable you to answer previously unasked questions. Then instrument the systems to get this data. A powerful mechanism for doing this is to identify the domain models of your system and publishing events every time an event happens in the domain (e.g. request id 123 received, response for request 123 returned - notice how these two "domain" events can be used to derive a new metric called "response time". Raw data >> pre-decided aggregations). ### General guidelines 1. **Cache aggressively** : The network is fickle, so cache as much as you can as close to the usage of the data as you can. Of course, your caching mechanism may also be remote (e.g. Redis server running on a separate machine), but at least you will bring the data into your domain of control and reduce the load on other systems. 2. **Consider unit of failure** : If an API or a message represents multiple units of work (batch), what is the unit of failure? Should the whole payload fail all once or can individual units succeed or fail independently. Does the API respond with success or failure code in case of partial success? 3. **Isolate external domain objects at the edge of the system** : This is another one I have seen cause a lot of trouble over the long term. We should not allows domain objects of other systems be used all over our system in the name of reuse. This couples our systems to the other system's modelling of the entity and we end up with a lot of refactoring every time the other system changes. We should always build our own representation of the entity and transform external payloads to this schema , which we then use inside our system. ### Security - **Sanitize input at every edge** *: In a distributed environment, any part of the system may be compromised (from a security standpoint) or buggy. Hence every system mist take individual care to sanitize its input at the edge instead of assuming that it will get clean/safe input.* - **Never commit credentials** *: Credentials (database username/password or API keys) should NEVER be committed to. code repository. This is an extremely common practice that is very hard to get rid of. Credential must always be loaded into the system runtime from an external, preferably secure storage*. --- I hope you find these guidelines useful in reducing the most commonly found mistakes in distributed systems code. I would to hear if you have some other considerations that you find simple to apply but very effective - we can add them here! --- ***Thanks for your suggestions!*** - Mayank Joshi for his note on exponential backoffs in retries. - Manjit Karve for the suggestions on security. - Sumit Satnalika for recommendation on keeping remote invocations outside transactions. - Raja Nagendra Kumar for pointing out that service mesh technologies like [Istio](https://istio.io/?ref=kislayverma.com) can take care of some things like retries, timeouts, and circuit breakers. Read Next - [Design review checklist for distributed systems](https://kislayverma.com/design-review-checklist-for-distributed-systems/) ### Launch Announcement: Rulette Home, Rulette Server, and more!!! URL: https://kislayverma.com/launch-announcement-rulette-home-rulette-server-and-more/ Last updated: 2020-07-22T06:49:54.000Z Today is a big day in Rulette land! I want to make several big announcements. ### The official home for Rulette [Rulette.org](http://rulette.org/?ref=kislayverma.com) is now the official home for all things Rulette! There's tons of new information and freshly written documentation to get started with Rulette and to get deeply familiar with it. Do take a minute to explore it. This and my mailing list/blog will carry all future updates about Rulette. Sign up to the mailing list and stay updated - more articles, tips-and-tricks, etc are coming! ### Rulette Server is GA ![](https://kislayverma.com/content/images/2020/07/rulette-view-rules-ui.png) The first version of [Rulette Server](https://www.kislayverma.com/rulette/getting-started-with-rulette-server?ref=kislayverma.com) is now available for general use. This has been a couple of months in the making and I'm happy to announce that with this release, non-Java users can have first class access to the power of Rulette. The server exposes REST APIs for every operation that the Java SDK can perform. API documentation is available on [SwaggerHub](https://app.swaggerhub.com/apis-docs/kislayverma/rulette-server-api/v1?ref=kislayverma.com) and a [Docker image](https://hub.docker.com/repository/docker/kislayverma/rulette-server?ref=kislayverma.com) is available for the container-happy folks. Even more awesome, Rulette server comes with a UI that you can use to navigate and manage your rule systems. Working with rule and collaborating with non-tech teams just got very easy! Don't believe me? Check out this [live instance](http://demo.rulette.org/?ref=kislayverma.com) (non-secure, but there's nothing malicious, cookies etc there, so don't worry) where you can play around with the UI. ### Working with Rulette ![](https://kislayverma.com/content/images/2020/07/working-with-rulette-book-cover.jpeg) I announced last week on social media that I had self-published [*Working with Rulette : Mastering Business Rule Management*](https://www.amazon.in/gp/product/B089S7NWS6?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B089S7NWS6&ref=kislayverma.com) on Amazon. This is a compilation of all the Rulette documentation and a case study in a step-by-step, concise package. You can get a Kindle copy from Amazon (thank you very much!) or join the mailing for a free PDF version. ### Rulette SDK Latest Version As I had mentioned in a [run-up post](https://www.kislayverma.com/rulette/gearing-up-for-the-big-rulette-release/?ref=kislayverma.com), Rulette SDK has received a lot of fix and new functionality over the course of developing Rulette server. If you are using Rulette now or plan to do, the new version (1.3.4) is absolutely worth checking out. ### Support Rulette A little support and external validation never hurt anyone! While I and the other contributors will continue to make Rulette an even more powerful tool for you, we sure would appreciate it if you show your love. If you use Rulette or like what you are seeing so far, here's a few things you can do to join the gang. - Consider using Rulette when you next encounter a case for rule management. Let me know if you want to know if your use-case fit. I'd be happy to have a quick chat. - Reach out to us and others about Rulette has made your life easy. We love hearing tales from out in the wild. - Request features, file bugs, and contribute code to the repositories ([SDK](https://github.com/kislayverma/Rulette?ref=kislayverma.com) and [server](https://github.com/kislayverma/rulette-server?ref=kislayverma.com)) - that's the power of open source! - Consider buying ["Working with Rulette" on Amazon](https://www.amazon.in/gp/product/B089S7NWS6?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B089S7NWS6&ref=kislayverma.com). The book provides a complete reference to using Rulette for modelling business rules and the royalty from sales keeps the demo instance running. - Support me on [Patreon](https://patreon.com/kislay?ref=kislayverma.com). Think of it as buying me coffee once a week :) - Check other avenues for lending financial support on the [home page](https://kislayverma.com/rulette/support/). Have questions? Want to find out more? Post your comments here or drop me an [email.](mailto:kislay.nsit@gmail.com) ### Platforms (and dogfood) everywhere URL: https://kislayverma.com/platforms-and-dogfood-everywhere/ Last updated: 2020-07-22T06:46:49.000Z I have often been asked about what type of systems make good candidates for being platformized, and my answer is always "All of them". The characteristics of a platform derive not from the nature of the problem it intends to solve, but from the inherent structure of any business space. There are problems that a business wants to solve, and it picks one of the many potential ways of solving it which it deems most suitable for itself. This is the company's "organization context". In the technical modelling of the problem space, however, engineers have to deal with real world nuts and bolts of the problem space (the "Domain Context") as well as the defined business process. This split almost always creates the potential for applying the subtle knife of platformization to any technical design. ![](https://kislayverma.com/content/images/2020/07/knife-of-platformization.jpg) In other words, there is dog food everywhere. Consider a fairly realistic scenario : A team is asked to build a system for sending marketing notifications to customers. The company wants to be a platform centric company. How should this team adopt the the platform design mindset and approach this as two problems : send marketing related notifications via a notifications platform - especially when the latter part is not explicitly stated. How should the team discover the hidden, broader mandate and recognize the platformization opportunity here? To achieve this split, the team has to perform a very different type of design exercise. They have to analyse the product specification and determine what parts of it are invariants and which parts of it are a function of marketing. This is harder than it appears at first glance because the team is immersed in the organization context for marketing, as are its stakeholders, its product mangers, and its engineering leaders. It is difficult to disengage from this implied context and design platform and product components separately. Even though the platform is only meant for consumption within the organization and the only know user at this point is the platform owner team itself, the [Golden Rule of Platforms](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com) says that the platform has to be built as if it was being built by someone outside the marketing notifications team and later they are going to use it just like any other tenant on the platform. As you might imagine, it takes a lot of discipline to draw and hold this line. It helps if the development team temporarily thinks that they are a separate company called "Notifications Pvt Ltd." and are trying to sell their software to "Marketing Pvt Ltd" and "Payments Pvt. Ltd." ![](https://kislayverma.com/content/images/2020/07/notification-platform.jpg) --- --- "But my job is not to make a platform, I just want to send notifications!" - If you have been thinking this, you are right. In fact, if your organization is in the middle of switching to a platform-mindset, this is probably the most frequently asked question and the biggest source of disgruntlement to engineers. When we face this question, we need to consider what we are trying to build. If it is a technical capability, then it stands to reason that we don't want to build the same thing again and again in the organization. Developers understand this, and technical constructs like authentication etc often get built as platforms spontaneously. Software engineers have been building platforms way before the word was cool. A core part of Unix development philosophy is to have tight programs with well defined purposes, easily connected via pipes - this is also the essence of building a technology platform. Whatever we building outside of pure technical capabilities, however, we are likely building it in the context of a business and we are building it as per the rules of that business. And the thing with rules of business is that they change. And when they do, it is up to the engineers to design the systems in a way that we can salvage as much as we can and enhance the rest, instead of having to go in for a big rewrite every time. Remember - [build momentum, not velocity](https://www.kislayverma.com/agile/being-fast-or-getting-faster-aka-build-momentum-not-velocity/?ref=kislayverma.com). Now we can say "that's just modularity and good design", and it would be right. But I propose that we will often find so called modular code to still be dependant on an assumed context in which it is operating w.r.t how and why it will be invoked, how its errors will be handled etc. Our modules are often decoupled in implementation, but dependant in the context of real world use. Platform design enforces a conscious evaluation of modularity to arrive at a split between what is business opinion and what is technical "fact". We can apply the same pure-technology mind-set to business problems as well and break it down into what is "core" and what is "opinion". Applying the product-platform split mindset to every design problem over time will result in diverging rates of change for different components. This means that we will end up changing lesser and lesser code over time to meet business needs. A common example of this failing, especially common in "internal" platforms, is for the team building the platform to assume itself to be the "centre of domain expertise" and deciding behaviours on behalf of the clients. e.g the "notification platform" team may decide that it is the best judge of who the best provider of notification infrastructure and not allow the users of the platform to make that choice. This may be fine for most use cases, but this simple choice has put the team owning the platform in the path of potential change, and [reduced the platform to a product](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com). If a need arises where a variant behaviour is required, the platform APIs will have to be changed in some way, and this might impact existing users. Had the team chosen to expose an overridable default as the way of choosing notification infra, such a thing could have been avoided. The implementation was clean, but it made assumptions about who can decide what. Every time we find platform teams making business choices beyond. suggested default, there is a good chance that the modularity of implementation is weakened by the assumption of context in which the implementation will run. In platform architecture, we are generally better off exposing more hooks than less. So the job may not be to send notifications at first glance, but over a longer time scale, the job is to deliver more and more value to marketing business, and anything we can do to easily keep up with evolving requirements is a good thing. The Golden Rule of Platforms and applying platform thinking to every design problem has other interesting effects too. The layered structure it creates ripples outwards as successive components apply the same principles and a dynamic boundary automatically emerges for the platform. At this boundary, special use-cases continually separate themselves out of the platform and generic capabilities make their way in. This is the kind of dynamic situation where eco-systems and collaboration can thrive and opportunities can be readily leveraged. This is the [edge of innovation](https://edgeperspectives.typepad.com/edge%5Fperspectives/2008/05/innovation-on-t.html?ref=kislayverma.com) . ### Scope of Business Rule Management Systems URL: https://kislayverma.com/scope-of-business-rule-management-systems/ Last updated: 2020-07-22T06:41:30.000Z When organizations evaluate business rule management systems, there are several way in which they can interpret the boundary of a rule system. The business teams work at the “what do I want to do” level of abstraction and for them the requirement is quite simple. e.g. they want a system where they can define that *a customer with life time value of more than Rs 5000 should not be charged cancellation fee if she cancels her order*. Or *a call from high priority client should be routed directly to the more highly skilled customer care agents who are free and can speak the customer’s preferred language*. The development team has a more ambiguous decision to make - what part of the statement mentioned above go inside the rule system and what should be kept outside. While the end result will be the same for the business team, the choice made will have important effect on how the rule system and the systems using this rule system evolve over time. It also decides what technologies are viable for implementation of both of these. In the above example, we may decide that the system should encapsulate the entire semantics of the rule, i.e., it should encapsulate the meaning and computation of customer lifetime value. More specifically, the interface of a rule system should be something like *getCancellationCharge(customer id)*. This is an excellent domain aware API which can be intuitively used from anywhere in the system. However, the implementation of this rule will also have to change if the order management system makes any changes to the customer LTV is tracked. [Rulette](http://rulette.org/?ref=kislayverma.com) make a conscious choice to keep all domain context outside. This means that Rulette cannot calculate LTV For you and it does not understand what it means. Rulette’s interface looks like this - *getOutput(String inputName, int threshold)*. If the input with name fieldName (“customer LTV” in this case) has a threshold more than 5000, output should be zero. The application which uses Rulette has to understand what customer LTV means as input and what zero means as output. Should a rule management system have to change if the implementation of the system about which the rule is written changes? If it does, can the rules be considered to be fully decoupled from use of rules? This is a decision you have to make for your application and organisation. ### Rule of thumb for using Rulette If you can frame your rules in terms of *IF something=fixed value AND something-else=other fixed value AND something-else-again between fixed value 1 and fixed value 2, THEN outcome*, then Rulette is an excellent choice for you. However, if you want to frame your rules as *IF something = derive some value from some input AND something-else = derive another thing from more inputs, THEN outcome derived from combination of inputs*, then Rulette has been deliberately designed to not serve your use case. You should consider using Apache Drools or something like that. ### Gearing up for the big Rulette release URL: https://kislayverma.com/gearing-up-for-the-big-rulette-release/ Last updated: 2020-07-22T06:39:24.000Z The last couple of weeks have been very hectic as I've been working full-speed on making [Rulette Server](https://github.com/kislayverma/rulette-server?ref=kislayverma.com) ready for its first big release. A lot of code has been written, but it turns out that code is easy (relatively speaking). I want to offer a top-notch user-experience for Rulette server from day-1 (something I didn't do well with [Rulette](http://rulette.org/?ref=kislayverma.com) SDK). So I've built a pre-launch checklist focussing on developer and first-time user experience that I'm going through now. I thought I'll share some of these things with you today, and share some early bird views. ### Fixing Rulette Turns out that although [Rulette](https://github.com/kislayverma/Rulette?ref=kislayverma.com) has been out there for over 5 years now, it still had a bunch of bugs, which had to be fixed first. Finding and fixing these was great because Rulette is a much better product now, but I'm also thinking testing-someting-something. ### Enhancing Rulette The big gap between Rulette and Rulette server turned out to be that the Rulette works well as an SDK (which it is-makes sense), but does not expose a lot of functionality that is super-cool to have in an API server/admin UI. You couldn't add/delete rules systems or rule inputs because it was assumed that the user of the SDK sets up the rule system separately behind the data provider implementation and then uses Rulette to simply access the data. But admin actions in the Rulette API/UI demand that all of this stuff be exposed via API. So I spent a lot of time adding these capabilities to Rulette itself. Then I exposed these properties over the REST API and built them into the UI, ended up redoing a whole bunch of stuff because I realized a little too late that providers had to be a first-class, public construct for namespacing purposes. Without this, two rule system created under different provider but with the same name would overwrite each other. ### Building the homepage I want a single place where Rulette SDK, server, documentation, articles, and news can find a common home. So I'm creating spaces for all these on my personal website and will get rulette.org to point to it. This has all the complexity of setting up a small website (figure out layout, customize pages, prepare and arrange content, optimize for mobile etc etc). ## Extensive documentation The documentation on Rulette SDK and server is kind-of slim so far. So I started writing detailed documentation which explains the concepts, the usage, and the internal design of Rulette. I started out writing these docs on the website, but this thing grew and grew and currently stands at \~50 pages of detailed documentation, UI walkthrough and case studies (one down, one more to go). Turns out writing good documentation is insanely hard - I come back after a break and realize that what I had written is not as clear as I thought it was. Re-phrase, re-organize, repeat. Checkout the current draft [here](https://docs.google.com/document/d/15W1yOJt7he5pM-bMFnYFt95I01XQx1yBSZ2pATA5Dn0/edit?ref=kislayverma.com#heading=h.749nfuak8e6l) \- I'm planning to re-do it in Apple Pages and make it available for download as a PDF. ### Docker and Public API I really like how [Zipkin](https://zipkin.io/?ref=kislayverma.com) (and others) offer a docker image right on their website for people to get started quickly. I wanted the same experience for Rulette server users - so I learnt a little bit about Docker and published the docker image for Rulette server on [DockerHub](https://hub.docker.com/repository/docker/kislayverma/rulette-server?ref=kislayverma.com). Figuring out Docker took a while, publishing the image itself was super easy. Similarly, Swagger API is available on [SwaggerHub](https://app.swaggerhub.com/apis-docs/kislayverma/rulette-server-api/v1?ref=kislayverma.com). ### Live instance Documentation is great, but nothing beats the real thing. I'm going to set up a live instance of rulette server with some dummy data so that users can play around with it and get a feel for its awesomeness. ### Support Making Rulette a saleable product has been top of mind for me for some time now. So I'm going to try several different ways of doing this with Rulette server release. Soliciting support on [Patreon](https://patreon.com/kislay?ref=kislayverma.com), self-publishing the user's manual on Kindle (coming soon), and taking direct donation on the website are some early ideas. If you have seen/used and liked Rulette, I'd love to hear more ideas around this. A whole lot is going, and the goal is to finish everything in one more week. Stay Tuned! ### APIs are not platforms URL: https://kislayverma.com/apis-are-not-platforms/ Last updated: 2026-07-22T12:47:15.000Z An [*Application Programming Interface*](https://en.wikipedia.org/wiki/Application%5Fprogramming%5Finterface?ref=kislayverma.com) (aka API) is the interface other people/developers use to access the functionality of your code. This is valid at the lowest levels of programming (microprocessor programming), at the highest levels (REST/GraphQL/gRPC/... APIs exposed by services), and everything in between (public methods of Java classes etc). With more and more organizations choosing to expose their business capabilities over publicly accessible APIs, there is a widespread conception that having an API is a clear indicator that a business is "going digital". And since the [distinction between products and platforms](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com) is already drawn poorly (as we have seen before), it has led to a massive explosion of so called "platforms" which are, in fact, nothing of the sort. --- Exposing public APIs represents an attempt at opening up our system for use by the external world. An organization might build and expose API for many reasons. The primary among these is to allow its existing customers or partners to access some parts of its system remotely. It might also wish to attract more customers or partners by touting this digital experience as a differentiating factor from its competition. In industries where technology has not found a strong foothold, this can be very effective. ![](https://kislayverma.com/content/images/2020/07/api-is-one-way.jpg) Once we have APIs in place, we are committing to other teams/organizations that they can access the behaviour of our systems in a certain way and that we offer well-defined guarantees around the functionality. This includes defining the specifics of the functionality, expectations around availability and other service level agreements like latency, throughput etc, and the cost of using the API. Done well, all of this represents a significant trust on and investment in technology. Even building this little digital window into its business can have dramatic effects for a company and a lot of hitherto old-school companies find even this limited digital transformation extremely challenging. Consider that you are a bank which has hundreds of branching running paper based banking processes. Every one of these processes has been refined over decades to cater to every customer touch-point with certain SLA. e.g. Opening a bank account may involve filling out a form, submitting many documents, repeated trips to the branch office, all taking 3-4 days. Now let's say you decide to expose a createAccount API /web portal where a customer can fill the form and upload all documents in one go. Customers now expect to open bank accounts much. more quickly and many more of them may sign up via. this method. What happens to the. manual account opening machinery now. All the processes put in place over the years have to be re-thought to service the new customer expectation. Also, what good is a *createAccount* API without a *checkBalance* API or a *transferFunds* API? If the bank continues to offer deeper online access to its capabilities, it is disrupting its legacy business very rudely and you can bet that a lot of feathers are being ruffled. If it doesn't offer these capabilities or offers some arbitrary subset of them, the entire digital customer experience is sub-par and the "digital transformation" stalls or fails. As you can see from this example, having an API by no means qualifies the organization as being tech-driven. The technology mindset has to be adopted wholesale, or it is unlikely to be as fruitful as expected. --- --- The platform narrative is more subtle, and thus gets obscured or misinterpreted in this API building frenzy. As we have [discussed before on this blog](https://www.kislayverma.com/category/platform-thinking/?ref=kislayverma.com), a platform is a set of tools that can be used to build new products and experiences. While APIs are critical to the platform journey (as the means to accessing the platform), a set of APIs is by no means a guarantee of platform architecture. [A platform is more than a set of APIs](https://www.youtube.com/watch?v=d8UEYNFrvjk&ref=kislayverma.com) . By themselves, APIs are a one-way street. We have certain capabilities, that we allow some external agencies to access. An API, in this sense, in a sales funnel for our core business. While this allows others to access our core product, a platform offers far deeper modes of co-operation that an API does. This is because[ platforms offer the opportunity for organization to partner](https://www.kislayverma.com/platform-thinking/why-you-should-build-a-platform/?ref=kislayverma.com) with each other at every step of every business process. While APIs limit interactions to the edge of the system (an existing end-user functionality becomes accessible over an API), a platform approach would open up the systems in a way that allows organization to easily hook into each other's decision making processes, leverage each other's capabilities more deeply, and create business value at the core of the business. This is done by integrating APIs from different organizations into a continuous chain of business decisions and resultant actions. ![](https://kislayverma.com/content/images/2020/07/platforms-is-more-than-api.jpg) Fig-2 : Platforms allow for very deep integrations among partner organizations in an ecosystem As Fig-2 shows, platform architecture actually externalizes the organization itself (or vice-versa, depends on how. you want to look at it) from the technical capabilities, thereby freeing all decisions to choose the best option from the entire eco-system with equal ease. The business processes of the organization can now use what was built by the internal teams, or easily bring in the technical capabilities of partner organizations to fill a gap in the value creation process. As some of you may have noticed, this is very similar to how we can build systems "inside" a platform organization by stitching APIs from different teams into end to end business processes. This is yet another way to understand why I think that there are no "internal" platforms. Platform architecture and processes are inherently externalizable - always ready to be deployed in a context other than the one in which they were originally developed. Over and above the useful capabilities it offers its users, [a platform is the set of control structures](https://www.kislayverma.com/platform-thinking/platform-nuts-bolts-enforcing-constraints-in-platform-architectures/?ref=kislayverma.com) required of operate the platform smoothly and prevent abuse. It is backed by an organizational structure and processes which focusses on creating and capturing economic value by empowering others, and it is the platform owner's commitment to mutually beneficial exchange of capabilities in an eco-system. The APIs of a platform are explicitly designed for being used as building blocks for other things. Similar arguments hold for team mind-set. Teams in platform-centric organization work HAVE to work with an enablement mindset of helping their customers build new things and exploiting any opportunities in the eco-system to best accomplish their respective missions. The technical architecture of the platform super-charges this mandate by making it extremely easy to collaborate technically. As a result, [a team can work seamlessly with any internal or external teams](https://www.kislayverma.com/platform-thinking/control-and-chaos-in-platform-systems/?ref=kislayverma.com) without having to worry about playing technology gatekeeper or process policeman. Participating in the ecosystem becomes the default behaviour instead of being an exception. --- An API, sadly, is just an API. It can be thought of as a product - useful within its own context, but ill-suited to the give and take of a dynamic eco-system. The API-is-platform fallacy is another instance of organizations falling for the hype surrounding platforms and trying to build one as cheaply as possible without understanding their true nature. **Read Next** \- [Marketplaces are not platforms](https://kislayverma.com/marketplaces-are-not-platforms/) ### Modelling tax rules with Rulette : Part Two URL: https://kislayverma.com/modelling-tax-rules-with-rulette-part-two/ Last updated: 2026-07-22T12:47:16.000Z In [part one](https://www.kislayverma.com/rulette/modelling-tax-rules-with-rulette-part-one/?ref=kislayverma.com) of this series, we looked at ground-up modelling, storage, and evaluation of taxation rules using [Rulette](http://rulette.org/?ref=kislayverma.com). By the end of that article, we had a [Rulette](https://kislayverma.com/rulette-a-pragmatic-rule-engine/) based system storing all rules in MySQL and we were able to evaluate the applicable tax values for various combination of rule inputs. In this post, we will look at how we can evolve this rule system to accommodate more evaluation criteria. ### Rules are meant to change Let us consider that the government changes the tax law to state that taxes to be paid on a sale now depend not only on the state where the manufacturer is located but also on the state where the customer is located. And this change of rules will come into affect a week from now, i.e. May 6, 2020. Everyone goes into a huddle again, and out comes an updated excel sheet which has lots more rules than our previous sheet (because we need to model all to-from combinations of states). e.g ![](https://kislayverma.com/content/images/2020/07/old-rule.png) becomes ![](https://kislayverma.com/content/images/2020/07/updated-rules.png) The other part of the problem is that the new rule set should only become live at 00:00 on May 6\. There should ideally be no downtime to the system. How do we achieve this change using Rulette? Let us focus first on solving the *cutover on May 6* problem. ### Achieving cutover on May 6 We will consider two straightforward ways of doing this. #### Create a new rule system We can create a fresh rule system by storing the new set of rules in a new table \`tax\_rule\_system\_v2\` and mapping it in the rule\_system and rule\_input tables in exactly the same way as we had in the previous example. For this rule system, the rule input table will get one additional entry for the destination\_state input. We now need to make a code change saying that for all evaluation dates prior to May 6, 2020, we should use one rule system, and for dates after May 6 2020 we should use another. This is a simple enough solution, but it leaves room for improvement on two fronts. - If tax rules change often, creating a new table every time is a lot of overhead. Not only do we have to set up the changed rules, we also have to copy over all the unchanged rules for completion. Then we end up loading all these rule systems into our application. - Another layer of decision-making is introduced for deciding which rule system to pick. What's worse, this layer lives in code and changes are, therefore, intrusive. Let's look at a more elegant solution. --- --- #### Enhance the existing rule system What if we incorporate the date based decision making step inside the current rule system? Dates and ranges are both supported out-of-the-box in Rulette, and we can therefore augment the current rule system with the time dimension. This will remove the need to maintain an extra step of decision making and our rules would themselves become time aware. Let's add two more columns to the 'tax\_rule\_system' table called 'effective\_from' and 'effective\_to'. Since all the current rules have been effective from the beginning and are valid till May 6, we can set all the rows to have NULL (NULL represent 'Any' or min value of range in Rulette) as 'effective\_from' and '2020-05-06 00:00:00' (YYYY-MM-DD HH:mm:SS is the default date format in MySQL) as 'effective\_to'. ![](https://kislayverma.com/content/images/2020/07/rules-with-effective-date.png) Now we need to make sure that everyone using the tax rule system is passing in a parameter called 'effective\_date' which is the date as of which the tax data will be returned. Since this will need code change from our users, we will have to wait till everyone migrates. We can also assume a default value of "now" to make this a non-breaking change for our users. Finally, we map a new rule input called 'effective\_date' in the rule\_input table to indicate that this rule system should have a new input of data type DATE and input type RANGE ``` INSERT INTO rule_input (`name`, `rule_system_id`, `priority`, `rule_type`, `data_type`, `range_lower_bound_field_name`, `range_upper_bound_field_name`) VALUES ('effective_date', 1, 5, 'RANGE', 'DATE', 'effective_from', 'effective_to'); ``` Reloading the rule system after this query will make our effective date related changes live. If anyone happens to try to find tax rate applicable on May 7, 2020, they will get nothing as we have no rules defined for that time. ### Modelling Destination State With either of the above solutions, adding this new dimension is very simple. If we created a new rule system, it already contains this dimension and looks like Fig-2. If we take the second option of adding the time dimension to the existing rule system, we want our rule system to finally look like this. ![](https://kislayverma.com/content/images/2020/07/rules-with-dest-state.png) To get there, we repeat the process for adding a new rule input we just performed for adding effective\_date. - Add a column called to the tax\_rule\_system table called 'destination\_state'. - Since this value is immaterial pre-May 6, set NULL ('Any') as the value for all rows. - Wait for users to change their code and start sending destination\_state in their tax queries - Insert new rows in tax\_rule\_system table to represent the new destination specific rules. These rules will be effective from '2020-05-06 00:00:00' to 'NULL' ('Any' aka forever). The blue row in Fig-4 is the older rule applicable before May 6, and the other rows are the. new rules available on and after May 6\. Notice the way *effective\_from* and *effective\_to* columns are set. - Insert a row in rule\_input table to represent that the destination\_state column contains values for a new VALUE input of data type STRING. ``` INSERT INTO rule_input (`name`, `rule_system_id`, `priority`, `rule_type`, `data_type`, `range_lower_bound_field_name`, `range_upper_bound_field_name`) VALUES ('destination_state', 1, 6, 'VALUE', 'STRING', NULL, NULL); ``` - Reload the rule system. My personal preference is to enhance the current system instead of creating a new one because what we are modelling is essentially an evolution of the existing tax regime. If a completely new law arose about a new notion of tax ,e.g. environment tax, then that should be built and evolved independently. ### Code Changes Regardless of the approaches taken from the above two options, one thing is sure - our system is now time aware, and hence effective date is now a mandatory input. If we want, we can now expose a version 2 of our *getTax* API. This is, of course, outside of the Rulette modelling process. ``` public class RuletteBasedTaxManager { private final RuleSystem taxRuleSystem; // Constructor public RuletteBasedTaxManager(String dpPropertiesFilePath, String ruleSystemName) { File f = new File(dpPropertiesFilePath); IDataProvider dataProvider = new MysqlDataProvider(f.getPath()); taxRuleSystem = new RuleSystem(ruleSystemName, dataProvider); } // API version 2 public Optional getTaxV2(Map inputMap) { if (!inputMap.contains('effective_date')) { throw IllegalArgumentException("Effective date is mandatory"); } return getTax(inputMap); } // Older API @Deprecated public Optional getTax(Map inputMap) { Rule applicableRule = rs.getRule(inputMap); if (applicableRule != null) { return Optional.ofNullable(applicableRule.getColumnData(rs.getOutputColumnName())); } else { return Optional.empty(); } } } ``` --- I hope this case study has demonstrated how Rulette is powerful not only for starting a fresh rules modelling effort but also in evolving an existing use case elegantly. We just reduced a major change in business environment to largely a matter of changing configurations. And that is the intent behind Rulette - our time as developers should be spent in problem solving, not in writing/maintaining tedious bits of code. Reducing rules to externally modifiable configuration allows us to [deliver business value ever faster](https://www.kislayverma.com/software-architecture/sidestep-architectural-ilities-and-deliver-business-value/?ref=kislayverma.com)[.](https://www.kislayverma.com/post/sidestep-architectural-ilities-and-deliver-business-value?ref=kislayverma.com) Rulette is open source on [Github](https://github.com/kislayverma/Rulette?ref=kislayverma.com) , so pop over, show some love and spread the word. Code review, bug reports/fixes, and any other forms of contribution are more than welcome. In follow-up posts, I will show how we can expose our rules over a REST API so that non-JVM clients can also access Rulette's powerful feature set and we can expose taxation as a generally available capability in the organization's larger ecosystem. ### Modelling tax rules with Rulette : Part One URL: https://kislayverma.com/modelling-tax-rules-with-rulette-part-one/ Last updated: 2026-07-22T12:47:16.000Z This is a deep dive into using [Rulette](http://rulette.org/?ref=kislayverma.com) for modelling business rules. It is a simplified (slightly) version of a real world use case that I have used Rulette for in production. We will try to model the tax that must be paid when certain types of items are sold by manufacturers located in different states. ### In the beginning... It starts with the forging of the great tax laws, where the government mandates that the tax that must be paid by a manufacturer on a sale depends on the state in which his facility is located (West bengal, Punjab...), the type of the item (T-shirt, shoes, bags etc), the material (silk, leather, gold...) and the price of the product. The product managers confers with the financial and legal team and puts together all combinations of these attributes in an excel sheet. The whole thing is a layered structure built on default rules with more specific rules applied on top of them (Karnataka charges 5% on shoe sales, unless they are made of leather and priced above Rs. 5000, in which case it charges 10%). Something like this. ![](https://kislayverma.com/content/images/2020/07/rulette-sample-rules-1-2.png) The dev team has to build a system to store these rules and determine what should be the applicable taxes on a sale. There are a lot of rules, so the system should be able to handle that, but the expectation is that they will be changed infrequently but read very regularly. Of course, we would like to make the management of all these rules as easy as possible and their evaluation as fast as possible. Let's say we already have a Java base tax system and we want to enhance that to handle the new tax regime. With this, let's jump into the solution(s). ### Hard-coded If-else Any set of rules can be expressed as a combination of if-else conditions - that's what all rule management is. So the naive way would be to hardcode all of these rules and return the corresponding tax values as the output. If there is a particular dimension we don't care about in a rule (e.g. Any MRP), we will represent it with the string "Any". ``` public String getTax( String sourceSate, String itemType, String material, Double mrp) { if (sourceState == "KAR" && itemType = "bag" && material="Any" && mrp ="Any") { return "5"; } else if (sourceState == "KAR" && itemType == "bag" && material=="Leather" && mrp > 5000) { return "10"; } else if........//About 100000 more time } else { return null; } } ``` This will work, but it will turn the codebase into the perfect hell for man and beast. This is difficult to understand, change and test for correctness. If anyone asks for tax rules in our system, there is no way but to open code - which makes conversations with product managers and business teams very difficult. They have no visibility into the technical representation of the rules. Additionally, from a computing perspective, this is a brute force lookup when trying to evaluate any outcome. We are iterating over all combination of all the inputs. The structure of the problem indicates that this could be improved if we were to use some tree/graph based solution. ### Configurable if-else A huge step up from this situation would be to capture the rule inputs and the operators (equals, greater than etc) in some sort of configuration or DSL which we can parse at start-up to build the entire set of if-else combinations. This will make it easy to manage and share the rules outside of code because we can transform the product managers excel sheet into the configuration independently and then reload them into the tax system. The make the code look simpler, the heavy lifting has moved from implementing the code to implementing the transformation of excel sheet to DSL.This does not remove the computational complexity - we still evaluate all combinations to find out which rules apply, the combinations are just generated using config. ``` public String getTax( String sourceSate, String itemType, String material, Double mrp) { List allConfigs = loadCOnfigs(file);//Assuming things are stored in a file for (Config config : allConfigs) { if (config.matches(sourceState, itemType, material, mrp)) { return config.output; } } return null; } ``` --- --- ### Rulette for modelling rules Rulette encompasses the learnings for the previous two attempts and presents them in a single, compact solution. We see that the most intuitive configuration/DSL to model the rules is the same format in which they were shared - a 2D matrix with each column representing a "rule input" (source state, item type, material, price) and each row representing a "rule", i.e a combination of these inputs mapped against an output value. We also see that the relation between column of the sheet (in a single row) is of "AND" type (also seen in the code above) and that any "OR" relationships are modelled as mutltiple rows (as shown by the else-if conditions in the code above). Rulette carries over these modelling insights into the heart of the application so that the dev and business teams are talking in the same language when it comes to rule modelling. #### Setting up Rulette Let's assume that we take the excel sheet and dump it into a MySQL table called '***tax\_rule\_system***' in '***tax***' schema. You don't necessarily have to do this (Rulette has an extensible data loading model which allows you to plug in any source of rules), but MySQL is officially supported out-of-the-box so we will use that in this case case study. This table is near identical to the excel sheet, and now contains all our tax rules. ``` CREATE TABLE `tax_rule_system` ( `id` BIGINT(20) NOT NULL AUTO_INCREMENT, `source_state` varchar(100) NULL, `item_type` int NULL, `material` varchar(100) NULL, `min_mrp` DECIMAL(12,3) NULL, `max_mrp` DECIMAL(12,3) NULL, `rule_output_id` VARCHAR(256) NOT NULL, PRIMARY KEY (`id`) ); ``` Let's create two metadata tables in the 'tax' schema (or any other schema the tax application has read access to). These tables help Rulette in making sense of the tax rules we stored earlier. ``` CREATE TABLE rule_system ( `id` bigint(11) NOT NULL AUTO_INCREMENT, `name` varchar(100) NOT NULL, `table_name` varchar(100) NOT NULL, `output_column_name` varchar(256) DEFAULT NULL, `unique_id_column_name` varchar(256) DEFAULT NULL, PRIMARY KEY (`id`) ); CREATE TABLE `rule_input` ( `id` BIGINT(20) NOT NULL AUTO_INCREMENT, `name` varchar(100) NOT NULL, `rule_system_id` int(11) NOT NULL, `priority` int(11) NOT NULL, `rule_type` varchar(45) NOT NULL, `data_type` varchar(45) NOT NULL, `range_lower_bound_field_name` varchar(256) NULL, `range_upper_bound_field_name` varchar(256) NULL, PRIMARY KEY (`id`) ); ``` The first table models the rule system as an entity and allows us to locate where the rules of a rule system are stored. In this example, we can perform the following mapping to map a rule system name "tax\_rule\_system" to the data stored in our 'tax\_rule\_system' table. It also identifies that that unique identifier for rules in that table is called "id" (the primary key) and that the output tax values live in the column named "rule\_output\_id". ``` INSERT INTO rule_system (`name`, `table_name`, `output_column_name`, `unique_id_column_name`) VALUES ('tax_rule_system', 'tax.tax_rule_system', 'rule_output_id', 'id'); ``` The second table interprets the columns of our rule system. Each row in this table defines the name of the input (source state, material, item type, mrp), its type (VALUE or RANGE), its data types (String/Number/Date), its priority (the order in which inputs are matched during evaluation). You can learn more about types and data types [here](https://www.kislayverma.com/rulette/rulette-design?ref=kislayverma.com) and about how rule evaluation happens in Rulette [here](https://www.kislayverma.com/rulette/rulette-rule-evaluations?ref=kislayverma.com). For now, just focus on the priorities being set for each input and the since the MRP value is a RANGE input, it is physically stored as two columns ('min\_mrp', 'max\_mrp') in the 'tax\_rule\_system' table which are mapped as the \`range\_lower\_bound\_field\_name\` and \`range\_upper\_bound\_field\_name\` columns in the last INSERT statement. ``` INSERT INTO rule_input (`name`, `rule_system_id`, `priority`, `rule_type`, `data_type`, `range_lower_bound_field_name`, `range_upper_bound_field_name`) VALUES ('source_state', 1, 1, 'VALUE', 'STRING', NULL, NULL); INSERT INTO rule_input (`name`, `rule_system_id`, `priority`, `rule_type`, `data_type`, `range_lower_bound_field_name`, `range_upper_bound_field_name`) VALUES ('item_type', 1, 2, 'VALUE', 'NUMBER', NULL, NULL); INSERT INTO rule_input (`name`, `rule_system_id`, `priority`, `rule_type`, `data_type`, `range_lower_bound_field_name`, `range_upper_bound_field_name`) VALUES ('material', 1, 3, 'VALUE', 'STRING', NULL, NULL); INSERT INTO rule_input (`name`, `rule_system_id`, `priority`, `rule_type`, `data_type`, `range_lower_bound_field_name`, `range_upper_bound_field_name`) VALUES ('mrp_threshold', 1, 4, 'RANGE', 'NUMBER', 'min_mrp', 'max_mrp'); ``` All of this setup is included in the [sample SQL script](https://github.com/kislayverma/Rulette/blob/master/rulette-examples/src/main/resources/sql/sample-rulesystem-setup.sql?ref=kislayverma.com) in the Rulette examples module. It also contains a bunch of sample rules which you can insert into the tax\_rule\_system table to play around with this case study. That's it! Rulette is now set up for use. Let's look at the code side now. #### Using Rulette in code The first step is to add Rulette library to our tax system. In Maven, we can do this: ``` com.github.kislayverma.rulette rulette-engine 1.3.2 compile com.github.kislayverma.rulette rulette-mysql-provider 1.3.2 compile ``` The first is the dependency is for the Rulette evaluation engine, and the second is for MySQL data loading plugin which we will use to connect with the database setup we just did. Now we need to tell Rulette how to connect to the database. Rulette into uses [Hikari connection pool](https://github.com/brettwooldridge/HikariCP?ref=kislayverma.com) which tis highly configurable. For this, create a properties file where it can be found and loaded by your application. A sample file looks like [this](https://github.com/kislayverma/Rulette/blob/master/rulette-examples/src/main/resources/sql/sample-rulette-datasource.properties?ref=kislayverma.com) . Now we can initialize the tax rule system as follows: ``` File f = new File("db properties file path"); IDataProvider dataProvider = new MysqlDataProvider(f.getPath()); RuleSystem rs = new RuleSystem('tax_rule_system', dataProvider); ``` This will first load the rule system definitions from our meta-data tables, and then load all the rules for the "*tax\_rule\_system*" in the memory in a trie format optimized for evaluation. Make sure that you instantiate each rule only once because each new *RuleSystem* object contains all the rules from the database in memory. If you mistakenly start instantiating a new object for every evaluation request, you will quickly run out memory! We are now ready to start evaluating rules. Let's say we need to know what tax tax rate is applicable generally in the state of Karnataka. We can do the following. ``` Map inputMap = new HashMap<>(); inputMap.put("source_state", "KAR"); Rule applicableRule = rs.getRule(inputMap); if (applicableRule != null) { System.out.println(applicableRule); } else { System.out.println("No rule found"); } ``` What about finding out how items made of leather are taxed in the state of Karnataka? ``` Map inputMap = new HashMap<>(); inputMap.put("source_state", "KAR"); inputMap.put("material", "leather"); Rule applicableRule = rs.getRule(inputMap); if (applicableRule != null) { System.out.println(applicableRule); } else { System.out.println("No rule found"); } ``` Note that the values of the map should exactly match the values of the database, so in an actual application you might be doing some data normalization before invoking the getRule method. Putting it all together, here is what the new code in the tax system might look like: ``` public class RuletteBasedTaxManager { private final RuleSystem taxRuleSystem; public RuletteBasedTaxManager(String dpPropertiesFilePath, String ruleSystemName) { File f = new File(dpPropertiesFilePath); IDataProvider dataProvider = new MysqlDataProvider(f.getPath()); taxRuleSystem = new RuleSystem(ruleSystemName, dataProvider); } public Optional getTax(Map inputMap) { Rule applicableRule = rs.getRule(inputMap); if (applicableRule != null) { return Optional.ofNullable(applicableRule.getColumnData(rs.getOutputColumnName())); } else { return Optional.empty(); } } } ``` Notice the syntax of how the output of the applicable rule is being accessed. As a library, Rulette does not understand what the output value "means in any use-case - it simply returns the best matching rule. This business-agnosticism can be used to build another level of indirection where we do not store the actual tax value inside the tax\_rule\_system table but rather a reference to an external data source somewhere else. This can be useful in case we have already some data in our system and we need a rule system to map different use-cases to them. The Rulette part will continue to work in exactly the same way while the using application can now use the returned output as a reference in other systems. This compact piece of code shown above all the complexity of modelling the business rules and evaluating input data against it in a blazingly fast manner. Using Rulette also means that you can talk to the business teams in their own language (anyone can understand the tax\_rule\_system table) and that you can add more rules without having to change you code. More of the ways in which Rulette can be used to manipulate MySQL based rules are outlined in the [examples module](https://github.com/kislayverma/Rulette/blob/master/rulette-examples/src/main/java/com/github/kislayverma/rulette/example/mysql/SimpleMysqlUse.java?ref=kislayverma.com) . This case study demonstrates how to model business rules using Rulette. But we all know that rules never stop evolving. In the [next post](https://kislayverma.com/modelling-tax-rules-with-rulette-part-two/) we will see how we can easily evolve this rule system to incorporate changes to the tax regime. ### Platform Nuts & Bolts : Enforcing Constraints in Platform Architectures URL: https://kislayverma.com/platform-nuts-bolts-enforcing-constraints-in-platform-architectures/ Last updated: 2020-07-21T19:18:46.000Z I have [discussed earlier on this blog](https://www.kislayverma.com/platform-thinking/control-and-chaos-in-platform-systems/?ref=kislayverma.com) about how there is a profound shift in the way constraints are enforced on business and technology usage as an organization adopts platform architecture. In platform architecture, process driven enforcement systems do not work well - they just get in the way because they put the platform owner in the path of any change that needs to be made. To achieve the platform's true potential, we want its **users to be able to navigate it independently** while the **platform owner is able to maintain the operational and functional safeguards** needed for a rock solid experience. And we want to be able to do this at scale across any number of users and any volume of usage. And we want to be careful about what we allow and what we don't, because too much of the former leads to instability and the too much of the latter to rigidity and inaccessibility. The only way to achieve all these goals is to **bake the rules into the system's usage pattern** and make it impossible for them to be broken. The constraints and best practices of the system should not be imposed on top of the platform, they should be a part of the platform and any usage should be defined in their terms. There are two types of constraints that a platform typically enforces - technical constraints and data constraints. We will review both here. --- ### Enforcing Technical constraints Technical constraints are technical rules (duh!) around the usage of the platforms that allow the platform owner to effectively administer and operate the system. These are typically not very different from the considerations we have while operating technical products, except that we have an additional "per tenant" dimension in all of them (**products MAY have multi-tenancy, but platform MUST have it**). The most common technical constraints are: - **Identity management** : Identity management in platform systems goes deep. The platform wants to identify its tenants. Tenants want to identify their sub-systems or human users, they might also want to identify specific resources (API, database, queues etc). Typically, identity management ends up being the first dog-fooded mini-platform with the overall platform. - **Authentication/Authorization** : Once we can assign identities, we want to enforce that the. users using the system are who they claim to be and they are doing only what they are allowed to do. - **Rate Limiting** : This governs the amount of usage of the platform by any tenant. This ensures that the entire platform is not tied up by the surge in traffic from one tenant. We typically use identity management to identify who is using the platform. - **Billing and Charge back** : All of the above along with terms of per usage cost result in the billing of a tenant. The same billing mini-platform may be used by the platform to bill its tenants and by tenants to monitor the [charge back](https://www.embotics.com/blog/5-factors-to-chargeback-showback-success?ref=kislayverma.com) between their sub-systems. Platforms might enforce rules around unexpected billing spikes. Enforcing these kinds of constraints is a well understood part of administering a platform since they are all widely use in product domain as well. While I am calling them constraints, we may also consider them technical capabilities in their own right because having them makes the platform more usable - it is not just the owner of the platform who wants a stable system but also the its users. --- --- ### Enforcing business/data constraints As I have written [earlier](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com), a lot of the daily discussion in our technical teams (rightly) centres around solving business problems. There is almost always a business context in which we are building a technical solution, and this backdrop has a significant impact on the way we think about how the system should behave and what is core to the system. Imagine designing for an employee management application. What are the first things that come to mind? - Employee age < 200 years - Salary never negative - Joining date <= resignation date - Unique employee id - Salary payout cycle This and many more such rules would go through our heads immediately. This is the classic product technology mindset. It tries to understand how the business is to be run, and implement those rules in code. Most of our focus is on ensuring that these rules are never broken. Now imagine that you are building a SaaS employee management platform. We do not know what our clients might want because we do not know the specifics of their business. We are solid on some things (Employees will likely be < 200 years in age and joining date <= resignation date), but not so much on other details. e.g. - What kind of unique employee id - numeric, prefixed by department name? - What is the basic employee's lifecycle : Offered->Joined->Resigned or Offered->Probation->Confirmed->Resigned? How can we enforce the rules when we do not know many of them? We cannot, and perhaps not surprisingly in the platform world, embracing this chaos is essential to build a viable system. Let's identify two terms : **Domain Context** and **Organization Context**. **Domain context is the overall, high level knowledge of the various rules of any business domain** (e-commerce, payments, CRM, Retail Banking). It identifies the invariants of that systems regardless of where or how it is being used (salary cannot be given to a person who does not work with an organization anymore). **Organization context identifies how a specific business operates**. It comprises of the entire domain context as well a myriad of additional rules about how one specific organization is set up and conducts its business. **Platforms are built with domain context in mind but with little to no reliance on specific organization context.** Which means that the only rules they should enforce are the rules that the entire industry complies with. As you might imagine, there are very few such rules. Much of the complexity of writing software comes from the business domain, and platforms cannot police that without becoming coupled with their tenants (which is, of course, the death of the platform). Some specific example of the kind of data constraints that should and should not be enforced follow. #### Entity Modelling Platform entities should be built only of domain concepts the values of which can be bound by the platform. Ideally, this limited set of attributes will still allow useful applications to be built on top. However, it is likely that tenants will want to store extra data specific to their use-cases against these entities. There are several ways of doing this, but in none of those ways does the platform ever allow itself to become aware of the details of these data model extensions, nor does it ever try to enforce any sort of validity rules against . The tenant is solely responsible for maintaining these fields. #### API/Messaging payloads The platform exposes standard APIs that use standard entity schemas as payloads. Tenants may, however, want to pass in extra information so that it can be relayed further downstream. A platform component should either let these pass through or if it exposes some other standard way of passing extra data, throw an error to force the tenant to use the standard mechanism. In either case, it is not the platform's business to interpret what the payload is outside of the fields defined by it. --- Platforms are all about external programmability. They always expose mechanisms that allow tenants to [plug-in new workflows](https://www.kislayverma.com/platform-thinking/platform-nuts-bolts-flexible-decision-making-with-rule-engines/?ref=kislayverma.com) (a rule defining how some thing is to be done) or [extend core entities](https://www.kislayverma.com/platform-thinking/platform-nuts-bolts-extendable-data-models/?ref=kislayverma.com) (a rule defining what a thing is) with custom metadata. The platform can maintain and enforce these rules. However, the responsibility of defining the rules still rests with the tenant. The key consideration for platforms in this regard is to **not get "conceptually coupled" with the use cases of their tenants**. e.g., the platform can expose a schema registry where a tenant can come and defines her schemas for custom fields. Now the platform can enforce the schema, but still in a dumb way. It does not understand "what" the field is, just that its value should fit some criteria. **Giving up attempts to control the tenant's business workflows and data is one of the most fundamental technical underpinnings of a platform implementation**. To the product thinker, this is also the most difficult to wrap her head around. And this is one of the areas where we have to be very very careful when we are exposing hooks for extensions. Imagine that you are a database administrator. In a sense you are the owner of the company's database platform. You can enforce that no queries run for more than some stipulated time, no tables grow beyond a given size, no single IP or set of IPs fires too many queries etc. However, you cannot enforce that teams that use your DB infra as a platform are putting the right values in the column of their tables. Those values are not an invariant of your "database domain" (in the sense that they are valid values in some tables, but maybe not "these" tables). The tenant teams are responsible for ensuring the sanity of their data. This is an example of the platform owner consciously relinquishing control over the tenant's business. --- ### TL;DR - Platforms perform policy enforcement via systems and controlled usage patterns. - Platforms should enforce technical considerations very rigidly since they tie in directly with the ability to operate a reliable system with transparency. - Platform should enforce business rules loosely because they don't understand or want to understand the specifics of the tenant's business. - They can provide mechanisms to tenant using which tenants can set up rules they want to enforce of their data. The platform can then enforce these rules without taking any responsibility for their veracity. An interesting fallout of the need to bake the constraints into the system is that it forces our hand on the technical choices in the system. A platform has to be built all at once, in one go, with much of the core functional including the checks and balances. It cannot be delivered piecemeal, because the users do not want an order system that does not do billing, and platform owner does not want to operate an order system which does not have permissions. All aspects of a platform including basic functional completeness and control mechanisms have to be delivered together. This is not to say that we haver to think of all features and capabilities up front, but that even the first version of the platform has to be useful as well stable. We will discuss this more in future posts. --- I hope this has helped you get a deeper understanding of the kind of control structure platform system design entails. I'd love to hear your experiences and thoughts about these types of systems - drop a comment below! If you are interested in further musings on the whats, whys, and hows of building platform architectures, sign up to my mailing list and stay tuned! ### Platform Nuts & Bolts : Extendable Data Models URL: https://kislayverma.com/platform-nuts-bolts-extendable-data-models/ Last updated: 2020-07-21T19:10:52.000Z The core of building Platforms rests in versatile entity management. Entities represent the nouns or the "truths" of our world. What those entities do or how they behave can vary a lot from situation to situation, but the key thing to identify is that very often, the same entity can feature in all of these scenarios. When we look at our systems from a business and real-world perspective, it is clear that we are working with a set of things that behave in a certain way. The business teams are focussing on getting things doing, and if two things behave in a similar way, they often think them to be the same (or at least same enough to be getting along with). There is a kind of street-wisdom in clubbing together things that behave in a similar way - even though they may not technically the same. Early design and architecture conversations often centre around identifying "nouns" (entities) and "verbs" (actions) in the PRD and trying to determine if we are actually dealing with the same entity or with two things that act in a similar way. This is important because it exposes the full underlying complexity of the problem and identifies actors thats might otherwise slip under the radar. However, there is the other side of this situation too - Developers like to divide things too much. It is ingrained in our training. Combining data and behaviour is one of the core tenets of object-oriented programming. If we look at something in a different context and the behaviour is wildly different, we reason the thing itself must be fundamentally different. The differing behaviours blind us from the underlying unity of the entity - developers often believe they are building a "cleaner" system when they build completely separate code paths to handle different behavioural paths. I have written about such a [scenario](https://www.kislayverma.com/post/products-are-not-platforms?ref=kislayverma.com) that my team ran into while designing logistics systems. We saw something that behaved in very different ways in different scenarios across multiple services, and we thought these were all fundamentally different things. Both approaches suffer from extremes - one will put any two things together as long as they work the same. This is good for defining interfaces, but not for entity modelling. The other will create multiple things from one thing because it looks to be doing different things. --- A platform approach bridges this gap by identifying and implementing centralized "nouns" to be what they are, and then coming up with ways to make them behave in different ways for different scenarios. One of the concerns in creating such standardized entities is how to store use-case specific information about them. e.g. a place can be a restaurant or an office or a tourist place or all of these at the same time. We can model a place as a "Place Entity" but how can we store this additional metadata about it? Alternatively consider an e-commerce order which needs to store different types of metadata depending on which country it was placed in (Credit card info in some countries, customer's IP address in some countries etc). In other words, ***how can we design entities with extendable data models***? --- ### A Bad Idea The most straightforward way to implement this would be if we could keep adding data fields to the platform entities data model as we came across different needs. It is is also a horrible, horrible idea. - **Not really a platform** : The biggest problem is that this approach relies on making changes to the platform to address each use-case. Remember that having a platformized system means that users should be able to build things without needing the platform to make changes. This pattern definitely break this platform paradigm. - **One man's data is another man's cruft** : Even if we were to add data fields to the entity schema to address a specific use-case, to all other tenants using this entity (who did not ask for this field to be added) this field would appear as a meaningless construct which still appears in their data domain and has to be dealt with in some way. (whether or not it can be built generically and imbibed as a first class attribute later is not the point here). This is, in a way, a violation of the [Interface Segregation Principle](https://en.wikipedia.org/wiki/Interface%5Fsegregation%5Fprinciple?ref=kislayverma.com) in the data model realm. - **Schema migration** : If some users of the platform schema (or some components in the larger platform itself) are applying strict checks to the schemas they are working with, they will all have to upgrade to the new schema to prevent outages. - **Tracking the origin** : How does the platform team keep track that this field was added for ad-hoc reasons and its specific meaning? Additionally, we can be assured that if an empty field is seen in the schema, one or more tenants will surely start abusing it to store completely unrelated things over time. Cleanup and refactoring of platform code becomes nearly impossible over time. - **More data transfer** : We are now moving more data in the form of an empty field. This may be mitigated by not moving null objects etc, but in general this is a problem. By now it should be clear that we need a cleaner strategy than this. --- --- ### Extensions local to the use-case A more reasonable approach to extending data models can be to not allow it in the platform at all and having each use case store its own additional data locally, referenced against the platform entity that it extends. e.g. A digital order management system can extend the basic order entity schema by storing all details specific to the digital business in its own database alongside a reference to the core order entity. When any user of the digital order service reads an order. data will be read both from the core order service as well as from the its local database, merged together, and served to the user. ![](https://kislayverma.com/content/images/2020/07/extendable-data-models-local-ext.jpg) The Digital OMS now has an option in terms of data modelling - It can wrap the base order inside the digital order schema or vice versa. The former will create a new entity which can be used anywhere in the digital order service's domain but nowhere in the platform since none of the platform services understand a digital order. This can sometimes create problems if we want to pass data around via platform services. The latter approach enriches the existing order entity with additional data (exactly like Java's 'extends' keyword does) and all of digital order service's users can continue working with a platform entity (though augmented). We can use this entity to call platform services, but the platform implementation has to take care to not be too strict about schema validations etc. The good part about this approach is that it is extremely intuitive (a lot of our cross-entity data modelling is done like this), has no dependency on the platform (the use-case owns the service and the local extension locally). The disadvantages are that if you were using capabilities like data change audit trails, privacy protection etc given out of the box by the platform, we now ave to implement them on our data too (as required). There will also be performance overhead in always reading from a remote order service for every read and implementation complexity in handling the failure cases and availability. ### Extensions inside the platform Now let us look at ways of extending the platform entity directly without having to change the entity repeatedly (to take the platform owner out of the development cycle) or by encumbering all platforms tenant with the specifics of one tenant. The bad idea that we discussed earlier can be actually be modified into a viable data model extension strategy. A direct way of doing this would be to add the capability for anyone to store key-value pairs in the entity. The platform will treat this "meta-data" like some random JSON (or binary etc) and never try to read or understand it. How to do that is left to the tenant who wrote that data. This remove the overhead of adding and maintaining multiple tenant specific field in the core entity's schema. ``` { "order number": 1, "customer name": "kislay verma", "customer address": "some address", "items": [{ "item id": "item number 1", "price": "100", "currency": "INR" }], "meta-data" : { "digital-order-info": { "digital details 1": "some digital details", "digital details 2": "some more digital details" } } } ``` This allows us to store arbitrary data points inside the core platform itself with a formal understanding that no platform components will ever read the extended data model. This is critical for preventing the platform from getting coupled to upstream tenant systems. #### Implementation Problems From an implementation perspective, however, there are still problems. The same entity in a tenant's ecosystem can be extended by multiple sub-systems. e.g. An order entity may be extended by both the warehousing system as well as the logistics system, and they both might want to track "dispatch date". How do we do this? Additionally, the interface segregation principle also applies here to keep the different tenant sub-systems decoupled - they should not see extensions that they did not make. We can achieve this by making our metadata field a nested key-value structure (map of map) with each tenant subsystem owning one key in the top level map. The platform does not understand the contents of the map but individual systems can read their own key and then work with only their own data and extended schemas. We can go one step further by making each key a named resource in our access management system and giving each tenant subsystem access to only its resource. ``` { "order number": 1, "customer name": "kislay verma", "customer address": "some address", "items": [{ "item id": "item number 1", "price": "100", "currency": "INR" }], "meta-data" : { "warehouse" : {..some warehouse specific data...}, "logistics" : {..some logistics specific data...} } } } ``` This keeps the tenant subsystem domains well decoupled, but what if the tenant's warehousing subsystem needs to share some extended data with the tenant's logistics subsystem? This is easily done by sharing the understanding of warehousing extended data model with the logistics subsystem and letting the latter read the former's key. This is a decision taken completely in the tenant's domain and the platform neither controls nor prohibits this. It is worth noting how little control the platform is exerting in maintaining the sanctity of data. Platforms trust their users to manage their business logic and enforce only the rules that tenant's ask them to enforce. This level of leeway would be blasphemous in a product with well defined structure. We would likely not even be talking about extendable data models. The advantage of this is an extremely simplified development experience for the tenants - no local database has to be maintained, and all entity data continues to live in a single place. All the platform capabilities that may have been lost in extending the data model locally are available in this model. There are performance consideration of the platforms side though. Some limit must be set on how much data can be stored in the extended data model, perhaps even at tenant sub-system level. Extra data transfer is now inevitable. Successfully executing this model also requires discipline by platform developers to ensure that the extended data model is NEVER, EVER read. --- If you like reading about [platform thinking](https://www.kislayverma.com/blog/categories/platform-thinking?ref=kislayverma.com) and [details of building technical platforms](https://www.kislayverma.com/blog/categories/platforms-nuts-bolts?ref=kislayverma.com) , sign up for my mailing list and stay tuned. I publish technical articles once or twice a week. ### Learning React in 24 hours URL: https://kislayverma.com/learning-react-in-24-hours/ Last updated: 2020-07-21T19:02:17.000Z After getting inspired by my "[Learning Golang in 24 hours](https://www.kislayverma.com/programming/learning-golang-in-24-hours/?ref=kislayverma.com)" exercise, I decided to broaden my developer toolkit further by learning the [React](https://reactjs.org/?ref=kislayverma.com) UI framework. Of course I had to keep the same theme, so this was going to be a "Learn React in 24 hours" binge and would result in a simple CRUD UI on top of the CRUD service that I had [built using Go](https://github.com/kislayverma/go-crud?ref=kislayverma.com) . --- There were a few things different, of course. I have been a backend engineer for almost all of my career. The last time I did any meaningful frontend work was about five years ago when I coded the user interface of my now defunct book and travel review website using [Backbone js](https://backbonejs.org/?ref=kislayverma.com) . Even that had been an attempt to learn new technologies by building something - I learnt [Play! framework](https://www.playframework.com/?ref=kislayverma.com) , [ElasticSearch](https://www.elastic.co/?ref=kislayverma.com) , basics of [Akka](https://akka.io/?ref=kislayverma.com) actors, and Backbone.js to build it. Since then, however, I have gone deeper into the backend stack and not touched UI stuff at all. Hence the learning experience comprised not just of learning React, but also of learning the frontend domain. Moreover, in the last 6-7 years, the frontend world has moved away from tools like Backbone, [Angular](https://angularjs.org/?ref=kislayverma.com) , [JQuery](https://jquery.com/?ref=kislayverma.com) and [YUI](https://yuilibrary.com/?ref=kislayverma.com) . These are all powerful tools, but they derive their power from being very good at facilitating an older paradigm of user interface development. The next generation of tools like React and [Vue](https://vuejs.org/?ref=kislayverma.com) increasingly look to blur the line between user interface development for web, mobile, wearables etc. Then there are PWAs, native technologies, and so on, all of which represent what can easily be called an architectural revolution in frontend technology. I had to cover a lot of ground to even see the starting line. To add further complexity, I don't know Javascript either. So the learning experience had yet another dimension - I struggled with the basics of Javascript, while also learning React at the same time. I thought of sitting down to learn the language first, but realized that it was not going to work - I had to build something, and why build something in "pure" javascript and then figure out React on top of that. So for better or for worse, I would tackle all of these thing at once. I had been warned repeatedly of the elaborate tool-chain in the React world, so I was very cautious to stick to the absolute basics. I used absolutely no frameworks other than what the [official react beginner's tutorial](https://reactjs.org/tutorial/tutorial.html?ref=kislayverma.com) used. I started with create-react-app and did not add any further modules beyond that. The IDE of choice was Atom. All the code is available on [Github](https://github.com/kislayverma/react-crud?ref=kislayverma.com) . Check it out - I'm always happy to get my code reviewed! Here are my take-aways from the experience. --- --- ### TL;DR All in all, I would say that while I managed to get a semblance of handle on basic React, I couldn't get as accustomed to Javascript as I would have liked. ### The Good #### Use what you learn to not regress The experience of building a basic screen felt very similar to what it had felt like building the website. I had forgotten all Javascript syntax that I picked up then, and was essentially starting afresh this time too. #### React is friendly to Backend-ers Learning React was actually a very nice experience. Foe the first time, moving around in Javascript code felt like familiar ground to me - the code is far better structured than I remembered from my JQuery/Backbone days. Object-oriented instincts find good footing here - there are classes and the state represents instance variables nicely. The syntax was a little more free-form, but close enough that I could grok the code with reasonable ease. In fact, defining classes and variables to directly back UI elements reminded me of working in JSF with backing data fields and re-rendering UI on changes to them. The things I learnt : Javascript lists and maps, if switch and for loop, basics of React components, event handler, state management and updating UI dynamically based on user interaction or other events, calling REST API and rendering the result, code organization and local builds. #### React is a vast ecosystem The learning mentioned above represents a miniscule part of the overall power of both Javascript and React. Although I did not cover much of the React toolkit, I feel that I can explore it more easily than I had with the older generation of frameworks. The huge amount still left to explore means that while learning by building is effective, one will have to build a lot to be get exposed to even a respectable part of the dictionary of these technologies. I have not yet delved into the build and deploy side of React - 'npm start' doesn't qualify :) ### The Bad While the functional syntax and constructs that Java has been adding make it easier (for me) to understand Javascript code with its function passing and handlers, weak typing remains a huge hurdle in understanding what is going on. As code grows larger, I find myself struggling more and more about what the different variables are. ### The Ugly The lack of auto-complete in Atom just completely, utterly sucks. I had to Google a LOT because there was no way to explore the methods being exposed by any object. ### What now? Many people have told me that I should explore Typescript if I want typed Javascript, but for the moment I am not enthused enough by the prospect. I'm going to pause here and call this a lukewarm experience and stop. My next project was going to be to build a mind-mapping software for organizing my notes and ideas, but since discovering [Roam Research](https://roamresearch.com/?ref=kislayverma.com) that need has been temporarily addressed and I have no real project in my sight for putting my newfound React skills to the test. When I finally get around to building the admin UI for [Rulette](http://rulette.org/?ref=kislayverma.com) , I will see how much of this can be put to use, or if I'll have forgotten everything (yet again). ### Control and Chaos in Platform Systems URL: https://kislayverma.com/control-and-chaos-in-platform-systems/ Last updated: 2020-07-21T18:56:41.000Z At the heart of platform architecture and strategy lies a very interesting duality - Platforms are about chaos and control at the same time. In this article, I will talk about how these seemingly competing forces are actually complementary and what this means for organizations building technology platforms. --- I say that platforms personify organizational and even ecosystem-wide chaos because of the possibilities they unleash at their edge. By providing powerful capabilities and the ability to compose these abilities into new or existing business processes, platforms can unleash a tide of innovation which would have hitherto been impossible or locked up inside an organization. Organizations can exchange value across platform boundaries by building customized, higher-order interactions among themselves and with their customers. This is as true of [business platforms aka marketplaces](https://www.kislayverma.com/platform-thinking/marketplaces-are-not-platforms/?ref=kislayverma.com) as it is of technical platforms. In a well designed platform this is done without explicitly involving the platform owner. This is the core definition of a platform anyway - that non-owners can use it create new experiences. This enablement is what I refer to as Chaos, and it is the desired outcome of [deciding to build a platform](https://www.kislayverma.com/platform-thinking/why-you-should-build-a-platform/?ref=kislayverma.com). --- However, platforms are also defined by standards. And standards are limiting by definition. A well designed platforms explicitly calls out exactly what all of its components do, how they can be extended and composed, how new things can be added, how its users are identified, permissible limits of use, cost per unit for use and so on. The "[Golden rule of platforms](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com)" is that there are no backdoor for anyone, so the defined standards are the only ways ANYONE get to use it. This is the controls aspect I had mentioned earlier, manifested in the real world via well defined standards and APIs. It turns out that Control and Chaos are not conflicting it all. If we want to fully harness the power of the platform but avoid a complete meltdown, we can only do so by enforcing some rules. The rules could be technical (identity management, rate limiting, standard API protocols etc) or they can be business (policing of the platform against illegal activities etc). Once these rules and the tools to enforce them are in place, the users can be allowed to do what they will with some assurance of stability. This is equivalent to saying that a platform has to be designed and implemented "all up-front" in terms of these meta-capabilities. Authentication, authorization, rate limiting, billing, reporting all have to be made available from day-1 if we want platform adoption and stability. Control in a platform system comes from standards and interfaces that everyone must adhere to. There is nothing else which says what can or cannot be done. --- --- The situation is very different in [product-centric organizations](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com). The way the true customers of the technology interact with it is often fashioned by a layer of internal business teams like sales, relationship management, operations etc. These teams define a panoply of processes that control and regulate customer access to the technology. In this way, they preclude the need for the product to become a platform because there is never the need to let the customers do "whatever they want". The internal teams decide what is being offered. ![](https://kislayverma.com/content/images/2020/07/platforms-internal-external-teams.jpg) Internal teams control behaviour in product scenarios In case of products internal to an organization, we can still see this internal team(s) phenomenon buffering the technology from customers. We might have product managers and business leaders defining every aspect of technology development and use by defining specific goals and funding specific programs (roadmaps, OKRs etc are all processes). While this is not a bad thing in itself, it also means that technology only develops to meet the current goals, and is not well equipped to handle fast changing business environment or even the internal shifts in focus that often come with changes in leadership. Development teams are not free of this process mentality either. Have you ever heard a developer say "Oh no one else will call that API I wrote. We will just manually hit whenever we receive a mail from XXXXX". I have. This is process filling in for system. This difference in how control is exercised has a profound implication on the responsibilities of people in companies that choose to pursue a platform technology strategy. As a product company becomes a platform company, internal teams become more peers and advisors to the customers rather than being the gatekeepers to the platform. The platform needs no gatekeepers beyond the publicly defined and systemically enforced controls already in place, and so the customers can start playing around with it on their own. They can do whatever they want without involving the internal business teams at all. The internal teams now have to compete with the other customers of the platform, or become advisors to them in the best use of the platform (depending on the context and role). For example, Amazon internally uses AWS to build Prime Video, Netflix also use AWS to build a rival, and may the best company win! ![](https://kislayverma.com/content/images/2020/07/platforms-peer-teams-1.jpg) Standard regulate everyone in Platform systems This means that any person or team that has relied on defining and enforcing processes in the use of technology now finds itself increasingly irrelevant. This is one of the biggest hurdles an organization has to overcome if wants to build true platforms. The nature of work can change so dramatically for so many people and teams (to the point of obliterating their roles) that there is intense resistance to adopting this new way of doing things. People want to continue exerting control using the time-tested (but suddenly ineffective) process driven methods. --- The key objective we are chasing in this discussion is this - *How can the platform owner continue to operate the platform reliably without having to review every customer use-case?* The control structures emerging from platform systems are very different from what we would normally consider structured processes. This leads us to the realization that building a platform is not just a technology activity. It is a mindset (much like being agile) that has to permeate all aspects of the organization and be absorbed across all functions, often painfully. [Let's unleash Chaos](https://www.kislayverma.com/technology/how-to-build-a-technology-platform/?ref=kislayverma.com)! ### Book Review : Wool by Hugh Howey URL: https://kislayverma.com/book-review-wool-by-hugh-howey/ Last updated: 2020-07-21T18:50:52.000Z ## Rating : 5/5 [![](https://kislayverma.com/content/images/2020/07/wool-cover-hugh-howey.jpg)](https://www.amazon.in/gp/product/0099580489?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=0099580489&ref=kislayverma.com) This review contains some spoilers. Wool is a series of 5 science fiction novellas by American writer Hugh Howey. The first novella was published as a short story in July, 2011\. This and the following 4 editions are usually considered the Wool series (as different from a prequel trilogy and from "Dust", also written by Hugh). > This is the story of mankind clawing for survival, of mankind on the edge. The world outside has grown unkind, the view of it limited, talk of it forbidden. But there are always those who hope, who dream. These are the dangerous people, the residents who infect others with their optimism. Their punishment is simple. They are given the very thing they profess to want: They are allowed outside. > > [From the author's website](http://www.hughhowey.com/?ref=kislayverma.com) Having read Wool, I have to believe that legends take time to build. That the only reason Hugh Howey is not already counted among the elite of sci-fi writing is that its been only a few years since this book (or series, if you prefer. I think of this as a single book both because the individual stories are so short and because they are too tightly knit to be thought of as separate) came out. I have definitely been shouting from my rooftop to anyone who cares to listen - this is one of the best sci-fi books I have ever read. Wool is a post-apocalyptic, dystopian novel set in America that describes a world where mankind lives in a silo and to think or hope of stepping outside is heresy. The premise is not unique in itself and the usual elements of similar plots (a population living in fear, a secret group guarding forbidden knowledge, lone individuals digging for the truth) are present here too. However, what makes wool special is Hugh's masterful writing and the way he has packed so much emotion in so few words. Wool is dense with pathos, with the constrained humanity of the denizens of its silo. Every single line has a purpose, either describing the nature of the world around the character or the deepest nature of the characters themselves. Of necessity, the tone is meditative, even during the more action-centric parts. The silo itself is described in some detail, thus setting the proper ambience for a sci-fi drama. In Juliette, Marner, the Mayor, Lukas, and Bernard, Hugh has crafted memorable, archetypal characters. Each represents a unique facet of life in the Silo and human nature. The first two stories literally reek of the younger generations claustrophobia (physical and emotional) and the insistence of the old guard to maintain status quo and go on as they have before. The last three, on the other hand, embody movement. You can feel the silo stirring with the struggle between knowing and not knowing. Each story is longer than the previous one to accommodate the expanding plot and number of characters. From the beginning, there is enough smattering of suspense to keep the reader hooked. The secrets are unveiled very slowly, every bit given almost grudgingly, so that reader shares the revelations of the characters and the revel in the shock of it all. By the end, the sense of shock and betrayal is visceral. A subtle aspect of the book, and something worth thinking about, is the nature of status quo. The story brings into sharp relief the way traditions become so pervasive that they lose all meaning and become an unthinking way of life. They are "just the way things are done", and to question them is to disturb the peace. While societies thrive in the familiarity (even the limited society of Wool), it is the interplay of stasis and change that underpins all conflict. In this manner, Hugh has captured an essential feature of our lives, magnified it a hundred-fold, and presented it in his microcosmic world. I had read this book some years ago and loved it just as well. But reading it in the current locked-down state of life and mind brought out the power of the story and the characters afresh. In these times, Wool is a tale of containment and liberation all at the same time - one that sheds far more light on the our world than most breaking news. I guarantee that this is one of the finest sci-fi books you will ever read in all its aspects (plot, language, characters, writing et al). Well done Mr. Howey! Read next : More [book reviews](http://category/books/?ref=kislayverma.com) ### Platform Nuts & Bolts : Flexible decision-making with Rule Engines URL: https://kislayverma.com/platform-nuts-bolts-flexible-decision-making-with-rule-engines/ Last updated: 2020-07-21T18:46:38.000Z After I received a lot of question from my readers about specific details, I understood that I had packed in a lot of information about platform principles and the mechanisms of building platforms in my [previous article](https://www.kislayverma.com/technology/how-to-build-a-technology-platform/?ref=kislayverma.com). I realized that I need to break down the process of building platforms a little bit more and give more hands-on details. In this "Platform Nuts & Bolts" series, I am going to do a set of follow up articles which look into the specifics of how exactly we can build platform systems. --- ### How do we know what to do? After I laid a lot of emphasis on state machines and workflows as a couple of critical tools on the road to building platformized software, a lot of people reached out to me asking what state machine libraries I recommended and what were the pros and cons of different workflow management tools like [JBPM](https://www.jbpm.org/?ref=kislayverma.com) , [Cadence](https://cadenceworkflow.io/?ref=kislayverma.com) , [Airflow](https://airflow.apache.org/?ref=kislayverma.com) etc. These are good questions, and I will get to them in later articles. Today I want to discuss something more fundamental, that is, what should a platform service managing an entity do when an action is taken on said entity. Let's take a concrete example. Let's imagine an order management service managing an order entity which can be cancelled. However, what exactly should happen on an order cancellation? We might want to change the state of the order and order items to "cancelled", we can issue a refund to the buyer from the seller's account, we might want to notify the buyer and the seller, we might want to free up inventory so that further orders can be taken against it. However, this is just one view of things. A properly platformized order management service may offer the above as default behaviour, but it must also [allow its tenants to customize it](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com). A tenant might want to notify the buyer but not the seller, another tenant might not want to free up inventory or issue refund immediately (for whatever reason), and so on. There is the further complication that even a tenant might different rules for different types of order. e.g. a high lifetime value customer can be refunded immediately but not others, digital inventory can be considered restocked immediately but not physical inventory etc. There are as many possibilities as there are ways to make business decisions, and an order management platform should ideally enable them all. How? --- ### The Tenant is King The most straightforward way is for the order management service to let the tenants specify each of these behaviours in the cancellation request. The order management platform exposes an API with hooks for each of these behaviours and then trusts the callers to do the right thing as per their use-case. e.g. We can build this by having each behaviour as a query parameter. Such a REST API might look like this. ``` http://www.awesome-platform.com/orders/{order id}/cancel?notifyBuyer=true¬ifySeller=false&doRefund=true&releaseInventory=false ``` We can immediately see the many things objectionable about this API. It leaks detail of its implementation details by surfacing the internal code paths as booleans, it is difficult to extend to more actions, and it is very difficult for the tenant to govern what is happening in which scenario, which is very important if a tenant is looking to build many different behaviours. The platform need not understand the upstream cases, but in this case, it offers zero visibility into the business rules. Also, as we will see below, the choices that the tenant needs to make are not limited to order cancellation. There are multiple platform services that will come into play in this process (payments, communications, inventory etc) and in this model, the tenant has to know, understand, and specify the behaviour of every single step of the workflow in every cancellation request. This is obviously not the best way to design an order cancellation API. --- --- ## Rule Engines FTW! A better way of achieving the kind of flexible behaviour we need is to not think of the multiple things that need to be done as independent, but rather club combinations of them into workflows each of which satisfies one or more specific tenant use-cases. A tenant has workflows defined for different order cancellation scenarios, and the problem now becomes one of picking the right workflow based on the scenario. The scenario is defined by characteristic of the order and order items, and which characteristics to use is (potentially) unique to every tenant. This kind of requirement is best managed using a rule engine (aka business rule management system aka rule system etc). We can define all the characteristics that a tenant wants to use in making the decision as the input of a rule engine and the output of this engine is the identifier of the workflow that is to be executed. We can now construct one such rule system per tenant and expose a vastly simpler order cancellation API which doesn't take external input but rather picks the right rule system for the tenant, evaluates the scenario to identify the right workflow, and triggers it via the appropriate means. At low level, we can use something like a factory or a resource locator to hide the details of how to trigger the workflow and to make it easy to add more workflows over time. ![](https://kislayverma.com/content/images/2020/07/order-cancellation-rules.png) Example of rules for order cancellation The above is a contrived but entirely plausible set of rules for handling order cancellation requests. As you can imagine, a different set of input criteria mapping to another set of workflows is also possible if we want to run the business a little differently. I want to be very clear that I'm not talking about the implementation details of any of these pieces. As the order management platform, we can use whichever rule system you like (I, of course, recommend [Rulette](https://www.kislayverma.com/post/rulette-a-pragmatic-rule-engine?ref=kislayverma.com) for most of the cases) and any workflow engine you like (it could be some local code, a tenant API, or anything in between). The idea is to identify the building blocks which let platform services exhibit flexibility, and rule engines are serve as very effective decision makers. In my opinion, the biggest advantages of this model is that all tenant rule systems are maintained in the platform, and each one of these serves as implicit documentation for the tenant's business rules (along with the specification of the actual workflows, of course). This can be very useful in understanding overall system behaviour and debugging, especially if our platform is large and has many tenants. Developers no longer need to look up the variant behaviours in code or rely on tribal knowledge - they are available almost like a configuration. The second good part about this model is of course that we can now have a very simple cancellation API which talk only in terms of the order entity and action being performed on it. ``` http://www.awesome-platform.com/orders/{order id}/cancel ``` This is clearly far simpler to maintain since it doesn't change if some new action is added to some tenant's cancellation process. Those details are hidden inside the workflow and in a one time rule configuration without impacting the API at all. ![](https://kislayverma.com/content/images/2020/07/rule-engine-per-tenant.jpg) Rule system per tenant with rules mapping to many workflows The last benefit of this model is that it allows us to define specific behaviour inside the components responsible for them. Let's look deeper into this. ### It's rule engines all the way down As I briefly mentioned above, while we might think that sending a notification to a buyer is a simple operation in the context of an order cancellation, it is actually full of choices in its own right. e.g. what kind of notification (email/SMA/in-app)? Which provider should be used? What template should be used etc?. Similar problems exist in the other aspect like payments (which payment gateway to use, what taxation rules apply, international versus local cards, pre-paid or not etc). If we use the first model of flexibility described above, ALL of these choices have to be made up front by the tenant (which is fine - they are his choices anyway) and they have to be sent in via the order management platform. This is where things become thorny, because it is not really the job of the order management system to accept or understand these extra parameters. At this point the cancellation API in OMS becomes totally incomprehensible. However, if we are talking about other platform services involved in the order cancellation process, we know that each of them have the rule system-workflow capabilities. Therefore, instead of having to specify the entire behaviour in order management system, we can break it across each of the platform services involved. We can build multiple rule systems across multiple services, each of which only deal with configuring the behaviour of that particular platform system and its workflows. the rules of each service will be defined in its own domain language and will map to workflows visible only to that service and its tenants. ![](https://kislayverma.com/content/images/2020/07/per-tenant-workflow-fanout.jpg) Workflows fan-out using domain-specifc rule systems in platform services We can visualize an order cancellation request first hitting the order management service and being satisfied by a workflow, every step of which hits a different platform service, triggering the rule engines and workflows of each of these services, and so on. The entire graph of these operations comprises the business workflow of the tenant order cancellation. --- I hope this article has thrown some more light on how we can use rule engines to not only make the behaviour of our platform services configurable but also to consolidate business rules so that are easy to access and understand. Cheers! ### Why I like the Vert.x Framework URL: https://kislayverma.com/why-i-like-the-vert-x-framework/ Last updated: 2026-07-22T12:47:18.000Z I have been playing around with [Vert.x framework](https://vertx.io/?ref=kislayverma.com) for over three years now. I started by trying it out for some side projects like [Throo](https://github.com/kislayverma/throo?ref=kislayverma.com) and liked it so much that I introduced it in [Myntra](https://medium.com/myntra-engineering?ref=kislayverma.com) as a viable contender for the asynchronous-framework-of-choice position next to [Play! Framework](https://www.playframework.com/?ref=kislayverma.com). Having used it to build multiple high-scale services in production and several side-projects, I have no hesitation in saying that this is my favourite framework for building any kind of Java application. Here's why. --- ### Modular, with lots of modules The usual understanding of a framework versus a library is that framework calls user code and user code calls libraries. Vert.x is not a framework in this sense of the word. It comes as a large number of independent modules with vertx-core being the only critical, shared element among them. It is essentially a set of libraries all of which share the same design principles and play well with each other. But they do not HAVE to be used together. We can pick the circuit-breaker and RabbitMQ connector, drop it into an existing Spring based Java application, and things will work perfectly fine. There is no all-or-nothing, and this makes de-risks the framework choice by making it it gradual and reversible. The extremely modular nature if the framework make it easy to gradually try-out different parts of the framework and get used to the shared design constructs among them all. Then we can decide how much of our system should be built using Vert.x and where we need other libraries and frameworks. And there are a LOT of modules. Check them out [here](https://vertx.io/docs/?ref=kislayverma.com) \- Vert.x folks have all the toys! ### Asynchronous from the ground-up Like node.js, Vert.x is built around the concept of an event loop and is asynchronous from ground up. This means that the default mental model for writing code in Vert.x is asynchronous (we can anyway write synchronous code in core Java, so why would we use a framework for that). Everything in Vert.x revolves around not blocking the event loop, which forces developers to maximize the application's scalability by fully embracing the async programming paradigm. Writing synchronous code is possible via blockingHandler etc but it feels like an afterthought. --- --- ### Really Fast and Light Because Vert.x is fundamentally asynchronous and can therefore scale pretty well out-of-the-box, there isn't a whole lot of multi-threading going on in its internals. This makes the system extremely efficient by removing much of the thread synchronization overhead which is a performance killer in so many other tools. Vert.x is very, very fast. Additionally, it has very low intrinsic runtime memory footprint. The application memory usage remains very low and stable. ### Concise Dictionary Vert.x has a small set of concepts that we need to learn to work with it. Everything is a handler, everything is asynchronous and backed by an event loop, etc. There are a limited number of constructs, and they behave the same everywhere. This comes from building an up-front, coherent design philosophy that can connect so many modules together seamlessly. This concise dictionary of concepts lends itself to very quick onboarding and a very consistent programming style. In fact, I was [recently](https://www.kislayverma.com/programming/learning-golang-in-24-hours/?ref=kislayverma.com) struck by how similar some of my Vert.x code looked to Golang code (Golang is very popular for its small but expressive language feature set). This allows developers to easily get accustomed to moving around in Vert.x based code. --- There are many other cool things about Vert.x like its polyglot nature but I did use them a lot so do not want to comment on them. --- ### What I wish was better #### An all-in option While the un-opinionated, non-compulsive nature of Vert.x was a great thing to begin with, I was looking for a more all-in, all-Vert.x option once my mind was made up about the framework. Something like Play! framework or Spring Boot which would abstract the best practices of the framework under conventions and let me get on with my actual work super-quick. I didn't find any compelling options at the time, so I am currently working on building one. [QVertx](https://github.com/kislayverma/qvertx?ref=kislayverma.com) is intended to be the quick-start service template for the Vert.x world. I have also seen some work around running Vert.x event loop inside Spring Boot itself, but it looks early stage right now. ### Better support on serialization formats Vert.x framework seems a little too tied to JSON as a serialization format, and working with XML and other formats proved to be a bit of a challenge. It can be done, of course, but I wish it was a little more straight forward. **Read Next**: An in-depth look into [asynchronous programming](https://kislayverma.com/content/files/2026/07/asynchronous-programming-2.html) ### Learning Golang in 24 hours URL: https://kislayverma.com/learning-golang-in-24-hours/ Last updated: 2026-07-22T12:47:19.000Z Malcolm Gladwell made popular the notion of spending 10000 hours to achieve mastery in any skill. However, this notion of 10000 hours is built upon a body of research that studied how much time top performers of several fields spent in "deliberate training" to achieve their level of skill. The number represents world-class expertise. However, when learning a new skill, this is not what we are targeting. We are looking at functional competence, just the basic ability to execute that skill. Playing chords well enough for 5 songs around a campfire qualifies as having learnt the guitar - it is not necessary to be able to play the [Freebird](https://www.youtube.com/watch?v=kgkYN3QjD5M&ref=kislayverma.com) solo with your teeth. Some have claimed that gaining this level of skill can taker as little as 24 hours. Learning a new programming language is no different. It takes considerable time and effort to master a programming language, and even more to effectively use it in the wild. This is my story of learning [Golang](https://golang.org/?ref=kislayverma.com) in 24 hours. --- ![](https://kislayverma.com/content/images/2020/07/gopher-go.jpg) I had previously tried to learn Golang by going through the rules (it's a language, so you gotta learn the grammar, right?) and using the excellent [Go Tour](https://tour.golang.org/welcome/1?ref=kislayverma.com). But even though Go has a very concise syntax (IIRC the entire Golang spec is \~55 pages, the Java spec can crush a grown man's skull with its weight), it still got boring after a while so I stopped. After this happened twice, I decided to see if the "learn a skill in 24 hours" thing worked and set myself a very specific goal. I would build a MySQL+ORM backed, Swagger documented, REST service to perform CRUD operations over a simple entity using Go - no more, no less. The specificity of the goal allowed me to focus on what I wanted to learn (not the whole language, but hopefully the most commonly used parts). I must have built a bazillion CRUD services in my life by now, so there was no "domain" learning to worry about - I wasn't building something new, just the same old things with a new tool. I am happy to announce now that this actually worked. I now have a functional understanding of Golang and can build some basic things with it. And it was fun too! Achieving each of those specific milestones kept the motivation strong, allowing me to become familiar with the language gradually, and engaged me enough that I ended up writing some logging and context propagation related middleware that I hadn't originally planned on doing (I can't help it! Why would someone just write a CRUD service when you can write a CRUD [service framework](https://www.kislayverma.com/programming/choosing-a-service-framework/?ref=kislayverma.com)). I had a kept up a tweet stream through this whole exercise to document the happenings as they happened. Check it out [here](https://twitter.com/kislayverma/status/1236188176965763077?ref=kislayverma.com). The complete code is on [Github](https://github.com/kislayverma/go-crud?ref=kislayverma.com) \- feel free to play around with it and critique it. Here are some of my takeaways from the process. Let me know what you think and if you had a different experience in the comments. These are not the pros and cons of Golang (or not just that), these are my notes on the whole experience of learning a new language by building something with it. --- --- ### The Good #### Always learn by Building Learning by building something rather than learning the rules was so much more effective. The playground it really cool, but learning syntax after syntax gets old real quick. The motivation of seeing something tangible get built cannot be over-stated. #### Blogo-sphere really helped There are many more "how-to" resources for specific things than there are for just learning the language. The amount of resources available to a beginner via blogs and forums is extremely empowering. A big thank you to everyone whose writing facilitated my blazingly fast copy-paste. I have given credits in the code as well, but these two series of articles really helped me. - [REST service in GO](https://medium.com/@supun.muthutantrige/lets-go-create-an-awesome-rest-api-in-go-part-ii-c3c9f19377da?ref=kislayverma.com) - [goroutines and http calls](https://medium.com/@gauravsingharoy/asynchronous-programming-with-go-546b96cd50c1?ref=kislayverma.com) After I completed the service and started getting into the language, I stumbled upon several talks by Rob Pike who is one of the creators of Golang at Google. If you haven't yet had a chance to listen to him speak, you absolutely must. My favourites were his talks on ["Concurrency is not Parallelism"](https://www.youtube.com/watch?v=cN%5FDpYBzKso&ref=kislayverma.com) and ["Simplicity is Complicated"](https://www.youtube.com/watch?v=rFejpH%5FtAHM&ref=kislayverma.com) #### Golang is Simple Building CRUD applications is probably 90% of a programmer's work , and after this I can confidently say that it is banal in any language :). However, the simplicity of Golang made the experience really, well, simple. Things that take many lines in Java can be expressed so neatly and compactly in Golang that it was a pleasure to "find by id" and "insert a record" again. The fact that there aren't too many different ways to do something in code gives the confidence that what you are writing in a certain way will work, or the compiler will barf. What I learnt : variable declaration and assignment, methods and functions, interfaces and implementations, code structuring using packages, basic deployment, array/list/map operations, loops and if conditions, goroutines and channels. While this looks very basic (except goroutines), it is actually a large subset of the language. There aren't too many things left to cover from the language perspective (obviously there is a whole lot more if you want to understand the internals of Golang). #### Golang is Powerful Golang is actually a NEW language developed to implement infrastructure related fesatures in the cloud. Instead of being an amalgamation of features from existing languages, it takes a fresh look at the most common needs of the world today and offers them out of the box. 1. Garbage Collector : GC is completely seamless in Go. There is no way for developers to tinker with it - no finalize() etc, it just works. 2. In built server packages : Golang is wise to today's API heavy world of today and give inbuilt package to start different types of servers. Firing up an HTTP server is about 5 lines of code. 3. Ground Up Concurrency: Concurrency management is baked right into the language, and the importance of this is impossible to exaggerate. Developers can build scalable concurrent systems without extra frameworks or programming paradigm shifts (as you would have to do in most languages when going from sync to async). 4. Circular Dependency detection : This was just awesome. Anyone who has dealt with keeping dependency graphs straight in large organizations will appreciate this. The compiler actually FAILS the build if it detects circular dependencies. This forces developers to be very careful about how the code is structured and what is upstream and what is downstream. This might cost some extra time during design, but definitely leads to more maintainable software in the long run. All of this comes right out of the language - I haven't even checked out too many frameworks yet! #### OSS FTW! As usual, the open source community makes everything so much simpler! I used several frameworks as recommended by the blogs I was following, and without fault the documentation on each of them was top-notch. I often went from opening the documentation to feature completion in no more than a couple of hours. - Request Routing : [Gorill mux](http://www.gorillatoolkit.org/pkg/mux?ref=kislayverma.com) - Context propagation : [Gorilla context](https://www.gorillatoolkit.org/pkg/context?ref=kislayverma.com) - ORM : [gorm](http://gorm.io/?ref=kislayverma.com)Documentation : [Swagger](https://github.com/go-swagger/go-swagger?ref=kislayverma.com) ### The Bad #### Weak debugging instincts As a beginner, debugging was hard because there was no visual intuition of what is happening when looking at the code. Nothing "looked wrong" because I didn't have a gut feel for what wrong looked like. IDE helps of course, and this diminished as I got more familiar with the layout of the code. #### Forgoing Conciseness for Explicitness I think Golang gets this right for the most part, but in some cases I felt that the syntax could be more explicit. 1. The upper/lower case convention for public/private access - It seems a little "magical" to me. I'd have liked something more explicit (e.g. putting an "\_" before private methods). 2. It is difficult to tell which struct implements which interface(s). In large code bases, this might cause discovery problems. ### The ugly The map.exists syntax is absolutely atrocious. Considering how many time this is done in any piece of code, I urge the language owners to clean this up. Read Next: My attempt at [learning React in 24 hours](https://kislayverma.com/learning-react-in-24-hours/) ### Defining messaging terms precisely URL: https://kislayverma.com/defining-messaging-terms-precisely/ Last updated: 2026-07-22T12:47:19.000Z Asynchronous messaging has become ubiquitous in software systems. "Publish an event" or "send a message" are commonly heard terms in all design discussions. However, less clear is the difference between words like *event*, *message*, *notification,* etc, which builds up into large-scale ambiguities when we start using messaging patterns to compose larger architectural patterns like workflow orchestration or event sourcing. In this article I'm going to lay out some of these frequently used terms, what they mean "exactly", and where/how they should be used. These are not new things, more famous people than me have said it repeatedly, so hopefully my repetition will contribute further to cementing of the industry jargon and bring deeper understanding and context to the discussion. ![](https://kislayverma.com/content/images/2020/07/msg-defs.jpg) ### The infra side of the world Let's first consider the infrastructure side of messaging world which physically enables messaging. #### Record A record is basically anything that we put on the message broker. From the broker's perspective, it is only a blob of bytes which must be delivered to one or more consumers with a certain delivery guarantee. Beyond this, a record carries no semantic meaning - most brokers never open a record put on them beyond some broker specific metadata (aka headers) which define how the broker should treat the message. This typically includes things like TTL, persistence properties, shard keys for distributed brokers (e.g. partition key for Kafka) etc. The idea of a dumb "record" is important because it helps us keep in mind there are perspectives where the intent of messaging means nothing. This is the infra platform's neutral perspective of the system - things in a Kafka topic are not intrinsically events or messages or commands, they are just network traffic. #### Message Broker/Message Bus/Event Bus By these or any other name, a message broker is the infrastructure that encapsulates the "queue" part of the messaging architecture. It may be an in-memory system (a wrapper around ArrayBlockingQueue in Java) or a persisted distributed commit log based system like Kafka or something in-between like RabbitMQ/ActiveMQ etc. The broker may give some guarantees around how records will be ordered. Regardless of the implementation, the message broker houses the data sent out by the publishers and makes sure it is delivered to its consumers. The broker also gives certain type of "delivery guarantees" about how a record is taken form publisher to consumer. There are three main guarantees : best-effort, at-least-once, and at-most-once. Each of these is a subject of considerable theoretical investigation and must be carefully evaluated by the users of the broker to fit their use-cases. #### Queue A queue represent a single logical "pipe" of records between publisher of asynchronous data and its consumer. Sometime a single physical pipe in the broker can represent multiple logical queues. Each broker has a different way of defining a queue. RabbitMQ maps an exchange + a routing key to physical queues and every queue created and mapped to a exchange+routing key combo gets a physically different copy of every record. Kafka, on the other hand, keeps a single copy of message in a "topic" which multiple consumers can subscribe to and consume independent of other consumers of the group. --- --- ### The application side of the world Moving on to the fun stuff now! Let's talk about the words application developers use when using asynchronous messaging. #### Event As I've [written before on this blog](https://www.kislayverma.com/programming/using-events-to-build-evolutionary-architectures/?ref=kislayverma.com) , An event is a broadcast, triggered by a system when something happens in its business domain that it wants to tell the outside world. The language of the broadcast is the business domain language of the publisher. Other systems that are interested in learning about these events have to subscribe to the broadcast and interpret the publisher's domain language to act on it. However, even though the publisher is not explicitly aware of who is consuming his event, the mere fact that there is an externally accessible event stream make it a part of the publisher's public interface and any change to event structure or meaning should follow the same change management protocols as changes to synchronous APIs. #### Message/Command/Notification A message represents directed asynchronous communication from one system to another and is typically expressed as a command (a "do-this" message), which is why "message" and command" are often used interchangeably. Since the communication is peer to peer, this is essentially the asynchronous version of a request-repsonse interaction and similar failure handling semantics can be employed (retries after sometime by re-sending the message, idempotency in the consumer etc.) #### Query This is a lesser used pattern in messaging - one where a system requests (queries) data from a remote system by sending a query over a queue. I have only encountered this pattern in orchestration scenarios using workflow orchestration tools like Camunda, JBPM etc to implement workflows for stitching together data and actions across multiple services. We may have a use-case to get data from one service and then invoke another service per-element of the retrieved data. The second piece is clearly a candidate for a message/command style interaction, but what about the first. Usually we would invoke a synchronous API on the first service to get the data. However, this suffers from well known brittleness of temporal coupling. However, if the workflow system were stateful, we could issue an asynchronous query (tagged with a unique id) to the first service and pause the workflow. The service would then respond back (also asynchronously) with the outcome of the query and maintaining the original id. The workflow system can now map the incoming response with the open request. This is a message style system ("do-search" request/response) which can scale significantly better. Another flavour of this is encountered in building actor model based systems. There this pattern is even more explicit because actors can only interact via messaging, and they are necessarily stateful. A workflow composed of actors is therefore stateful and asynchronous by definition. What if one actor wants another actor to give it some data. Ideally, we would not have this dependency among independent actors, but let's say it can't be removed. In Akka, we would use the "ask" pattern for this, which is nothing but an asynchronous message sent to the second actor with the first actor waiting for response. The second actor responds with a direct message to the first actor containing the required data. ### Conclusion This covers the exact concepts of messaging architectures. These words have a specific meaning, work in specific ways, and result in very different architectural outcomes. Having this specificity in our heads helps in making clearer decisions about what an emitted "record" is expected to achieve. I've seen teams emit generic business events 5 times with slightly different syntax to onboard 5 different use-cases when in fact a single event would have sufficed. I've worked in teams that refused to build P2P messages because they had an event stream and everything had to be done via that. A thing often becomes what you keep calling it, and I believe having an explicit understanding of messaging paradigms helps us build better systems with clearer responsibilities. Read Next: [Change Data Capture versus Domain events](https://kislayverma.com/domain-events-versus-change-data-capture/) ### Saving the day with Continuous Refactoring URL: https://kislayverma.com/saving-the-day-with-continuous-refactoring/ Last updated: 2020-07-21T18:13:56.000Z Have you run across this scenario before? 1. Dev team keeps getting tight deadlines for feature delivery 2. It responds by shipping tech debt-laden code 3. This tech debt is never paid off because there is never time to clean-up 4. Goto 1 a few times 5. Tech debt gets worse till it starts actually blocking new changes or breaking things in production 6. Dev team campaigns for a major redesign because the current system is beyond saving 7. This will take a lot of time, but the business team is promised technology indistinguishable from magic at the end 8. The re-arch is executed by putting everything else on hold 9. It takes far longer than imagined and yields far less impressive results, 10. Goto 1 a few times 11. Business teams lose all faith in the dev team and go back to using excel/Company shuts down The above sequence of events is unfortunately far too common. While there are many causes and many corresponding courses of action to prevent this, I want to look at what the dev teams can do. The biggest problem to me is that repeated big re-arch. Like startups, big redesigns mostly fail. They fail at containing scope, they fail at delivering on their promises, they fail at levelling up the organization's capabilities, often they fail at being complete at all. A truly agile organization should be surprised if it finds itself needs such a large one-shot change - whatever happened to [discovering and implementing solutions incrementally](https://www.kislayverma.com/agile/agile-for-innovation-going-beyond-execution-excellence/?ref=kislayverma.com)! However, there is something that dev teams can do to prevent this, and that is to adopt a culture of continuous refactoring. More than sprints, scrums, or daily stand-ups, continuous refactoring can make a dev team agile by improving what they work with, i.e., the code base. Martin Fowler has written written a great book and many articles about Refactoring (including [this interesting aside](https://www.martinfowler.com/bliki/EtymologyOfRefactoring.html?ref=kislayverma.com) on the etymology of the word) so I will assume we all know what it is. Let's jump into why I advise "continuous" refactoring. [![](https://kislayverma.com/content/images/2020/07/refactoring-fowler.jpg)](https://www.amazon.in/gp/product/B007WTFWJ6?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B007WTFWJ6&ref=kislayverma.com) ### Why refactor continuously #### Consistently keeps the code base of high quality For a software developer, a well maintained codebase is its own reward - like a personal Monalisa or a zen garden. It is a pleasure to work in, it is easy to get on-boarded to and to understand, and a team with such a unicorn is generally likely to be more productive. #### Makes everyone familiar with code Software is never "done", so it is important to stay up to date with its "current" form to work effectively with it. If every on the team in tinkering around in the code base looking for improvement opportunities, the odds are that more than one person has seen the obscure by-lanes of it and can help in case something happens to that part. This is especially relevant for large codebases that have built up over a long time and seen multiple team members come and go. Refactoring and being on-call are the two most powerful tools I can think of for spreading broad knowledge of all aspects of the code among all. team members. #### Sets up the next generation of the architecture Everyone wants microservices right? If you are saddled with a big-ball-of-mud monolith, most likely you can't wait to break it apart . But how to do this safely? The large, hairy beast has tentacles that you probably don't even know of. Who knows what breaking-off a module will do? The actual splitting is simple enough, it is the unpredictable fall out that we must be wary of - we do not want to end up with a distributed big-ball-of-mud! There is one sure shot way to set this up - start refactoring this legacy code base. You will learn things about the code that you didn't know, previously muddled boundaries will emerge, dependencies between various parts will become clearer, bugs will be found and squashed, and low-hanging performance improvement will be made. You will develop a deeper understanding of where to go than would otherwise have been possible. Who know, may be at the end of all this, you won't want micro-services because you will have an awesome, super-efficient monolith. And most importantly, the business will keep running through all of this! Even when we are working with new code, continuous refactoring reveals emerging boundaries and behaviours are in our code and this insight makes a bigger architectural change far more likely to succeed because we are not coming at it requirement from a "this-sucks-lets-rewrite-everything" mentality but from actual signs emerging in the code base from attempts to solve real business problems. Experienced developers can generally identify these patterns and set up the team towards powerful changes (e.g. split this module into a separate service, let's put circuit breakers in the central client library). When everyone on the team is doing refactoring, more and more people start getting this sixth sense of sensing incoming change. --- --- ### How to refactor continuously There are many ways of doing this and a ton of advice on the internet. I want to focus on two aspects, one technical and one organizational. #### Write tests ![](https://kislayverma.com/content/images/2020/07/red-green-refactor.png) The oft-ignored adjunct to this oft-repeated advice is - Write tests. Refactoring doesn't sit in some sort of technical vacuum - it is a means to an organizational end. I have read different perspectives about this, but to me, the objective is to increase shipping velocity. What we do today should make future changes faster/easier/safer. Now if we agree that refactoring today facilitates future changes, then it is a no-brainer that having testable code facilitates refactoring. We don't have to think only in terms of changing code, we can think in terms of adding tests as well. We may not do the refactoring, but due to the extra tests, anyone else can do it safely tomorrow. Writing test IS refactoring. #### How to make time "We''d never get time for this". I agree, and it is sad but true. It takes a lot of trust to get exclusive refactoring time, especially around parts of code which aren't actually broken (but could be improved). I prefer the widely popular "buffer your estimates" approach. Ask for an extra day to finish the task. It is easier to refactor what you were going to modify anyway, stakeholders are less likely to complain about a day or two worth of estimation as much, and we will be able to continually chip away at cruft. This is easier than getting time allocated for "tech only" changes separately. Doing this also set bounds on how much time we will spend on refactoring - it would make things better but only in the scope of the given business feature that this is being funded via. We are far less likely to boil the ocean in these situations because there is a larger goal to be achieved - you refactor only what you are working with at the moment. ### Conclusion It is, in my opinion, unfair to expect that a team would "obviously" do continuous refactoring. It is not obvious, and it is seldom a priority in face of the pressure of delivering features. It is a culture, and like all cultures, should be explicitly talked about, discussed, and encouraged. If we can interpret our work in terms of [making things a little better everyday](https://www.kislayverma.com/programming/make-it-better-every-day-of-the-week/?ref=kislayverma.com), the gains add up really fast. \[newsletter\_form type="minimal" lists="undefined" button\_color="#27AE60"\] ### Distributed Systems as Data Pipelines : Throughput, Capacity, and BackPressure URL: https://kislayverma.com/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/ Last updated: 2026-07-22T12:47:20.000Z [The last we spoke about asynchronous programming](https://www.kislayverma.com/programming/asynchronous-programming-a-cautionary-tale/?ref=kislayverma.com), I paused with the notion of thinking about distributed systems as data pipelines. Let's dig a little deeper into this, and talk about the concepts of throughput, capacity, and backpressure in building resilient systems. We have seen how asynchronous programs "stash-away" uncompleted tasks till they are notified by someone of their completion. This can be interpreted as building a "queue" of unfinished tasks between the asynchronous caller and callee. In fact, if we are using [thread-pools for asynchronous programming](https://www.kislayverma.com/programming/asynchronous-programming-with-thread-pools/?ref=kislayverma.com), we have to explicitly use a queue to transfer tasks and then wait for result. In [event-based systems](https://www.kislayverma.com/programming/event-based-asynchronous-programming/?ref=kislayverma.com) the same interpretation can be made by thinking that having invoked the caller and waiting for result is no different from queueing a task and waiting for completion. ![](https://kislayverma.com/content/images/2020/07/event-loop-a-b-1.jpg) This is very similar to message-based communication between two independent systems (over a message broker like Kafka or RabbitMQ). A system invokes a remote system/method asynchronously and awaits the completion of the invocation. The waiting terminates when a completion signal/message is received. ![](https://kislayverma.com/content/images/2020/07/async-comm-a-b.jpg) Theoretically, there is no difference between the two models. And the common element between the micro and macro scenarios is the concept of the queue. Both use queues as buffers to temporally isolate the caller from the callee. So all distributed systems, when visualized end-to-end, are stream processing systems, regardless of their implementation details. Asynchronous systems, distributed or not, can ALWAYS be interpreted as stream processing systems (since a handover-interrupt model is built into them). With this in mind, we can now start applying queueing theory to how we look at scaling software systems. ![](https://kislayverma.com/content/images/2020/07/systems-thinking.png) --- ### Throughput and Capacity If we have a hose with water being pumped in from one side and used at the other, what would we do to push as much water through as possible? The first thing is to make sure that water goes out of the hose as fast as possible. This can be achieved by making the consuming side of the hose as frictionless as possible. Remove all obstructions, make it fast to get water out. If all the water being pushed in can be taken out at the exact same rate, then there is no "bottleneck" and we can push more and more water. In software terms, this is the equivalent of increasing throughput, i.e., taking less time per unit work. When we think of reducing response times or using faster algorithms, we are thinking in terms of throughput. Throughput is usually measured in time-per-request, Teams spend a LOT of time and effort in this type of optimization, writing better code, smarter algorithms, adding caches, etc. Let's say a user invokes system A that needs to invoke system B to do its job. Let's assume both A and B are single-threaded and synchronously programmed. The user wants to call system A as often as possible. But due to its synchronous and single-threaded nature, A can only accept one request at a time. So we cannot add any more users, and the only way to get more out of the system is to make both A and B very very fast so that they can at least get through more requests. The other way to push more water downstream is to use a bigger/wider hose. This one is interesting because now it is realized that the water tap can open really wide, and the other side of the hose can push out all of that water in the same time that it takes to produce it. Notice how we are now measuring in terms of work-per-unit-time (the opposite of throughput). In software terms, this is handling concurrency or building capacity. Capacity is the measure of how many simultaneously submitted tasks a system can serve. As the analogy suggests, this is a very different beast, and asynchronous programming patterns hold a very important place here since they allow for at least the acceptance of a large number of tasks without getting locked up. Consider that systems A and B from our example above are still single threaded, but A is now asynchronously programmed so that its single thread is not blocked when calling B. This system can support multiple users from the get-go because not being blocked allows it to take more work. Also, every optimization in terms of increasing speed/decreasing latency now has a much wider impact across all open requests. When only one request could get faster at a time, now the entire set of requests being buffered by system A reap the benefits. We have a much more scalable system now. There is now the problem of too many requests now reaching B all at once - we will get to it later in this article. --- --- ### Little's Law The concepts of throughput and capacity find joint expression in [Little's Law](https://en.wikipedia.org/wiki/Little%27s%5Flaw?ref=kislayverma.com) . > At steady state, the average number of items in a queuing system equals the average rate at which items arrive multiplied by the average time that an item spends in the system. In other words, the amount of water in the hose is a factor of amount-of-water-per-unit-time being pushed in and the time-per-unit-water being removed. In further other words, we cannot define a steady-state system only in terms of throughput, we also need to include its capacity or the amount of unfinished work that can hang around in it. Why did we add "steady state" there? A system that is already crashing and burning, water hoses exploding or popping out is not very interesting - it is already broken. "Steady state" means that we are trying to describe a system with steady flows and behaviour, aka stable SLAs. If we look at a distributed system like a mesh of water hoses of different sizes connected to each other, we want to be able to tell how much water is flowing where if everything is going well. This is what SLAs are (how fast + how much), and Little's law allows us to define these elegantly. By drawing boundaries around any set of connected components, we can monitor their entry/exit rates and volumes and figure out how they will behave under variable load. Yes, I said around "any set of components". Not only can apply the law around the whole components, we can recursively apply it to sub-sets of components to analyze internal stable states and sum of all these is equal to the superset. ![](https://kislayverma.com/content/images/2020/07/applying-littles-law.jpg) Let's consider data caching as an optimization strategy. What does it optimize? It reduces the amount of time needed to fulfill a data read request, so we know that we are optimizing for throughput. But have we taken care of all failure patterns now? If we use something like Redis (a single-threaded system), we know that we cannot handle a surge in parallel calls. Now we can go back and perhaps add a read-only slave to our Redis setup, thereby doubling the capacity. Now we are going through requests twice as fast, and the steady-state capacity of our system just went up twofold. So now that we know all this theory, how do we build a stable, high scale, distributed system? --- The only way to build a stable distribute system is to make sure that ALL systems and connections in it are stable. This is where true complexity of microservice architectures emerges - there are system dependencies (often circular), there are fluctuations in the performance of each system, variations in load due to things like traffic or retries-on-failure etc. Beyond a point, it becomes very difficult what will happen when something unexpected happens? The way out is to build communication at each step in such a way that the overall system automatically converges towards stable state due to feedback (positive or negative - usually the latter). What kind of feedback can we design to achieve this? [Theory of Constraints](https://www.leanproduction.com/theory-of-constraints.html?ref=kislayverma.com) in manufacturing processes has the notion of in-progress-inventory as cost. Once you release some work on a factory floor, it is a liability, eating manpower and materials till it is complete and can be sent out to be sold. If a downstream machine isn't working, there is no point for workers in upstream processes to be doing a lot of work and generating in-progress-inventory - it will never complete. The factory floor uses this feedback loop to control how much work can be taken at any time. We can apply the same concept to the software system pipeline. --- In general, we always want to have "some" surplus capacity (so that we can gracefully handle fluctuations in traffic) without incurring a prohibitive cost for it. The simplest way of adding capacity is to add more hardware to it. But this is often not cost-effective at scale. As we have seen above, asynchronous programming helps us build elastic capacity (because we have a queue as a buffer between two parts of the system) in face of variable load, and we can turn a lot more of our attention to improving the throughput of individual components instead of constantly worrying about capacity of the system. However, elasticity goes only so far. Accepting more requests than we can handle is always a bad idea, in software as in life. Let me repeat - if our system is a mesh of pipes, we cannot push through more data than the narrowest point allows. This is a big problem in asynchronous design. The buffer between caller and callee is essential for asynchrony, but the same buffer prevents the caller from understanding that the callee is being overwhelmed by the workload being sent it's way. In the synchronous model, the calling system would start seeing thread-blocking, etc, but asynchronous systems don't have this feature(bug??). We can set timeouts on the submitted tasks to start shedding load, but this means that we will keep on taking requests only to fail them later (leading to a bad user experience). Circuit breakers can help here, but is there is a more elegant way to achieve flow-balancing dynamically? --- ### Backpressure Back-pressure is a mechanism to resist the flow of data through software systems so as to maintain a stable flow in face of local variations in capacity and throughput. We want to regulate the flow of water by not letting too much water enter at too much speed, thereby risking the water-hose mesh. To do this, let's flip the analogy and try to pull data through the system rather than trying to push it. Let's say that conceptually, the queue lies inside the producer of the task pipeline, and it is finite in size (NEVER EVER USE INFINITE BUFFERS. EVER. NEVER.). The caller reaches out to the callee, in this case, to get a new task from the buffer to process. The throughput is thus explicitly controlled by the consuming side (which makes sense, he who has to do the work should decide how fast he can go), and the capacity (a measure of open requests) is determined by the calling side by deciding how many unprocessed tasks we want to keep, aka the queue size. If the queue is full because the consumer is not running fast enough, the caller has a choice - he can drop the oldest (and presumably the least relevant) tasks from the queue to make room, or stop taking any more requests immediately (since they will likely fail anyway). This is different from setting task timeouts because we are not saying how long a task should take, but how unfinished tasks the system as a whole can have and still hope to complete them within the SLA. We can also make the callee own the queue, and let him own both the rate as well as the number of outstanding requests. This centralizes the logic to some extent but it is possible that this system gets so loaded during a traffic surge that it cannot even make the decision of load-shedding. There are [three ways to achieve backpressure](https://medium.com/@jayphelps/backpressure-explained-the-flow-of-data-through-software-2350b3e77ce7?ref=kislayverma.com) : 1. Tell the caller to stop giving new tasks: This is usually the best and the most effective way. If a downstream system under load can proactively request a breather, a dynamic equilibrium between producer and consumer can be achieved very quickly. RxJava uses this paradigm. 2. Block the caller when he tries to give a new task: This is slightly different because here the caller faces an error situation on producing a new task and he has to take an independent decision on how to handle this error after having taken on the task. Patterns like rate limiting fit in here. 3. Start dropping some tasks: Depending on the context, the callee can start dropping the oldest or the newest tasks from the pipeline. This silent behaviour is not my favourite because it gives less visibility into the system unless the caller is monitoring task completion properly, but this is the same thing that gives it value - the callee can independently implement this if the caller cannot be influenced to change behaviour. ### Conclusion This was a lot of heavy stuff! Let's tl;dr it. 1. Distributed systems and asynchronous systems both behave very similarly and can be understood as computational pipelines. 2. Throughput is the rate of doing things in the system 3. Capacity is the number of tasks that can remain open in the system without breaching the SLA. 4. SLA is a joint promise of throughput and capacity given by the downstream system to the upstream system. 5. Back-pressure is a mechanism of propagating the knowledge of downstream failures/problems. 6. Scaling and stabilizing a system is a constant back-and-forth between addressing throughput issues and capacity issues. Try this - Go back to a bird's eye view of your architecture and imagine that all IO-bound processes are asynchronous. What would now be the bottlenecks? Have you explicitly identified fallback behaviours for their callers? What would happen if these fail? And so on and so on. What I expect you will see from this exercise is a combination of throughput and capacity bottlenecks each feeding into each other - and now that you are armed with Little's Law and back-pressure, you will be able to eliminate them to get to a stable, beautiful system. Read Next: More article on [domain-driven design](https://kislayverma.com/content/files/2026/07/domain-driven-design.html) \[newsletter\_form type="minimal" lists="undefined" button\_color="#27AE60"\] ### The full-stack team of this decade URL: https://kislayverma.com/the-full-stack-team-of-this-decade/ Last updated: 2026-07-22T12:47:21.000Z Have you heard of the "full stack team"? Odds are that you have - it is the cornerstone of the agile methodology that is very much in vogue these days. A full stack team is a self-contained, autonomous team whose members collectively have all the skills needed to solve the problem they have been set upon. Such a team can build backends and APIs, manage its own data stores (for all practical purposes), and build its own UIs/frontends - thereby removing any need of dependency on other teams for fulfilling its mission. These teams are now found all over the industry, and limiting the lines of communication and co-locating the skills in this way has turned out very well in general. aka Two-Pizza team. ![](https://kislayverma.com/content/images/2020/07/two-pizza-team.jpg) In the coming decade, a full-stack team is going to require two more critical skills to remain full-stack. These are data engineering and data analysis/machine learning. --- ### What are these things Business Intelligence and Reporting (BI) has historically been an independent function and team, separate from the more "operational" teams which own the software that runs the business. The latter generates business data (customer, orders, invoices, bookings whatever) and the former uses this data to generate insights into the business (RoI, TAT, personalization, margins across different dimensions etc). This would typically be done by shipping all the operational data into a "data lake", and then unleashing specialized analysts upon this vast trove of data to do all sort of slicing and dicing. The outcomes from this analysis would flow out to organization leaders, then back to operational teams as the next set of features they had to build - thereby completing the feedback loop. ![](https://kislayverma.com/content/images/2020/07/bi-ops-team-stacks.jpg) The organizational divide mirrors the technological divide. Frontline teams use "OLTP" data storage and build high-performance systems to please the customers of the business directly. Microservices, sub-second latencies, asynchronous messaging, and the latest UI frameworks feature prominently in the discussions of these teams. BI teams focus nearly exclusively on moving large amounts of data around using frameworks like Pentaho, Spark, etc, joining data from all over the organization into "OLAP" stores like Hadoop, Hive, Amazon Redshift, etc, and catering to the analytical requirements of internal customers like business analysts and data scientists. --- --- ### Why touch what ain't broke? There are two main reasons why these worlds are now converging, and they tie in neatly with the distinctions mentioned above. #### Centralized BI is a bottleneck There are two kinds of analyses a BI team usually does. One is deep-dive into business data to derive insights into what the business is doing and what can be done to move it forward. The other is a more mundane, operational type of reporting around revenue, operational business metrics, monitoring the impact of some features being rolled out, etc. While the former type seems like the place where ML/AI etc would belong, they are being increasingly used in the latter type of analysis as well. As an organization grows, the latter type usually grows so much that the former begins to suffer and a centralized BI team becomes a bottleneck in running the business. The feedback loop I described above starts to get too slow for the modern agile organization. Look at what is happening on the OLTP side. We are creating small teams and giving them greater and greater freedom to execute on independent roadmaps using their favourite technologies and microservices architectures. But on the BI side, the complexity is becoming greater and greater as the BI team struggles to keep up with a faster rate of external change. As an organization scales, it is near impossible for a central team to manage all this data, understand it, and give meaningful insights unless at least some of the workload is taken off. The most straight forward of doing this is to move some of the operational data responsibilities back to the OLTP teams. This organizational single responsibility principle would vest all operational capabilities (including data management and analysis) in the operational teams and deeper analytical work in the erstwhile business intelligence team, who now become just some external consumers of the business data, not sitting in the line of business. But how is this to be done, considering the technical chasm between these teams? Fortunately, there is good news on that front as well. #### Technical Choices are converging As the hunger for "real-time everything" expands across the board, the tools and technologies used by the BI and operational teams are beginning to converge. Instead of the clunky batch jobs of old, more and more data is moving over messaging systems like Kafka. These data streams can be processed equally well by asynchronous microservices for interprocess communication or by tools like Flink and Spark for computational and analytics. As the technology lines blur, a data engineer with specific business domain knowledge in, let's say, warehousing and shipping, can use his existing tool-kit do a lot of analytics and reporting from within the operational system boundary. In a sense, we can now expand the bounded context of our services to included analytics as well. Additionally, the use of big data is becoming more and more commonplace and "familiar". Writing a data pipeline is like writing just any other kind of code, albeit one that has a different set of requirements. As developers get more and more access to commoditized/managed versions of big data platforms, it only makes sense that the data production and data analysis are getting more localized. Similar logic applies to the machine learning eco-system. The technology eco-system is becoming more mature, and the big companies are open-sourcing their internal infrastructure. AI/ML engineers have long bemoaned that data is the biggest challenge for them - and it makes sense that as data manages goes into the of individual teams, so can the use the use of this data for use in ML models etc. --- I'm not suggesting that deep research work or cross-domain data crunching will be subsumed within our two-pizza teams. There will always be a huge number of cases where uses of specific technologies and data sets will require teams sitting outside operational boundaries going niche work with niche skill-sets. However, I expect basic (definition of basic varying from company to company) competency in data processing and analysis to become part and parcel of every developer's and every team's bread and butter. Gone are the days when developers could toss their data over an ETL wall and call it a day. I believe this is a good thing. Teams that can own their data and process/analyze it to understand their business will be able to bring greater value to the business as compared to the teams that cannot. Consequently, they will retain more of their autonomy and agility in a world where data is quickly becoming king (if it isn't already). They are also likely to move faster since they have removed an external team's dependency from their go-to-market path. If you are looking to scale your organization and team in this decade, I would say that providing your developers with these skills and tools should be among your top priorities. The biggest problem I see in this path is recruitment. In the early days of autonomous teams (and even now), a DBA was always too scarce a resource to commit entirely to a small team. They were often shared among multiple teams. Similar will be the case with ML engineers - there just aren't enough of them - despite it being the most sought-after field for students. Hence for some time at least I expect that ML/AI engineers will be a shared, precious resource among multiple teams and a source of much political footwork among managers. **Read Next**: [The difference between autonomy and independence, and why your small teams are not moving as fast as you expected](https://kislayverma.com/independence-autonomy-and-too-many-small-teams/) \[newsletter\_form type="minimal" lists="undefined" button\_color="#27AE60"\] ### Book Review : The Innovator's Dilemma URL: https://kislayverma.com/book-review-the-innovator-s-dilemma/ Last updated: 2020-07-21T17:42:52.000Z > Harvard professor Clayton M. Christensen says outstanding companies can do everything right and still lose their market leadership -- or worse, disappear completely. And he not only proves what he says, he tells others how to avoid a similar fate. > > Focusing on "disruptive technology" -- the Honda Super Cub, Intel's 8088 processor, or the hydraulic excavator, for example -- Christensen shows why most companies miss "the next great wave." Whether in electronics or retailing, a successful company with established products will get pushed aside unless managers know when to abandon traditional business practices. Using the lessons of successes and failures from leading companies, "The Innovator's Dilemma" presents a set of rules for capitalizing on the phenomenon of disruptive innovation. > > [Goodreads](https://www.goodreads.com/book/show/2615.The%5FInnovator%5Fs%5FDilemma?ref=kislayverma.com) *The Innovator’s Dilemma – When new technologies cause great firms to fail* is a seminal book by the Harvard professor Clayton M. Christensen. It was first published on May 1, 1997 by Harvard Business Review Press. It deals with the theory of disruptive innovation (Innovations that lead to the emergence of new markets) and the precise mechanics of the disruption such developments cause in the business of the established companies of the time. The specific focus is on why the technology leaders are repeatedly, consistently beaten by upcoming start-ups. Disruption Theory is essentially a study of value networks. Companies of different sizes define their capabilities and markets differently. Over a period of time, industry leaders establish their competencies in markets ever increasing margins and therefore, profitability. This is not a mistake at first sight but rather a direct result of sound management practices. Large companies simply need larger markets to keep growing. This, however, gives room to smaller companies to occupy the lower margin markets with poorer products. It doesn’t impact the large firms initially, but as the start-ups improve their product, they reach the stage where their offering is good enough for everyone. While the industry leaders are probably still offering better products, they are now too good and lose the battle simply because the new market defined by the start-up’s product does not require them. Christensen proposes that it is this difference in perception of value and the ingrained culture of the firm that causes established firms to fail. Even though they have all the know-how, internal processes do not allocate enough value (resources, time etc.) to a technology aimed at niche audiences. This is what eventually leads to disruption. Christensen argues that the only way for an established company to take on the start-ups is to spin-off an independent subsidiary which is sufficiently free of the parent company’s culture to be able to compete with the newcomers on their terms. --- --- The first two chapters of the book are in depth case studies of disruptive innovations in the hard-drive industry and the excavator equipment industry. Having deeply studied the effects and symptoms of disruption, the author systematically builds up a very strong case for his theory (the [S-curve](http://innovationzen.com/blog/2006/08/17/innovation-management-theory-part-4/?ref=kislayverma.com) makes an appearance) and ends it with some advice on handling disruptions for managers. The power of disruption theory, the reason why it is so scary for managers, is the almost oversimplified approach it takes to assessing whether or not a venture will work. None of the massive data collection and research that typically characterizes technology management. The mantra is simple - - Newbie in sustaining technology – FAIL - Industry leader in disruptive technology – FAIL Businesses can fail for many reasons. But the above two are surely destined to fail. *The Innovator’s Dilemma* is a very lucid presentation of a powerful theory. I have never seen a solution extracted from a problem as elegantly or presented as forcefully. I would have liked the author to take some more case studies for a more holistic coverage, but the extent to which he has explored the two that have been taken is immense. He writes in an even, professorial tone that is easy to grasp. He has not tried to use the theory predictively, or to pit it against alternate theories. Though some general observations have been regarding managing disruption, this is not a “How to” guide. That comes later, in [The Innovator Solution](https://www.amazon.in/gp/product/B00E257S7C?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00E257S7C&ref=kislayverma.com) . It would be interesting to apply the disruption theory to today’s internet technology leaders like Google, Facebook etc.. Since these corporations explicitly try to maintain a start-up like culture, IMO the “values” argument doesn’t apply to them in quite the same way as the author has applied it to the hard disk industry. Companies that routinely use the “product champion” structure (flying squads of super star innovator’s devoted to an off-beat product) would also push the theory in its original form. Still, the power of simplicity is evident in its success in the real world. Today, “Disrupt” is the buzz world in technology. TechCrunch, a widely known website covering technology, holds an event called “Disrupt ”. Many other have taken the theory and built it into a larger body of ideas that finds wide acceptance and application. The Innovator’s Dilemma deserves its position as absolutely required reading in business schools around the world. If you are interested in technology or business or both, this insightful book is a must read. Read Next - More [book reviews](https://kislayverma.com/category/books/) ### Sidestep architectural "-ilities" and deliver business value URL: https://kislayverma.com/sidestep-architectural-ilities-and-deliver-business-value/ Last updated: 2020-07-21T17:36:32.000Z - Usability - Scalability - Maintainability - Extensibility - Reliability - Security - Portability - ... --- It is often said that software architects deal in "ilities". There are many "ilities", all of which are jointly used to describe the attributes good software design should have, a sort of multi-dimensional matrix for evaluating software quality. As is to be expected with all things software, not all of these dimensions are equally important in all scenarios, and architects are often required to make tradeoffs between one or more of them to solve the problem at hand. e.g. one might sacrifice usability for security or portability for scalability. Which one is better "depends" on whether you are building banking software (security) or an API gateway (scalability). ilities are a great way of analyzing specific components deeply. A single component (or a set of component serving a single purpose) is likely to have very well defined requirements, and we can express this in terms of specific software attributes like maintainability, security etc. This is useful in driving focussed conversations around how this component should be built and evolved. However, when we design a large system made up of many components, each of which may have different requirements but are still expected to work together (e.g. a large scale microservice architecture), it is difficult to extrapolate from specific technical dimensions and express overall rules that should always be upheld across the org. At such a macro level, we need to define architecture in terms of what business values the most, and there is no way of doing this with ilities. They are a good way of examining specific systems at a very technical level, but they cannot give guidance about how to build large scale system architectures. It is difficult to set down broad level architectural principles using them. So ilities should be used at the appropriate level of abstraction. They are useful when analyzing a single component or a small, coherent group of components. As long as the components under the scanner have a single purpose, typically we can pick out specific capabilities that best fit the requirements of the component. As should be clear by now, I find ilities kind of boring - useful but often too straight-laced. I feel that defining high level architecture in these terms cuts the analysis into overly narrow and technical slices and does not take into account the way business value is delivered. Over time, I have started evaluating software design in a slightly orthogonal way. I have picked out two system characteristics that I "personally" value the most, and these are the rules that I most often run any software design by. I think these two together strike the right balance between delivering value and technical competence in a broad enough way that they can be applied to any system scale. And like all good ilities, they are open to case-by-case interpretation and tradeoff. --- --- ### Increasingly reduced time to market Pragmatic Dave Thomas says - "When faced with two or more alternatives that deliver roughly the same value, take the path that makes future change easier" (Check out his [talk](https://www.youtube.com/watch?v=a-BOSpxYJ9M&list=LLvSBLfAD5azJXnefcCOA29A&index=4&t=742s&ref=kislayverma.com)(s) and [blog](https://pragdave.me/blog/2014/03/04/time-to-kill-agile.html?ref=kislayverma.com) \- extremely funny and insightful). No matter what beautiful things we might do as software architects, business wants things in production as soon as possible so that [money can be made](https://www.goodreads.com/book/show/113934.The%5FGoal?ref=kislayverma.com). Eventually, that's all that matters to the organization. "Move fast and break things" is rightly one of the most famous slogans in startups - deliver as fast as you can, even at some risk of mistakes. This is the part where we focus on the business. So I think about whether the current architecture will make it easier or harder to ship things in the future. There are specific dimensions along which this can be analyzed: 1. What decisions in this architecture are hard to reverse - identify and minimize them. 2. Is this feature [building towards a platform](https://www.kislayverma.com/technology/how-to-build-a-technology-platform/?ref=kislayverma.com) or is it a one-off. If the latter, can it be split into [platform and product](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com) so that we can later build on top of this ([Build momentum, not velocity](https://www.kislayverma.com/agile/being-fast-or-getting-faster-aka-build-momentum-not-velocity/?ref=kislayverma.com)) 3. Does this increase communication between teams - more comms -> more execution alignment required -> slower delivery speed. Would this require some sort of transfer of responsibility across teams or reorg. This aspect is always a little political/sensitive in nature. 4. Does the design have good abstractions behind which we can change things or are we leaking details leading to coupling - the latter obviously makes future change harder. ### Resilience I cheated a little bit here - Resilience is often considered one of the ilities. I like "Move fast and break things". "Move fast and break everything" - not so much. So while we give a lot of thought to delivering rapid changes to the system, we also need to think very carefully about how do we contain problems that will inevitably be introduced as we get faster and faster. There are two things to look out for here. One is to ensure that all failures are contained, and the other is that failures are detected quickly. #### Failure Containment 1. Can this system take down adjacent systems is load on it spikes? 2. Can adjacent systems take this system down if they get a load spike? 3. How will adjacent systems function if this system goes down and vice-versa? Are there fallbacks and ways to recover? 4. What will customers not be able to do during the outage? As you can see, all of this is familiar technical territory for software builders. Another thing to note is that I'm not so concerned about "why" a system would fail, just with what would happen when it fails. Because no matter what we do, a system will fail at some point - bad code, infrastructure issues, unexpected load, neglect over time - the list of potential causes is endless and pretty much none of them is completely avoidable. So at a high level, I like to focus on defining what happens when things fail rather than explicitly preventing failure itself. #### Failure Detection The second step is building observability into the systems to quickly detect and root cause cases that we failed to anticipate. Microservice architectures especially are complex system and the whole is more than the some of parts. [Charity Majors](https://twitter.com/mipsytipsy?ref=kislayverma.com) has written much about observability that I can never hope to match, so i will just point you towards her for the specifics, but the basic idea is to enough sufficient instrumentation of the system such that we know very quickly what has broken and where/why. --- I'd love to hear what you think of this broader architecture evaluation rubrik (at least for the large scale architectures) as compared to specifically using "ilities". Please leave your comments below and as always, if you like what you read, subscribe to the mailing list for more good stuff. \[newsletter\_form type="minimal" lists="undefined" button\_color="#27AE60"\] ### Book Review: Zero To One URL: https://kislayverma.com/book-review-zero-to-one/ Last updated: 2020-07-21T17:23:42.000Z > If you want to build a better future, you must believe in secrets. > > The great secret of our time is that there are still uncharted frontiers to explore and new inventions to create. In Zero to One, legendary entrepreneur and investor Peter Thiel shows how we can find singular ways to create those new things. > > Thiel begins with the contrarian premise that we live in an age of technological stagnation, even if we’re too distracted by shiny mobile devices to notice. Information technology has improved rapidly, but there is no reason why progress should be limited to computers or Silicon Valley. Progress can be achieved in any industry or area of business. It comes from the most important skill that every leader must master: learning to think for yourself... > > Zero to One presents at once an optimistic view of the future of progress in America and a new way of thinking about innovation: it starts by learning to ask the questions that lead you to find value in unexpected places. > > [Goodreads Blurb](https://www.goodreads.com/book/show/18050143-zero-to-one?ac=1&from%5Fsearch=true&qid=GtTnLCuwf5&rank=1&ref=kislayverma.com) I felt that reading *Zero to One* was like reading two different books. In the first book, Peter Thiel looks back towards the early parts of the 19th century and compares the positive, ambitious attitudes of those times to the more humdrum, incremental worldview that the current society, philosophy, politics, and companies hold most dear. He openly disparages the current propensity to deem future as unknowable and any attempt at making great plans as hubris. In his opinion, diversification, incrementalism, MVPs, are all signs of an indecisiveness attitude towards the future. Peoples, societies, and most importantly startup founders ought to have a clear vision of the future in their mind, and should continually drive towards that vision, instead of trying to remain "lean" or "flexible", which in the author's mind is merely another word for being noncommittal. He creates some interesting frameworks with which to assess what the future might look like and what kind of attitudes we should cultivate to drive towards a clearly envisioned future. His emphasis throughout this first part of the book is on encouraging the reader to envision a world materially different from the current one, and then to build it. In the second book, Peter offer practical, founder centric advice on distributing equity, company culture (companies don't "have" culture, they "are" culture), the importance of sales and distribution, and a brief detour into how humans and computers are complementary and there is no real to humans because "AI is coming". This last is, in my opinion, only partially true. The impact would be very real on people who are doing low complexity analytical jobs - but there is no proof yet either way, so we shall see. The last chapter is a bit of a mix, with advice about building your own brand as a founder and realizing that a founder is as empowered today to change the world as anyone has ever been in history. Personally, I found the first part to be a lot more passionate and heartfelt, even though there are parts where I disagree with the author. Peter's bias for action, his desire to materialize a future he envisions, and frustration at why the rest of the world does not, comes through very clearly. The second part was not very relevant to me as I do not have any experience of the startup life, hence I could not relate very well to it. And even with my untrained eye, I would advise that you go to [Ben Horowitz's "The Hard Thing About Hard Things"](https://www.kislayverma.com/books/book-review-the-hard-thing-about-hard-things/?ref=kislayverma.com) if you are looking for very specific advice about how to start, run, and grow a technology company. A huge point in favour of this book is that it is concise and expresses its ideas pithily (though not without eloquence). Coupled with the power of some of its ideas, it is entirely worth spending an afternoon in reading this. Below are detailed notes from my reading of the first part of the book to highlight the core ideas of the book. My comments are italicized, while the rest is either taken verbatim from the book or is meant to be as loyal as I could keep it while rephrasing it. --- --- ### Future - A later time is not future, a changed world is future. - Horizontal progress/Globalization means more of what is already there - replicate the same things everywhere in the world. - Vertical progress/technology - make new things, make the future. This doesn't happen automatically - it has be done deliberately. - Peter gives a brief history of tech crash at the turn of the millenium - and explains how he believes that at this point people turned away from the hubris of technical optimism. - The crash turned the tide of popular attitude towards more "feasible" ways of making progress. "Feasible" in those times meant safe or incremental, and globalization, aka more-of-the-same has been the name of the game ever since. - The other lesson from the crash was for companies to stay "lean" and "flexible" - another survival metaphor aimed being able to do whatever works rather than doing what you wanted to do in the first place. - *Peter is not a fan of surviving unless there is a goal in service of which you want to live another day - a generic ability to "pivot" is probably not his thing* ### Monopolies and Competition - What valuable company is nobody building? - Creating value and capturing value are different - all companies must try to be monopolies to capture as much value as possible. - Competition is gospel but impractical as it removes all possibility of making money - In a perfectly balanced, highly competitive market, there would be no profits and all competitors would be equally commoditized. - Monopolies try to express themselves in extremely broad terms to avoid attention. Highly competitive companies pretend to own the market by defining the market very narrowly. - "In business, money is either an important thing or it is everything". Only somewhat successful companies can afford to think about things other than money. - Patents and copyrights are legal support for monopoly - create something new and you get monopoly on it for some time. - This is the way monopolies move the world forward, not by being monopolistic, but because they had to necessarily create something new in order to be monopolistic. They had to go from zero to one. - The best kind of competition is not optimizing existing things (this leads only a ruthless, competitive market and profits for now) but to create entirely new things which result in fresh monopolies and may destroy existing monopolies in the process. *This is a fundamentally different way of looking at competition and how value is created by innovation.* - Competition focusses on past and present, not the future. It doesn't allow for a broad enough perspective that will allow a company to redefine itself as it grows. - Market share in a well defined market should not be a goal - if that is the only goal, that implies that there is no difference between the rivals. If you see too much competition - perhaps it is time to get out. ### Building a monopoly - Monopolies must think about enduring as a monopoly. Current metrics can help you focus on present but distract you from the future (Groupon, Zynga suffered from this). - To build a monopoly, find a small market, monopolize it, and then expand outwards deliberately. - This is similar to the idea of "[1000 true fans](https://kk.org/thetechnium/1000-true-fans/?ref=kislayverma.com)". - Entering a large market and not dominating it will lead to immediate competition from copycats - not a monopoly. - To endure, Monopolies can build the following moats : Proprietary value proposition which is at least 10X better than others, network effect, economies of scale, branding. - If you are starting up, don't think about disruption - hopefully you are building something new enough that you don't have to think about incumbents. Disruption, if it happens, should be incidental. ### Success is Luck? *Peter does not believe that success is luck, and although he tries to keep a balanced sound, the undercurrent is that of a person who believes that people who try deliberately will achieve success. He is not a fan of covering his bases or diversification, and uses a very interesting framework to discuss this - a combination of definite/indefinite and optimism/pessimism. To be Definite is to have a specific plan, and to be Indefinite is to be ready for anything (but nothing in particular). Peter falls strongly on the "definite" side of the world.* *He repeatedly shows a longing for the good old "definite" days of America, where people used to dream big about the years to come and repeatedly lambasts the current education, investment, and corporate scenario which promotes "multi-sided mediocrity". And in this context he suffers from a massive survivorship bias. Perhaps "suffers" is the wrong word - he obviously knows about the risks of determinism but doesn't care. His push is not towards being risky and single tracked, but rather towards "intelligent design" is the way we look and function in the world.* ### Power Law - 20% of companies in any VC fund give 80% of the returns. - Given this, we might imagine that "spray and pray" is a good strategy, considering no one can really tell which companies will succeed. Peter thinks this is wrong. - Since there so many companies, every company in the fund must be deliberately chosen to generate those astronomical returns. - All bets should be targeted to win instead of being targeted to hedge - *intelligent design at play again.* ### Secrets 1. Our society behaves like there are no knowable secrets left. 2. Everything is either known or unknowable - and this results in all kinds of extremisms - religious fundamentalism (dogma or mystery, anything in between is heresy), environmentalism (destruction of nature or mystery, nothing in between), free marketism (share price or theoretically efficient market, nothing in between). 3. If the world has no secrets or hidden realizations, there would be no injustice (since everyone would be equally enlightened) or enquiry (it wont be needed). Our world is not such a world, 4. *According to Peter, there is much yet left to discover for those who are willing to look, an attitude which ties in neatly with the "definite optimism" I have already called out so often in this article.* 5. *I would like to bring in a note from the science of complexity here. There is no such thing as "all" the secrets, especially if "soft" domains like capital markets are counted. The world is a complex information processing and producing ecosystem, so there are always things that are new and unknown. Peter's argument is not this, however. He is yet again looking back to the world of explorers and pioneers and talking about unearthing things from a fixed pool of secrets. While you could emotionally resonate with his version of our banal, non-curious existence, it is not really a factual argument.* Read next : More [book reviews](https://kislayverma.com/category/books/) ### Book Review : The Hard Thing About Hard Things URL: https://kislayverma.com/book-review-the-hard-thing-about-hard-things/ Last updated: 2020-07-21T17:16:53.000Z Now Reading [![](https://kislayverma.com/content/images/2020/07/hard-thing-about-hard-things-cover-1.jpg)](https://www.amazon.in/gp/product/B00DQ845EA?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00DQ845EA&ref=kislayverma.com) > "...In *The Hard Thing About Hard Things*, Ben Horowitz, cofounder of Andreessen Horowitz and one of Silicon Valley's most respected and experienced entrepreneurs, draws on his own story of founding, running, selling, buying, managing, and investing in technology companies to offer essential advice and practical wisdom for navigating the toughest problems business schools don't cover. His blog has garnered a devoted following of millions of readers who have come to rely on him to help them run their businesses. A lifelong rap fan, Horowitz amplifies business lessons with lyrics from his favorite songs and tells it straight about everything from firing friends to poaching competitors, from cultivating and sustaining a CEO mentality to knowing the right time to cash in. > > His advice is grounded in anecdotes from his own hard-earned rise—from cofounding the early cloud service provider Loudcloud to building the phenomenally successful Andreessen Horowitz venture capital firm, both with fellow tech superstar Marc Andreessen (inventor of Mosaic, the Internet's first popular Web browser). This is no polished victory lap; he analyzes issues with no easy answers through his trials..." > > [Goodreads Blurb](https://www.goodreads.com/book/show/18176747-the-hard-thing-about-hard-things?ref=kislayverma.com) ### Marketplaces are not Platforms URL: https://kislayverma.com/marketplaces-are-not-platforms/ Last updated: 2026-07-22T12:47:22.000Z Today we return to one of my favourite technical topics : [building technical platforms](https://www.kislayverma.com/technology/how-to-build-a-technology-platform/?ref=kislayverma.com). Since I started writing on this topic since early last year, I have spoken to a lot of confused product managers who didn't quite get what I was talking about, and had a completely different mental model of a platform. I have also met a lot of developers who say "we are building a platform for X or Y or Z" when they really are building a product for X or Y or Z. This has now become a bit of a pet-peeve of mine, so I want to take another shot at clarifying matters. This post is going to sound a little rant-y. ![](https://kislayverma.com/content/images/2020/07/you-keep-using-that-word.jpg) Quick thought experiment - When you hear the word "platform", what is the first thing that comes to your mind. Other than railway platforms. ![](https://kislayverma.com/content/images/2020/07/railway-platform.jpg) Not this platform! For most people, likely it is Amazon, Uber, AirBnb, Doordash or something on these lines. What is common to all of these - they are all two or more sided marketplaces. What else is common among them - NONE of them are platforms. Before the outrage begins, let's consider definitions. --- --- ### What are these things? A market place is a place/system where 2 or more parties come together to exchanges good/services for some compensation (usually money). It forms a neutral ground which matches parties looking to do a transaction (Uber matches drivers with riders) or allows them to discover each other (Airbnb guest can find hosts by searching the listings). It may also facilitate the said transaction by offering some services (e.g. Amazon will deliver something from a seller's warehouse to a buyer's doorstep) What then is a platform? A google search gets us this. ![](https://kislayverma.com/content/images/2020/07/marketplace-vs-platform.png) And this is the cause of all the confusion. The idea of a "platform" has become completely intertwined with "matching parties" in currently prevalent business-speak, which interprets it in the sense of a stage (a "platform" to showcase your talents/goods/services to potential patrons/buyers/clients). The original, technical context of a "platform" is vastly different. A platform is a tool/set of tools which allows parties other than the platform owner to build new products/services/experiences, in exchange for payment to the platform owner. The platform and its capabilities are just that, general purpose, reusable capabilities. It offers nothin of specific value to anyone other than a builder who can leverage the platform to build what she wants. Read in this context, Amazon is a platform only in the business sense - but I cannot use Amazon to build anything new. AWS, on the other hand, has been the birthing ground for so many new things in the digital world. You can pick any of the dozens of things on offer, combine them with other things from elsewhere, and create your own magic (e.g. Netflix runs off AWS). Another cause of the confusion is the hands-off stance of both marketplaces and platforms towards who is using them and for what. Marketplace say "we don't own the transaction, we just facilitate it" which is very similar to the technical platform stance of "we don't know what you are building, but here are some capabilities to help you along". So I will repeat one last time - ***unless you can use a thing to build something new and completely unrelated to it, it is not a platform***. In tech-speak, [***a platform is an externally programmable system***](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com). I have one more argument left, one that might be even more clear to business folks. ### Platforms have no network effect, marketplaces do Marketplaces and Social Media "platforms" today offer the most accessible example of what [Brian Arther](https://en.wikipedia.org/wiki/W.%5FBrian%5FArthur?ref=kislayverma.com) calls the "[Law of Increasing Returns](https://economicsconcepts.com/law%5Fof%5Fincreasing%5Freturns.htm?ref=kislayverma.com)". Them that has, gets. The more sellers there are on a platform, the more buyers will come there. The more buyers there are, the more sellers will come. If we can get a critical mass of demand and supply of good and services on a marketplace, this "network effect" causes a virtuous cycle with more and more people gravitating towards it just because everyone else is on it. Platforms have no such effect. A well made, useful platform might get good traction because of word of mouth and an increasingly large community of users as compared to its competitors in the market, but there is no commercial flywheel saying more makers will choose a platform because some makers are choosing it. There is no demand and supply on a platform - hence there cannot be any feedback loop. I hope this little piece helps you deciding what is a platform and what isn't. [Product are not platforms](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com). Marketplaces are not platforms. The warehouse management system you wrote for your company is likely not a platform . SaaS businesses are often platforms. Operating systems are platforms. etc etc. Happy hunting! Read Next : [APIs are not Platforms](https://kislayverma.com/apis-are-not-platforms/) ### Domain Events versus Change Data Capture URL: https://kislayverma.com/domain-events-versus-change-data-capture/ Last updated: 2026-07-22T12:47:23.000Z The building of [change data capture](https://en.wikipedia.org/wiki/Change%5Fdata%5Fcapture?ref=kislayverma.com) (CDC) and event based systems have recently come up several time in my discussions with people and in my [online trawling](https://open.spotify.com/episode/7MCz4MN4eKH2qlnjLhiMN4?si=74a6GueyS3-JTWPDksAnkQ&ref=kislayverma.com). I sensed enough confusion around them that I figured this was worth talking about here. CDC and event based communication are two very different things which look similar to some extent, and hence the confusion. Beware - using one for the other can lead to very difficult architectural situations. --- ### What are these things? Change Data Capture (CDC) typically alludes to a mechanism for capturing all changes happening to a system's data. The need for such a system is not difficult to imagine - audit for sensitive information, data replication across multiple DB instances or data centres, moving changes from transactional databases to data lakes/OLAP stores. Transaction management in ACID-compliant databases is essentially CDC. A CDC system is a record of every change ever made to an entity and the metadata of that change (changed by, change time etc). I have written about events on this blog before and have described them as announcements of something that has happened in the system domain, with relevant data about that occurrence. At a glance, this might seem to be the same as CDC - something changes in a system and this needs to be communicated to other systems - which is exactly what CDC is about. However, there is a key distinction to be made here. Events are defined at a far higher level of abstraction than data changes because they are meaningful changes to the domain. Data representing an entity can change without it having any "business" impact on the overall entity that the data represents. e.g. There can be several sub-states of an order that an order management system might maintain internally but which do not matter to the outside world. An order moving to these states would not generate events but changes would be logged in the CDC system. Vice-versa, there are be states that the rest of the world cares about (created, dispatched etc) and the order management system explicitly exposes to the outside world. Changes to or from these states would generate events. The difference can be stated explicitly in terms of system boundaries. When we design microservices or perform any system decomposition, we are trying to identify and isolate bounded-contexts or business domains from each other. This is the basis of all [domain-driven design](https://en.wikipedia.org/wiki/Domain-driven%5Fdesign?ref=kislayverma.com). CDC is about capturing data changes within a system's bounded context, usually in the terms of the physical model. The system is recorded changes to its own data. Even if we have a separate service or system which stores these changes (some sort of [platformized](https://www.kislayverma.com/platform-thinking/why-you-should-build-a-platform/?ref=kislayverma.com) audit store), the separation is an implementation detail. There is a continuity of domain modelling between the actual data and changes to it, hence both belong logically inside the same boundary. ![](https://kislayverma.com/content/images/2020/07/cdc-event-bounded-context.jpg) Events, on the other hand, are domain model level broadcasts emitted by one bounded context to be consumed by other bounded contexts. These represent semantically significant events in a language that the external systems can understand and respond to. That they are published over the same messaging medium, use similar frameworks, maybe get persisted somewhere etc are all implementation details. --- --- ### What about CQRS What about building a [CQRS](https://martinfowler.com/bliki/CQRS.html?ref=kislayverma.com) style system? For the uninitiated, CQRS (Command Query Responsibility Segregation) is an architectural style where the data model and technologies used for writes (Command) are different from those used for reads (Query). Such a design is typically used when there is a large difference in write patterns and to be supported and the read patterns to be supported. I have given a brief example of such a system in my [case study on the nuts and bolts of using asynchronous programming](https://www.kislayverma.com/programming/asynchronous-programming-a-cautionary-tale/?ref=kislayverma.com) . Updates to the command model and propagated to the read model, typically (but not necessarily) asynchronously. Can we use CDC for this? or should the command module emit events that are read by the query module to build its data model? ![](https://kislayverma.com/content/images/2020/07/building-cdc-cqrs.jpg) I would argue that since the command-query model separation is the internal design of the system, both models lie inside the same bounded context, and using CDC logs would not be inappropriate. Both producer and consumer are at the same level of abstraction (both are data stores, though one may be MySQL and the other [ElasticSearch](https://www.elastic.co/?ref=kislayverma.com) ), so using DB level changelogs is not a bad idea. This is, of course, just an opinion. Using events here would not be bad either, especially if different teams manage the models. The command module should always emit events anyway, if nothing else, then for [lending evolutionary characteristics to the overall architecture](https://www.kislayverma.com/programming/using-events-to-build-evolutionary-architectures/?ref=kislayverma.com) . ### Building CDC and Eventing Systems In modern distributed setups, change data is typically published over a messaging medium like [Kafka](https://kafka.apache.org/?ref=kislayverma.com) and can then be consumed by other systems which want to store this data. A very popular and efficient way of building CDC systems is by using tailing the internal log files of databases (MySQL and other relational DBs always have this for transaction management, ElasticSearch has a change stream in its newer versions) using something like Filebeat and then publishing the logs over Kafka. The other side typically has Logstash type plugins to ingest data into another system which persist this changelog. Consumers may also be Spark/Flink style streaming applications that consume and transform this data into a form suitable for other use cases. ![](https://kislayverma.com/content/images/2020/07/building-cdc.jpg) This is obviously not always possible since not all databases have changelog files to stream. For these systems, we must resort to adding code to the application layer itself to emit the changelog. Making sure that there is no case where data gets changed but the log is not emitted is a very hard problem to solve (essentially an atomic update problem: how to make sure that DB update and event emission over Kafka both happen or nothing happens). Lossless-ness is critical in a CDC system. ![](https://kislayverma.com/content/images/2020/07/building-cdc-no-acid.jpg) To build an event-based system, we would have the event generation logic at the applications layer like we did for CDC in the case of databases that don't have log files. That is the only place where we can translate the language of the database to the language of the domain. Same as for CDC, preventing event loss in the publisher is key to the design. However. some people propose to use the CDC stream as a system's event stream, and this is where I completely disagree due to all the reasons I have mentioned above. This would couple other systems to our system's physical data model, and we would have to forever keep our public entities the same as the database model. This severely reduces the expressiveness of our domain model. Consider an order getting canceled. The CDC system will record something like > ChangeLog{"order number" : "12345", "changed field" : "state", "old value": "in progress", "new value" : "cancelled"} If I were to express this in my domain language of what can or cannot happen to orders, I would ideally something like > OrderEvent {"order number" : "12345", "event type" : "order cancelled"} But this abstraction would just not be possible if we physically couple the language of transmission to CDC language. --- ### Summary One of the things to remember in building software is this: sometimes things that look similar and use similar tools to function are not the same. Especially when working with logical and physical models, we should be careful to isolate the implementation detail from that which is being implemented. Look hard at the publisher and consumer of the record that we are publishing - if they are both defined at the "data store" level, we are probably talking CDC. If that more of business constructs (bounded contexts) like order, courier, invoice etc, we are likely in events-ville. **Read Next** : [Messaging terms defined precisely](https://kislayverma.com/defining-messaging-terms-precisely/) ### Rulette : A pragmatic rule engine URL: https://kislayverma.com/rulette-a-pragmatic-rule-engine/ Last updated: 2026-07-22T12:47:24.000Z > What should be the discount offered for Nike shoes or on Nike running shorts if the customer is a high value customer versus a new comer to the system? > What is the applicable tax rate if a cotton t-shirt is made by manufacturer in a state and sold in another state? > Which customer care agent should a call be routed to if it relates to an already escalated issue by a customer who only speaks Hindi? --- Situations like those described above are no doubt familiar territory to any software engineer. Much of everyday software engineering is the codification of these types of rules in programming terms and then keeping up with the constant evolution that the business wants to do to them. And as you might imagine, a variety of tools have sprung to help programmers solve the problem. ![](https://kislayverma.com/content/images/2020/07/rulette-domain-1.png) As a general class of software, these tools are called "Rule Engines" or "Rule Systems" and play an important role in business rule management. They are available in most programming languages and offer a variety of ways of achieving rule management. The big daddy among them all, without a doubt, is [Drools](https://www.drools.org/?ref=kislayverma.com), which offers extremely powerful, rule modelling, management and evaluation capabilities. No matter how how complicated the rule set, there is probably a way to do it in Drools. So why did I build [Rulette](http://rulette.org/?ref=kislayverma.com)? What is Rulette, you say? Rulette is my first [open source project](https://kislayverma.com/content/files/2026/07/open-source.html) that I have been working on (fitfully) for the last 6 years. From the [Wiki](https://github.com/kislayverma/Rulette/wiki?ref=kislayverma.com) : > Rulette is a lightweight, domain-agnostic rule modelling, storage, and evaluation engine. I'm going to do this vis-a-vis Drools because it is THE most feature-complete rule engine out there, and also because looking for Drools alternatives is what actually drove me to creating Rulette. I will cover why I think Rulette is a better (at least simpler) alternative in most cases, but I will also talk about When Rulettte should NOT be used. ### Business rules usually aren't that complicated Drools's DSL offers extremely flexibly ways of modelling very complicated rules. Hell you can actually write java code inside the DSL to tackle the most difficult use-cases. Why would we give up this superpower? While product managers would like us to believe that the rules governing a business can get insanely complex, "typically", this just isn't so. The people running the business have to be able to make sense of what happens when, and an over-complicated set of rules just isn't practical in doing so. Most time, the rules can be expressed in a spreadsheet! A few columns each representing the inputs, mapping to another column which states the output. Anything more complicated than this is more exception than the rule. ![](https://kislayverma.com/content/images/2020/07/rulette-sample-rules.png) So it turns out that we don't really need the super awesome modelling power of Drools. For most of your needs, the requirements reduces to a series of "AND" conditions and and handling the "Any"/"Other"/Default use-case. Rulette does exactly this in a simple and efficient manner. Each rule is a series of "AND" conditions on the rule inputs and the entire set of rules is arranged as a trie for efficient lookup. ### End to End Rule Management While we typically won't need the extreme flexibility offered by Drools, we still need to "manage" the rules - create them, store them, update them etc. Drools has out of the box capabilities for doing these admin activities, but it has its own storage format and quirks to go with it. Most other rule engines don't even come with these capabilites - they focus more on rule evaluation to determine the output and leave out the rule administration. I wanted the best of all of these, so Rulette not only evaluates the output using a given rule set, but also has built-in rule management (CRUD) APIs. You can create new rule, update them, read them etc in the same consistent way that you use to evaluate them. This also allows Rulettte to enforce consistency checks on rule semantics (e.g. don't create the exact same rule twice) in-band to prevent any data corruption which may happen if you were to manage the rules outside of Rulette and then try to load them. Rulette also comes with a plug-and-play storage SPI that can be implemented for any data store. As long as you implement the data layer interface exposed by Rulette, you can store your rules in whatever datastore or manner you want. This last is important because often when a rule engine is being introduced into an existing application, the rule-set already exists in some shape or form and it is not alway possible to change that storage. But it is almost always possible to write a new adapter between the data and the Rulette rule format and get started. ![](https://kislayverma.com/content/images/2020/07/rulette-idata-provider-1.png) I started off by completing a MySQL implementation (thats the store I was working with), but I have since seen a JSON file based implementation, a Redis backed implementation, and even a REST API backed implementation of rule storage (this team couldn't access the data directly, but had access to the APIs around it). ## Talking business language The way Drools models rules (Each rule is broken down into a tuple of key-value combination of rules input names and values) does not translate easily to business speak - I found that explaining any errors that arose in the application was kind of difficult without using specific additional tools/UIs. While I do not believe that having "[users write the rules themselves](https://martinfowler.com/bliki/RulesEngine.html?ref=kislayverma.com)" is practical or even desirable, I do feel that having minimal impedance mismatch between rules makers and rule implementers can help the dialogue. So I wanted Rulette to align the business team's mental model of their rules and the way the developers do the technical modelling. So I explicitly designed the rules domain model to be tabular and having data types and ranges as first class citizens. These are the most widely used constructs in defining rules, and having these capabilities out of the box help in talking about things like price, date, discount natively. This is a design choice that, in my opinion, makes things simple for everyone. Several times, we were able to directly ingest/export excel sheets into our systems and analyze the input and output validated in a very intuitive manner. ### Simple setup, High Performance Using Drools involves setting up a server and having your applications connect to it in a specific manner. For many use-cases, this is too elaborate a setup. Rulette simplifies this, at least for Java based applications, by being implemented as an extremely lightweight library (no external dependencies except joda-time) which loads the rules in-memory. This make the rule evaluation process simple and super fast - You can get hundreds of thousands of evaluations done per second since each one is just a trie-traversal. --- --- ### What next Right now, Rulette is a useful library for implementing and managing your business rules. But there are some things that will make it even more enterprise-friendly. #### Distributed Rulette The single biggest room for improvement that I see today is that Rulette only works on one machine. This has two adverse results. One is inconsistent data in distributed setups. e.g. If your service uses Rulette internally, all instances of the service will have their own in-memory version of the rules and they might go out of sync if any rule is updated - there is no way to keep the distributed setup in sync. The only way to do this today is to periodically reload the rule-set on all machines. Essentially Rulette is acting like a local cache instead of a distributed cache. The other manifestation of the problem is that we are limited by the number of rules that can be loaded in memory of a single machine. A more distributed version of Rulette would be able to store and evaluate rules across multiple servers. #### Server Rulette server is an attempt to make Rulette's capabilities available to non-Java applications. There is already a [Rulette server](https://github.com/kislayverma/rulette-server?ref=kislayverma.com) available, built on the RestExpress framework which exposes the rulette APIs over a REST interface. But both the design and the API leave much to be desired. #### UI Everyone loves a good UI - and Rulette doesn't have one today. A good admin UI to visualize and administer rules would make Rulette a very attractive proposition indeed. --- ### Caveat Emptor If you truly are in the unfortunate situation of having more complicated rules that a series of AND conditions can't express, you have my sympathy - I and Rulette cannot help you. Also, if your rule-set is so large that it cannot fit in memory on a single machine, you cannot use Rulette. This problem is easily to solve by adding more memory, and I haven't actually seen this in the wild. You can check out [a talk I gave](https://www.youtube.com/watch?v=o1ENIoKI5fA&ref=kislayverma.com) ([deck](https://www.slideshare.net/KislayVerma1/rulette-a-pragmatic-business-rule-management-library?ref=kislayverma.com)) and the project Wiki to get more details on the [internal architecture](https://kislayverma.com/rulette/rulette-design) and [the way rules are evaluated](https://kislayverma.com/rulette/rulette-rule-evaluations). I have also written a case study of using Rulette to [model taxation rules](https://www.kislayverma.com/rulette/modelling-tax-rules-with-rulette-part-one/?ref=kislayverma.com). I would love to hear your thoughts and suggestions about how we can make Rulette even more awesome. Contributions to the code are welcome. Please reach out to me in the comments below if you want to learn more about use-cases or if you have any questions. I hope that when next you come across a reason for using a rule engine, you will remember Rulette and spread the good word! Read Next : [More articles on Rulette](https://kislayverma.com/category/rulette/) ### The Children of Hurin URL: https://kislayverma.com/the-children-of-hurin/ Last updated: 2020-07-21T16:43:44.000Z > Six thousand years before the One Ring is destroyed, Middle-earth lies under the shadow of the Dark Lord Morgoth. The greatest warriors among elves and men have perished, and all is in darkness and despair. But a deadly new leader rises, Túrin, son of Húrin, and with his grim band of outlaws begins to turn the tide in the war for Middle-earth -- awaiting the day he confronts his destiny and the deadly curse laid upon him.Deftly balancing thrilling battles with moments of introspection, Tolkien's vivid and gripping narrative reaffirms his primacy in fantasy literature > > [Goodreads review](https://www.goodreads.com/book/show/597790.The%5FChildren%5Fof%5FH%5Frin?ac=1&from%5Fsearch=true&qid=fmtz9nlNVL&rank=1&ref=kislayverma.com) *The Children of Hurin*, as one might imagine, is the story of children of Hurin of the house of Hador. This tale of Turin and Nienor is set in the world of Silmarillion (pre Lord of The Rings) and forms part of the story of the struggle of Elves and Men against Morgoth, who was the first villain of Middle Earth and the master of Sauron. The book centres around the life of Turin, the firstborn son of Hurin. Hurin was one of the greatest warriors among men of the first age. He was captured by [Morgoth](http://tolkiengateway.net/wiki/Morgoth?ref=kislayverma.com) while playing rearguard in King Turgon's retreat after the debacle of [Nirnaeth Arnoediad](http://tolkiengateway.net/wiki/Nirnaeth%5FArnoediad?ref=kislayverma.com) and tied to a chair on top of a mountain to watch as Morgoth's curse destroyed his family. With this background, the book cuts over to the story of Hurin's son Turin and daughter Nienor. Turin was a proud, headstrong man - the perfect tragic hero. His home was over run by Easterlings after Nirnaeth Arnoediad and his mother Morwren had him escape to Doriath to the protection of King Thingol. There he learnt elvish lore and skills, but all the while he was pursued by Morgoth's curse. Coupled with a proud nature and general disinclination towards taking advice from anyone, he ran into situation after situation where he let all his friends and well-wishers down and made every situation he was in worse by his presence. Everyone fell under the spell of his great skill at arms, but Turin's constant refusal to listen to reason led to disaster after disaster - capped eventually by falling in love with his sister Nienor (Morgoth's curse lead to a series of events where he never knew her and she lost her memory) and her committing suicide. The drama is entirely Shakespearean, with everyone you could root for doing something stupid or having some really bad luck. But Tolkien being Tokien, there is a greater design than just a series of misfortunes. While the curse on Hurin's family drives them to greater and greater misfortune, it also leads Turin's to killing the [Glaurung](http://tolkiengateway.net/wiki/Glaurung?ref=kislayverma.com) the dragon, who had destroyed elven armies and kingdoms. This deed gets him into the annals of great men of his time. By the time the tale ends, Turin and Nienor are dead, and Hurin is released by Morgoth just in time to find his wife Morwren dying by their graves. Like I said, a properly Shakespearean tragedy. I have dismissed Turin's character so far, but every time I read Tokien's stories of early middle earth, I am reminded of Lex Luthor and the real reason he hates Superman. Superman belittles humanity simply because he exists. No matter how great a man may be or how humble Superman may be, the latter is just more "Super". How much of Turin's pride (and that of other men of the Silmarillion universe) comes simply because elves exist as the elder children of the Valar? Can we rightly judge their stubbornness in doing things that are their own decision rather than being elven-wise? The highlights of the Children of Hurin are covered briefly in Silmarillion among the many episodes among the houses of men. In that larger context. it makes for an interesting interlude. However, full blown to a standalone book, there just isn't enough content in the story to be engaging. I say this as a die-hard Tolkien fan - skip this one unless you want the badge of having read absolutely all of Tolkien. Read next : More [book reviews](https://kislayverma.com/category/books/) ### Asynchronous Programming : A Cautionary tale URL: https://kislayverma.com/asynchronous-programming-a-cautionary-tale/ Last updated: 2026-07-22T12:47:25.000Z This is part 4 in a series of posts about [asynchronous programming](https://kislayverma.com/content/files/2026/07/asynchronous-programming-3.html). The previous posts are linked at the end of this article. In the previous posts of this series on asynchronous programming, I have outlined two ways of writing asynchronous code and the underlying concepts of how this paradigm helps us achieve greater system scale than the one-thread-per-request-model. And while I have shared some caveats to using asynchronous programming in these articles, today I want to share a cautionary war story about how I and my team once got so excited about using asynchronous programming and then so shocked by the results. --- About 3 years ago, we were working on [Myntra](https://medium.com/myntra-engineering?ref=kislayverma.com) ’s order management system (OMS) at the time which was the source of truth for all order related information and actions in the company. It was a Java REST API based system which used MySQL as store and used the classic single-thread-per-request model. The database schema was optimized for operations on the orders primary key but over time had accumulated a bunch of indexes for facilitating reads on other columns. However, as our scale grew and grew, we realized that the myriad read patterns were overwhelming even the multiple slave databases we threw at them because the MySQl schema was too ill suited to handle them. ![](https://kislayverma.com/content/images/2020/07/async-oms-old-arch-1.png) Thus began the quest for implementing a CQRS like architecture where the main OMS would only handle transactional operations and a read-only cache service (later name Armor because it shielded OMS from a lot of the read traffic) would store data in a format suitable for arbitrary reads. The OMS (command part of CQRS) would emit events on all data changes and the cache service (query part of CQRS) would be kept in near real time sync by ingesting these events. There were many considerations w.r.t. data management, consistency and latency but those are not relevant here. Let’s look at the technology choices instead. ![](https://kislayverma.com/content/images/2020/07/async-oms-new-arch.png) “Search at scale” is synonymous with [Lucene](https://lucene.apache.org/?ref=kislayverma.com) and we chose [Elasticsearch](https://www.elastic.co/?ref=kislayverma.com) as the database for Armor. What about the service itself? Looking at the requirements, we saw that the system had two aspects: 1. Ingest OMS events to keep the cache up to date 2. Serve the read calls from clients. There are many beautiful things about Elasticsearch, but at the time, one of the most appealing to us was we could use REST APIs to query it. This meant that the door to using asynchronous techniques was wide open for us!!! Lack of reactive database drivers is the most common sticking point in building end-to-end asynchronous systems, but here we were free because even our DB calls looked like REST calls. Fantastic! If we look at the requirements in the light of this capability, we can see that most of the Armor service was going to be IO bound: 1. Read from queue -> call some REST APIs for transformation -> write to ES 2. Receive read call -> query from ES -> return data I decided to use the super awesome [Vert.x framework](https://vertx.io/?ref=kislayverma.com) to build a completely asynchronous system. Except for data serialization and some minimal transforms, every single thing was non-blocking. And it worked beautifully. Initial launch showed that Armor could take of our entire traffic on a business-as-usual day with far less hardware that we had anticipated. So we decided to start the load tests. Since I already said this is a cautionary tale, I will pause here a moment and let you guys form your own theories about what could have happened. --- --- At small loads, we saw a trivial small rise in the API response times and the event consumption kept up perfectly. Data in Armor was fresh and its readers happy. Good signs. As we increased the load, however, the events were still being consumed almost immediately and the API latencies only increased minimally, but after a while the service would just freeze, often the Elasticsearch cluster would crash too. This happened every single time beyond a certain throughput. We had expected degradation, for sure, but what could be causing complete failure cliff? After a few dozen LOG.infos and memory dumps, a picture emerged that none of us had anticipated (no one had any true experience with asynchronous programming at scale). Let’s assume that there is only one thread in the system (there were, of course, more than one, but that only made everything that much worse). Because the whole system was asynchronous, this thread would read data from the event queue (async), call a couple more APIs (async), then call Elasticsearch (async) to write the data. ``` QueueReader.readMeassage() .onSuccess(msg -> { SomeServiceClient.getData() .onSuccess (someResponse -> { SomeOtherServiceClient.getData() .onSuccess(someOtherResponse -> { JsonObject finalPayload = someMinorDataMassaging(); ElasticsearchClient.writeData(finalPayload) .onSuccess(response -> LOGGER.info(“Success”)); }) }) }); ``` The asynchrony means that the thread never really stopped anywhere and ended up reading events from the queue as fast as it possibly could (unless interrupted by some IO completion). This in turn generated such a flood of *concurrent* requests to the other APIs and to Elasticsearch that they froze or collapsed under the load. But Armor did’t know about this (remember asynchronous) so it still kept picking up messages and trying to do its thing. The non-responsiveness of downstream systems, though, meant that the in-memory stash of call stacks of unfinished requests would now increase steeply in Armor, quickly leading to memory exhaustion and unresponsiveness. We had thought that even if downstream system fail for any reason, setting proper timeouts and circuit breaker would protect Armor (protecting armor — ironical. I know.) but we found that under load, Armor created unfinished requests in its memory much, MUCH faster than what timeouts could shed. --- Once we determined the behaviour of the system, the way out was straight-forward. Concurrent requests were killing the system, so we put a limit on the number of open requests that the system could have. This is the classic throttling pattern, where a counter was incremented when a request was started and decremented when it was completed. If the counter was already at max value, the thread trying to make the request would be forced to sleep for some time and try again later. This immediately call flood, and now we began to see more controlled flow of data change events through the system — we could change the value of the counter up or down to adjust our throughput based on what the downstream systems could handle. --- So what is the moral of the story ? For me, it is that asynchronous programming is such a paradigm shift in system architecture that it should be analyzed very different from “synchronous” system. We analyzed response times but never thought how many concurrent requests there would be at any point because in a synchronous system, the calling system is itself limited in how many concurrent calls it can generate, because of threads getting blocked for every request. This is not true for asynchronous systems, and hence a different mental model is required to understand causes and outcomes. Any large software system (especially in the current environment of dependent microservices) is essentially a data flow pipeline and any attempt to scale which does not expand the most bottlenecked part of the pipeline is useless in increasing data flow. We thought of pushing a huge amount of data through our pipeline by making Armor alone asynchronous and failed to distinguish between a matter of Speed (doing this faster) from a matter of Volume (doing a lot of it at the same time). The latter is what asynchronous programming is all about — it works better than the blocking code model because instead of getting stuck at pending IO, it “enqueues” it for deferred, interrupt driven processing. Asynchronous programs should be always be analyzed in terms of queuing theory. A formal statement of this is [Little’s Law](https://en.wikipedia.org/wiki/Little%27s%5Flaw?ref=kislayverma.com) , which explicitly distinguishes between throughput and capacity. > At steady state, the average number of items in a queuing system equals the average rate at which items arrive multiplied by the average time that an item spends in the system. In subsequent posts, we will look at Little’s law, throughput, capacity, and throttling/backpressure as they pertain to asynchronous programming when visualized as stream processing. A big shout out to my partners in crime on this escapade — Navneet Agarwal, Sanjay Yadav, Rahul Kaura, [Neophy Bishnoi](https://medium.com/@neophyb?ref=kislayverma.com) , and all the rest of the [Myntra](https://medium.com/myntra-engineering?ref=kislayverma.com) order management team. --- These are the previous posts of this series. - Part -1 : [Overview of asynchronous programming paradigms](https://kislayverma.com/overcoming-io-overhead-in-micro-services/) - Part-2 : [Asynchronous Programming using Thread Pools](https://www.kislayverma.com/programming/asynchronous-programming-with-thread-pools/?ref=kislayverma.com) - Part-3 : [Event-Based Asynchronous Programming](https://www.kislayverma.com/programming/event-based-asynchronous-programming/?ref=kislayverma.com) ### Event-Based Asynchronous Programming URL: https://kislayverma.com/event-based-asynchronous-programming/ Last updated: 2026-07-22T12:47:25.000Z In the last two posts in this series, we looked at the various [asynchronous programming paradigms to reduce thread blocking](https://kislayverma.com/overcoming-io-overhead-in-micro-services/) , and then took a deep dive into [asynchronous programming done via thread pools](https://www.kislayverma.com/programming/asynchronous-programming-with-thread-pools/?ref=kislayverma.com) . Now let us look at the true non-blocking style of coding, and how we can use IO-interrupts as events to build systems that hold up in the face of massive load. As before, let me repeat that whenever I say “blocked thread” in this discussion, I mean threads blocked/waiting on IO. We should also remember that we are not talking about event-based communication between two systems. We are talking about using IO-interrupts (IO started, IO completed etc) as events to control the behaviour of threads within a system. ### What is event-based programming Event based programming approach relies on having no blocking code in our application. This means that a thread initiates an IO operation (e.g. a thread in service A calling service B’s REST API), and then switches over to doing other things. When the IO operation completes, this thread is notified (interrupted) to come back and handle the result of its operation (deserializing the response). From the perspective of calling thread, the whole IO stage becomes outsourced to someone else, who will notify it when the job is done. The thread then starts executing the code path from where it left off before. From 30000 feet, this description looks very similar to our discussion of asynchronous programming via thread pools. But there is one vital difference. While thread pools were being used to isolate blocking IO from the calling thread, here there is NO blocking code whatsoever. We don’t need a thread pool on the other side of the calling thread accepting the request and notifying the calling thread back. However, that leaves the question of who issues the interrupt on completion of the IO operation. The answer is — the operating system. \*nix systems and their derivatives have long had the support for accepting a IO hook from an application, keeping track of the IO lifecycle, and notifying the application when IO is complete. By leveraging this, an application can eliminate IO management from its concerns and simply focus on triggering IO and handling the result. ![](https://kislayverma.com/content/images/2020/07/event-based-async.png) ### Handling task handover The way IO happens is that the application opens a socket and then issues commands to start sending and receiving data over that socket. Once the request data is sent, a typical application using one-thread-per-request model would just sit and poll the socket to see when the result is received. This is the origin of blocking IO. Non-blocking, event-based applications, however, handover this waiting stage to the OS. The OS comes with a polling program (epoll/selector/others depending on the distribution) which can handle polling of sockets for data very efficiently. The application adds its own socket to the list of sockets being polled and gives it a hook (aka a callback) that epoll can use to inform the application when the socket receives the response and the inbound data stream is ready for processing. The application thread is now free to process other tasks. ALL application threads are now free to process non-IO tasks (because there is no blocking IO whatsoever) giving the application the ability to handle massive scale. ### Handling task completion Epoll will keep polling all of the sockets registered with it, typically using a single thread. Once a socket receives data, epoll invokes the callback given to it by the application with the received data and the thread execution context (also stashed here by the application when handing over to epoll). Of course, epoll doesn’t understand what the data is, just that there are some bytes that the application can process. This callback interrupts an application thread to handle the output (deserialize, run business logic etc). ![](https://kislayverma.com/content/images/2020/07/event-based-async-cal-flow.png) Note that in this event-based style, the task handover and thread interrupt are happening across the Application-OS boundary. This exchange is often called the *event-loop*. Node.js made event-loops famous by being one of the first application development frameworks to leverage NIO to build single-threaded(!!!) applications that could nonetheless serve a massive volume of traffic so long as the lion’s share of work was IO-bound. These days, there are frameworks in most languages that will do this ( [Vert.x](https://vertx.io/docs/vertx-core/java/?ref=kislayverma.com) is a great example). ### Elegance and Scalability I love the event-based style because it is so much more elegant from the perspective of application design than the work-stealing style. The application does not have to tinker around wrapping blocking calls in thread pools or deal with sensitive application behaviours resulting from thread pool sizing. The entire application can behave asynchronously without having to deal with the specifics of IO management — we just initiate IO, don’t stick around, and come back to handle the output when it is ready. Scalability in style! --- --- ### Coding Time! Let’s revisit the code for making an HTTP API call in the one-thread-per-request model. ``` public class Client { public Response get(String url, Request request) { // API calling logic } }public class CallingClass { private Client client = new Client(); public void call() { String url = “some-api-url”; Request requestData = new RequestData(); Response response = client.get(url, requestData); LOG.info(“Got data {}”, response); } } ``` The thread running `CallingClass` code will block on the *`client.get()`* till it returns the data.The code in the NIO style looks very similar to the work-stealing style, except that there isn’t any task queue or worker pool of threads. ``` public class AsyncClient { public Future get(String url, Request request) { // API calling logic } }public class CallingClass { private AsyncClient client = new AsyncClient(); public void call() { String url = “some-api-url”; Request requestData = new RequestData(); Future responseFuture = client.get(url, requestData); responseFuture.onComplete() { // callback handler for successful future completion LOG.info(“Success with data {}”, response); }.onFailure() { // callback handler for failed future completion LOG.error(“API call failed with response {}”, response); } LOG.info(“Moving on immediately”); } } ``` This code will log “Moving on immediately” after handing over the IO control to the OS. epoll will interrupt the thread running CallingClass on receiving the API response. The application then parses the data to understand whether we have a success or a failure and then invokes the corresponding handler bound to the *Future*. Note how this code is far simpler than the code of the [work stealing style](https://www.kislayverma.com/programming/asynchronous-programming-with-thread-pools/?ref=kislayverma.com), even though it suffers from the same [callback-hell problem](http://callbackhell.com/?ref=kislayverma.com). ### Limitations to scale Awesome as the event based programming paradigm is, there are limits to what we can achieve with it. The limits to scalability in this model come from the following factors. #### Memory Overhead Every time a thread hands over to the OS an IO task, it also hands over the data in its execution stack for safe-keeping (so that it can resume from the same point on receiving completion interrupt). This info is kept in the memory and as more and more threads stash away their data, we can start running out of memory. This only happens at a very high scale, but then again, we don’t do this style of programming for a low scale. #### Data Copy Overhead When we use NIO, data and control switch between the operating system’s user-space (where application code runs) and its kernel space (where epoll runs). The data copy from user space to kernel space, and then back to user space can be very expensive at the OS level. Light-weight threads avoid this problem by keeping all their data in the user space throughout, and worker thread pools don’t have this problem because they never hand over control to the OS. #### Blocking code in other parts of the application I once read somewhere that “*the best way to do something in node.js is to do nothing in node.js*”. This statement says a lot about what non-IO operations can do to the scalability of an application. While our IO is non-blocking, all the rest of the application is still blocking, starting from serializing and deserializing IO data. As a result, the overall throughput of the application is now defined by and limited by how much CPU bound work it has to do. #### Too many interrupts A single application thread can potentially handle thousands of requests. However, what a single thread cannot handle is a flood of interrupts resulting from the simultaneous completion of those requests. We may end up in a situation where the application thread gets interrupted so often that it cannot serve any request completely without adding a lot of latency. ### Maximizing CPU usage It is true that a single application thread can suffice, using NIO, to serve way more traffic than traditional applications. However, server hardware typically has more than one CPU core and if we run just one application thread, we are leaving a lot of hardware capacity on the table. Single-threaded applications are also more susceptible to the last two problems mentioned above (performance degradation due to blocking code and too many interrupts). The [multi-reactor pattern](https://vertx.io/docs/vertx-core/java/?ref=kislayverma.com#%5Freactor%5Fand%5Fmulti%5Freactor) offers a way out by running more than one event loop thread (twice as many threads as there are CPU cores is often recommended as a rule of thumb). This adds a lot of capacity to the system by making full use of all the cores so that blocking code and the number of interrupts are less of a problem. ### Not everything can be non-blocking The event-based programming paradigm is premised on having APIs which do not block a thread for IO operations to complete. Typically these places are network calls to remote APIs, DB queries, File read/write, etc. In practice, it is rarely possible to make ALL our code non-blocking. Databases, especially RDBMS, usually have poor support for NIO (primarily because of the way transaction support is implemented). Application containers too, need to support asynchronous programs. If my REST service returns a `Future`, the application server (e.g. Tomcat) should be able to understand this and have corresponding event-driven code of its own to handle the client requests. This, unfortunately, isn’t widely adopted. As a result, we cannot always have a system built entirely on events. To work around this, we often see a combination of NIO (where the APIs allow), thread pools (to convert blocking code into async mode), and plain old blocking code in most applications ### Did we really change anything? You would have noticed that epoll still has to run a thread to poll all the sockets which is essentially a blocking operation. So aren’t still, in a sense, using thread pools (a thread pool with only one thread)? This is correct, but for the fact that we (i.e., the application) are not doing this anymore. The application is completely event-driven, and the OS is extremely efficient at handling low-level operations like socket polling. Of course, this puts a limit on how many operations we can do (like the data copying concerns highlighted earlier) but overall the event-based style results in applications that are vastly more scalable than one-thread-per-request applications and much better designed than thread pool style applications. Read Next : More articles on [asynchronous programming](https://kislayverma.com/content/files/2026/07/asynchronous-programming-4.html) ### Asynchronous Programming with Thread Pools URL: https://kislayverma.com/asynchronous-programming-with-thread-pools-2/ Last updated: 2026-07-22T12:47:26.000Z In my [previous post](https://www.kislayverma.com/programming/overcoming-io-overhead-in-micro-services/?ref=kislayverma.com), I described strategies for improving thread utilization in an IO-heavy environment. I will take a closer look at the thread-based asynchronous programming approach in this post. Whenever I say “blocked thread” in this discussion, I mean threads blocked/waiting on IO. This is the waste we are trying to get rid of — threads blocked on the CPU can only be unblocked by addition of more hardware. This strategy allows us to achieve massive system scale even when working with blocking code. ### What is thread-based asynchronous programming The thread-based asynchronous programming approach, also called “*work-stealing*” or “*bulkheading*”, allows one thread pool to hand over a task to another thread pool (let’s call it a work thread pool) and be notified to handle the result when the worker thread pool is done with the task. From the perspective of the calling thread, the system now becomes *asynchronous* as all of its work on a single call path is not being done sequentially — it does something, then hands over IO related tasks to one or more worker pool, and then comes back to resume execution from that point onwards (having done some completely independent task in between). Threads on the worker pool still gets blocked for IO, but now only the threads of this pool get blocked, thereby limiting the cost to the system. Other code paths of the system which do not involve IO activity are scaled-up by the freed caller thread. System throughput increases considerably because the calling thread doesn’t sit around waiting for IO to complete — it can perform other computations. ![](https://kislayverma.com/content/images/2020/07/thread-based-async-prog-1.png) A good analogy for understanding this behaviour is that of a checkout counter in a shopping mall. A small number of checkout counters are able to handle a large mall of visitors so long as not every one comes for checkout at the same time. Only a small number of workers (those at the checkout counter) are blocked on the checkout function — other workers are free to assist shoppers. How many shoppers could be accommodated in the mall if a worker had to be attached to a shopper from the moment they entered the mall till checkout? A more technical analogy is that of a connection pool, e.g. database connection pool or TCP connection pool. In a service, we could have all threads that want to call another service create their own RPC connections (let’s ignore the connection creation cost) and fire their own API calls. However, so long as not all threads need to access the other service at the same time (i.e. there are other things the system has to do), we can create a small worker pool of RPC connection and funnel all API calls through them and free the calling threads of this blockage. By multiplexing the calls over this small thread pool, we can free up a lot of other threads more doing non-IO related work. This is exactly what happens when we use [Apache Async HTTP client](https://hc.apache.org/httpcomponents-asyncclient-4.1.x/index.html?ref=kislayverma.com) or others of its ilk. ## Handling task completion We have so far spoken about off-loading of work to worker threads. The other equally important aspect of the asynchronous model is the interrupt-based program execution pattern. Having offloaded its task to a worker thread, the calling thread needs to know where it was in its call path when it receives the result of the task from the worker. But tracking runtime state is a problem. Where is it to be kept and how? This problem is typically solved by introducing *callbacks* or *callback handlers* which are methods to be invoked by the worker thread on completion of the task given to it. The calling thread registers these callbacks to the *Future* returned by the worker pool and the language/framework can now easily track and invoke them on the calling thread by issuing an interrupt to it (to get it to stop whatever it was doing) and instructing it to execute the relevant callback. The handoff look something like this: Thread 1 calls worker pool to give it a task.Thread 2 in worker pool executes the task and invoke callback.Thread 1 gets an interrupts and switches to executing the callbackDifferent languages give different callback provisions, but variants of *onComplete and onFailure* are the most common. As the name suggests, these are invoked on success and failure of the task given to the worker pool. --- --- ### Coding Time! To make things clearer, let us look at some (pseudo) code in Java. In the one-thread-per-request model, an HTTP API call to external service might look like this. ``` public class Client { public Response get(String url, Request request) { // API calling logic } }public class CallingClass { private Client client = new Client(); public void call() { String url = “some-api-url”; Request requestData = new RequestData(); Response response = client.get(url, requestData); LOG.info(“Got data {}”, response); } } ``` The thread running this the *CallingClass* code will invoke the API via the client, then wait for the response to come back from the remote server so that it can unmarshal it to the response object. It would then log it and go on executing further instructions. All very familiar. In the work-stealing style, the client contains an internal thread pool to which all requests are submitted. [*ExecutorService* ](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ExecutorService.html?ref=kislayverma.com)is the recommended way in Java though you can, of course, hand roll your own. The caller thread is returned a [*Future*](https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html?ref=kislayverma.com) (check out the differences between futures and promises elsewhere - deliciously confusing!) which indicates that a task will be done in the future and the caller notified. Threads of the client pool now execute the call with same blocking behaviour but hidden from the caller thread, which has, in the meanwhile, go on to serving some other request. When the response object is ready in the client or the call is known to have failed, the *Future* is completed and the calling thread interrupted to execute completion/failure handlers of the *Future*. ``` public class AsyncClient { // Create thread pool of size 5 with task timeout of 300 ms private ExecutorService workerPool = new ThreadPoolExecutor(5, 5, 300, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<>(10)); // The task to handover to the work pool private class ApiCallable implements Callable { private String url; private Request request; public ApiCallable(String url, Request request) { this.url = url; this.request = request } @Override public String call() throws Exception { // API calling logic } } public Future get(String url, Request request) { return workerPool.submit(new ApiCallable(url, request)); } }public class CallingClass { private AsyncClient client = new AsyncClient(); public void call() { String url = “some-api-url”; Request requestData = new RequestData(); Future responseFuture = client.get(url, requestData); responseFuture.onComplete() { // callback handler for successful future completion LOG.info(“Success with data {}”, response); }.onFailure() { // callback handler for future completion failure LOG.error(“API call failed with response {}”, response); } LOG.info(“Moving on immediately”); } } ``` This code will log “Moving on immediately” before any of the other log messages, and if we log the thread name inside the callback handlers and inside the *call()* method of *AsyncClient*, we will see that they are being executed on different threads. Any number of threads can execute the calling class code, but as long as the *AsyncClient* can complete a given task in less than 300 ms, they will all remain free to do other things while their *Future* is not completed. This is how asynchronous programming helps us achieve massive system scale. ### Let’s do this, then? Inspite of the massive scale systems designed with dedicated thread pool can handle, there are a few problems that we need to be aware of before jumping onto thread-based asynchrony with both feet. Some are practical problems related to code writing and maintenance, others are somewhat more philosophical. #### That code looks weird! To most programmers weaned on old-school Java programming, this callback style of coding smells like Javascript, and so [Evil](https://benmccormick.org/2018/07/04/evil-javascript/?ref=kislayverma.com) by definition. This code can get seriously complicated to understand and debug once you have a few parallel or serial task handoffs happening to create nested callbacks (aka [callback hell](http://callbackhell.com/?ref=kislayverma.com)). This is one of the biggest problems in all asynchronous programming in any language. Another smaller problem (in Java-ville, at least) is that [*ThreadLocal*](https://docs.oracle.com/javase/7/docs/api/java/lang/ThreadLocal.html?ref=kislayverma.com) variables no longer work. Since the calling thread hands off the work to another thread and moves to other tasks, any context stored as a *ThreadLocal* (think request context in many services) is lost. The only way to propagate it is to explicitly pass it along as a parameter in the task handoff, which often results in ungainly APIs that accept an explicit yet opaque “context” parameter.e.g. Instead of the neat ``` Future response = client.get(url, request); ``` we get this, where it isn’t clear what the context has to do with anything. ``` Future response = client.get(context, url, request); ``` #### Did we really change anything? Going back to the shopping mall/connection pool analogy. there are two behaviours worth noticing: 1. Members of a worker pool are committed to a specific task/set of tasks — even if no shoppers buy anything, the checkout counters still have to be manned. or if we add a caching layer upstream resulting in fewer DB queries, the DB connection pool would still be maintained. 2. Size of a pool is determined not only by the amount of work it needs to do, but also by the nature of the work. e.g. If all shoppers start coming to the checkout counters quickly, more workers would be needed to handle them. These behaviours point us towards the main drawback of the work stealing approach — the size of each worker pool has to be continuously tweaked manually as the operating environment of the system changes(scale, new tasks, modified older tasks etc). The throughput of a system built using thread pools can be deterministically computed only if the nature of the game remains exactly the same. We have to constantly monitor if the nature and volume of tasks allocated to each worker pool are changing, and if so, what should be the new thread allocation. This need for constant supervision leads us to a still deeper insight — we have not changed the fundamental programming model at all!!! The calling thread is merely pretending to be unblocked but essentially its block has been pushed across to some other thread. IO is still blocking and threads are still needed for each IO task — even though our original problem statement was to remedy this very thing. Work stealing gives us significant improvements in system scale and resilience towards runtime fluctuations, but it is focussed on isolating and containing the problem of thread-blocking. It does not remove the root cause but rather moves the blocking around smartly to mitigate the damage. A fundamentally different approach to eliminating this problem is to use a true non-blocking IO paradigm (aka NIO. aka event-based asynchronous programming, aka reactive programming), like [node.js](https://nodejs.org/en/?ref=kislayverma.com) or [Vert.x](https://vertx.io/?ref=kislayverma.com), where threads are never blocked on IO and we have no need of creating and maintaining worker thread pools. We will look at this paradigm in a follow-up blog post. The term “work stealing” highlights that the worker pool “steals” some work from the calling thread. The term “bulkheading” comes from a shipbuilding analogy where the bottom of the ship is diving into watertight [*bulkheads*](https://en.wikipedia.org/wiki/Bulkhead%5F%28partition%29?ref=kislayverma.com) which prevents water from spreading all across the ship in case of a hull breach — the programming equivalent of a worker pool isolating all other threads from having to do a certain type of task. Read Next : [Event Based Asynchronous Programming](https://kislayverma.com/event-based-asynchronous-programming/) ### Overcoming IO overhead in micro-services URL: https://kislayverma.com/overcoming-io-overhead-in-micro-services/ Last updated: 2020-07-21T14:42:30.000Z One of the biggest overheads of adopting a micro-service architecture is the cost of inter-service communication. The overhead comes in many forms : the latency overhead in network calls, failure of deep call stacks and error handling in distributed states etc. But to my mind, one of the most insidious costs is paid by each service in the resources that are wasted in waiting for completion of network IO. ![](https://kislayverma.com/content/images/2020/07/IO-overhead-in-microservices-1.png) You know how the story goes — Service A makes a call to Service B, and the thread on which the call was made waits around till the response from Service B is received, after which the sequential execution of code begins again. Also known as the one-thread-per-request model, this is the prevalent programming model in most programming languages and frameworks (barring very few). ### What’s wrong with a blocked thread? In computational terms, a thread is a very expensive resource. Some people find this statement strange, since the textbook definition of a thread is a “lightweight process”. How can a thread be expensive? The answer to this lies in hardware and programming models. A thread is a unit of computation, and only one thread can run on a CPU core at a given point of time. This means that though we can extract a lot of juice from our CPU cores and OS using smart scheduling algorithms, having a lot of threads running in an application will eventually mean that most of threads are just stuck, waiting for their turn at being scheduled at the CPU, and eventually this leads to the application grinding to a complete halt which can only be resolved by a restart. #### IO blocks are the worst If your application does a lot of calculations and therefore needs a lot of CPU, there is no way to scale it without adding lots of cores. What we are fretting over here are threads blocked on IO (data transfer over the network/database IO/file IO etc). Threads involved in these do not need the CPU, and yet interfere with scheduling by getting blocked.As it turns out, this is not really an unsolved problem today, and two mainstream computational models are available to address this. ### User Space/Lightweight Threads This model separates *user space threads* (threads started and used by our application) from *kernel threads* (those managed by the OS and running tasks on the CPUs). This model maps multiple user space threads onto a single kernel space thread to achieve a sort of multiplexing over finite number of CPUs. [Quasar framework](https://docs.paralleluniverse.co/quasar/?ref=kislayverma.com) in Java and co-routines of Golang and Kotlin make use of this paradigm to achieve high concurrency. User space threads can be swapped in and out effectively because doing so does not involve the full context switch as is required for a Kernel thread. As a result, when a user space thread of Service A is blocked on calling Service B, the scheduler swaps it out for another user space thread very cheaply without disturbing the underlying kernel thread — thereby reducing the amount of switches going on in the system. ![](https://kislayverma.com/content/images/2020/07/resource-funnel-1.png) The whole architecture looks like a funnel, with a large number of user space threads multiplexing over a small number of kernel space threads, which in turn multiplex over an even smaller number of CPU cores. The advantage of using this model is that the programming language and the OS do all the heavy lifting around making execution light weight. The programmer has to pay limited attention to how this is achieved and she can continue writing her program in the usual, linear way. The learning curve is limited to learning the right programming language (e.g. Go) or using the needed framework. The cons are two fold. If all the application code is blocking, then at some point of time the user space scheduler will start running into the same “too many threads” problem as more and more threads are created. Additionally, there are certain method calls in this model that will block a Kernel thread even when they are invoked from a user space thread. The programmer has to be careful to identify and avoid them. --- --- ### Asynchronous Programming This model, made popular of late by Javascript’s [Node](https://nodejs.org/en/?ref=kislayverma.com) framework, refers to a style where program execution is not linear. Threads submit their tasks for execution (in some way) and move on to other tasks without waiting for its completion. They are then notified when the task is complete and they can resume execution of code from that point onwards.The advantages of this model are obvious, but the actual implementation details are tricky. How do we “not” block when the programming language uses a linear execution paradigm? How do we again start executing “from that point onwards”? How is the thread stack and memory to be managed? Two flavours of asynchronous programming have arisen to handle the various difficulties to different extents. #### Thread Based Asynchronous Programming ![](https://kislayverma.com/content/images/2020/07/thread-based-async-prog-1.png) The [*thread based model*](https://kislayverma.com/programming/asynchronous-programming-with-thread-pools/)*,* also known as the *work stealing model*, tries to achieve a semblance of asynchronicity by defining different thread pools for different tasks and having threads hand over tasks to the correct pool for actual execution and unblocking themselves. The designated pool carries out the blocking task (incurring the same blocking overhead as the traditional programming model, but over a limited set of threads) and then notifies the original thread of completion. #### Event Based Asynchronous Programming This model uses kernel capabilities to make processing truly non-blocking for all thread. A thread starts an IO task and registers it with the kernel, stashes away the entire call stack, and goes on to do other things. The kernel watches all the tasks (thereby reducing blocking behaviour to one thread in the entire system) and notifies the submitting thread with the result when the task is complete. No application thread is ever blocked — node.js servers famously employ only a single thread! Both these models rely on *callbacks* to handle the “interrupt” of task completion. The control flow of the program is not linear anymore but goes goes from task initiation to the callback for task completion. The code typically looks “*functional*” or “*streaming*” in nature, with callbacks serving as nodes and work stealing handoffs/OS interrupts serving as edges of the stream. In large scale programs, the callback programming style itself becomes a drawback, as it becomes more and more difficult to debug with control flow hopping all over the place. ### Which one? Both. Both these style of scaling applications have their own advantages, and can be applied individually or together to develop massively scalable applications. Golang uses the user space scheduling model effectively in its channel and co-routines to achieve massive levels of concurrency. Node.js is the poster child of event driven style and serves at many places as the tool of choice for building API gateways. [Akka](https://akka.io/?ref=kislayverma.com) uses both light weight threads and asynchronous code to run tens of thousands of actors on a single machine. In subsequent posts, I will explore the asynchronous programming models in greater detail. **Read Next** : [Asynchronous programming with Thread Pools](https://kislayverma.com/programming/asynchronous-programming-with-thread-pools/) ### Being Fast or Getting Faster? (aka Build Momentum, not Velocity) URL: https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/ Last updated: 2026-07-22T12:47:27.000Z With technology becoming a large, often critical part of any large business today, businesses always want their technology NOW — to prove product market fit, to leapfrog ahead of competitors in new offerings, and to stay ahead of newcomers by building an unassailable technical moat. Needless to say, the market always wins, and pure technological concerns are often left behind in favour of achieving the next critical business goal. As engineers, we don’t always do, or get to do, the best technical thing — we often settle for meeting the end goal. Repeat this story of developers building the just-enough solution writing not-so-good code using the not-best-suited tech stack over a couple of decades and we end up with a collective belief that ***good code and short timelines are opposites of each other,*** and never the twain shall meet. I want to question this folk wisdom by calling out that there are caveats to this, and there are now well proven ways where we can (and teams have) delivered consistently good technology in great timelines. The thing with *“good code takes longer”* is that it holds good only if we are in the building game for a short period. Short term is fine for early start-ups, who don’t know if they will exist the next year. They should focus on doing whatever it takes to get a foot in the door and finding their first customers. But if a team is past that stage and knows that it is there to stay (at least for while), different rule apply. As a team, we should of course be as fast as we can today. But I would argue that the actual goal should be more ambitious— ***the goal should be to get faster over time***. ***Build momentum, not velocity.*** ![](https://kislayverma.com/content/images/2020/07/build-momentum-not-velocity.jpeg) Let’s be clear, doing good work does take longer. The current problem has to be thought through, future problems have to be anticipated, architecture has to be figured out for functionality and scale, technologies have to be evaluated and selected etc etc etc ad nauseam. All this is a lot of work. What this means in the context of our current discussion is that if we want to be fast right away, the only way to do it is to do a shoddy job. ***Simply “be fast” is a poor goal for a team***. Now consider what happens if we explicitly state our goal in terms of getting faster over time. How can we get faster when business problems coming at us are of increasing difficulty? The way I see it, the only choice is to ***not do some things*** so that we get to the finish line quicker. However, we cannot drop the feature requests because that’s the job. So we must drop some technical things. And the only way to drop technical things but still do a good technical job is to make sure that we are able to build on what we have. We build something once, take our time about it, and re-use it every time in the future. ***Build momentum, not velocity.*** --- --- Some of you will now realize that I’m getting at building [platforms](https://kislayverma.com/category/platform-thinking/) (Yes. [Again](https://www.kislayverma.com/technology/how-to-build-a-technology-platform/?ref=kislayverma.com). I know). The above statement is a paraphrasing of the [golden rule of platforms](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com) (which is to build once and re-use in every relevant use case). But the idea of “get faster via DRY” is more widely relevant and platformization is only one of its by-products. We can apply it at multiple level of abstraction with some pretty cool results. At the code level, good code leads to libraries and abstractions which can be re-used for faster intra-component feature development. Team members never have to write the same thing twice because it is already available in an easily consumable way. Clean interfaces and clear, extendable behaviours ensure that we only work on the critical new stuff. This level of abstraction imposes the least penalty on current speed. At a component level, the getting faster principle leads to [service based architecture](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com) (or similar things) where technical capabilities are made available for reuse across multiple teams. It is slow at first, but if we do it well and do it right, next steps become ever faster as we isolate the changing parts from the non-changing ones and can focus on the changes needed now. Again, we are sacrificing current speed for future turbo-charge.At an even larger scale, we are looking at [reusing entire domains](https://www.kislayverma.com/platform-thinking/why-you-should-build-a-platform/?ref=kislayverma.com) across the organization to come up with new business propositions. This is the platform play at it most potent — an organization which can compose business ideas whole cloth out of existing tools/teams/processes can respond and innovate faster than its rivals in the market. This kind of strategic advantage is hard to beat. Obviously, getting into this mindset and executing well on it is a very very hard thing to do. ***Build momentum, not velocity.*** None of the above means that the things that were built earlier are absolutely never modified again. The idea is to modify them as little frequency as possible. The idea is to “do fewer things to move faster” as we move forward, and we have to be willing to pay the occasional fine as long as the average feature delivery time is shrinking. Notice that the above technical principles are not emerging from an explicit technical directive, but from a kind of *management principle* of getting faster over time! Ergo, this is an organization pattern. ***We are extending the size of the team and trying to identify the things that team has already done and should not do again***. The core capabilities and processes are well known, and can be used in varying contexts without fundamentally altering the org. The undercurrent powering this is a conscious eye for “composability” of capabilities. If a system (tech/non-tech) cannot be used to compose something bigger than itself, it will end up being re-invented elsewhere (i.e. that which is platformized will defeat that which isn’t). This is clearly not the way to being faster. We should take the time now to ensure if the class, the service, the infra that we are building has enough functionality to provide business value and enough flexibility to mould itself to other use cases. So don’t repeat yourself. Let the time you spent in the past be an avalanche to power your future ***Build momentum, not velocity***. **Read Next** : [Leverage agile methodology for more than execution excellence](https://kislayverma.com/agile-for-innovation-going-beyond-execution-excellence/) ### Working in the seams of code with Resource Locator URL: https://kislayverma.com/working-in-the-seams-of-code-with-resource-locator/ Last updated: 2026-07-22T12:47:28.000Z The term “Seams” was introduced in popular language by Michael Feathers in his excellent book [*Working Effectively with Legacy Code*](https://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052/ref=sr%5F1%5F1?crid=3CDGVYC6CT709&keywords=working+effectively+with+legacy+code&qid=1554780196&s=gateway&sprefix=working++effectively+%2Caps%2C358&sr=8-1&ref=kislayverma.com) as a place where we can alter behaviour in a program without editing in that place. Alternatively, a seam is a place in the structure of an application where two components meet, and hence a place where the interaction between them can be tested. ![](https://kislayverma.com/content/images/2020/07/seam.png) The term *seam* comes from the tailoring world as a reference to a place where two pieces of cloth meet and are stitched together. Since a seam is where two components of the application interact under well-defined rules, it makes a great place for testing since we can replace one component with something else (mock) without impacting the behaviour of the other. Intercepting the workflows across the seam makes for excellent integration testing. From a software development perspective, this is an elaborate way of saying “code to interfaces, not implementations”. However, there is another dimension to this concept. There are two ways in which software evolves — one is by a change in functionality and the other is by the addition of more functionality. In both cases, though, our concerns around testability remain the same. This is because when we test an interaction at a seam, what we are effectively testing is an abstraction that should hold regardless of changing implementations. The way we could handle a change in functionality seems simple enough — modify the called side of the seam and make sure that all tests work after the change. Makes sense. How do we then add more functionality or multiple flavours of a given functionality? Changing the current implementation violates the [*open-closed principle*](https://en.wikipedia.org/wiki/Open%E2%80%93closed%5Fprinciple?ref=kislayverma.com) of object-oriented programming and can surely lead to breaking code (even if inadvertently). The safest thing would be if could write some new code for this functionality and somehow invoke it with no changes to the existing code. But how to do this? --- --- I like to introduce new functionality by writing multiple implementations of the same interface on the called side of the seam. This introduces boundaries orthogonal to the earlier boundary. However, these are very different from what we were talking about before. These act more like isolation chambers or silos — there is no interaction across these, stuff on one side isn’t even aware that there are others like itself on the other side, and we can keep adding more of these, thereby continuously widening the variety of functionality offered. So to add new functionality, we just add one more implementation and stick it between the existing ones. There are our *seams of extension*. However, now the caller has a problem. How should it invoke one or more of the many implementations available without changing a lot of code? ![](https://kislayverma.com/content/images/2020/07/sems-of-execution.png) Enter [*Resource Locator*](https://en.wikipedia.org/wiki/Service%5Flocator%5Fpattern?ref=kislayverma.com) (aka *Service Locator*). We introduce a resource locator to which all implementations register itself, and we expose this to the caller instead of having him bind directly to the interface (via dependency injection etc). The resource locator now allows the caller to get access to “named” instances of the implementation (names being unique) and hence putting the caller in control of what code to invoke. The locator thus acts a bridge across the seam, allowing the caller the choice of picking the appropriate implementation out of a multitude of them. Once the name instance is return by the locator, the caller can invoke it in a similar manner as before, and we should not see any behavioural difference. ![](https://kislayverma.com/content/images/2020/07/seams-resource-locator.png) The overall effect this has on the code structure is that we can continue to add more and more functionality to our code without having to modify existing code. It also keeps intact the contract between caller and callee, so that the testability across the seam is preserved. In fact, mocking code for testing purposes becomes even easier since now there is a central access point (the resource locator) that can be manipulated to give whatever implementation is needed for testing. There is a change in the nature of the coupling as well — the caller was previously unaware of the implementation on the other side of the seam, but now it has to ask for some specific implementation. i.e. it is directly aware that there are multiple available flavours to choose from. This awareness is not always desirable, hence this kind of design should only be used for scenarios where dynamic conditions on the calling side dictate what implementation to use, and hence we cannot bind the implementation to the caller at compile time. It is also useful when building code superstructures that require multiple implementations of the same interface to be chained by some framework code (request filters in web servers are a good example of this). The resource locator+seams combinations allow us to write more and more implementations and adding them to the chain by adding their names to some configuration that the framework can read and process. **Read Now** : How [continuous refactoring](https://kislayverma.com/saving-the-day-with-continuous-refactoring/) can help your team and codebase. ### Agile for Innovation - Going Beyond Execution Excellence URL: https://kislayverma.com/agile-for-innovation-going-beyond-execution-excellence/ Last updated: 2020-07-21T14:19:33.000Z *tl;dr — Incremental requirement definition and technical design in agile sprints allows us to use Agile methods for exploring innovative solutions for our customers. Also, your product manager and architect will love you.* What I am saying is nothing new. It is sold as “Agile Transformation” for organizations by hundreds of coaches and consultants. This is just my tech-centric, mostly jargon-free version of it. ![](https://kislayverma.com/content/images/2020/07/agile-going-beyond-execution.jpeg) In most of the teams that I have worked with, the Agile method is used by developer teams as a tool for improving execution excellence and visibility. We break down tasks into stories that can be delivered in a 2 week sprint, retrospect at the end of the sprint to assess what went well and what did not, and try to get better over time. Full [kaizen](https://en.wikipedia.org/wiki/Kaizen?ref=kislayverma.com) for all things code. However, two stages of the software lifecycle are often left outside this system. One is the product manager’s job of writing the spec, and the other is the Architect’s job of defining the overall architecture of the system. These are not bound by sprint boundaries and are often expected to be fully completed before dev teams start writing code. This BRUF (Big Requirement Up Front) and BDUF (Big Design Up Front) is what I want to talk about in this article. The product manager gets the worst part of the deal, especially for large projects. She is expected to spec out in very great detail exactly what the dev teams are to build. She is needed to do the impact analysis to convince managers that the project is worth doing, think of a broad vision of how the project is going to evolve over a multi-year horizon for the technical architect, define the customer workflows with designers, and figure out the innumerable minutiae of the frontend/backend behaviours BEFORE THE TECH TEAMS EVEN LIFT A FINGER! Never mind that the business teams often only have a rough idea of what they want. Never mind that we are not even sure whether any of this works in the real world. All of this tremendous body of work is required to be finished at the outset before engineering even thinks that the project has started (it is still “being figured out by them” till then). The architect fares a little better. She is the first consumer of the requirements document and has to come up with a design for solving the problem. The problem is that design is expected to be done in great detail and all at once. Never mind that the spec itself is a figment of someone’s imagination at this stage. Never mind that most likely there will be a ton of changes as we go into execution. The design has to be frozen and signed off in its entirety. Already, we have introduced two HUGE sources of inflexibility in a supposedly *agile* system. It gives us a false comfort that we have figured out the entirety of what we want to build (requirement) and how we want to build it (Architecture/Design). There is no more ambiguity, no further exploration of the problem or solution spaces required. All that remains is efficient execution for which we want to use agile-like techniques. Let’s start cranking the wheel. ### Agile is as Agile does But first, a short aside on what it means to be *agile*. I take “Agile” at face value — make the organization *literally* agile by enabling it to move fast towards a known target and to quickly change direction towards a new and/or moving target. Agile can serve for great execution, but once we free ourselves from the fake comforts of BRUF and BDUF, we can employ agile for so much more. --- --- ### Agile for Innovation The problem is that we are applying agile principles only in narrow pockets instead of applying them end-to-end. Once we internalize the idea that software is meant to be changed, that it is never “done”, then we will realize that it doesn’t really make sense to try to define the entire product right in the beginning. A better model would be to employ agile as a method for learning about the problem and solving it one step, one sprint at a time. This is not to say that we shouldn’t take a big leap when we see it, just that we shouldn’t commit to a course when we have much to learn. The incremental execution methods of agile can, in this exploratory mode, be turned into innovation as we move towards a deeper understanding of our customer’s problems without over-committing ourselves to any one course of action. We can simply allow the best solution to emerge over time. ### The proposal At the beginning of the project, the product manager should be required to present/document the current state of the union and the broad strokes of the complete solution. We should know the different aspects of the proposed solutions and the functional constraints/characteristics of all of them. We should know what metrics we are trying to improve. The architect now provides the broad technical structure of the system and the architectural precepts as she sees them now. Opportunities for re-use and platformization can be identified here, alongside the absolute top-level components and their interactions. This is the extent of up-front work that should be done. Now we enter the sprint cycles and keep iterating over the following steps. 1. The team (including the product manager) identifies one or two most critical aspects of the problem that they should tackle in this sprint. “Tackling” includes giving full features, MVPs, workarounds, bug-fixes etc. 2. The tech team (including architects) deep dive into these few problems and come with detailed technical solutions. 3. The tech team delivers the solution. 4. The product manager adds the stakeholder feedback and other improvements to the backlog. 5. Engineers add tech debt to the backlog. 6. Lather, Rinse, repeat. See how we have given up a fixed goal and are able to address the next emerging set of priorities at every sprint boundary. We are now building software that learns from the real world even as it shapes it. There is room to understand problems a little more and to gradually come to see the solution. There is even time to backtrack and try something else. ### But if I do this, then… When I discuss this with engineering teams, they almost always complain about the lack of clarity around what is to be done and an unclear vision. Also that they will end up doing throw-away work. Also that they will amass tech debt due to frequent changes. Let’s talk about these as all are legitimate concerns. ### Lack of Vision A vision for the product is the idea that it will solve X/Y/Z problem and that we will save money/time/effort etc as we build it. It is not a crystallized set of tasks. In fact, any attempt to do so does not set a vision but actually constrains it to only those tasks. Visions is an organic, amorphous thing defined by the metrics it wants to improve and the rough means of doing so. Hence, it can only be defined in a rough way. Please spare the product manager the quest for endless details before the start of the project. ### Wasted Effort Throw away work always happens since software is always changing. The real question is how much to throw away and how often. Would you rather build a behemoth that is way off requirements and then re-do large chunks of it or would you build what is needed, refining things as you go with some small waste per cycle? I prefer the latter, personally. I would argue that while it develops with a certain amount of wasted work, an architecture that is built incrementally is much more robust as it is built by identifying the emergent qualities of the system and not all-at-once in an ivory tower. It is much more suited to withstand change and evolve gracefully, especially since it itself evolved over time. I am assuming, of course, that the dev team gets resources and pays enough attention to the technical aspects of the codebase as it evolves. If such is not the case, then all is lost anyway, no matter what approach we take. --- ### Caveat Emptor And here I must express an often raised lament against Agile methodology — its short-term thinking. Much has been written about how agile reduces everything to sprints and no one ends up caring about the big picture. I agree with that sentiment and it behooves us to be wary as we go about our work. As with all things in life, there is a balance — and it rests will senior folks to keep an eye out for it. We do not need to know everything upfront — but we do need to know enough that we can see the nebulous half-form of the final thing. We can iterate every 2 weeks but should have some idea of our environment and what we are iterating towards. This bi-focal mindset is the very essence of being a good “senior” in any field of work, and software development is no exception. I would love to hear from you about your experience in using agile techniques beyond just execution excellence. Drop a note! **Read Next** : [More article](https://kislayverma.com/category/agile/) on Agile and agility. ### Choosing a service framework URL: https://kislayverma.com/choosing-a-service-framework/ Last updated: 2026-07-22T12:47:32.000Z tl;dr : Side-step framework wars by defining the implementation and operations standards for your services. Then permit any tools which conform to this standard. **The standard IS the service framework, not the technology in which it is implemented**. ![](https://kislayverma.com/content/images/2020/07/service-framework-cover.jpeg) A lot of this article is inspired by my experiences at [Myntra](https://medium.com/myntra-engineering?ref=kislayverma.com) and [D.E. Shaw](https://www.deshawindia.com/InformationTechnology.shtml?ref=kislayverma.com) and by the numerous articles and talks from engineers at Netflix. Spring Boot recently became the [Java development framework of choice at Netflix](https://medium.com/netflix-techblog/netflix-oss-and-spring-boot-coming-full-circle-4855947713a0?ref=kislayverma.com) . This comes after years of the OSS community embedding bits of Netflix OSS like [Hystrix](https://github.com/Netflix/Hystrix/wiki?ref=kislayverma.com) , [Eureka](https://github.com/Netflix/Eureka/wiki?ref=kislayverma.com) into frameworks like Spring. Internally, however, Netflix used home-grown frameworks to build their services. A similar situation exists at many companies large enough to have many services and many teams. They want standardization of development, deployment, and monitoring styles. One of the easiest ways of achieving this is by adopting a *service template* or *service framework*. ### What is a service framework A *service framework* is exactly what it says it is — a template/blueprint for writing new services. The template gives out-of-the-box scaffolding and tools which facilitate rapid application development by providing what may be thought of as a “ [paved road](https://www.slideshare.net/diannemarsh/the-paved-road-at-netflix?ref=kislayverma.com) ”. This allows developers to focus only on writing their business logic and saves them the bother of how to structure the applications for build and deployment, or how to integrate against the company’s request tracing system. Despite the emergence of multiple open-source service templates ( [Spring Boot](http://spring.io/projects/spring-boot?ref=kislayverma.com) , [Play! framework](https://www.playframework.com/?ref=kislayverma.com) , [RestEasy](https://resteasy.github.io/?ref=kislayverma.com) …), companies often end up implementing their own because they want just a little bit of customization and it’s easier to roll your own (in the short term) instead of modifying something open source. The problem with this approach starts when teams want to try out some other framework or a different programming language. Now we are stuck with teams publishing metrics or other telemetry in their own way, or not being able to use the build and release systems and building their own etc. To counter this behaviour, “architecture committees” start coming out with lists of accepted technologies, explicitly whitelisting what can be used. This then annoys developers who want to play around with their favourite tools. ![](https://kislayverma.com/content/images/2020/07/you-shall-not-java.jpeg) We want our development teams to use the best tools/frameworks/language for their work. To achieve a certain basic amount of consistency and standardization in such a poly-everything world, how do we pick the one “best” framework? How do we get the best of both worlds — standardization of tools as well the ability to experiment with technologies? ### Standards over Frameworks The answer lies not in the choice of frameworks but in imposing constraints around the output generated by their use. I argue that as long as we can define and enforce certain guidelines around behaviour of the system (basic internal structure, deployment, observability), it doesn’t matter what framework or technology is being used to build systems. If defined *just* tight enough, not only can these guidelines unlock technology governance and experimentation in the short term, they can also drive technology consensus in the long term. In essence, **we want the standard to be the service framework, not the technology in which it is implemented**. ### Defining a standard I believe that before we start debating template choices and doing feature comparisons, it is important that an organization define the basic characteristics of an application, regardless of the language or framework it is built-in. These characteristics are the minimum set of guidelines that an application should adhere to so that systems look sort of homogenous when viewed at org-level, team level, or service level. The guidelines could include mandates around: 1. Code bootstrapping and structure. 2. Integration with application platform components like log aggregation, service discovery, CI/CD etc. 3. Published telemetry. These guidelines are the minimum feature-set that any service template used in the organization needs to support. The constraints around experimentation are now set, and only this set of rules has to be adhered to. Conversely, this standard, no matter how minimal, MUST be followed. This makes using new frameworks frivolously an expensive proposition and should add the requisite friction and discipline over the long term. --- --- ### Pieces of the puzzle A service framework consists of three major parts: 1. The service scaffolding defines the overall structure of the code and deployables. 2. Utilities (middleware etc) to integrate against application platform. 3. Utilities to integrate with the framework/protocol being used (e.g. Servlet filter and interceptors, HTTP clients). As should be clear, the more evolved a company’s application/devops infrastructure is, the wider the template definition will be, and the more the work needed to use new technologies. This is because more utilities are now needed to satisfy #2 above. ![](https://kislayverma.com/content/images/2020/07/svc-framework-pieces-of-the-puzzle.png) What follows is my opinion of a basic service template definition. I have put only a few things to set the stage. More rules can be added on a case-by-case basis. #### Application Bootstrapping 1. Developers should be able to start a new service with nearly no work: Either via a CLI-based bootstrap tool, via maven’s pom inheritance (similar to Spring Boot), or some other mechanism. Forking a skeleton service and then manually changing all configurations (like application name, deployment paths etc) is the worst way of doing this. 2. The bootstrapped service should immediately deploy and run locally. 3. It should be structurally ready for deployment via the CI/CD pipeline. #### Application Structure The overall application structure should be as follows. ![](https://kislayverma.com/content/images/2020/07/svc-framework-application-structure.png) 1. Service layer: This is the first layer of application code in the synchronous call path and defines the semantics for the respective API mechanism. 2. Message Listener layer: This is the asynchronous access path and equivalent to the service layer in that sense. 3. Manager layer: This is the business logic layer and the place where most of the application code would go. 4. DAO Layer: This is the data access layer. All layers are seeded by default (Base\*) implementations which provide sensible defaults for CRUD operations. #### Application Integrations 1. Separate utilities should be provided for integration with each piece of applications infrastructure. They should be separate so that they can be included individually as per need and not all at once as a big package. 2. Utilities should encapsulate the standards defined by each infrastructure component and best practices around them. 3. The utilities should be baked into the Base\* structural classes as required by the minimal standard defined below so that everyone extending those classes has the integration behaviour by default. 4. All integration utilities should be integrated with a common profiling utility so that they too, publish a set of useful metrics out of the box. At the very least, utilities should be provided for synchronous and asynchronous invocation of other services and for ad-hoc instrumentation of code. #### Service invoking component 1. Shouldn’t abstract the details of the remote API like protocol, path etc. The details of the target service should be visible to the caller to be able to configure the interactions effectively. 2. Should abstract the user from the service discovery mechanism (DNS and/or central load balancer also counts as service discovery). 3. Must provide configurable connection pooling and timeout configurations to invoke remote service. 4. Must provide a configurable circuit breaker to safeguard the user in case of remote service failure. 5. Must propagate/generate request correlation id via the standard mechanism (header/payload etc). #### Message publishing component 1. Shouldn’t abstract the details of the messaging system like protocol, path etc. The details of the message broker should be visible to the application to be able to configure the interactions effectively. 2. Should abstract the user from the service discovery mechanism (DNS and/or central load balancer also counts as service discovery) used for discovering the message broker. 3. Must provide configurable connection pooling and timeout configurations to publish to a message broker. 4. Must provide a configurable circuit breaker to safeguard the user in case of message broker failure. 5. Must propagate/generate request correlation id via the standard mechanism (header/message properties etc). #### Instrumentation Utility 1. This is the core component using which all instrumentation is published.It integrates with the metrics management system of choice and with the prescribed standards. 2. It should be able to publish counts and times. 3. Instrumentation failure should not cause application failure. 4. This component applies no data aggregation sampling. That is done by the metrics system. #### Other useful platform components Beyond these basics, some utilities that would be very useful to have baked into your service would be: 1. Service Discovery 2. Authentication and Authorization 3. Distributed request tracing 4. Rate Limiting 5. Log Aggregation As you build and evolve these foundational capabilities, simply upgrading your integration utilities via template version upgrades can keep the developers at the cutting edge of the platform and facilitate adoption of best practices. ### Application Deployment 1. Application deployment should observe the [12-factor principles](https://12factor.net/?ref=kislayverma.com). 2. The deployment process for test environment and for production should be the same. 3. Only difference should be the configuration.CI/CD pipeline should identify certified artefacts and only promote them to production. 4. Artefact management should be done as described by the CI/CD system. 5. Packaged application structure should comply with the demands of the CI/CD system. ### Application Telemetry The following data points should be recorded by the service template out of the box. Message Listener and Service layers should have largely identical telemetry behaviour since both are access gateways for the application. #### Service Layer 1. Request count per API 2. Response time per API (sampling, avg, 90p, 99p etc will be defined and calculated by the monitoring infra) 3. Response code counts per API (e.g. counts of 2xx, 3xx, 4xx, and 5xx responses for an HTTP based API). #### Service invoking component 1. Invocation count per API 2. Observed response time per API. 3. Observed response codes per API(e.g. counts of 2xx, 3xx, 4xx, and 5xx responses for an HTTP based API). #### Message Listener Layer 1. Processing count per message type 2. Processing time per message type (sampling, avg, 90p, 99p etc will be defined and calculated by the monitoring infra). 3. Response code (processing output) counts per message type (e.g. count of success/thrown exception code in a java application). #### Message Producer Component 1. Published message count per message type. #### DAO Layer 1. Count of DB queries per query type (SELECT, INSERT,UPDATE,DELETE) 2. Time taken for queries per query type (SELECT, INSERT,UPDATE,DELETE). ### Conclusion The above rules should offer a good starting point for building a simple service in a basic infrastructure setup. Note that I have not mandated too many platform component beyond a monitoring system. A basic [statsd](https://github.com/etsy/statsd/wiki?ref=kislayverma.com) setup can suffice an organization for a long time for this.If we follow the general rules laid out above, we should be able to come up with an increasingly broad but effective requirements for technology adoption. This will definitely shed more light on the path of adopting a service template which can drastically improve developer productivity and promote adoption of tools and best practices in one go. Let the framework wars begin !!! Read Next : [Code review guidelines for distributed systems](https://kislayverma.com/code-review-checklist-for-distributed-systems/) ### How to build a technology platform URL: https://kislayverma.com/how-to-build-a-technology-platform/ Last updated: 2020-07-20T17:01:27.000Z This is a rumination on how we are thinking about building Myntra’s logistics capabilities. The attempt is to develop a mental framework for building software systems and teams that can be applied to any problem domain. Let’s try to build a logistics technology platform for the world! ## What platform? SaaS platform for logistics companies. Out-of-the-box modelling and default behaviours for logistics entities. Standard method of customizing core behaviour to create new experiences without modifying the platform. ## Why platform? What are the problems in building a software system for the long term? 1. Changing processes : Frequent changes and even complete overhauls of the logistics operations model. 2. Evolving business : No one can predict *how* the business will evolve. 3. Reducing go-to-market : Expectation of ever increasing agility and reduced time to market despite the above. Given this, how do we write software that maintains/scales well and helps us keep pace with the business? We have two important observations here. 1. Technical solutions built for specific business problems are not reusable in a fluid business landscape. 2. We encounter different behaviours for the same entity more often than we encounter new entities. [A platform approach solves this unknown-unknowns dilemma](https://www.kislayverma.com/platform-thinking/why-you-should-build-a-platform/?ref=kislayverma.com) neatly by creating a reusable set of business tools which can be arranged in different configurations to achieve new and varied outcomes. ## But we already have a platform! No we don’t. We often use “platform” to mean “this system/suite of systems lets us do everything related to xxxxx”. Replace xxxxxx with taxation, discounts, logistics etc. This is not a strictly correct definition of a platform. Ideally, a platform allows *others* to do xxxxx in a minimally opinionated way. What we have is a [product that is trying to fill the shoes of a platform](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com). Our software is built around the business *as we know it today*. Changes to business will either result in ever increasing “if-else” or recurring rewrites.It is difficult to change behaviour of the system without touching core codebase. It is near impossible to evolve software at the edges. We have something like this. ![](https://kislayverma.com/content/images/2020/07/core-non-core-in-products.png) 1. A set of entities and behaviours that we consider *core*, but are then forced to modify for each use case. Different behaviours are considered core for their respective use case. 2. A shallow business logic layer with fuzzy boundaries since we cannot properly define what is core and what is not. 3. A trivial API layer which delegates everything to downstream components and does not abstract callers from the underlying architecture. Because entities and their behaviours are locked together intrinsically, we end up with a system which is being pulled in directions it was never meant to go in. ## Envisioning a true logistics platform ### Characteristics of the platform Taking a leaf out of [Jeff Bezos play book](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com), we define our platform to have the following characteristics: 1. Built first, then reused forever : Has to be built agnostic of specific business needs, and then used for addressing business needs. 2. Externally programmable : Must expose hooks to build customizable experiences on top of it. 3. No exceptions to the above two. ## Getting there As we go about building a platform, how will we know if we are moving in the right direction. A few key benefits can be tracked to evaluate this. #### Technical considerations Components are increasingly isolated by rate of change. Some change often and some perhaps not at all.Business decoupling : Fewer and fewer components should be impacted by any business requirement.Single source of data and modelling for business entities.Homogenous development experience and consensus on design and development practices. #### Non-technical considerations Well defined owners of different technical components. No shared responsibility.Increasing team autonomy (aka reduction in inter-owner dependencies).Agility and reduced time to market for business features. --- --- ## Bird’s Eye View Here’s a quick look at Myntra’s technical ecosystem and where our logistics systems are to fit in it. ![](https://kislayverma.com/content/images/2020/07/myntra-lms-birds-eye-view.png) The green boxes are business facing products. Under them, we have a stack of tools to serve an increasing degree of abstraction. Each layer is a platform in itself, compliant to all platform building guidelines. While there is a sense of increasing abstraction from hardware management to business capabilities, this is not really a “stack” since all layers of the “stack” are independently available for use at any level. e.g. Service discovery can be used by CEP system as well as last mile shipment service. The non-logistics pieces are included only for context. Each of them merits a deep discussion all by themselves, but we have a platform to build. So onward! ## Separating product from platform We identify 2 constructs : platform services and product servicesPlatform services are business agnostic capabilities. Product service is any specific experience or workflow built on top of one or more platform services. ![](https://kislayverma.com/content/images/2020/07/separate-product-from-platform.png) #### Platform Services 1. They are not aware of how they are being used in the business context. 2. They provide standard read/write constructs and guaranteed SLAs. 3. They don’t understand the content of workflows beyond their trigger conditions. #### Products 1. Products deliver business value by building optimized end-user constructs like API gateways for data aggregation, UI for manual interaction, and workflows for process automation. 2. They may themselves be made up of multiple components. 3. They may extend platform entity data models by storing local versions of that data and/or storing additional data against it. If we do this right, our software should look like this. Notice the shrinkage of core and increase in size of logic and API layer. ![](https://kislayverma.com/content/images/2020/07/core-non-core-in-platform.png) 1. Small entities with minimal essential states and business validations. 2. Ability to register more states and validations for different use cases, thereby adding extensibility to the entity model. 3. Ability to externally configure workflows to stitch actions across multiple core entities. 4. Handling of infra concerns like rate limiting, authentication etc. The business logic layer would increasingly drop out from within core services onto external workflow management systems. ## Interactions between platform and products 1. Since platform service are unaware of the products built on top of them, they should ideally not have code to invoke those components directly. 2. Platform services can drive cross-service interactions by triggering workflows configured by the product services. 3. Platform service broadcast all their events (business events) over some messaging medium. Other platform services or products can hook into these events to [build functionality in a decoupled manner](https://www.kislayverma.com/programming/using-events-to-build-evolutionary-architectures/?ref=kislayverma.com). 4. While the above is ideal, as a compromise, products can configure the platform service workflow mapping so as to call any other single API (instead of always calling the workflow service API). This can be done for several reasons — the product service may want to encapsulate the workflow logic in its code or the business logic involved does not require a workflow (multi-step process) and can be achieved by a simpler piece of code. 5. Components built for the experience layer may be pushed down into the platform layer as they evolve to serve more use cases in a generic manner. ## Discussing a platform service The main concern in building a platform service is how to make it extensible without modifying it repeatedly, or at the very least to be able to modify it with little danger of impacting existing functionality. We adopt a state machine and workflow driven approach to this. ![](https://kislayverma.com/content/images/2020/07/platform-service.png) Platform service exposes API which has some core functionality. Additionally, it can trigger a custom workflow defined by its users. ![](https://kislayverma.com/content/images/2020/07/platform-service-internals.png) Every platform service is composed of these parts. #### API Layer 1. Authentication 2. Authorization 3. Rate limiting 4. Call routing to correct component. Some of these concerns may be outsourced into an external API gateway, thereby reducing the complexity on the service itself. #### Entity core business logic This logic is uniform across across all tenants/clients etc and pertains to maintaining the sanctity of that entity. We only work with the service’s own entity here. #### Configurable state machine 1. All writes are validated via an extensible state machine. Every entity MUST define a base set of states which can be extended but never reduced. This also defines the set of actions supported on the entity. e.g. Trip service defines CREATED->STARTED→COMPLETED as the core set of transitions that must happen in that order. Myntra trips team might want to define an extra state to make this : CREATED→PENDING-START→STARTED→COMPLETED. Both of these will be configured and invoked while trip updates are happening for Myntra and Store trips respectively. 2. Writes that don’t involve a state transition cannot be validated by platform service. #### External workflow trigger 1. An external API can be invoke to trigger additional workflows as configured for some combination of entity attributes. 2. It only calls the given external end-point with the input given to the service and output emitted by the service. The responsibility of interpreting this data and enriching it further rests with the called API. 3. Since the triggered workflow may call other services which in turn have their own workflows defined, a single event in one service can create a workflow fanout. 4. The platform service owner is responsible for guaranteed invocation of the configured workflow but not for the contents of the workflow. 5. Development and operational ownership of the workflow rests with the team which created it and mapped it in the platform service. This separation of concerns must be honoured EVEN IF BOTH PARTIES ARE THE SAME PERSON. #### Event PubSub This module broadcasts every successful event via a messaging medium. The publication must be guaranteed if an event has happened. ## Building a product experience 1. Product building effort should be centred around solving business problems. 2. We want to build efficient workflows(ui/backend) using platform components as building blocks. 3. New components and entities relevant to this product should be built outside of the platform. 4. All architecture, design practices like scalability, micro-services, security, maintainability etc apply independently to product building just as they apply to any software development. #### API experience 1. Teams building services on top of the platform integrate with platform services via the APIs, workflow service, and event bus. 2. These APIs may be pass-throughs to platform service, thin wrappers on platform services (minor changes to API structure and language), or heavy duty APIs which compose multiple platform services. 3. These services may directly use workflow service and other parts of the “lower platform” offerings for its own purposes. i.e. Not everything must go via the logistics platform layer. 4. The APIs exposed here need not be generic. The focus should be on building effective/optimized API for specific use-cases. #### UI experience 1. UIs can be built directly on the platform services. However, it might be difficult to bridge the gap between the user experience and platform since the latter does not do any cross entity aggregation, nor build any APIs for specific product use cases. 2. Typically the UI team would build a backing API layer to take care of data aggregation and massaging. 3. The backing API would be built as per the API experience guidelines mentioned above. 4. Most product changes are expected to impact only the UI and the backing API layer. Platform components should only be impacted in extreme scenarios. ## The human perspective Let’s look at how we are doing on our metrics so far. Since we have discussed only tech guidelines, let’s look at technical metrics. #### Technical metrics 1. *Components are increasingly isolated by rate of change. Some change often and some perhaps not at all* \- Looking good. Platform service seem to change rarely. Product layer changes often without impacting the platform (Except in terms of scalability etc). 2. *Business decoupling : Fewer and fewer systems should be impacted by any business requirement* \- Changes to business UX or workflows impact only the product layer, and even there the change is isolated by good design. UI changes typically only impact the UI layer and the backing API’s read layer. Operational workflow changes are dealt with mostly by reshuffling of defined technical workflows. 3. *Single source and modelling for business entities* \- The universally acknowledged entities and their data models are all located in the platform. Meta data about them may be scattered across products, but it is only of local relevance. 4. *Homogenous development experience and consensus on design and development practices* \- Since the base data model is uniform and so is the tooling to build workflows, state machines etc, an effective guideline is already in place about how to build new systems. e.g. Shipments look roughly the same in every system, and can be manipulated in a consistent manner. The constraints enforced on product-platform interactions and an increasingly deep uniformity in development tools set a design and implementation standard, any deviation from which is easily detected. This is either corrected or explicitly acknowledged as a genuine requirement. If it is a genuine requirement, it might be a good candidate for assimilation into the platform. ## Organizing teams for platform architecture [Conway’s law](https://en.wikipedia.org/wiki/Conway%27s%5Flaw?ref=kislayverma.com) is [real](http://www.melconway.com/research/committees.html?ref=kislayverma.com). Now that we are able to visualize decoupled software, we need a team structure which can leverage this to achieve agility and scale. An org structure explicitly designed for this essential since developer discipline to do the right thing is hard to enforce over any large team size. We can’t rely on activist developers to keep a large company on the straight and narrow. [Amazon’s two-pizza team](http://blog.idonethis.com/two-pizza-team/?ref=kislayverma.com) is a good working model to achieve this. 1. Create small teams with ownership of very specific technical or business problems. 2. One team officially does not care about teams. 3. A team will not be disturbed randomly for other stuff simply because they don’t have “enough work” in this sprint. It is the responsibility of the team to push the boundary on their particular charter. 4. All teams MUST comply with the strict guidelines around how to write software (product-platform separation etc) which prevent technical anarchy. 5. We also mandate that all internal, re-usable software built by a team should be pushed into the platform if possible. Otherwise, the team must demonstrate that the software isn’t re-usable by definition. ## A logistics example Instead of having a single big LMS team which spans across the entire business and technical domain, we create 5 teams with narrow but deep responsibilities. #### Logistics platform team 1 1. Shipment 2. Master Bag 3. Container 4. Shipment Tracking #### Logistics platform team 2 1. Network (Hub, lanes, and mappings) 2. CourierCourier contracts (including HLP, store etc) and handover configurations 3. Serviceability 4. Staff 5. Trip 6. Geo Data #### Logistics platform team 3 1. Courier Integrations and manifestation 2. Inbound API Gateway 3. Outbound client integrations 4. Capacity Engine 5. Promise Engine 6. LMS inbound API gateway 7. LOVE (Screens till DC handover) 8. Hub App 9. Shipment ,master bag, and container workflows #### MT LMS last mile team (Product) 1. HLP and store handshake workflows 2. LOVE last mile screens 3. MDA The idea is to decouple teams as much as we decouple systems to move ever faster. ## Closing Notes A lot of these principles are being tried out in Myntra right now and are the stuff of hot debates all across our tech teams. We hope to establish a framework for platform architecture thinking which will stand the test of time and guide us on this exciting journey. Read Next : More articles on [Platform Thinking](https://kislayverma.com/category/platform-thinking/) ### Using Events to build evolutionary architectures URL: https://kislayverma.com/using-events-to-build-evolutionary-architectures/ Last updated: 2026-07-22T12:47:33.000Z Evolutionary architecture is software architecture that can be incrementally, continuously, and rapidly changed to deliver new functionality. While this has been common wisdom at lower levels of software engineering ([SOLID principles](https://stackify.com/solid-design-principles/?ref=kislayverma.com) are used to achieve something similar at code level), it has of late been possible to achieve the same kind of agility at macro level as well, using various strategies like containerization, microservices, and devops tools like CI/CD. Today I want to talk about how we can use events to build evolutionary architectures and how events essentially represent the open-closed principle (OCP), but at architectural scale. ### What are events An event is a broadcast by a software system about something which has happened within its boundary. The system performs an operation, and on success of that operation, tells the whole world (usually via asynchronous messaging) that the operation has happened. The system will also pass along enough data in the event to make it meaningful to the external world. e.g. An order management may publish an ORDER\_CONFIRMATION event every time an order is confirmed, and ITEM\_CANCELLED event every time an ordered item is cancelled. ### Events vs Messages While they are often used inter-changeably by developers who are building asynchronous communication between two system, [*events* and *messages* are fundamentally different](https://kislayverma.com/defining-messaging-terms-precisely/) and give rise to very different kinds of behaviours in software systems. An event is a record of a certain action having happened in a system and is therefore defined in the language of the publishing system. The publisher cares not at all about who might be listening and merely guarantees that a certain set of event data will be emitted over a certain medium of transmission. A message, on the other hand, is a peer-to-peer construct. The publisher of the message targets the message at a specific consumer system and the contents must be defined in the language of the consumer. Such a message would not be meaningful to others, even if they were to listen in. In a sense, a message sent by system A to system B is API invocation done asynchronously. ### Event based architecture If both event and message travel over an asynchronous transport medium (e.g. Kafka, RabbitMQ), how does it matter which one is which? It matters when we think about interactions between many distributed systems and who knows about who in such a world. If we use events to propagate information across our distributed system, we come up with a very loosely coupled architecture where there is minimal knowledge of each other across systems. All systems either broadcast events corresponding to activities in their world or consume events from other systems to trigger workflows in their own world. As a publisher, a system does not know who will consume its events. As a consumer, a system is not aware of where the event came from, just that it should perform something when it receives such an event. e.g. An order system might emit an ORDER\_CONFIRMED event, which may be consumed by an invoicing system and an accounting system. The invoicing system will now generate an invoice and emit INVOICE\_GENERATED event. Listening to the INVOICE\_GENERATED event, the order system may send an email to customer. The order system sees one publish and one consume but does not trace a causality between the two. ![](https://kislayverma.com/content/images/2020/07/event-based-arch.png) In the micro-service world, events give rise to the *choreography style* of building workflows. Essentially, this is no explicitly defined workflow at all but service are mapped to respond to certain set of events. The interaction described above is an example. An end-to-end workflow is achieved without describing it as such because we are able to compose it from independent event-service interactions. No one needs to know the *complete* flow as it does not really exist. ### Message based architecture In a message based architecture, the order system would emit two messages : GENERATE\_INVOICE (to the invoicing service) and BOOK\_REVENUE (to the accounting system) with order identifier as reference and then wait (callback based) on the invoicing system response. The invoicing system, after generating the invoice, sends back an acknowledging message for the GENERATE\_INVOICE message, on receiving which the order system sends an email to the customer. ![](https://kislayverma.com/content/images/2020/07/event-based-arch-1.png) Note how systems are aware of each other in this paradigm. They may be decoupled in time due to the use of asynchronous messaging, but they are coupled at the domain handover boundary. However, since systems are aware of each other, we can build nuanced experiences around handshakes (the ack sent by the invoicing system in our example above is such an example) and error handling which would not be possible in the event driven world. In the micro-service world, messages give rise to *orchestration style workflows*. A service or an orchestrating system (often a workflow engine like JBPM or its more modern avatars like [Conductor](https://netflix.github.io/conductor/intro/?ref=kislayverma.com) and [Cadence](https://github.com/uber/cadence?ref=kislayverma.com)) captures the sequence in which a set of services should be invoked to achieve an end-to-end output and it invokes them via messages (or APIs, as the case may be). ESB based systems are a version of messaging architectures. --- --- ### Events are OCP Now it should be clearer why I think of events as a form of open-closed principle (OCP). OCP says that our code should be open to extension but closed to change. i.e. anyone who wants to add additional functionality to existing code should be able to do so *from the outside*, without having to touch the code itself. In an event based architecture, all a system is responsible for is performing its function and emitting the corresponding events. It doesn’t know which other systems are consuming these events or how. So if we were to change the implementation of our current invoicing system, or to build different invoicing systems for different types of orders, or don’t want to send notifications for some types of invoices, we could do it all without touching the order system itself. Whole new things could be developed outside of the order system to enrich the order management platform without touching the order system. The is the open-closed principle at work on an architectural scale. ### Evolving an event based architecture Let’s talk a little more about how we would evolve an architecture based on events. We have already seen how we can change everything around a system without touching the system itself. Now what would we do if we wanted to change this system itself (the order system in our previous example)? How to manage the impact on other system? As it turns out, there is no/minimal impact. As far as all the other systems are concerned, this system does not exist. For them, the event stream *IS* the fact of life, and as long as the events continue to flow in, it doesn’t mater to them whether they are coming from the same system or from the next version of it or from a entirely new system. Even if we build a new system which does not abide by the current event structure or semantics, it is often only a matter of understanding the new event data and massaging it into the consuming systems own language. This kind of decoupling is very powerful when we want to quickly move around our technical constructs. A widely employed strategy for building new versions of software is the [strangler pattern](https://www.martinfowler.com/bliki/StranglerApplication.html?ref=kislayverma.com) where you progressively migrate and deploy functionality from one version of a software to the next one, all the while keeping the structure of the events same. As long as we keep the event flow backward compatible, no one need to know that something is changing. This pattern is often used in migrating from monoliths to micro-services. ### Event-ful Pitfalls! However, this degree of decoupling comes at a price. There are some problems that must be kept in mind when adopting event driven architectures. The most important problem is one of tracking business workflows. Since systems do not collaborate with each other but rather with events, it becomes difficult to track what the status of any business process is. Long pipelines like order processing become very difficult to track and manage. Answering “define the complete process of order fulfilment” can have you running all over the engineering department! The other, lesser problem is around error handling. If one system suffers from an outage and loses some messages, there is no straightforward way to re-generate/replay them. The publisher gives no guarantee that it can re-publish them. Persistent messaging system like Kafka help to a certain extent, but guaranteeing their uptime and resilience, even more than the core services, becomes a mission critical problem for the whole company. In some cases, we are all right accepting a little coupling in order to attain business cohesion and debuggability. Simply enabling an event stream for your system will decouple you from all other use-cases which you do not deem to be “core” for your domain, while you retain the freedom to sign up for message driven use cases when you must. How to distinguish these situations will, of course, vary from use-case to use-case. The key thing, as always, is tradeoffs. Sprinkle carefully for a juicy architecture! **Read Next** : [Defining messaging terms explicitly](https://kislayverma.com/defining-messaging-terms-precisely/) ### Why you should build a platform URL: https://kislayverma.com/why-you-should-build-a-platform/ Last updated: 2026-07-22T12:47:33.000Z In my [previous post](https://www.kislayverma.com/platform-thinking/products-are-not-platforms/?ref=kislayverma.com), I discussed what a platform and how it is different from a product or a service. Today I want to talk about the reason why a company should look at building a platform. Before we get into the why’s, I want to clarify that I am talking about technical platforms here. Many people and articles out there confuse this with bi-directional marketplaces (Amazon, Uber) or other products with network effects. These are [not the same things](https://kislayverma.com/marketplaces-are-not-platforms/). Platforms are identified by their multi-faceted application. Airbnb today is unlikely to be used for anything other than renting homes. It may have a compelling network effect and may grow faster and faster as more home-owners (supply) and vacationers (demand) sign-up, but it is unlikely to venture into say, travel bookings, very easily. Similar is the case with Amazon.com. It is a platform only in business-speak (as in “you can discover hundreds of sellers and millions of customers on our *platform*”). Amazon has built a platform internally, but the website/app remains a product built on top of the platform. So there may be overlaps and arguments here, but we should identify that there is a difference. ### The Law of Conservation of Attractive Profits [Clayton Christensen](http://www.claytonchristensen.com/?ref=kislayverma.com) ’s book “ [The Innovator’s Solution](https://www.amazon.in/Innovators-Solution-Creating-Sustaining-Successful/dp/1422196577/ref=sr%5F1%5F1?ie=UTF8&qid=1542094517&sr=8-1&keywords=the+innovator%27s+solution&ref=kislayverma.com) ” coined the term “**Law of conservation of attractive profits**”. A good discussion of this can be found [here](https://stratechery.com/2015/netflix-and-the-conservation-of-attractive-profits/?ref=kislayverma.com) . It offers good insight into the question of platforms by saying that value creation happens when people re-arrange the existing economic systems and processes into ways that are more and more beneficial to them. When a product is not “good enough” or in its infancy, Prof. Christensen says, value is created by having an integrated, locked-in technology stack that is highly optimized for specific purposes. If you don’t do this, the overheads caused by generalization will prevent the product from getting better fast enough and hold it back from getting more customers and climbing the value chain. However, as the product and its performance get better and better and it becomes good enough, there is no further value in increasing its performance or other already existing qualities because the customers already find it good enough and will not pay premiums for further improvement of the same capabilities. This is called “commoditization”. The focus of value creation now shifts outside the product and towards the surrounding eco-system. Allowing the product to integrate with external services and offerings evolves your product in completely new dimensions which customers may find attractive. To do this, however, you need to open up your product stack in such a way as to allow interoperability with external systems. In other words, you have to start platformizing your product. Even now there are multiple levels that we can go to. We can: 1. Open up our product just enough to integrate against some external services. 2. Open it a little more and allow any outsiders to interact with us in well-defined ways that serve our core product. 3. Go all out and open up the entire set of capabilities and start using them as outsiders. The first option offers minimal incremental value as you can only do what you can think of. The second option offers more benefits as your ecosystem can now contribute to your product. The third option, of course, is what keeps the internet and VCs abuzz and offers potentially new business opportunities beyond the existing product. Since the rewards are in increasing order of options, the cost of achieving them is obviously in the reverse order. i.e. Enhancing a product just enough to integrate against some external services (adding “social” offerings etc) is relatively cheap and risk-free. Opening up the product so that others can integrate over standard protocols and interfaces (e.g. offering webhooks or APIs) is a more time-consuming thing and might involve significant changes to the product design. The last option demands a complete change in the organization’s mindset and is very difficult to pull off and potentially fatal to the organization if not executed properly. --- --- ### When should a company build platforms? I really like the “good enough” argument as a rule of thumb for deciding whether you should be building a platform or a product. A company should [never start out with building platforms](https://techcrunch.com/2015/11/28/the-platform-paradox/?ref=kislayverma.com) , unless of course it is explicitly a company that builds platforms. In the early stages, it should focus on its core customer offering and try to achieve a product-market that eludes so many startups out there. Only when you have a product which customer demonstrably like (demonstrated via incoming revenue), should you try to explore platform options.What good should you expect from building a platform. There are many articles outside there extolling the virtues of platform, so why not add my own two cents :) #### Tapping the eco-system A well-designed platform makes integration with external systems easy to the point of being trivial. And once this has been achieved, we can immediately see how business value is generated at the edges of our platform not by us alone, but by us and our partners in the business eco-system jointly to mutual benefit. The edge of the [platform is designed for speed and flexibility](https://stories.platformdesigntoolkit.com/design-apis-for-disobedience-7894f930e2cc?ref=kislayverma.com) to produce new experiences, while the core is designed for stability and re-use. This allows us to plug-in anywhere in the wider economy and tap opportunities that we previously could not have exploited. #### Reduced time to market in changing business landscape If the business landscape is expected to change very often or have lots of nuanced use-cases, building a platform will give a reduction in time to market as changing business requirement will only a require a fresh re-stitching of platform capabilities rather than complete re-invention of the product. This is, of course, an over-simplification but I can speak from personal experience that the more re have in the way of re-usable business and tech components, the faster an organization can respond to business changes. #### Ease of evolution As discussed before, Platforms enable evolution at the fringes of the existing set of capabilities. More and more capabilities and features can be added to a platform without impacting the existing ones. This is possible because the parts of a platform are not connected or critically dependent on each other (remember the tool-kit analogy). They stand independently and can be strung together as required.As a result, we can add more and more capabilities very easily to a platform. We can even design so that our partners in the industry or users themselves can contribute new capabilities. This is where network effect can start kicking in. The more people can build with us, the more they will build with us (hopefully!). #### Uniformity of tools and experience If you have followed the Golden Rule of Platforms, you are likely going to use your platform to build a variety of products yourself. From a technical perspective, a well designed platform offers capabilities which all have a unified developer and user experience, so it is easy to move from one piece to another, exploring and picking as you like. The end product will have a cohesive feel to it, since the standards followed in the bedrock are the same. **Read Next** : [How to build a technology platform](https://kislayverma.com/technology/how-to-build-a-technology-platform/) ### Make it better, every day of the week URL: https://kislayverma.com/make-it-better-every-day-of-the-week/ Last updated: 2026-07-22T12:47:33.000Z [Read this](https://blog.cleancoder.com/uncle-bob/2014/04/03/Code-Hoarders.html?ref=kislayverma.com) Exactly. Spot on! As programmers, if we haven’t made our codebase a little bit better (or a whole lot better) for every new feature we add, we have failed. If no one but us can understand what a piece of code is doing or modify it in meaningful ways without bringing down the house, we have failed. If we take a shortcut/hack under the pressure of deadlines or whatever but don’t make it easy to spot/remove, we have failed. As engineering managers, if we don’t deeply understand what makes our components good or bad, we have failed. If we can’t set up processes and tools to monitor the health of our code, we have failed. If we can’t motivate our teams to love their code and to want to make it better, we have failed. More words don’t make a better song. More brush strokes don’t make a better painting. More features/LoC don’t make a better product. In his own (perhaps small) way, every good software engineer is a visionary. He pre-empts the future by building it today. When he can’t predict the future, he leaves room in the present to handle it tomorrow. Our legacy is not the code we write today, but the code others will write using it tomorrow. So what did you improve today? **Read Next**: [Get faster instead of being fast](https://kislayverma.com/being-fast-or-getting-faster-aka-build-momentum-not-velocity/) ### Explicitly Yours URL: https://kislayverma.com/explicitly-yours/ Last updated: 2026-07-22T12:47:34.000Z I shared this note internally in Myntra in 2015 when I was working on some particular feature of their Order Management System (OMS). Sharing it now with internal details redacted. --- I broke OMS yesterday. Didn’t think I would, but still did. And I am mad as hell about it. **tl;dr** — Always be explicit about the inputs when exposing APIs. Relying implicitly on the caller or some other global condition is a horrible horrible idea. OMS is a large project. We maintain the entire life cycle for Myntra’s orders, and [shit ain’t easy](https://steve-yegge.blogspot.com/2009/04/have-you-ever-legalized-marijuana.html?ref=kislayverma.com) . The challenge, however, is not in building the individual sub-parts, but in building the flow. There are two ways to approach this. One is to build tightly coupled parts which only make sense as a whole. This approach chooses maintainability to the detriment of flexibility and reuse. Essentially, the whole can be implemented as a single piece (in the logical extreme, a sequence of method calls invoked from one place). The internal parts rely heavily on where they are located in the workflow and each part sets the context for its successors. The other approach is to build an overall scaffolding and reusable pieces which can be plugged in as needed. This approach makes the reverse trade-off. While writing a component, we never know what is happening around us and so we have to code in an isolated manner. This means tremendous flexibility, but may mean that the overall flow isn’t optimized (Read from DB too many times, write to DB too many times to manage state in isolation). Over its years, OMS has evolved (decayed???) into a mix of both of the above. We have a lot of semi-independent components which work alone. Some of them need loving from their neighbours and some don’t. Of these, there are some psychos that pine away internally but wont say that a [ll they need is love](https://www.youtube.com/watch?v=WWP80rXP4cM&ref=kislayverma.com) . What is wrong with this API: *CancellationUtil.issueCancellationRefund(Order order);*On the face of it, nothing. Looks intuitive. Reads like a story (issue-refund-for-the-order-object-passed-to-you). A good API. Except that it isn’t. The actual implementation of the API assumes that certain fields of the input object have been manipulated in a certain way (the caller should take the applied coupon code from order and set it into all order items). The problem is not that it needs such a manipulation. The problem is that the API doesn’t make it obvious that it relies on some very specific pre-processing of its inputs. That is to say, it is aware of its location in a workflow. Something like this would be better: *`CancellationUtil.issueCancellationRefund(Order order, String appliedCoupon);`* Or even better: `CancellationUtil.issueCancellationRefund(int orderNumber);// Everything is done inside the API.` At long last, we come to the point of this rant. IMPLICIT EXPECTATION IS THE ENEMY. You know you are in a bad place when absolutely unrelated changes start breaking you. This is why our “trivial” changes sometimes break catastrophically. This is why we are afraid of refactoring (I certainly am!). This is why we prefer rewrites to maintenance. Let’s face facts. Writing and maintaining great documentation is hard :). But what we can and must do, ALL THE TIME, is look at our code real hard, and remove any hints of subtle, hidden agreements from its interfaces. Bad looking APIs are ok. Bad performing APIs are ok (performance can probably catch up), but that gentlemen’s agreement between caller and callee isn’t going to honour itself. **Read Next** : [Extending code vis the Resource Locator pattern](https://kislayverma.com/working-in-the-seams-of-code-with-resource-locator/) ### Products are not Platforms URL: https://kislayverma.com/products-are-not-platforms/ Last updated: 2026-07-22T12:47:34.000Z **tl;dr** — A product is a conceptual whole intended to serve a specific purpose. A platform is a set of independent-yet-interlocking tools which can be externally orchestrated to create products. This post is in continuation to [my redux](https://www.kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/?ref=kislayverma.com) of Steve Yegge’s blog post about platforms and builds on my thoughts and learnings from trying to convert Myntra’s supply chain technology to a more platformized avatar. Here I want to talk about how products and product thinking differ from platforms and platform thinking. The logistics team at Myntra was recently discussing how to model master bags (big bag of individual shipments shipped as a single unit), and we initially came up with entities like “first mile master bag”, “last mile master bag” etc. They would each exist in different legs of transportation and would behave very differently (have different state machines etc). Only when the duplication of capabilities became very obvious did we start talking about the general concept of “master bag” which could be used by all logistics legs. The debate rages on today :), but new business requirements coming in of late are bearing out that intuition about a central, re-programmable entity which can cross domain boundaries and change behaviours as it does. It is worth looking at that example (I often do) to understand why the final design wasn’t the first thing to spring to mind. There are two aspects to this. First, we didn’t start out to model master-bags per se. That conversation was a by-product of a new business requirement to use master bags in the last mile when they had so far only been used in the long haul stages. As such, we set out to solve a specific problems and a context neutral, behaviour-less entity didn’t make much sense in the discussion. After we agreed to build the central entity, the problem would still have to be solved using this new component. The requirement to build the product was blindsiding us from the beginning, taking us away from the platform. The second aspect is the way the team thinks. We are the team that build products for logistics operations. Our design instincts are to encapsulate behaviour and data inside products which serve a well defined purpose. So we wanted to round up behaviours of a specific kind of master bag into one component, another kind of master bag into another. They should be different because they behave differently. The “master bag service”, as it is to become (development to start shortly) does not contain much logic of its own. Much of its complexity comes from the various hooks that had to be built in it so that specific logistics workflows and products could extend its basic domain and states. Beyond some very basic things, it does not impose any restrictions of its own. The entity has almost no behaviour of its own. All of this goes completely against the grain when you are used to thinking about well-defined, stand alone systems. And so we come to the point of this little back-story. A product is the final business offering — the end product that we hope to satisfy our customers with. It does not really matter to the customer how it is built, just that it should work delightfully. It is the castle made of Lego bricks — built to serve a specific purpose very efficiently. A platform, on the other hand, is the set of Lego bricks itself- a variety of pieces that fit together in ever varying forms. That is what a well designed platform feels like. A set of tools which are all very different but work very well together. Just look at AWS. You may go there only for the compute, but it is so easy to start using RDS or SQS once there that it becomes very easy to get pulled into using it all. This is why teams that are used to building products have a tough time switching to thinking in terms of platforms — a platform never feels like a complete, cohesive thing. It is too diffuse, and never seems to serve a final business purpose, and so they never really buy into it, at least not easily. They are not able to conceive of a widely different set of tools as all belonging to a single toolbox. I have found that a platform conversation is more accessible to developers if it is about a purely “technical” concept like authentication or rate limits or configuration management. Then design purity automatically pulls them towards designing an “identity platform” or some such. However, when taking about business entities like items or payments, people find this definition of purpose much more difficult. This is because the business context starts playing a bigger role and they start looking for product boundaries which will serve the business purpose. A platform offers none of that at the outset, so it is unlikely that we will stumble upon it during the regular course of product engineering. It has to be conceived very deliberately, and pursued with the collective will of the organisation if it is to be built at all. Read Next : [Why build platform architecture?](https://kislayverma.com/why-you-should-build-a-platform/) ### Distilled : Steve Yegge's platform rant URL: https://kislayverma.com/distilled-steve-yegge-s-platform-rant/ Last updated: 2020-07-20T12:24:07.000Z I came across this now legendary [rant by Steve Yegge](https://gist.github.com/kislayverma/d48b84db1ac5d737715e8319bd4dd368?ref=kislayverma.com) 5–6 years ago. While it was amusing and interesting then, I have now been trying to convert Myntra’s supply chain technology to a platform for over the last year and a half and have found it really relevant when thinking about what a platform is, what it means to build one, and how would you know if you succeeded in building one. I have revisited the article over and over again over the last few months. What consistently amazes me about it is that it couples deep technical and organizational insight with damn good writing. It is a long post, and I love reading it, but I finally decided to extract the parts which most appealed to me in it. One reason for this is to crystallize my own understanding of it, and another is to freeze my understanding of it as of this point. I guess it might be interesting later on revisit this and see how much things have changed. The follow are what I believe to be the key parts of the article. 1. SOA-driven design enables Platforms. 2. “All teams will henceforth expose their data and functionality through service interfaces” — this unlocks capabilities for re-use. This is the beginning of a platform. 3. “All service interfaces, without exception, must be designed from the ground up to be externalizable.” — This truly unleashes the platform built under the previous dictat by implicitly mandating all softer aspects of an externally used service (rate limiting, identity, security, documentation, uptime etc). Organizing into services taught teams at Amazon not to trust each other in most of the same ways they’re not supposed to trust external developers. 4. The big realization Bezos had was that he can’t always build the right thing, that he can’t build one product and have it be right for everyone. 5. Bezos realized that he couldnt provide everyone with the right “products” : interfaces and workflows that they liked and felt at ease with. 6. A platform-less product will always be replaced by an equivalent platform-ized product. 7. Conversely, a platform needs a killer app. 8. Facebook — that is, the stock service they offer with walls and friends and such — is the killer app for the Facebook Platform.Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. 9. The Golden Rule of platforms is that you “Eat Your Own Dog food”. 10. The Golden Rule of Platforms can be rephrased as “Start with a Platform, and Then Use it for Everything.” You can’t have secret back doors for internal apps to get special priority access, not for ANY reason. 11. A platform is an “externally programmable” system. --- --- The first 3 points are talking about the technical aspects of building a platform. Points 4–8 talk about how platforms fit in the business context. These are the things a business should try to evaluate when thinking of how to build. Should you build a platform? How would it aid the business in the way that it plans to grow? Is it worth the extra technical overhead that probably would not be there if you built a more tightly knit “product”? Points 9–11 talk about the organizational thought-process for building platforms and for me, they are really, REALLY, the heart of the matter. Even as Steve’s post suggests, Google isn’t bad at platforms because they dont have the technical capacity (obviously) or business need. They are bad at it because the organization does not think about platforms as independent things on which mutiple products could be built. Teams are focussed on delivering value via products, which are then mostly not re-usable in the business context and have to be re-invented again and again as the business changes to try and do different things. Point#11 is the litmus test of a platform — can others use the offered capabilites efficiently, without jumping through 20 hoops to build meaningful products and deliver business value. If it cannot be used “programmatically” by complete “outsiders”, it is not a platform. ### Update Here’s [another one](https://gist.github.com/kislayverma/6681d4cce736cd7041e6c8214469d2fd?ref=kislayverma.com) by Steve, this time talking more about Amazon and his time there. This one is pretty amazing too. **Read next** : [More articles](https://kislayverma.com/category/platform-thinking/) on platform thinking and architecture. ### It Depends #59: Supercharge your reading with 4Later URL: https://kislayverma.com/it-depends-59-supercharge-your-reading-with-4later/ Last updated: 1970-01-01T00:38:57.000Z 4later helps you manage your reading list | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #59: Supercharge your reading with 4Later Hello Everyone! Welcome to the 59th episode of It Depends. Hope you and your loved ones are doing well. And a warm welcome to the 114 new readers who joined since the last edition. Welcome aboard folks - hope you have as good a time reading this newsletter as I have writing it. Introducing 4Later: Reading Supercharged The last couple of months have been intense. Not just because of the day job (where the team shipped a couple of super impactful things), but because I’ve been working on something that I’ve wanted to build for some time now. I’ve been thinking about a better way of managing my reading process on the internet. This need has become more pronounced as I’ve started sending out curated links in this newsletter. My current process is something like this. Discover articles somewhere - social media, other newsletters, some rabbit hole or the other. Either read it right away or store it in my Roam graph with a "read later" tag. Revisit this list when I have time and go through the articles. Take notes on the articles in Roam as I read the article. Go through this list and select the best ones to share in It Depends. This copy-pasting of things everywhere has always bothered me. It feels clumsy and is disruptive to the flow of whatever I’m doing at that time. The note-taking is also disconnected from the reading due to the toggling between the reading window (usually the browser) and the Roam app. A smaller thing is that while I share a few articles on It Depends, I read a whole lot more, much of which is also great. I’d like to share that as well. Sharing to Twitter seems obvious, but that also has the same multiple copy-paste problem. There didn’t seem to be any tool which would seamlessly meld the reading list, the notes, and the sharing process. So I decided to build 4Later - a tool which lets you bookmark links as you find them without disrupting whatever you are doing, and then go back to them later when you have time. It includes an integrated reading + note-taking experience and lets you share the links and notes to your followers on Twitter or to the community of users on 4Later itself. The main tools are the [android app](https://play.google.com/store/apps/details?id=com.asugoapp&ref=kislayverma.com) and the [chrome extension](https://chrome.google.com/webstore/detail/4later/hdekpokknaocegfjamldpgfbfajppknp/overview?ref=kislayverma.com), both of which provide the bookmarking and read later capability. To discover more people to follow, you can visit the “Discover” tab on the website, though I’m bringing that to the app and the extension very soon. Using 4Later over about a month has had some impact. My Twitter doomscrolling has reduced somewhat since I always have interesting things to read sitting ready. My note-taking has also increased, though now these notes are split from my main graph at Roam, so some copy-pasting remains. But Roam is increasingly becoming a more passive knowledge store (something more for correlation and insights) while my active working shift towards 4Later. All the notes in today’s “From the internet section are taken straight from my 4Later reading list. Seems to be working - at least for me! I’d love for you to give 4Later a shot and see if it improves your reading workflow. I’d love to hear about bugs, feature requests, and most of all your thoughts on how this tool can become even more “invisible-yet-embedded” in your knowledge work. Thank you [Buchi](https://twitter.com/buchireddy?ref=kislayverma.com) and [Paras](https://www.linkedin.com/in/paras4all/?ref=kislayverma.com) for your bug reports - I’m incorporating that stuff soon. From the internet This rant by Chiff Harris about [how bloated our programs and system have become](https://www.positech.co.uk/cliffsblog/2022/06/05/code-bloat-has-become-astronomical/?ref=kislayverma.com) resonated deeply with me. In college, I was obsessed with killing windows processes and running as trimmed down a version as I could. Sure don't worry about micro-optimizations but much of the way we program today is so wasteful of computing resources it ought to be criminal. In building 4Later, I had to have an auth system in Spring Boot that worked for both the embedded website built using ThymeLeaf and the chrome extension (which doesn’t allow redirects etc which are critical for doing OAuth). [This tutorial ](https://www.callicoder.com/spring-boot-spring-security-jwt-mysql-react-app-part-2/?ref=kislayverma.com)on building a JWT based auth finally helped me get through. Slightly heavy on the business details side, but a very detailed tutorial nonetheless. Most tech companies do special, often months-long prep work to support high scale events. [This video by Ganesh Subramanian of Cred](https://www.youtube.com/watch?v=aKI7sQAtQOc&ref=kislayverma.com) shares a template of how to do this prep well. That’s all for this week folks! \-Kislay Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #58: Evolving software using all SOLID principles at once URL: https://kislayverma.com/it-depends-58-evolving-software-using-all-solid-principles-at-once/ Last updated: 1970-01-01T00:37:03.000Z And SOLID principles as evolutionary guardrails | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #58: Evolving software using all SOLID principles at once Hello Everyone! Welcome to the 58th episode of It Depends. Hope you and your loved ones are doing well. First, apologies for skipping the last 2 weeks. As I said the last time, there was a whole lot going on (including my own 37th birthday - YAY!), so I couldn’t get to writing at all. Next, a warm welcome to the 60 new readers of this newsletter who joined over the last 3 weeks. The last two articles on [building scalable systems](https://kislayverma.com/software-architecture/on-scalable-software/) and [robust distributed systems](https://kislayverma.com/software-architecture/building-robust-distributed-systems/) got picked up by a bunch of other curated newsletters. It makes me happy to see that my writing is adding value to some people’s lives. If you love reading *It Depends*, don't forget to share it with your friends, and consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Evolving Software: SOLID principles as a continuum SOLID principles are powerful tools for building a system with low coupling between its components. A quick recap on these principles: SRP: Single Responsibility Principle OCP: Open CLosed principle Liskov Substitution Principle Interface Segregation Dependency Inversion If you don't know what these terms mean, I recommend[ this primer](https://www.baeldung.com/solid-principles?ref=kislayverma.com) \- go check it out and then read the rest of this article. Here, I want to talk about how all the SOLID principles are interlinked. They all apply simultaneously in any situation. Breaking one will also break multiple others. In my opinion, they should be read as a continuum, rather than a set of independent principles - one always needs some of them to achieve the other(s). Personally, I start out with a mindset that I don't want to modify existing code. Who knows how I might break it? Ideally, I want to just inject my new logic into the currently running system in the specific places where I need to. So SRP is kind of my favourite principle. But all other principles come in to uphold this one. Let's consider the windows machine example from the primer I linked above. Here’s the class for reference. ![](https://kislayverma.com/content/images/2022/04/applying-solid-principles-300x245.png) The machine has a Keyboard, a CPU and a Monitor. If we give the machine the responsibility of creating these objects, it needs to not only know how it does its own functions, it also needs to know how to create these different subcomponents. This breaks SRP. How can we outsource the knowledge of building the monitor? The easiest way is to let someone else build the monitor object and give it to the machine class during construction (much as it happens in the real world). This is the dependency inversion principle - we are using one SOLID principle to achieve another. But if anyone can construct a monitor and pass it to a machine, the machine needs to be sure that the monitor is compatible with the implementation of the machine. Otherwise, the machine has to handle the differences between various types of monitors. This again breaches SRP. The machine wants others to create monitors, but all monitors must do exactly what the machine expects, regardless of how they do it. How do we get this? Enter Liskov Substitution Principle, which requires subclasses to do exactly this. The machine class exposes some interface or base class that comprises the "specs" of the monitor. Every monitor must do exactly that, and nothing else. This ensures that the machine can be given any implementation of the base monitor class to work with and nothing in the machine code needs to change. Yet again, we see how one principle supports another. To the extent that the behaviour of the machine is controlled by the behaviour of the monitor, we have already achieved OCP because we can pass in different monitor implementations to do the same thing in different ways. But how can we modify the behaviour of the machine itself? One way would be to open up the machine's code and add some conditional or additional business logic there to cater to our new requirements. But if we do this, the final artefact still breaches SRP in a way since it changes for multiple behavioural reasons now. To prevent this, we must redesign the machine component in a way that allows others to subclass it and override it with new behaviour. The old code is still in play for existing use cases, but wrappers can now be built around it to support new use cases. Here, OCP is helping maintain SRP in the long run. Let's look at the monitors themselves. In isolation, they can have many many attributes. Monitor manufacturers deal with tons of complexity. But all of that is not relevant to the machine. So the machine-facing part of the monitor implements a much narrower set of specifications that the manufacturing-facing part. This is the interface segregation principle, where the monitor object implements two different sets of interfaces for two different use-cases. As this example shows, SOLID principles cannot be applied one by one. They have to be applied all at the same time to achieve the decoupling we want in our systems. An evolutionary take An interesting way to look at this is in terms of system evolution. Everywhere in the world, evolving systems develop greater degrees of specialization for every type of component, and they develop a rich collection of different types of components. The cross-play of both these axes results in the immense diversity of living systems we see around us. I have written before about the [mechanics of software evolution](https://kislayverma.com/software-architecture/the-mechanics-of-software-evolution/). That article painted a higher-level picture of system evolution. Let’s consider how we can guide this evolution inside our components. Software programs are living, growing systems. SOLID principles are the guiding forces that let the system evolve specialization and diversity in a healthy way, instead of collapsing into a mess of chaos. ![](https://kislayverma.com/content/images/2022/04/solid-pre-evolution-300x241.jpg) SRP and OCP create specialization. SRP is the restriction that prevents a component from becoming too muddled internally. But due to external business pressures, the same component MUST do different things. The pressure builds, and OCP relieves it by allowing a deeper subtree of more and more specialized subclasses which can satisfy business needs. We saw in the above machine example how the concept of monitor evolved from inside the concept of the machine due to SRP. Similar things can happen with CPU and mouse and so on. Here too, SRP is the forcing constraint, and the Liskov Substitution Principle, Interface Segregation, and Dependency injection jump in to satisfy the constraint by creating diverse types of components, most of which can work with each other. Once outside, monitor and the other concepts take on a life of their own, each developing its own hierarchy of specialized subclasses. And hence the cycle repeats, creating an increasingly large but consistently decoupled system. From the internet “The ceaseless pursuit of force multipliers is the only possible route to superlinear productivity improvements as an organization grows”. Codahale [talks about diminishing returns](https://codahale.com/work-is-work/?ref=kislayverma.com) as team sizes grow. A topic [close to my hear](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/)t, but this article is as awesome as it gets. Polly Labarre covers the philosopher Peter Koestenbaum on the attributes of a leader and what it takes to lead in the modern corporate world. TL;DR - More than anything else, leadership is about [the will to lead](https://www.fastcompany.com/38853/do-you-have-will-lead?ref=kislayverma.com). Codahale makes another appearance! Since I’ve been writing about distributed systems and scalability, here's a [USL based library](https://github.com/codahale/usl4j?ref=kislayverma.com) to measure the scalability of your system. Handle with care :) That’s all for this week folks. Have a great weekend! \-Kislay Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #57: Critiquing crypto, trunk-based development, and Paxos made simple URL: https://kislayverma.com/it-depends-57-critiquing-crypto-trunk-based-development-and-paxos-made-simple/ Last updated: 1970-01-01T00:36:03.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #57: Critiquing crypto, trunk-based development, and Paxos made simple Hello Everyone! Welcome to the 57th episode of It Depends. Hope you are doing well and staying safe. What with Holi, me visiting my parents' place in Delhi, and a massive OKR planning explosion at the day job, writing has been on the back burner a bit these past two weeks (and the next week doesn’t look very hopeful either). I’ve been reading [“The Politics of Bitcoin”](https://www.amazon.in/gp/product/B01M22NTCT&tag=kislayverma-21?ref=kislayverma.com) by [David Golumbia](https://twitter.com/dgolumbia?ref=kislayverma.com) so perhaps I can share a summary of that - let’s see how it goes. So this week it’s a shoutout to 54 new folks who signed up for this newsletter, and straight on to the best of the internet. From the internet About 15 researchers and critics, including [Molly White](https://twitter.com/molly0xFFF?ref=kislayverma.com), [Stephen Diehl](https://twitter.com/smdiehl?ref=kislayverma.com), [Grady Booch](https://twitter.com/Grady%5FBooch?ref=kislayverma.com), have critiqued [Kevin Roose](https://twitter.com/kevinroose?ref=kislayverma.com)’s NYT piece on cryptocurrency. [An interesting, often funny, and thoroughly brutal read](https://www.mollywhite.net/annotations/latecomers-guide-to-crypto?ref=kislayverma.com). The internet is a beautiful place. There is apparently a whole website about [trunk-based development](https://trunkbaseddevelopment.com/?ref=kislayverma.com) with everything explained much better and more deeply than anything else I have seen on the topic. [Mahesh Balakrishnan](https://twitter.com/maheshb?ref=kislayverma.com) is becoming a regular feature on this list. Here he is again, [making Paxos simple](https://maheshba.bitbucket.io/blog/2021/11/15/Paxos.html?ref=kislayverma.com). That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #56: Building robust distributed systems URL: https://kislayverma.com/it-depends-56-building-robust-distributed-systems/ Last updated: 1970-01-01T00:35:12.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #56: Building robust distributed systems Hello Everyone! Welcome to the 56th episode of It Depends. Hope you and your loved ones are doing well. First up, a huge shoutout to Satchit (that's the mail id at least) for showing this newsletter some love on [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com). It’s always a massive shot in the arm. Thanks to you and other loyal readers, It Depends is now at 2111 subscribers, so keep up the support ([Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com),[ Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) are open too) and keep spreading it among people who you think would like it. Since I gave reasons to build distributed systems last week, it is only fair that this week I talk about building them well. And then on to the best of the tech internet for some great weekend reading. Building robust distributed systems You can read [the original article](https://kislayverma.com/software-architecture/building-robust-distributed-systems/) directly on the blog. I have written before on this blog about[ what distributed systems are](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/) and how they can give us[ tremendous scalability](https://kislayverma.com/software-architecture/on-scalable-software/) at the cost of having to deal with a more complicated system design. Let’s discuss how we can make a distributed system resilient to random failures which get more common as the system gets larger. [Systems theory](https://kislayverma.com/books/book-review-thinking-in-systems-a-primer/) tells us that the more interconnected parts of a system are, the more the likelihood of large failures. So to build a resilient system, we need to reduce the number of connections. Where this cannot be done, we need to implement ways to “temporarily” sever connections to failing parts so that errors do not cascade to other parts. ![](https://kislayverma.com/content/images/2022/03/connected-components-300x300.jpg) Every component has to assume that every other component will fail at some point and decide what it will do when such failures happen. Lastly, we need to build some buffers in the system – some ways to relax, if not remove the demands placed on it so that there is slack to handle unexpected conditions. Minimize inter-component dependencies Components of a distributed system communicate with each other for data or functionality. In both cases, we can reduce the requirement of connectivity by pushing the data/functionality into the calling component instead of being accessed remotely. Building a high scale distributed system forces us to abandon a lot of the “best practices” of standard software engineering. The key thing to remember is as we adopt the complexity of distributed systems to attain scalability, we also need to keep the “distribution” in check as much as we can. Duplicate Data If we access some data from another component frequently, we can duplicate it in our component to not have to retrieve it at run time. This can massively reduce runtime dependency and help improve latency on our component. Data that is frequently accessed but changes with some regularity can be cached temporarily with periodic cache refreshes. Data that changes even less frequently or never (e.g. Name of a customer) can be stored in our component directly. We might have to do some extra work if/when this data does change, but this added small overhead is usually worth it for the increased resilience. Denormalize Data Denormalization is a special form of duplication that happens within a component. If we are using relational data stores, we can reduce the cost of looking across multiple entities by duplicating data in the main entity. The principle of localizing scattered data for better performance applies here as well. Libraries To mitigate functional dependency on another component, we can package the remote component as a library and embed it within our component. This is not always possible (it might be written in some other language or be too large to be a library) and comes with its own set of problems (change in functionality requires library upgrades across multiple components), but if the functionality is critical and frequently accessed at a high scale, this is a viable way of breaking the inter-component connection and making it local. Isolate errors Error isolation is important for two reasons. One is that individual errors are more common in distributed systems (simple function of lots of moving parts). The other is that if we cannot prevent errors from cascading throughout the system, then we lose the very reason for building a complex in the first place. The primary construct of error isolation is SLA. Every component declares some quality parameters it will honour in performing a function. these parameters can include latency, error rate, concurrency, and others. Beyond this SLA, components invoking it assume it to have failed and need to take suitable action on their own. If the component itself detects that it is unable to maintain its SLA, it can preemptively tell its callers to back away and come back later. To[ maintain overall system health](https://kislayverma.com/software-architecture/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/), it is better to fail fast rather than succeed with breached SLA. Both components (the one invoked and the one invoking) must put in mechanisms for this. Protecting the caller **Timeouts**: If the called component doesn’t respond within its SLA, the caller must timeout (give up) and use some fallback mechanism instead (even if it is throwing an error) to maintain its own SLA and prevent a cascade of SLA breaches. **Retries**: Since the network is unreliable, many errors in a distributed system are just random. The caller can retry the operation if its own SLA permits it to do so. The prerequisite for retries is the idempotency of the operation. i.e. it should not change state or do it only once even if it is invoked twice. **Circuit Breakers**: If calls to a component are failing continuously, the caller can sever the connection and stop calling it for some time by “opening the circuit”. Since the caller already has some backup behaviour for error scenarios, this saves the caller precious resources which would have been wasted. Stopping the calls also reduces the load on the called component and give it some breathing room to recover. Circuit breaker libraries have mechanisms to poll the troubled component periodically and restart the call flow if its performance seems to have returned to normal. Protecting the called **Random Backoffs**: While retries reduce errors, a small performance blip in a heavily used component can cause all of its callers to retry at once. This “retry storm” can create spikes in load and prevent this component from recovering. To prevent this, retries should be done with a random time gap between them so that load is staggered. **Backpressure**: If a component detects itself under too much load and about to breach its SLA, it can preemptively start dropping new requests till its performance comes under control. This is much better than accepting requests which it knows it can’t serve within SLA or without the risk of a complete crash. Build buffers in the system Asynchronous communication [Asynchronous communication](https://kislayverma.com/tag/asynchronous-programming/) channels like message buses allow remote components to be invoked without a very tight SLA dependency. By letting the messages be consumed when the called component is ready instead of right away, the system becomes a little more elastic to the demand of increased workload. Elastic provisioning Scalability eventually boils down to making the best use of available hardware. But a simple way of giving the system room to breathe can be to allocate more hardware if see the scale growing. While this is only feasible up to the extent of the cost we can bear, it gives us the last line of defence against unpredicted variations in load. You can read about some low-level details of using these techniques in my articles on[ code review](https://kislayverma.com/programming/code-review-checklist-for-distributed-systems/) and[ design review](https://kislayverma.com/programming/design-review-checklist-for-distributed-systems/) for distributed systems. From the internet Alex Komoroske describes [Schelling points in an organization](https://medium.com/@komorama/on-schelling-points-in-organizations-e90647cdd81b?ref=kislayverma.com). I recently discovered this concept and have been finding it endlessly fascinating - I think you will too. “Programming is like shovelling shit, at some point, you must start shovelling”. [Jamie Brandon’s](https://twitter.com/sc13ts?ref=kislayverma.com) (un)[learnings from a decade of coding](https://www.scattered-thoughts.net/writing/things-unlearned/?ref=kislayverma.com) is full of gems like this. One of the most pragmatic things I have read of late. Evan Bottcher has good insights on [platforms](https://martinfowler.com/articles/talk-about-platforms.html?ref=kislayverma.com) \- check it out. That’s all for this week folks. Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### It Depends #55: On building scalable systems URL: https://kislayverma.com/it-depends-55-on-building-scalable-systems/ Last updated: 1970-01-01T00:34:56.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #55: On building scalable systems Hello everyone! Welcome to the 55thd episode of It Depends. Hope you and your loved ones are doing well. This newsletter is almost at 2100 subscribers - so a big thank you to everyone who has stuck around for my irregular writing. If you have been loving *It Depends* so far, share it with others who would like it too and consider showing your support on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com),[ Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or[ Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com). This week I am presenting a fairly comprehensive look at understanding what scalability is and what do we need to know to build scalable systems. And then on to the best of the tech internet. On building scalable systems You can read [the original article](https://kislayverma.com/software-architecture/on-scalable-software/) directly on the blog. In software engineering, scalability is the idea that a system should be able to handle an increase in workload by employing more computing resources without significant changes to its design. Why don’t systems scale Software, though “virtual”, needs physical machines to run. And physical machines are bound by the law of physics. The speed of light limits how fast a CPU can read data from memory. The information-carrying capacity of wires determines how much data can be moved from one part of a system to another. Material sciences dictate how fast a hard disk can spin to let data be read/written. This means that there are hard limits to how fast our programs can run or how much work they can do. We can be very efficient within these limits, but we cannot breach them. Therefore, we are forced to use software or hardware design tricks to get our programs to do more work. The problem of scalability is one of designing a system that can bypass the physical limits of our current hardware to serve greater workloads. Systems don’t scale because either they use the given hardware poorly, or because they cannot use all the hardware available to them e.g. programs written for CPUs won’t run on GPUs. To build a scalable system, we must analyze how the software plays with hardware. Scalability lives at the intersection of the real and the virtual. The key axes of scalability Latency This is the time taken to fulfil a single request of the workload. The lower the latency, the higher the net output of the system can be since we can process many more requests per unit time by finishing each one fast. Improving latency can be understood in terms of “speed up” (processing each unit of workload faster) and is typically achieved by splitting up the workload into multiple parts which execute in parallel. Throughput The other axis of scalability is how much work can be done per unit time. If our system can serve only one request at a time, then[ throughput](https://kislayverma.com/software-architecture/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/) \= latency X time. But most modern CPUs are multicore. What if we use all of them at once? In this way (and others), we can increase the total of “[concurrent](https://www.youtube.com/watch?v=oV9rvDllKEg&ref=kislayverma.com)” requests a system can handle. Along with latency, this defines the total things happening in a system at any point in time. This can be thought of in terms of “scale up” or concurrency. [Little’s law](https://kislayverma.com/software-architecture/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/) gives a powerful formulation of this which lets us analyze how a system and its subsystems will function under increasing load. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.58.12-AM-300x17.png) If the number of items in the system’s work queue keeps growing, it will eventually be overwhelmed. Capacity This is the theoretical maximum amount of work the system can handle. Any more load and the system starts to fail either completely or for individual requests. Performance is not Scalability A system with high performance is not necessarily a scalable system. Ignoring scalability concerns can often result in a much simpler system which is very efficient for a given scale but will fail completely if the workload increases. An example of a performant yet non-scalable system is a file parser that can run on a single server and process a file up to a few GBs in a few minutes. This system is simple and serves well enough for files that will fit in the memory of one machine. A scalable version of this may be a Spark job that can read many TBs of data stored across many servers and process it using many compute nodes. If we know that the workload is going to increase, we should go for a scalable design up-front. But if we are not sure of what the future looks like, a performant but non-scalable solution is a good enough starting point. Quantifying scalability Amdahl’s Law Other than the problems with implementations, there are theoretical limitations to how much faster a program can become with the addition of more resources.[ Amdahl’s law](https://en.wikipedia.org/wiki/Amdahl%27s%5Flaw?ref=kislayverma.com) (presented by[ Gene Amdahl](https://en.wikipedia.org/wiki/Gene%5FAmdahl?ref=kislayverma.com) in 1967) is a key law that defines them. It states that every program contains part(s) that can be made to execute in parallel given extra resources, and part(s) that can only run serially (called serial fraction). As a result, there is a limit on how much faster a program can become regardless of how many resources are available to it. The total speedup is the sum of time taken to run the serial part plus the time taken to run the parallel part. The serial part, therefore, creates an upper bound on how fast a program can run with more resources. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.49.44-AM-300x75.png) This means that by[ analyzing our program structure](https://www.youtube.com/watch?v=EfOXY5XY9s8&ref=kislayverma.com), we can determine the maximum amount of resources that it makes sense to dedicate to speed it up – any more would be useless. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.49.56-AM-296x300.png) Universal Scalability Law (USL) While Amdahl’s law defines the maximum amount of extra resources which will improve a program’s performance by allowing parts of the program to run in parallel, it ignores a key overhead in adding more resources – the communication overhead in distributing and managing work among all the new processors/machines. [USL](https://wso2.com/blog/research/scalability-modeling-using-universal-scalability-law/?ref=kislayverma.com) formalizes this by adding another factor to Ahmdahl’s Law which incorporates the cost of communication. This further reduces the net gain we can get from the addition of resources and provides a more realistic measure of how much a program can be sped up. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.50.36-AM-300x77.png) Real-world tests show that in the worst case, these communication overheads build up exponentially as each new resource is added. Program performance improves in the beginning due to more resources, but this improvement is eventually overwhelmed by the communication overhead. ![](https://kislayverma.com/content/images/2022/03/Screenshot-2022-03-06-at-8.50.19-AM-300x183.png) Strategies for scaling systems Vertical scalability Vertical scalability says that if our computer is not powerful enough to run a program, we simply procure a computer that is. This is the simplest approach because we don’t have to make any change to the system itself, just the hardware it runs on. Supercomputers are a manifestation of this strategy of scalability. This is essentially throwing money at the problem to avoid design complexity. We can build more powerful computers, but the cost of doing so gets exponentially larger. And there are limits to even that. We certainly can’t build a computer powerful enough to run the entirety of Google. So the vertical scalability strategy can take us a good way, but it is not enough in the face of most modern scale requirements. To scale further, we need to make fundamental changes to our program itself. Horizontal scalability The simplest computer program is one that runs on one computer. The limits of vertical scalability indicate that the fundamental bottleneck in serving web-scale workloads is that our programs are bound to one computer. Horizontal scalability is the process of designing systems that can utilize multiple computers to achieve a single task. If this can be achieved, then scaling the system is a simple matter of adding more and more computers instead of being forced to build a single extra-large computer. The challenges of “distributing” a program can be far harder than the actual logic of the program itself. We must now deal not just with computers, but with the wires between those computers. The[ fallacies of distributed computing](https://en.wikipedia.org/wiki/Fallacies%5Fof%5Fdistributed%5Fcomputing?ref=kislayverma.com) are very real, and demand that horizontal scalability be baked into the very fabric of the program instead of being tacked on from above. Distributed systems In embracing horizontal scalability, we embrace[ distributed systems](https://kislayverma.com/tag/distributed-systems/) – systems in which various computing resources like CPU, storage, and memory are located across multiple physical machines. These are complicated[ architectures](https://kislayverma.com/category/software-architecture/), so let’s go through some main approaches. Distributing data Stored data typically represents the state of the system as it would be if there were no active processing going on. To store web-scale data, we have no choice but to split its storage across many machines. While this means that we have no storage limitation, the problem now is how to locate on which server is the specific data point located. e.g. If I store millions of songs across hundreds of hard disks, how do I find one particular song? Various techniques are used to solve this problem. Some of them are based on selecting which server to use in a smart, predetermined way so that the same logic can be applied while reading the data. These are simple techniques but somewhat brittle because the apriori logic needs to be updated constantly as the number of storage servers increases or decreases. Shard id or modulo based implementations are an example of this approach. ![](https://kislayverma.com/content/images/2022/03/scalability-data-sharding-300x235.jpg) Some other techniques are based on building a shared index to locate data across the set of servers. Servers talk to each other to find and return the data required while the actual reading program is unaware of how many servers there are. Cassandra’s peer to peer approach is an example of this. ![](https://kislayverma.com/content/images/2022/03/scalability-data-clustering1-252x300.jpg) Distributing Compute A computation or the running of a program typically modified the data owned by the system and therefore changes its state. Being able to leverage the CPU cores of multiple machines means that we have far more computing power to run our programs than with just one machine. But if all our CPUs are not located in one place, then we need a mechanism to distribute work among them. Such a mechanism, by definition, is not part of the “business logic” of the program, but we may be forced to modify how the business logic is implemented so that we can split it into parts and run it on different CPUs. ![](https://kislayverma.com/content/images/2022/03/scalability-distributed-compute-300x127.jpg) Two situations are possible here – the CPUs may simultaneously work on the same piece of data (shared memory), or they may be completely independent (shared-nothing). In the former, we not only have to distribute the compute to multiple servers but also have to control how these multiple servers access and modify the same pieces of data. Similar to multi-threaded programming on a single server, such architecture imposes expensive coordination constraints via distributed locking and transaction management techniques (e.g. Paxos). The shared-nothing architecture is far more scalable because any given piece of data is only being processed in one place at a point in time and there is no danger of overlapping or conflicting changes. The problem becomes one of ensuring that such data localization happens and in finding where this piece of compute is running. Replicating data ![](https://kislayverma.com/content/images/2022/03/scalability-data-replication-300x300.jpg) This is a hybrid scenario where even though our data fit on one or more machines, we deliberately replicate it across multiple machines simply because the current servers are not able to bear the compute load of reading and writing this data. Essentially there is so much processing going on that to be able to distribute compute, we are forced to distribute data as well (or at least copies of it). Using read replicas of databases to scale read-heavy systems, or[ using caches](https://kislayverma.com/software-architecture/architecture-patterns-caching-part-1/) are an example of this strategy. Considerations in distributed computing When we build a distributed system, we should be clear about what we expect to achieve. We should also be clear about what we will NOT get. Let’s consider both of these things while assuming that we are designing the system well. What we won’t get Consistency [Eric Brewer defined the CAP theorem](https://kislayverma.com/summary/working-around-the-cap-theorem/) which says that in the face of a network partition (the network breaking down and making some machines of the system inaccessible), a system can choose to maintain either availability (continuing to function) or consistency (maintaining information parity across all parts of the system). This can be intuitively understood by considering that if some machines are inaccessible, either the other should stop working (become unavailable) because they cannot modify data on the inaccessible servers, or continue to function at the risk of not updating the data on the missing machines (becoming inconsistent). Most modern systems choose to be inconsistent rather than fail altogether so that at least parts of the systems can function. The inconsistency is later reconciled by using techniques like[ CRDTs](https://crdt.tech/?ref=kislayverma.com). Simplicity A distributed system design is inevitably more complex at all levels from the networking layer upwards than a single-server architecture. So we should expect complexity and try to tackle it with good design and evolved tools. Reduction in errors A direct side effect of a more complicated architecture is an increase in the number of errors. Having more servers, more inter-server connections, and just more load on this scalable system is bound to result in more system errors. This can look (sometimes correctly) like system instability, but a good design should ensure that these errors are fewer per unit workload and that they remain isolated. What we must get Scalability This is obvious in the context of this article. We are building distributed systems to achieve scalability, so we must get this. Failure Isolation This is not an outcome but an important guardrail in designing a distributed system. If we fail to isolate the increasing number of errors, the system will be brittle with large parts failing at once. Ideal distributed system design isolates errors in specific workflows so that other parts can function properly. Why are distributed systems hard? In a word – coupling. While there are many types of coupling in software engineering, there are two which play a major role in hindering system scalability. Both of them derive from a single server program’s assumption of “global, consistent system state” which can be modified “reliably”. Consistent state means that all parts of a program seem the same data. Reliable modification of the system’s state means that all parts of the system are always available and can be reached/invoked to modify it. But as we have seen, the CAP theorem explicitly outlaws the consistency-availability-invocability in a distributed system. This makes the leap from single server architecture to distributed architecture very difficult. Let’s look at both these types of coupling. Location Coupling Location coupling is when a program assumes that something is available at a known, fixed location. e.g. A file parsing program assumes that the file is located on its local file system. or a service assuming its database is available at a given fixed location. or a subpart of a system assuming that another subpart is part of the same runtime. ![](https://kislayverma.com/content/images/2022/03/scalability-location-coupling-300x206.jpg) It is difficult to horizontally scale such systems because they do not understand “not here” or “multiple”. Additionally, their implementation might assume that reaching out to these other components is cheap/fast. In distributed systems, both aspects are critical. A subcomponent doing a part of the computations may be running on some other server entirely and therefore difficult to find and expensive to communicate with. A database may be many servers working as a sharded cluster. Location coupling is therefore a key problem in being able to horizontal scalability because it directly prevents resources from being added “elsewhere”. Breaking Location coupling ![](https://kislayverma.com/content/images/2022/03/scalability-breaking-location-coupling-300x181.jpg) The trick to breaking location coupling lies in abstracting the specifics of accessing another part of the system (file system, database, subcomponent) from the part which wants to access it behind an interface. This means different things in different scenarios. e.g. at the network layer, we can use DNS to mask the specific IP addresses of remote servers. Load balancing techniques can hide that there are multiple instances of some particular system are running to service high workloads. Smart clients can hide the details of database/cache clusters. An interesting way of becoming agnostic to the called system physical location is by not trying to locate them but by leaving all commands in a common, well-known place (like a message broker) from where they can up the commands and execute them. This, of course, creates location coupling with the well-known location but ideally, this is smaller in magnitude than having all parts of the system being coupled to all others. Temporal Coupling This is the situation where a part of a system expects all other parts on which it depends to serve its needs instantaneously ([synchronously](https://kislayverma.com/tag/asynchronous-programming/)) when invoked. In the context of scalability, the problem with temporal coupling is that all parts must now be “scaled up” at the same time because if one fails, all its dependent systems will also fail. This makes the overall architecture sensitive to local spikes in workload – any change in load on any part of the system and the whole system can crash, thereby removing much of the benefits of horizontal scaling. Breaking temporal coupling The most common approach to breaking temporal coupling is the use of message queues. Instead of invoking other parts of a system “synchronously” (invoking and waiting till the output appears), the calling system puts a request on a message bus and the other system consumes this You can read more about[ messaging concepts](https://kislayverma.com/software-architecture/defining-messaging-terms-precisely/) and how[ events can be used to build evolutionary architectures](https://kislayverma.com/software-architecture/using-events-to-build-evolutionary-architectures/). The event/message-driven architecture can massively increase both the scalability and resilience of a distributed system. From the internet [Sander Mak](https://twitter.com/Sander%5FMak?ref=kislayverma.com) on how the team at Picnic thinks about [scaling up their architecture and processes.](https://blog.picnic.nl/software-architecture-for-scale-ups-9c11cb6e1c7e?ref=kislayverma.com) This Wiley review of [academic literature on systems thinking applied to engineering](https://onlinelibrary.wiley.com/doi/epdf/10.1002/sres.2808?ref=kislayverma.com) is comprehensive and fascinating, even if a little daunting to get through. [Jessica Kerr](https://twitter.com/jessitron?ref=kislayverma.com) explains why we should aim for [better software, not better coordination](https://jessitron.com/2021/08/02/better-coordination-or-better-software/?ref=kislayverma.com). A topic close to my heart and something that I have written about several times in this newsletter (see [this](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) and [this](https://kislayverma.com/organizations/reduce-collaboration-by-good-design/)). That’s all for this week folks. Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #54: An introduction to Caching (Part-2) URL: https://kislayverma.com/it-depends-54-an-introduction-to-caching-part-2/ Last updated: 1970-01-01T00:34:37.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #54: An Introduction to Caching (Part-2) Hello Everyone! Welcome to the 54thd episode of It Depends. Hope you are all doing well. We continue our look into caching as an architecture pattern by going into some more nuanced topics like cache arming and the thundering herd problem. And then on to the best of the tech internet. An introduction to Caching (Part-2) You can read [the original article](https://kislayverma.com/software-architecture/architecture-patterns-caching-part-2/) directly on the blog. In [part 1 of this series](https://kislayverma.com/software-architecture/architecture-patterns-caching-part-1/ "part 1 of this series"), we looked at the different types of caches and the various ways they can be used to scale up applications. Now let’s look at some nuances of using caching. Scaling caches Like any other part of the system design, caches come under load as the scale of the application increases. External caches are servers like any other and can buckle under the read/write traffic being sent their way. Even in-memory caches can suffer degraded performance due to read locking if too many application threads try to access them, although this is much harder to get to and easier to mitigate. let’s look at some problems and solutions scaling caches. Scaling to more traffic A cache is essentially a data store, and the problem of scaling for traffic is a well-known one in the database domain. Rising traffic can cause scalability problems by increasing the CPU usage or by choking the network bandwidth available to a server. The most straightforward way to scale for an increase in traffic is to have multiple servers which can serve the traffic. In databases, we typically configure a master-slave (aka leader-follower) topology where all writes go to a single server which replicates them across all the other servers. This way, all servers have all the data and the application can connect to any of them to read them. This reduces the load on each server by a factor of the number of servers. ![](https://kislayverma.com/content/images/2022/02/caching-read-replicas-for-scaling-300x228.jpg) Since writes to cache data are much rarer than reads, the overhead of replicating the writes to all slave servers is usually acceptable. Both [Twemproxy](https://blog.twitter.com/developer/en%5Fus/a/2012/twemproxy?ref=kislayverma.com "Twemproxy") and [Redis Sentinel](https://redis.io/topics/sentinel?ref=kislayverma.com "Redis Sentinel") are examples of implementation which use redundancies to scale caches. Scaling to larger data size We have already discussed this under external distributed caches. If we want to store more data, we really don’t have a choice but to distribute it across more than one server. This directly brings the cache into the world of distributed systems with all its attendant pros and cons. ![](https://kislayverma.com/content/images/2022/02/caching-clustered-cache-300x242.jpg) Note that distributing data across multiple servers solves for both data and traffic scaling since no single server faces as much traffic as in the case of a single cache instance. However, we can look at only doing redundant deployments if our problem is just traffic and not data volume. Data replication is a simpler problem than data distribution. Both [Twemproxy](https://blog.twitter.com/developer/en%5Fus/a/2012/twemproxy?ref=kislayverma.com) and [Redis Cluster](https://redis.io/topics/cluster-tutorial?ref=kislayverma.com "Redis Cluster") are examples of distributing data to scale to larger data volumes. Cache Stampede In high throughput systems, a scenario emerges in the use of caches which, if not handled properly, can bring down the entire system. This has happened to pretty much every large scale company like Facebook, Doordash, Instagram etc. I have personally encountered it while building the Promise Engine at Myntra. Let’s say that we are using a read-through cache where application processes first try to read data from a cache, and if not found (cache miss) they tried to load it from the source database/system. If this is a high throughput system involving a large number of concurrent accesses to the cache, then even if a single key is missing, a large number of processes will try to access the database to read this data at the same time. This now triggers a flood of traffic to the database, which may now collapse under it since it always expects a cache to sit in front of it and is not designed to take such a heavy spike of traffic. This means that a single heavily accessed key being missing from the cache can trigger a complete system collapse. Such is the fine line of high scalability! Prevent cache stampede Before we try to solve the problem, please evaluate if this is a problem for your system. If your cache were to vanish and the traffic were to hit your database at once, would the database hold up with perhaps only a temporary spike in latency? If so, you do not need to worry about a cache stampede. It is worthwhile doing a simulation in your production environment to verify this. If you think this is going to be a problem for you. there are a few approaches you can take. Keep cache always full This approach treats the cache as the source of truth. The idea is that since a cache stampede is triggered by a cache miss, if you can load all the data into the cache using a refresh ahead strategy, then a cache miss will never happen and hence cache stampede will be avoided. While this is possible for small datasets, keeping large data set fully loaded at all times is not always feasible for cost reasons. Nominate one process to fill the cache Let’s say a cache miss occurs when 1000 processes are trying to access a key concurrently. To prevent all of them from rushing to the database, we can implement a mutex lock/leader election mechanism to elect one process which goes to the database to get the data and refresh the cache. The other processes can either wait for the cache to be filled before trying again (leading to temporarily increased latency) or they can all throw an error to their respective callers (leading to a brief spike of errors). Obtaining locks in a distributed environment is a complicated but solved problem. Zookeeper and Redis both offer convenient ways of doing this, but you can roll your own using lock entries in a table if you think it is easier (it isn’t). Probabilistic early expiration This is a smarter solution than most teams require, but the idea is to balance the above two approaches. If we cannot load all the data in the cache but still want to prevent cache misses, then the only way is to reload the keys intelligently before it expires from the cache. This paper outlines one of the strategies for preemptive loading of keys before they expire. TL;DR Cache stampede is one instance of the new failure modes introduced into an architecture that uses caching. In designing high scale applications, this outcome of cache miss should be carefully considered before using caches. From the internet [James Long](https://twitter.com/jlongster?ref=kislayverma.com) on the [future of SQL on the web](https://jlongster.com/future-sql-web?ref=kislayverma.com), or absurd-sql. [Chelsea Troy](https://twitter.com/HeyChelseaTroy?ref=kislayverma.com) has a series about [quantifying and managing technical debt](https://chelseatroy.com/2021/01/14/quantifying-technical-debt/?ref=kislayverma.com). I agree with some of it and don’t agree with some, but it’s an extremely insightful series of articles. A look at [how Twitter’s public API was rebuilt](https://www.infoq.com/presentations/twitter-public-api/?ref=kislayverma.com). That’s all for this week folks. Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #53: An introduction to Caching (Part-1) URL: https://kislayverma.com/it-depends-53-an-introduction-to-caching-part-1/ Last updated: 1970-01-01T00:34:22.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #53: An Introduction to Caching (Part-1) Hello Everyone! Welcome to the 53rd episode of It Depends. Hope you are all doing well. This week, we are taking a deep dive into the world of Caching, and then surfing the best waves of the technology internet. An introduction to Caching (Part-1) You can read [the original article](https://kislayverma.com/software-architecture/architecture-patterns-caching-part-1/) directly on the blog. Performance has always been a key feature of technical systems. Today on the internet, sub-second latencies are the norm. It costs companies money if their pages load slowly because potential customers won’t wait longer than that. On the other hand, there is more and more data from many different sources which have to be loaded into a rich user experience (think of the number of things going on a typical FB page). This data gathering problem is further exacerbated by the trend towards microservices. Given all this, how are we to build super-fast systems? What is caching? Caching is the general term used for storing some frequently read data temporarily in a place from where it can be read much faster than reading it from the source (database, file system, service whatever). This reduction in data reading time reduces the system’s[ latency](https://kislayverma.com/software-architecture/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/). This also increases the system’s overall throughput because requests are being served faster and hence more requests can be served per unit time. ![](https://kislayverma.com/content/images/2022/02/caching-why-use-cache-300x285.jpg) Microprocessor architectures have long employed this technique to make programs run faster. Instead of reading all data from the file system all the time, microprocessors employ multiple levels of cache where it is slower to read from lower levels than from the higher levels. The game is now one of making sure that the data read most often is in the highest level cache, and so on. Getting this right can make a dramatic difference to the speed of a running program. Typically, caches hold a copy of the source data for some time (called expiration time or time-to-live (TTL)) after which the data is “evicted” from the cache. As more data is loaded into a cache of finite capacity, some strategies may be applied to decide which data is retained and which is evicted. Where can we employ caching **The system must be read-heavy** – As should be clear by now that caching is only a solution for scaling the reading of data. So if you are building a system that reads data significantly more times than it writes data, caching can be a powerful technique for you. Write or compute-heavy systems have relatively less to gain from caching. **Tolerance to stale data** – Caching can only be applied if we can tolerate reading stale data (at least for some time). Since the cache is a copy of the source data, it is possible that the source data changes and the cache doesn’t know of it. Some systems can tolerate this e.g. the likes counter on your Instagram reel can be a little behind the actual count for some time. Other systems like accounting systems cannot tolerate working with stale data. If working with stale data is not acceptable in our system, then caching is not a viable option for scaling. **Data doesn’t change frequently** – A corollary to the above is that caching works best for data that doesn’t change frequently. This will invalidate the cache and push the system’s tolerance for data staleness. There are strategies to cache frequently changing data (like write-through caching discussed below), but they are usually more expensive to execute. **Limited to the amount of data that cache can hold** – In most large scale use cases it is not possible to store all of the data that you are using into the cache. e.g. You may not be able to load the profile data of all your customers into your cache because then the user data cache would be as large as the user database and that can get expensive. In these situations, we need to be smart about what is cached (the most frequently used data points) and what is retrieved from the source. Levels of Caching As I said earlier, caching is a general concept, not restricted to use only in web architectures. We can also employ it at various levels of the system architecture. Some of these are done by application developers explicitly, others are done behind the screens by tools and frameworks being used. I have called out some of the most commonly seen levels of caching in internet architectures, but there may be several more in between. **Microprocessor** – We have already covered this above. Microprocessors and Operating systems work together to cache data into registers and other caches to make our programs run faster behind the screens. **Databases** – Most databases employ some sort of internal caching mechanism to keep “hot” data in memory. e.g. MySQL loads small but frequently accessed tables entirely in memory. This is also typically hidden from developers, but understanding these mechanisms can be helpful in debugging database performance issues under high load. **Application** – Applications typically cache the data they own in cache tools of their choice. This is a developer-driven activity and one where we can exert the most control. **Scatter-gather** – This is a flavour of application-level caching but for applications that gather data from multiple sources (each of whom may have their internal caches). This level of caching is widely used on the internet, typically in backend-for-frontend type applications and has a significant effect on the end-user experience. **CDN** – This is “internet level caching” where we cache entire pages and distribute them across geographies so that they be read from servers very close to the end-user. Caching Strategies Depending on the type of application, the type of data, and the expectation of failure, there are[ several caching strategies](https://docs.oracle.com/cd/E15357%5F01/coh.360/e15723/cache%5Frtwtwbra.htm?ref=kislayverma.com#COHDG5177) that can be applied for caching. Read through ![](https://kislayverma.com/content/images/2022/02/caching-read-through-cache-300x199.jpg) This is the simplest and most commonly used strategy. The application tries to read the data from the cache. If it finds the data (known as “cache hit”), all is well. If it doesn’t find the data in the cache (known as a “cache miss”), it goes to the source to fetch it and loads it in the cache. If the cache is full, then some policy based on the nature of the data (e.g. Least frequently used, most recently used) is used to identify the data which should be removed from the cache to make room for the incoming data. In this strategy, tasks that encounter a cache miss will have a higher latency than those that get a cache hit. Write through ![](https://kislayverma.com/content/images/2022/02/caching-write-through-cache-300x210.jpg) In applications where the cache can hold all the data and is expected to be always fresh, we can use the write-through pattern. In this, every write is first done to the cache, and then to the source. This means that the cache is always in sync with the source. The cache becomes the source of truth for the application and it never reads the data from the source. On the flip side, this requires the full data to be loaded in the cache at the outset. It also introduces higher latencies on the write operations, and higher write load on the cache system, which may, in turn, impact its read performance. Write Behind ![](https://kislayverma.com/content/images/2022/02/caching-write-behind-cache-300x201.jpg) Similar to the write-through strategy, the application first new data to the cache. But after that, the application process returns to its main duties. The cache itself or some other process runs periodically and batch-writes the cache data into the source. This is an effective strategy for cases where we do not want to bear the latency cost of writing to the source in the main application process and the cache is reliable enough that we are sure of not losing the data before it is pushed to the source. This strategy requires that writes to the source never fail while dumping data from the cache, or there be a resolution mechanism to resolve inconsistencies. Refresh Ahead ![](https://kislayverma.com/content/images/2022/02/caching-refresh-ahead-cache-300x194.jpg) In this strategy, we pre-emptively refresh all or part of the data of a cache as it is reaching its expiry time. How to decide what to reload is up to the application. Note that the application may still face cache misses if not all data can be reloaded, and there this technique is usually combined with “read-through caching but with the idea that reloading process should make cache misses rarer. This is not a very common technique because it requires setting up a process to identify expiring data and reloading it based on some smart logic. This is usually not needed by applications. Choosing a cache implementation Now that we know of the various caching strategies, let’s consider what kind of cache implementation to actually use. While technically any data structure/medium that is faster to access than its source version can be used for caching, typical cache implementations are key-value stores of some sort. Three types of cache implementations are popular. In-memory ![](https://kislayverma.com/content/images/2022/02/caching-in-mem-cache-300x225.jpg) This is the case where the reading application loads the data into its main memory (as a hash table or map) and uses it as the cache. This makes for the fastest possible access since the data is available literally like a program variable. It is also the simplest possible implementation since it does not introduce any new elements into the system architecture. Many libraries are available to abstract the implementation details of caching/eviction etc from user code. There are also several downsides to this style. The cache lives inside the application, so if the application goes down, the cache vanishes and has to be rebuilt while launching the application. The memory footprint of the application increases and the amount of data that can be cached is limited by that. This type of cache is also local to the application server. If you have multiple instances of the applications running, each of them will have its own cache (waste of memory) and these may be temporarily out of sync with each other if one instance reloads its cache while the others still haven’t. External ![](https://kislayverma.com/content/images/2022/02/caching-external-single-server-cache-300x203.jpg) We can use a standalone system like Redis or Memcached as an external cache. It is essentially like having an external server that all nodes of an application talk to and which stores the hash table instead of storing it inside the application. This introduces a new element to manage in the architecture but creates a central cache that is durable and ensures that all instances of an application see the same cache value. This type of cache has the problem of failure tolerance. If the cache server crashes for any reason, the application will fail. This is solved by some implementations by having redundant caches which are kept in sync with a “leader server” but that can step in if the leader fails (Redis Sentinel uses this mechanism). This gives failure tolerance at the cost of design complications. External Distributed ![](https://kislayverma.com/content/images/2022/02/caching-external-ditributed-cache-300x175.jpg) Both the in-memory cache and the external cache suffer from some scale problems. The amount of data that can be stored in either of them is limited to the memory size of a single server. As large scale systems emerge and the volume of data to be cached increases, this becomes a bottleneck. To overcome this, we can use an external yet distributed cache implementation e.g. Redis cluster. In this architecture, the data is distributed across multiple instances of the cache servers. More servers can be added to this “cluster” as data size grows, making this architecture horizontally scalable. Data distribution among the servers is typically managed by the cache implementation itself. The reading application can continue to treat the cluster as a single entity when reading data. This is a full scale distributed architecture and comes with all the associated problems like node failure, split-brain, data redistribution etc. It is also the only feasible architecture at the highest web scale. TL;DR Caching is a powerful scalability technique that can be used in many different scenarios and in many different flavours to speed up the performance of our applications. In the next week’s newsletter, we will look at some more nuance in the use of caches and a specific but deadly failure pattern in systems that rely on caching. From the internet Robin Hanson talks about [3 types of general thinkers](https://www.overcomingbias.com/2021/12/three-types-of-generalists.html?ref=kislayverma.com) and the perils of ideological fervour. This post from the Slack engineering team about [how they design their APIs](https://slack.engineering/how-we-design-our-apis-at-slack/?ref=kislayverma.com) had gone absolutely viral some time ago. I believe it is worth revisiting. Paula Paul explains why she thinks [viewing architecture an activity is better than viewing an architect as a person](https://medium.com/simply-technology/architect-is-a-team-activity-not-a-person-5b0b7719ae0?ref=kislayverma.com). That’s all for this week folks. Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #52: The Pentagon of Entity Models URL: https://kislayverma.com/it-depends-52-the-pentagon-of-entity-models/ Last updated: 1970-01-01T00:34:18.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) Already a subscriber? Jump ahead to the good stuff! But if someone forwarded this to you, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #52: The Pentagon of Entity Models Hello Everyone! Welcome to the 52nd episode of It Depends. Hope you are all doing well. This week, I look at an interesting pattern of system evolution and then share the best of the internet as usual. The pentagon of entity modelling You can read [the original article](https://kislayverma.com/software-architecture/the-pentagon-of-entity-models/) directly on the blog. I recently read an article by Matt Ricard about the "[Heptagon of Configuration](https://matt-rickard.com/heptagon-of-configuration/?ref=kislayverma.com)" in which he discussed how configurations evolve in a cycle. It struck me that there's another thing that follows a similar routine - entity attributes. Entity attributes often grow into entities in their own right. Tax number grows into tax records and engines. boolean flags grow into multi-valued state machines etc The entity evolution cycle ![](https://kislayverma.com/content/images/2022/02/pentagon-of-entity-models-300x247.jpg) This is the transition as I've seen it go in my experience. Some new, urgently required properties of an entity are modelled as a hack in a configuration somewhere. This configuration is then pulled into the entity's main data mode, typically as a boolean attribute. Boolean attributes evolve into named/enumerated types, sometimes with more associated data. Enumerated types become full-fledged entities linked to their previous host entity Entities evolve into complete business domains with their systems, process, almost organizations. Why? This iterative process happens because as we explore more use-cases, a more and more complex domain model emerges. The pentagon of entity model evolution is just a sign of developers continuously trying to keep up with a deeper understanding of the business but not yet knowing enough to model its nuances fully. [I interpret technical debt similarly](https://kislayverma.com/programming/uncertainty-and-learning-as-tech-debt/) \- as wisdom in hindsight. The system's implementation is essentially just catching up with the developer's understanding of the problem domain. This is a perfectly safe, natural way for systems to evolve. But if you want to crank the wheel a little faster, the only way to move fast but stay on track is to [spend more time understanding the business domain](https://kislayverma.com/software-architecture/the-mechanics-of-software-evolution/) upfront. Understanding the domain better can help us predict future needs and build the necessary extensibility, if not the actual capabilities in our designs right away. From the internet For me, [this article/talk by David Rosenthal](https://blog.dshr.org/2022/02/ee380-talk.html?m=1&ref=kislayverma.com) is the last word in debunking the utter bullshit of the crypto-blockchain ecosystem. It is a technical, meticulous, and brutal takedown of the so-called decentralization of Blockchains. Brian Footer and Joseph Yoder of the University of Illinois (UC) have written a paper on the [architectural pattern known as “big ball of mud”](http://laputan.org/mud/?ref=kislayverma.com). Odd as it may seem, it proves that everything can have pros and cons if you look objectively enough. [Efe Karakus](https://twitter.com/efekarakus/status/1487473629512278023?ref=kislayverma.com) wrote this fantastic Twitter thread on [building client-side platforms](https://twitter.com/efekarakus/status/1487473629512278023?ref=kislayverma.com). That’s all for this week folks. Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #51: The Mechanics of Software Evolution URL: https://kislayverma.com/it-depends-51-the-mechanics-of-software-evolution/ Last updated: 1970-01-01T00:34:14.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #51: The Mechanics of Software Evolution Hello Everyone! Welcome to the 51st episode of It Depends. Hope you are all doing well. Last week, some of you asked about when I was going to get back to the podcast. I am trying to gather the energy for it - perhaps this Sunday will prove to be the lucky day. Onwards. The mechanics of software evolution You can read the [original post](https://kislayverma.com/software-architecture/the-mechanics-of-software-evolution/) directly on the blog if you’d prefer. Have you ever heard engineers in your team complain about only building “business features” and never doing any “tech work”? There are some ways in which this complaint is legitimate, but I feel that there is an underlying unity to both these things. I’d like to explain this by applying an evolutionary lens to changes in software. Let’s start by identifying the kind of change requests we typically see in software. Most of the time, changes come as feature requests for enhancing whatever capabilities already exist. Once in a while, changing business needs aggregate into large change requests which require new things that the system has never had before. In the latter case, the need for ground-up thinking is obvious. New capabilities or abstractions are being explicitly requested by the external world, and we must deliver. While such large changes are tricky to deliver on, they are also the more straightforward ones (in terms of what the output is expected to be). There is an explicit directive to “evolve” the system. Changes add up However, the former case often also contains the seem need, though it is embedded a little deeper. Each small feature is innocent in isolation, but applying a[ systemic lens](https://kislayverma.com/books/book-review-thinking-in-systems-a-primer/) to it can sometimes reveal a more fundamental gap in technical capabilities. This gives us the opportunity to devise a more holistic solution that not only addresses the current requirement but also add some new fundamental capability to the system. This is what all the advice around[ “understand requirements clearly”](https://kislayverma.com/software-architecture/system-design-from-one-level-up/) is talking about. We have to understand the immediate requirements properly, but we also have to read between the lines a little bit to see where the customer need is coming from and try to take the system there directly instead of traversing a morass of many small, disjointed changes feature requests. Strategy is not just the arena of business. It plays an equally important role in how technology evolves. Simon Wardley has formally adapted this relationship of strategy and evolution into his[ “Wardley Mapping Framework”](https://kislayverma.com/books/book-review-wardley-mapping/). As a CEO looks at market trends in aggregate and builds a strategy to evolve his company to keep up with them,[ the engineer has to look beyond the obvious requirement](https://kislayverma.com/programming/make-it-better-every-day-of-the-week/) today to see if there is an emergent theme underneath a set of feature requests when seen in aggregate. Evolving software at the edge of chaos Let’s phrase the design-for-the-future approach in evolutionary terms. ![](https://kislayverma.com/content/images/2022/02/systemiic-evolition-using-features-272x300.jpg) Change requests, however small, are the environmental pressure for software evolution. Teams that can identify the driving forces behind seemingly small requests and develop coherent abstractions in their systems in time will have adapted the best to this pressure. They will live to see another day. Teams that consistently fail to do this will perish, something alongside their entire organizations. Before the rallying cry of YAGNI etc starts, there is obviously a fine line to walk here. An over-engineered system is just as bad as an under-engineered when it comes to being a fit for the business landscape. We have to find a balance where we allow the emergent themes to be manifested somewhat clearly before we solve for them. Too far to one side is the chaos of hacks and piecemeal changes, too far to the other is too many useless abstractions that slow down everything else. When done well, a system grows new layers of abstraction and complexity just in time to prevent small needs from becoming big problems. A constant evolution mindset is best put into effect when designing anything, but we also have to deal with existing code that is getting outdated due to changes to the ecosystem.[ Continuous refactoring](https://kislayverma.com/programming/saving-the-day-with-continuous-refactoring/) can be a good way to encounter this. Refactoring is a great opportunity to identify scattered yet recurring patterns in code and see if there is an opportunity to aggregate them into something more concrete. It also gives a good sense check of whether the perceived “theme” is real (something happening multiple times in code) or just wrong intuition. Large change requests are top-down evolution, but this is the kind of purposeful bottom-up evolution[ I have written about before](https://kislayverma.com/meta-thinking/the-sense-of-purpose-in-a-complex-system/). If we don’t adopt this features-as-evolutionary-pressure mindset, scattered solutions to small requests will pile up and result in[ increasing tech debt](https://kislayverma.com/programming/uncertainty-and-learning-as-tech-debt/) or an eventual large change request to the system. This “stop-everything-and-rearchitect” scenario is expensive and risky (since such efforts are liable to fail or underachieve). TL;DR My recommendation is to use feature requests as a breeding ground for the next generation of the system’s architecture. By continuously evaluating what we are being asked to change, we can jump the gun and get to the next level faster and often more safely. From the internet [Jamie Brandon](https://www.scattered-thoughts.net/?ref=kislayverma.com) writes [against SQL](https://www.scattered-thoughts.net/writing/against-sql/?ref=kislayverma.com), or rather against SQL being practically the only representation of relational model/set theory. From Ron Sobol of team Granulate, here’s a deep dive into [scheduling in Kubernetes](https://granulate.io/a-deep-dive-into-kubernetes-scheduling/?ref=kislayverma.com). Mahesh Balakrishnan shares some sage advice for senior engineers and managers from his [journey of building a production database](https://maheshba.bitbucket.io/blog/2021/10/19/42Things.html?ref=kislayverma.com). That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### It Depends #50: The crypto-web's fatal flaw URL: https://kislayverma.com/it-depends-50-the-crypto-web-s-fatal-flaw/ Last updated: 1970-01-01T00:34:08.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #50: The crypto-web's fatal flaw Hello Everyone! Welcome to the 50th episode of It Depends. Fif-teee. The big five-oh. It’s been a long, slow, and irregular road, but we got here, so YAY!!! The crypto-web’s fatal flaw You can read the [original post](https://kislayverma.com/decentralization/the-crypto-webs-fatal-flaw/) directly on my blog if you’d prefer. ![](https://kislayverma.com/content/images/2022/01/jievani-weerasinghe-NHRM1u4GD_A-unsplash-300x184.jpg) “The form of law which I propose would be as follows: In a state which is desirous of being saved from the greatest of all plagues—not faction, but rather distraction—there should exist among the citizens neither extreme poverty nor, again, excessive wealth, for both are productive of great evil . . . Now the legislator should determine what is to be the limit of poverty or of wealth.” - [Plato](https://gordoncstewart.com/2012/02/18/plato-on-wealth-and-poverty/?ref=kislayverma.com) I have long held that as compared to the outsized influence it enjoys in our lives, most of Wall Street actually plays a minimally important role in the world. It is the place from where the ultra-rich set the engine of the world to turn for their ideological and financial benefit. This is hardly a unique opinion and does not merit further repetition. Web3 started with the promise of breaking the control of the few by decentralization. But by now it is well acknowledged that blockchain technology has the fatal flaw of not being scalable by design. So for all practical purposes, try as it might, it cannot build the decentralized world for everyone that it promises. Coupled with the problem that[ complex-tech-at-scale fundamentally gravitates towards centralization](https://blog.fabiomanganiello.com/article/Web-3.0-and-the-undeliverable-promise-of-decentralization?ref=kislayverma.com), what I get is the sinking feeling that the internet may never be truly decentralized. But for me and many others, there is a deeper, more fundamental problem with this eco-system. The crypto-web is irretrievably tied up with money. Nothing in it exists that is not defined or measured by money. Indeed, the fundamental construct of the crypto-world is a distributed ledger! By linking everything up to money in the form of coins/tokens (by design), and with absolutely no oversight (also by design), it invites the kind of actors that focus on the financial aspects to the exclusion of all else. The barrier to entry is only computing power so the already rich can get in easily. The more such actors come in, the more difficult it gets for people with less computing resources to get in. And so, progressively, the crypto world looks just like wall street, where people who make nothing sell everything at prices that are beyond the reach of everyone except the super-wealthy. For me, the crypto-web is the direct philosophical descendant of wall street. For all the talk about empowering creators and so on, the most money to be made in this economy is not by creating digital art but by buying and selling it via NFT (or other means too – I’m no expert). Crypto communities focus exclusively on prices instead of outcomes. Creating new things is not fundamental to this system, but continuous trading of anything possible is existential (no trading -> no mining -> dead blockchain). Also very reminiscent of financial markets. As an engineer, I think decentralization becoming synonymous with blockchain/crypto is a bit of a tragedy. It is like choosing implementation detail before thinking through any other solutions. To my mind, the only salvageable use-case of technology here is distributed identity management of some sort (public/private keys on the blockchain perhaps), most other things can be done in better ways. There are other ways of doing decentralized data, identity, and so on. The[ Solid project](https://solidproject.org/?ref=kislayverma.com) is giving this a good (but extremely sluggish) shot. The biggest criticism I hear of projects like this is that there is no killer app (true), or that it will never work for the masses because[ people don’t want to run servers](https://moxie.org/2022/01/07/web3-first-impressions.html?ref=kislayverma.com). To this, I can only say that in what sense is crypto working for the masses? The reality, IMO, is that the crypto-web is generating money from the get-go, which makes everyone forget the original problem statement – a fairer, digital existence for everyone on the planet. Maybe the utopia of a truly decentralized internet where people own their own data will never come to pass (due to any number of reasons). But reaching the promised land of Crypto would be far, far worse. In that dystopia of pure finance, the technology will no do what it claimed to do, it will be far more profitable to trade than to produce, nothing you buy will really exist, and everything will exist only to be sold. Not my idea of heaven. From the internet The first [standard for assuring a picture’s authenticity](https://petapixel.com/2022/01/26/the-first-standard-to-assure-a-photos-authenticity-has-been-created/?ref=kislayverma.com) has been proposed! Exciting times in the fight against fake news and propaganda. If you, like me, wish you had somehow understood Maths a little better in your school/college, you’re in luck. [3Blue1Brown](https://www.youtube.com/channel/UCYO%5Fjab%5FesuFRV4b17AJtAw?ref=kislayverma.com) is a fantastic Youtube channel with explanations of maths concepts. I spent a few hours rabbit-holing this week :) Here are some ideas on how to [scale the practice of architecture](https://martinfowler.com/articles/scaling-architecture-conversationally.html?ref=kislayverma.com) by [Andrew Harmel Law](https://twitter.com/al94781?ref=kislayverma.com) That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #49: The Marketplace Scam (seller beware!) URL: https://kislayverma.com/it-depends-49-the-marketplace-scam-seller-beware/ Last updated: 1970-01-01T00:32:46.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #49: The Marketplace Scam (seller beware!) Hello Everyone! Welcome to the 49th episode of It Depends. Hope everyone is doing well and staying safe. It's been a long break (apologies for that), but hopefully, I'm back to stay :) ![](https://kislayverma.com/content/images/2022/01/jezael-melgoza-HYQvV8wWX18-unsplash-300x187.jpg) *“Every great magic trick consists of three parts or acts. The first part is called “The Pledge”. The magician shows you something ordinary: a deck of cards, a bird, or a man. He shows you this object. Perhaps he asks you to inspect it to see if it is indeed real, unaltered, normal. But of course… it probably isn’t. The second act is called “The Turn”. The magician takes the ordinary something and makes it do something extraordinary. Now you’re looking for the secret… but you won’t find it, because of course you’re not really looking. You don’t really want to know. You want to be fooled. But you wouldn’t clap yet. Because making something disappear isn’t enough; you have to bring it back. That’s why every magic trick has a third act, the hardest part, the part we call “The Prestige”.”* \- [Christopher Priest (The prestige)](https://www.goodreads.com/author/quotes/23419.Christopher%5FPriest?ref=kislayverma.com) [Marketplaces](https://kislayverma.com/platform-thinking/marketplaces-are-not-platforms/) are everywhere on the internet. And they all typically start like this. Find a fragmented market for service/product where the service providers are difficult to discover and compare for the customer. Build a product that brings a large number of them under a single place to facilitate discovery and order. Convince service providers that they will make more money by joining you. Convince customers that this is a much better way of shopping, often by giving crazy discounts. …you know how it goes. The advantages to both the service providers and the marketplace owner are obvious. The former gets more demand, and the latter gets a share of the transactions. The customer gets a super convenient experience, and very often marketplaces have some kind of provider rating mechanisms that help the customer make a more informed choice. So far, so good. Everyone seems to be winning. The scam emerges But this is only the first of the marketplace magic trick, the pledge if you will. Because while providers are enjoying the increased revenue coming via the marketplace, the marketplace itself has already moved on to the next step of the playbook – identifying customer needs. Because it controls the entire supply of customer data (behavioural, transactional, everything), it can now build a very deep understanding of where the customers are, what they want, and when they want it. Initially, the data is used for targeted pricing, advertisements, better-informed product decisions, etc. This stage sees intense competition among marketplaces. Many marketplaces do not come out of this alive. Those that do typically command a tremendous scale of both demand and supply. This stage is typically too good to be true for both customers and service providers. Service providers haven’t changed their business much but are making a lot more money. Customers are getting insane discounts and near-instant gratification (as compared to the past). Unfortunately for the service providers, this is when marketplaces bring in the next devastating piece of the strategy – replacing the service provider. The thinking goes that why should the marketplace owner share anything with service providers when it can itself become a service provider? Providing the service is, after all, just a skill that anyone with time and resources can acquire (fairly easily in most domains). So fashion marketplaces like Myntra start their own clothing brands, food marketplaces like Swiggy start their own kitchens, convenience/delivery marketplaces like Dunzo start their own stores and warehouses, travel marketplace like Uber try their hand at self-driving cars, and so on. Now the scam is beginning to fully reveal itself in the power relationship between the marketplace and the service providers. While the providers were needed in the early days of the platform to solve the marketplace cold-start problem (demand needs supply, supply needs demand), they become a cost as the marketplace builds its own service capabilities. So the marketplace now tries to get a larger and larger share of the proceedings from the providers. Theoretically, providers that don’t like this could walk away, but by now the whole system has become too dependent on the marketplaces by now. Customers are no longer loyal to stores and brands but are addicted to the convenience of the marketplace (e.g. I almost completely stopped going to my local grocery store and started ordering from Dunzo, Grofers etc). The marketplaces have captures the demand, and are now moving to capture an increasing share of the supply. Sure they can’t capture all of the supply, but they don’t need to as long as they capture the most lucrative bits. This playbook has now been repeated and perfected by so many marketplaces that it is now the obvious, logical path for any such business. There is no reason to believe that any marketplace will “not” follow it in the future. In my view, this is not a way of running an economy that can be sustained in the long run. There is a whole socio-economic commentary to be done here about the transfer of wealth and power from SMEs to a smaller elite, but I will abstain from that here. Instead, I want to focus on a separate aspect of this whole system which makes “the prestige” possible. Let’s talk about ownership of data. Ownership of Data The core reason why marketplaces can make the shift from convenience providers to super-efficient service providers is because of the data they have about customer behaviour. This data being accessible only to the marketplace essentially means that the people providing the actual service have no way of knowing whether they are doing the right things and what else they could do. To be fair, this data at this scale did not exist before the marketplaces came along, but now that it does exist, it skews the marketplace-seller relationship tremendously with no recourse for the latter. Sure, most marketplaces offer “seller insights” or other such tools for service providers, but that is typically only the tip of the iceberg in terms of information and is typically the marketplace’s view of what the providers should know. once they enter the marketplace world, providers have no leverage to dictate how the business should be run. They surrender much of the agency they had in running their business and have to become robotic followers of whatever the marketplace wants them to do. The scam eventually turns to customers as well, but spenders are harder to replace so it usually takes longer. ![](https://kislayverma.com/content/images/2022/01/Screenshot-2022-01-23-at-5.36.46-PM-300x97.png) I don’t have an answer to the scam yet, but I do know that the scam exists. End-to-end ownership of functions, in my admittedly limited understanding, is beneficial for individual players but seems to be actively reducing the diversity of actors in the economic ecosystem and creating massive centralized entities, which are eventually powerful enough to act unilaterally and arbitrarily in the system. That is not good. A more sustainable version of this economic model can “probably” be built by adopting a decentralized model of information ownership where all sale and inventory data is owned by service providers and hence can be retracted anytime, logistics information known to shippers, rating and review know to other neutral observers, and all of them willingly sharing data to build a marketplace experience for the customer. The current D2C and creator economies are a move towards this structure, but I think that the internet architecture (as it is built today) is yet to truly catch on to this. Most of the creator economy work still happens on centralized platforms, most of which are also closed from a data perspective. True empowerment of creators will require them to physically own their data, and marketplaces to receive consent for receiving this data. The current marketplace model pretends to be free and open but it is actually the reverse, creating eventually unbalanced power relationships between marketplaces, customers, and sellers. Those of us who are building on the internet can and should do better. From the internet [Why you can have millions of goroutines but only thousands of Java threads](https://rcoh.me/posts/why-you-can-have-a-million-go-routines-but-only-1000-java-threads/?ref=kislayverma.com) is an excellent deep dive into concurrency models by [Russell Cohen](https://twitter.com/russellrcohen?ref=kislayverma.com). This is actually one of my favourite interview questions, and the article lays out the details beautifully. This wired article on [understanding exponential change](https://www.wired.co.uk/article/exponential-age-azeem-azhar?ref=kislayverma.com) by Azeem Azhar is opening eye-opening in many ways. Don't forget to get his book if you like the article. [ Glenn Engstrand](https://twitter.com/gengstrand?ref=kislayverma.com) writes about [consistency, coupling, and complexity at the Edge](https://www.infoq.com/articles/consistency-coupling-complexity/?ref=kislayverma.com). That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #48: Theory building, visual probability, index while read, and hiring a manager URL: https://kislayverma.com/it-depends-48-theory-building-visual-probability-index-while-read-and-hiring-a-manager/ Last updated: 1970-01-01T00:29:52.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #48: Theory building, visual probability, index while read, and hiring a manager Hello Everyone! Welcome to the It Depends #48\. I hope all is well with you and your codebase. Through the last two weeks I’ve been stuck inside the hilarious world of Bertram Wilberforce Wooster and his gentleman’s gentleman Jeeves for the umpty-upth time. The obvious fallout being that I haven’t managed to write anything. So this week, it’s just the writings of better men and women than me for your reading pleasure. Hope to be back on the writing track very soon. From the internet This paper called [Programming as Theory Building](https://pages.cs.wisc.edu/~remzi/Naur.pdf?ref=kislayverma.com) by Peter Naur (of the “Backus-Naur form” fame) speaks very close to my heart on how practice and knowledge co-evolve and on how we share knowledge. Definitely worth reading if your work involves knowledge sharing in any way. I’m a huge sucker for visual representation of mathematics (probably that’s the way I can grasp it even a little bit) and [Seeing Theory](https://seeing-theory.brown.edu/?ref=kislayverma.com) is really one of the best visual expressions of probability theory I have ever seen. Definitely check it out. Bernard Leong shares his “[index while read](https://www.bernardleong.com/index-while-read/?ref=kislayverma.com)” approach to reading better. Charity Majors is her usual scintillating self in describing her [approach to hiring a manager](https://leaddev.com/hiring-onboarding-retention/how-hire-engineering-manager-within-or-without?ref=kislayverma.com) for your dev team. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### It Depends #47: Guidelines for writing useful libraries URL: https://kislayverma.com/it-depends-47-guidelines-for-writing-useful-libraries/ Last updated: 1970-01-01T00:29:32.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #47: Guidelines for writing useful libraries Hello Everyone! Welcome to the 47th episode of It Depends. Hope everyone is doing well and staying safe. I expected [last week’s missive on competitive programming](https://kislayverma.com/organizations/competitive-programming-is-useless/) to generate a certain amount of controversy, and so it did, on both [Reddit](https://www.reddit.com/r/programming/comments/p98kcn/competitive%5Fprogramming%5Fis%5Fuseless/?ref=kislayverma.com) and [Hacker News](https://news.ycombinator.com/item?id=28274310&ref=kislayverma.com). There were rational discussions and brickbats in equal measure, but 100 followers each were gained by this newsletter (1772 subscribers now) and the [It Depends podcast](https://open.spotify.com/show/5gY1eGUE0RmNHj5t1HKpJI?si=kIlECDw9S3iLHeLozsSrvw&dl%5Fbranch=1&ref=kislayverma.com) (163 followers now) both, so all in all things turned out alright. Today I want to share some guidelines on writing good libraries, and then on to the best of the internet as usual. What is a library? *A library is a collection of implementations of behavior, written in terms of a language, that has a well-defined interface by which the behavior is invoked -* [Wikipedia](https://en.wikipedia.org/wiki/Library%5F%28computing%29?ref=kislayverma.com) So a library is an artifact containing the implementation of some functionality but hiding it behind an API. “Host” systems can use the library to achieve the functionality by simply invoking the API instead of having to understand the implementation. Libraries are created to share code between multiple systems. e.g. In the Java world, [Rulette](https://github.com/kislayverma/Rulette?ref=kislayverma.com) is a rule-engine library published to a central repository. Anyone who needs to use a rule engine in their system can pull in the library from the repository and use it. A library is different from a framework in that while your code calls library functions, a framework will typically call your code. e.g. I can write code that uses the MySQL-connector library like [HikariCP](https://github.com/brettwooldridge/HikariCP?ref=kislayverma.com) to connect to a database. [Spring Boot](https://spring.io/projects/spring-boot?ref=kislayverma.com), on the other hand, is a framework that provides the structure within which I must write my code. Spring Boot invokes my code, while my code may invoke the library for connecting to the database. There are a few basic characteristics of a well-designed library. It should be easy to understand and use. Its behavior should be easily modifiable where that was the intent of the library author. Behaviours not intended to be modifiable should be completely hidden from users. To these ends, here are a few guidelines that have helped me in writing libraries. Make it small One of the biggest problems in using a library is the number of dependent libraries it requires. LIbraries with too many dependencies are often large in size (causing the size of the host system to bloat) and may cause clashes with other libraries being used in the host system itself (complicating the host system). Resolving conflicting dependencies which cause errors at runtime is one of the worst debugging experiences IMO. The fewer dependencies a library has, the easier it will be to use. Be opinionated A library should do a specific thing in a specific way. While the exact definition of “specific” is up to the author, and one can make things as “configurable” as one wants, it is usually better to build a tight, opinionated library than a large, multi-faceted monster that tries to do too many things in too many different ways. A minimal but sufficient feature set, APIs, and configurations, all go a long way in making a library easy to select (among alternatives) and use. Write the user’s code first I have written about [designing from one level above](https://kislayverma.com/software-architecture/system-design-from-one-level-up/) before. For a library, the “one level above” is the user code which is going to use it. So first write a few samples of how host systems might use the library and experiment with a few different scenarios. We can run this by our actual users or even sit with them and ask them to write how they might want to use the library. This will give good insights into which APIs/configurations are necessary and which are not. Identify internal components and identify the extensible ones With the external environment set, we need to design the internals. If we want library users to be able to modify certain the behaviour of certain parts, we first need to identify these “parts” so that we can design for extension. So having an opinion means we decide what we will allow being customized, and in the design process, we identify exactly where these customizations will go. These components/interfaces need to be designed with special care as they will be exposed to users and therefore hard to change later on. Allow injection of implementations of extensible components Since we want a library to be usable by any type of host system, it is usually a bad idea to assume a runtime environment when writing libraries. e.g Writing an HTTP client library using annotations like autowired (spring example) will make it unusable in non-spring systems. But it is ok if you are deliberately writing a library to be used only in a specific environment. It’s a fair opinion to reduce complexity by assuming runtime. IMO, it is better to provide a builder (or other similar) pattern so that these configurations and custom implementations can be injected when the library is being set up for use. Provide as few ways (ideally only one) of using the library. Again, opinionated design will reduce the complexity of the library API. An aside on platform thinking The guiding principles of designing libraries are very similar to the high-level principles for [designing platforms](https://kislayverma.com/category/platform-thinking/). Libraries, like platforms, are not things by themselves. They exist to allow others to build things using them. The same principle of composability and extension can be seen at high and low levels in platforms and libraries respectively. The fact that a library is shipped as a “closed” artifact makes sure that even the owner of the library is forced to use it like any other user. This is the [Golden Law of Platforms](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/) – eat your own dog food. If the library wants to allow any modifications to its behaviour via configuration/extension, it must provide well-defined hooks for this. This is [External Programmability](https://kislayverma.com/platform-thinking/external-programmability-the-second-law-of-building-platforms/) – the second law of building platforms. From the internet [Scale in or scale out](https://blog.dream11engineering.com/to-scale-in-or-scale-out-heres-how-we-scale-at-dream11-f88ef5e71cbc?ref=kislayverma.com)? The Dream11 team talks about how they scale their systems. This is a very interesting read from a company that handles massively spiky traffic. If you have been hearing about Domain Driven Design but don't yet know what it is, [this video](https://www.youtube.com/watch?v=pfMGgd%5FNDPc&ref=kislayverma.com) is a great place to start. Autonomous teams are all well and good, but a key part of their success is empowering them to exploit their autonomy. [This Ivey Business Journal article](https://iveybusinessjournal.com/publication/empowering-autonomous-teams/?ref=kislayverma.com) explains how. On a lighter note, here are [Craig Larman’s laws of organizational behaviour](https://www.craiglarman.com/wiki/index.php?title=Larman%27s%5FLaws%5Fof%5FOrganizational%5FBehavior&ref=kislayverma.com). They would be funnier if they weren’t so painful. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #46: Competitive programming is useless URL: https://kislayverma.com/it-depends-46-competitive-programming-is-useless/ Last updated: 1970-01-01T00:28:18.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #46: Competitive programming is useless ![](https://kislayverma.com/content/images/2021/06/headphone-icon.jpeg) You can listen to this episode on [Spotify](https://open.spotify.com/episode/2sodHpx6OIB0HzWdfZujN7?si=bf4925c0e9134812&ref=kislayverma.com), [Google](https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81MTZiNzI0NC9wb2RjYXN0L3Jzcw/episode/OWFmZTQzZmMtNTNhMi00MGE4LThkMGItYjkyNGZlMjRiZTEx?ref=kislayverma.com), or [Apple](https://podcasts.apple.com/in/podcast/it-depends/id1571969334?ref=kislayverma.com) Hello Everyone! Welcome to the 46th episode of It Depends. Hope everyone is doing well and staying. Today we are going to talk about competitive programming, and then it is the best of the internet as usual. Competitive programming is useless This post is a rant. I know that this is not true of every company and or every engineering student. But it is widespread enough in my experience that I find it worth ranting about. ![](https://kislayverma.com/content/images/2021/08/competitive-programming-300x187.png) tl;dr Competitive programming is a good tool for building the programming muscle. An extreme pursuit of competitive programming is worse than useless. Unfortunately, companies and students are both headed in that direction at the moment instead of looking for engineers with broad interests. From competency to fetish Competitive programming started out as a good thing. When I was in college, there was no [leetcode](https://leetcode.com/?ref=kislayverma.com) or equivalent websites. I think GSoC had just started out and only the “rockstar” programmers used to participate in that. Or maybe this was just the situation in my college – I don’t know. By and large, my class got by without writing much code, much less code of the type one might encounter in a real job. I and a few others who genuinely enjoyed programming ended up doing a variety of projects on our own and learning things that way. The leetcodes and [geeksforgeeks](https://www.geeksforgeeks.org/?ref=kislayverma.com) of the world filled a critical gap between textbooks and hands-on exercise. They provided a convenient place to see the kinds of questions asked in interviews and practice solving them. And then something went wrong. With growing access to these questions, interviewers started asking harder and harder questions in the coding rounds in college (and generally <5 years exp.) interviews. The expectation for these rounds is currently, IMO, meaningless. We left competency behind and are now well into fetish-land. As a response, college students now pursue competitive programming obsessively to stay on top. In this weird arms race against prospective hires, companies keep asking harder and harder questions in a misguided attempt to raise the bar. The students respond by doing nothing else but solve every single available question on every single competitive programming website. The junior engineer interviewing process, as it exists today, has a systemic problem. It doesn’t matter how high newbies are above a certain bar of logical, coding, and algorithmic competence – it is all the same. This is a classic case of a metric being gamed. A premium is placed on being able to solve super complex data structures questions. To meet this unnecessarily high benchmark, college students do whatever it takes. If the question had instead been “can this person become a great engineer in our company”, perhaps the outcomes might have been different? Being able to solve typical data structures, algorithms problems is a signal in the larger interview process. By lowering the unduly high bar on DS type questions, organizations can make room for students to exercise their curiosity and develop their passions for something unique to them – their favourite technology, their favourite tech stack, their favourite industry. This will help them find people that can be trained, are self-motivated, and have an actual interest in technology beyond a gamified version of it. A diluted signal Now all this would be fine if this would help people become infinitely better programmers (it doesn’t) or at least distinguish themselves from the pack. But there is no indication that this is the case. Students with 37 million zillion stars on coding ninjas, extra-super-advanced level on leetcode, or uber-coding-lord status on codeforces regularly fail the interview process because EVERYONE around them is at that same level! And in the pursuit of that level, they have ignored a lot of other fundamentals they should have learned or things they could have explored and tried out on their own. When asked what kind of technologies they find interesting, several students have told me over the last few years that they are only excited by competitive programming and have no other interest in software engineering or technology as such. I can understand students from lower rung colleges following this strategy. Assuming for a minute that students from lesser tier colleges are less smart (possibly untrue but again a discussion for another day), the better students can distinguish themselves from their peers by extreme achievements in competitive programming. But at the better colleges, everyone is doing the exact same thing, and as a result, there is no benefit for anyone. It is again the responsibility of the organizations to call this out and focus on other aspects of being an engineer. At least on other academic subjects if nothing else. But almost always, the first interview round is a super-hard data structures challenge which lesser mortals can’t get through. So later rounds are always evaluating people who are competitive programming biased. This funnel will never allow a different breed of software engineers to pass through. It is an accepted idea in the industry that employee performance appraisals are subjective. Most companies make only superficial attempts at making them objective because different employees contribute in different ways. This subjectivity/uncertainty is part of evaluating anyone for any role. Unfortunately, we have reduced the fresher interview process to a computing game. This is not good for anyone, and the sooner we stop it, the quicker we might be able to find good engineers instead of human-like robots. From the internet Aline Guisky describes how her team built an [event-driven architecture to clean up noisy machine learning labels](https://medium.com/riskified-technology/event-driven-architecture-can-clean-up-your-noisy-machine-learning-labels-f49363403f89?ref=kislayverma.com). [Matt Ricard](https://twitter.com/mattrickard?ref=kislayverma.com) has a funny but all too relatable take on the [evolution of configuration](https://matt-rickard.com/heptagon-of-configuration/?ref=kislayverma.com). This got me thinking about how domain models evolve, so stay tuned for more on that. Adventures in rolling-my-own-open-source: the storj.io team wanted a lightweight version of gRPC, so [they made DRPC](https://www.storj.io/blog/introducing-drpc-our-replacement-for-grpc?ref=kislayverma.com)! [Ole Rydland](https://twitter.com/orydland?ref=kislayverma.com) has a lovely [introduction to socio-technical architecture](https://www.oleandreasrydland.com/socio-technical-architecture/?ref=kislayverma.com). This is a great place to start before you move on to the heavier stuff. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #45: The rise of Edge Computing URL: https://kislayverma.com/it-depends-45-the-rise-of-edge-computing/ Last updated: 1970-01-01T00:28:10.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #45: The rise of Edge computing ![](https://kislayverma.com/content/images/2021/06/headphone-icon.jpeg) You can listen to this episode on [Spotify](https://open.spotify.com/episode/25EqpOJ5mFB3DXvdoGjoRZ?si=efb5d0828aa04eff&ref=kislayverma.com), [Google](https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81MTZiNzI0NC9wb2RjYXN0L3Jzcw/episode/ZjljMjM5M2EtYjEzMC00NGYyLWE0ODktY2Y3MzZmZGY4NWYx?sa=X&ved=0CA0QkfYCahcKEwjw98HdpLDyAhUAAAAAHQAAAAAQAQ&ref=kislayverma.com), or [Apple](https://podcasts.apple.com/in/podcast/it-depends/id1571969334?ref=kislayverma.com#episodeGuid=f9c2393a-b130-44f2-a489-cf736fdf85f1) Hello Everyone! Welcome to the 45th edition of It Depends. Hope you are all doing well. About a month ago I had spoken to the folks at Traceable.ai about my thoughts on observability and operating large-scale distributed systems. The discussion was published this week on their [Talking Observability podcast](https://anchor.fm/talkin-observability/episodes/Observing-is-not-debugging-e157lsk?ref=kislayverma.com). There’s a [video version](https://lp.traceable.ai/webinars.html?commid=499011&ref=kislayverma.com) as well (you’ll have to sign up for that). This is my first time featuring on any publication and I’m feeling quite smug about it :) This week’s feature is a guest post from Maaz Humayun on the rise of edge computing. Maaz is a senior engineer at Amazon. He spent five years with Amazon Appstore working on high-volume web services that power search, ordering, and entitlements on first and third-party devices. More recently, he’s joined the Amazon Luna team, where he’s working on game-streaming tech. In his free time, he enjoys reading about SaaS platforms and new trends in software development. The Rise of Edge Computing You would be hard-pressed to find an industry the internet has not yet transformed. Banking, health, publishing, entertainment; the list goes on. And we’re all better off for it. Internet-enabled services are faster, cheaper, and more reliable. The only thing that’s outpaced the technological progress of the internet is our expectations of it. So you want that YouTube video to stream in 4k without buffering, no pixelation, and crystal-clear audio quality? Why, yes, I’ll have that, thanks. The internet has gone through massive changes to keep up with growing demands. A decade ago, companies had to maintain network infrastructure and fund an IT department to keep everything working smoothly. This all changed with cloud computing. Today, all you need is a great idea and an AWS account to create a product/website that’s globally available and infinitely scalable. Moving to the Edge ![](https://kislayverma.com/content/images/2021/08/Edge_computing_infrastructure-300x261.png) So what do we expect will change in the future? Instead of answering the question directly, let’s ask ourselves “what will stay the same?” Users will continue to expect services to get faster and cheaper. Developers will want to iterate quicker and focus their efforts on writing core business logic instead of tinkering with infrastructure. Edge computing will help us evolve the internet to satisfy these requirements. We already use content delivery networks (CDNs) to optimize latency for applications. The concept is simple. Information can’t travel faster than the speed of light. So, to reduce latency, we need to move the data closer to the user. CDNs have several points of presence (PoPs) — also called edge locations — deployed near concentrated population centres. A CDN will cache popular content at the edge location based on customer usage patterns. If a device requests content cached at the edge, the CDN can serve the data directly, without the request going to the origin server (which could be thousands of miles away). However, these CDNs have traditionally been very — for the lack of a better word — dumb. Customers are limited to configuring content-management policies — how and when to expire data from the cache. But a new wave of CDNs led by Cloudflare and Fastly have gone a step further by adding general-purpose compute instances at these PoPs. If you’re a developer, this means you can insert any code between the end device and your app server. Computing at the Edge In 2017, Cloudflare launched ‘Cloudflare Workers’, which lets customers run arbitrary code on the Cloudflare platform. Workers use Google’s high-performance [V8 engine](https://v8.dev/?ref=kislayverma.com) to launch [V8 Isolates](https://developers.cloudflare.com/workers/learning/how-workers-works?ref=kislayverma.com) that execute your code. Unlike containers, Isolates are fast to spin up, which reduces cold-start time, and they are computationally cheap so you can run thousands of them on a single physical machine. To see Workers in action, watch this [YouTube video](https://www.youtube.com/watch?v=48NWaLkDcME&t=557s&ref=kislayverma.com) which shows a developer intercepting calls to his domain and modifying the response based on the URL. Fastly has taken a slightly different approach to serverless computing. Instead of building their compute platform on top of existing technology, they created an optimized WebAssembly compiler and runtime called [Lucet](https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime?ref=kislayverma.com). Fastly claims that Lucet can instantiate WebAssembly programs in under 5 microseconds using only a few kilobytes of memory. By comparison, Chromimum’s V8 engine has a larger memory footprint and can take 5 milliseconds to initialize programs. Fastly released Lucet as an open-source project under the [Bytecode Alliance](https://bytecodealliance.org/?ref=kislayverma.com) so you can check out the source-code [here](https://github.com/bytecodealliance/lucet?ref=kislayverma.com). Lucet already seems to be gaining adoption as a way of executing WebAssembly outside a browser environment. Shopify uses Lucet to host partner programs, called “Apps”, on top of their infrastructure. Understandably, this saves Shopify partners considerable effort because they don’t have to set up their own servers. You can read more on Shopify’s engineering [blog](https://shopify.engineering/shopify-webassembly?ref=kislayverma.com). Maintaining State The Achilles heel of serverless computing has been the inability to persist data between requests. In other words, “serverless” has become synonymous with “stateless”. Sure, you can connect to a database over the network, but you must reinitialize your database connection every time you bootstrap your function. More, you need to deal with networking latency because the database and function could be running in different data centres. Cloudflare is innovating on the data-storage front with a product called [Durable Objects](https://blog.cloudflare.com/introducing-workers-durable-objects/?ref=kislayverma.com). A Durable Object is attachable persistent storage for your serverless function — just imagine someone plugs a pen drive into your serverless function in the cloud. Each Durable Object is globally unique and offers transactional guarantees. By co-locating the compute and data, we significantly cut down both latency and cold-start time. As you might imagine, this is well suited for real-time applications like gaming, chat, and online collaboration tools. In fact, [here](https://blog.cloudflare.com/building-real-time-games-using-workers-durable-objects-and-unity/?ref=kislayverma.com) is a sample application that shows how you can use Workers and Durable Objects to build a simple 3D multiplayer game. It’s also interesting to note that the serverless + storage architecture forces us to rethink our application design. In our new paradigm, the data constructs we create closely mirror our business constructs. For example, each Durable Object can maintain the state of a specific context, like a chat or document. There is no need for a centralized database that hosts data across the entire user base. Best of all, this architecture lets the edge layer transparently migrate our compute instance close to the user to optimize latency. A natural next step and one that will pose some exciting challenges is coordination between edge nodes. Imagine a future where roads have intelligent traffic lights that have sensors to detect traffic flow in real-time. These lights could adjust the flow of traffic to avoid congestion or idle wait times. Such a system is most effective if each sensor constantly shares its data with all nearby sensors in the network. But how would we orchestrate such a system? We wouldn’t want to send all the raw data back to a central server that’s hundreds of miles away. Instead, we want all the decision making to happen at the edge. Perhaps each node is listening to data updates from all nearby nodes and making decisions independently? Alternatively, a group of nodes might elect a leader that orchestrates traffic between them, and leaders might communicate with each other through a similar mechanism. I don’t know how we’ll solve this problem, but I know it will help improve traffic. Global Regions Working in the cloud, we’ve grown accustomed to the idea of discrete geographical regions. Developers have to balance tradeoffs like cost and latency to decide where to deploy their applications. Want to expand to a new country? You need to deploy the entire stack to the closest region. A region-aware architecture forces developers to make decisions about geography even if their product doesn’t require it. However, the edge inverts this problem. There are no regions insofar as there is just one — “Earth”. When you deploy your code, it is deployed globally in minutes. The application is fast everywhere from day one, and you don’t pay for unused servers. In the future, developers will have to balance yet another constraint when architecting their applications — politics. Several countries are writing laws governing data flow. For example, China has mandated that data of Chinese citizens cannot leave the mainland. Typically, this translates to companies hosting a “China stack” siloed from the rest of the world. It doesn’t require a giant leap to imagine that other countries may someday follow suit. Of course, it would be cost-prohibitive for companies to launch a new stack for each country. In a regionalized architecture, the onus is on the developer to manage the flow of data in compliance with each county’s laws. Counter-intuitively, a global architecture helps developers because we can set jurisdictional boundaries at the object level. For instance, Cloudflare allows you to set [jurisdictional restrictions](https://blog.cloudflare.com/supporting-jurisdictional-restrictions-for-durable-objects/?ref=kislayverma.com) on Durable Objects that control where your data is stored. Remarkably, all of this is accomplished by specifying the jurisdictional restriction as a string, like so: *let id = OBJECT\_NAMESPACE.newUniqueId({jurisdiction: "eu"})* Where we are going? We keep hearing about how much new data we generate each year. But let’s think about the directional flow of said data. Most data today flows from the inside-out, i.e. from the cloud to the edge. Billions of people use YouTube, Netflix, Instagram. However, most bits flow to customers consuming content. With the proliferation of IoT devices, wearables, autonomous cars, the flow of data will invert. Eventually, we’ll start to see most data flow from the edge to the cloud. Because most data will be machine-generated, it won’t all be useful. Instead of sending back terabytes of raw data to our application server, it will be more efficient to process data at the edge and only send post-processed data. Not only does this improve latency, because we’re sending less data across the network, it will also reduce costs because we’re using less network bandwidth. As edge computing becomes more mainstream, our edge devices can become smaller and cheaper. We won’t need to ship devices with powerful hardware because the edge can do the heavy lifting. For example, a smart speaker can send raw audio to the edge server, which will strip out unnecessary bytes before sending the byte-stream to the app server. Cloudflare recently announced a [partnership](https://www.cloudflare.com/nvidia-workers/?ref=kislayverma.com) with Nvidia where they plan to introduce AI/ML at the edge. For use-cases like autonomous driving, the edge creates an optimized network for cars to communicate with each other. Imagine a road with hundreds of vehicles that need to talk to each other. It would be highly inefficient for the data to flow all the way back to a centralized server, only to be received by a vehicle a few feet ahead. With an edge network, data will only travel to the closest edge node, significantly reducing latency. Add not Subtract The rise of edge computing and programmable networks does not mean the death of the cloud as we know it. There will always be use-cases inappropriate for the edge, like training complex ML models, storing shared user data. Both paradigms will co-exist and work in tandem, much like SQL and NoSQL today. The future of edge computing looks promising and exciting. Already, we’re starting to see several edge computing startups try to capitalize on the coming revolution. While we can imagine all the unique ways edge computing will change the world, I suspect that the reality will be far more surprising. References [https://blog.cloudflare.com/serverless-performance-comparison-workers-lambda/](https://blog.cloudflare.com/serverless-performance-comparison-workers-lambda/?ref=kislayverma.com) [https://www.youtube.com/watch?v=48NWaLkDcME&t=557s](https://www.youtube.com/watch?v=48NWaLkDcME&t=557s&ref=kislayverma.com) [https://blog.cloudflare.com/introducing-workers-durable-objects/](https://blog.cloudflare.com/introducing-workers-durable-objects/?ref=kislayverma.com) [https://en.wikipedia.org/wiki/Software-defined\_networking](https://en.wikipedia.org/wiki/Software-defined%5Fnetworking?ref=kislayverma.com) [https://stratechery.com/2021/cloudflare-on-the-edge/](https://stratechery.com/2021/cloudflare-on-the-edge/?ref=kislayverma.com) [https://www.cloudflare.com/en-in/press-releases/2021/cloudflare-partners-with-nvidia/](https://www.cloudflare.com/en-in/press-releases/2021/cloudflare-partners-with-nvidia/?ref=kislayverma.com) [https://blog.cloudflare.com/cloudflare-workers-unleashed/](https://blog.cloudflare.com/cloudflare-workers-unleashed/?ref=kislayverma.com) [https://softwarestackinvesting.com/decentralization-effects/](https://softwarestackinvesting.com/decentralization-effects/?ref=kislayverma.com) [https://shopify.engineering/shopify-webassembly](https://shopify.engineering/shopify-webassembly?ref=kislayverma.com) [https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime](https://www.fastly.com/blog/announcing-lucet-fastly-native-webassembly-compiler-runtime?ref=kislayverma.com) [https://github.com/bytecodealliance/lucet](https://github.com/bytecodealliance/lucet?ref=kislayverma.com) [https://hhhypergrowth.com/what-are-edge-networks/](https://hhhypergrowth.com/what-are-edge-networks/?ref=kislayverma.com) [https://www.youtube.com/watch?v=QdWaQOgvd-g](https://www.youtube.com/watch?v=QdWaQOgvd-g&ref=kislayverma.com) From the internet Ron Jeffries adds valuable historical context to what “Agile” is by describing [the mindset of how the agile manifesto was originally written](https://ronjeffries.com/articles/021-01ff/outcomes/?ref=kislayverma.com). Alfonso De La Rocha explains how [IPFS can serve as the storage backend for the blockchain](https://adlrocha.substack.com/p/adlrocha-ipfs-for-storage-the-blockchain?ref=kislayverma.com) ecosystem. Here are two guides ([here](https://medium.com/ssense-tech/hexagonal-architecture-there-are-always-two-sides-to-every-story-bc0780ed7d9c?ref=kislayverma.com), and [here](https://8thlight.com/blog/damon-kelley/2021/05/18/a-color-coded-guide-to-ports-and-adapters.html?ref=kislayverma.com)) to understanding the hexagonal aka ports-and-adapters architecture. I love this pattern because I feel provides so much more explicit guidance than saying “layered architecture” in a hand-wavy manner. Charles Lambdin discusses some popular laws (Murphy’s, Parkinson’s etc) and shows how [bureaucracies blunder](https://charleslambdin.com/2021/01/12/bureaucratic-blunderland/?ref=kislayverma.com) in infinitely varied ways. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #44: Uncertainty and learning as tech debt URL: https://kislayverma.com/it-depends-44-uncertainty-and-learning-as-tech-debt/ Last updated: 1970-01-01T00:27:47.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #44: Uncertainty and learning as tech debt ![](https://kislayverma.com/content/images/2021/06/headphone-icon.jpeg) You can listen to this episode on [Spotify](https://open.spotify.com/episode/6chYoPPGCbRMv08JpvAiEh?si=32567f6cb3d04ca7&ref=kislayverma.com), [Google](https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81MTZiNzI0NC9wb2RjYXN0L3Jzcw/episode/NTBhMWZlOTItYTRiMC00MTk4LTg1N2MtMjhmNzczNmVkNTk0?ref=kislayverma.com), or [Apple](https://podcasts.apple.com/in/podcast/it-depends/id1571969334?ref=kislayverma.com#episodeGuid=50a1fe92-a4b0-4198-857c-28f7736ed594) Hello everyone! Welcome to the 44th edition of It Depends. Hope you are all doing well. First up, a HUUGE shoutout to Ms. Nimmi Bandaru who is now supporting this newsletter on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com) (only my second patron). Thank you Nimmi, this means a lot for a newbie like me - hope I can continue writing what you like to read. Today, we are discussing tech debt, and then the usual best of the internet for your learning pleasure. Uncertainty and Learning as Tech Debt ![](https://kislayverma.com/content/images/2021/07/tech-debt-300x265.png) Tech debt represents an accumulation of conscious or subconscious decisions which can now be identified as bad decisions. Basically, a shoddy job that makes taking the next steps harder. The more tech debt you have, [the harder you have to work to make new changes](https://adlrocha.substack.com/p/adlrocha-the-risks-of-technical-debt?ref=kislayverma.com). My approach to system architecture and evolution is via continuous learning and improvement. I believe that all things change all the time. So all parts of a system will carry tech debt some time or the other. The developer team’s job is to continuously identify these parts, prioritize the critical deficiencies which are blocking the way forward, and consciously deepen their understanding of the system by taking on controlled tech debt if required. This article is just a few observations about this phenomenon, mostly focussed on change and uncertainty. Every decision is tech debt in the making The reason we see so much tech debt, even in good teams and organizations, is usually not because of bad decisions. Decisions were probably good when they were made, but the world keeps changing. Parts of even good decisions will end up being tech debt at some point in the future. No one designs bad systems deliberately. If a decision looks bad today, it does not matter why it was taken. Maybe the team didn’t know any better, or the world changed on them. The outcome is the same either way – we need to do something about the situation. The ideal architecture is only ideal at a point in time. Perhaps not even then because there are almost always things we do not know that might lead us to design our systems differently (Conway’s Law). Tech debt is great when taken deliberately A lot of discussion around technical debt is only about technology quality. It does not talk about learning. Deliberately taking on tech debt allows us to learn faster by shipping faster. The problem is that we forget that we made the feature as a learning step. The context of that deliberate decision is lost. So when the next iteration comes around, the decision looks like a bad decision rather than a feature in process of growing. Too often, engineers think on the lines of “we built it badly and need to fix it” and not like “which part of this system should be evolved next”. The more we remember as a team, the more we can think of tech debt in terms of WIP features rather than an already finished piece that was designed badly. The location of tech debt matters The place where technical debt is seen in the overall system matters. At the edges of the system, where the most amount of learning is happening (like customer-facing product teams), tech debt is tolerable, even somewhat desirable (as mentioned above in the context of learning). It is less tolerable in the deeper layers of the system because the deeper in the tech stack a system is, the more dependencies it has, and therefore the more damage a bad decision can do. [The more stable the core systems are, the more fearlessly we can mess about in the other places](https://kislayverma.com/agile/being-fast-or-getting-faster-aka-build-momentum-not-velocity/). [Platforms architecture](https://kislayverma.com/category/platform-thinking/) actually encourages this type of dual-thinking. It [establishes standards for the platform components and allows product components to do whatever they want](https://kislayverma.com/platform-thinking/control-and-chaos-in-platform-systems/), however they want it. This is a point in time argument – there is no “core” as such. Since all systems are always evolving, even core components will change. When that happens, we should [apply this argument to them too](https://kislayverma.com/platform-thinking/platforms-and-dogfood-everywhere/). Tech debt is a balance The trade-off isn’t between speed and quality. The tradeoff is learning and executing on that learning in the long run. I wrote earlier about [ditching the urgency](https://kislayverma.com/agile/ditch-the-urgency/) to execute in the learning phase. This is where it is okay to take on tech debt. Build something small and fast to see what the users do with it. Now while the user feedback is coming in and we are trying to understand it, [clean up the most undesirable of the bad decisions](https://kislayverma.com/programming/saving-the-day-with-continuous-refactoring/) you have in the system. BOTH THE ABOVE STEPS ARE CRITICAL. It is inevitable that we will go back and forth to some extent. The origins of tech debt are important I mentioned earlier in this article that why we have a bad decision right now doesn’t matter. What matters is to fix it. This is true from an operational perspective but not from a growth perspective. For the engineering team, identifying the origin of the tech debt is a critical part of the learning process. Looking back on their decisions, can they identify the bad decisions that they then thought were good? What led to those decisions? This will typically reveal some sort of information gap – not having enough technical skill, not having enough knowledge about the product or the customer, not understanding the direction of the organization etc. These gaps can then be filled actively. This is hard because teams are biased against their former selves due to the new knowledge they have gained since a decision was made. From the internet This article from Cloudflare describes [what edge computing is and isn't](https://blog.cloudflare.com/cloudflare-workers-serverless-week/?ref=kislayverma.com). Cloudflare is doing some superb work in this field and yours truly is trying to make a blog on the technology happen soon. Stay tuned! Larry Sanger explains [what decentralization requires](https://larrysanger.org/2021/01/what-decentralization-requires/?ref=kislayverma.com) to work out in the wild. James Urquhart discusses some [critical elements of Flow architecture](https://medium.com/digital-anatomy/five-facets-of-flow-strategy-96a737243ee5?ref=kislayverma.com). I know this [Dan North article on testing](https://dannorth.net/2021/07/26/we-need-to-talk-about-testing/?ref=kislayverma.com) is on every curated tech newsletter right now, but I had to include it because it is just that good. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #43: Book review and Highlights from "Accelerate" URL: https://kislayverma.com/it-depends-43-book-review-and-highlights-from-accelerate/ Last updated: 1970-01-01T00:27:32.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #43: Book review and Highlights from "Accelerate" Hello Everyone! Welcome to the 43rd edition of It Depends. Hope you are all doing well. Today I will try to save some time for you folks by sharing the highlight of my reading of “Accelerate” by Nicole Forsgren, Jez Humble, and Gene Kim. And then on to the best of the internet! Highlights from “Accelerate” ![](https://kislayverma.com/content/images/2021/07/accelerate-cover-199x300.jpg) Accelerate has consistently been described as one of the best books when it comes to DevOps and building technical agility in organizations. I finally got around to reading it and it was every bit as good as I had expected it to be. The book essentially presents the conclusions of a multi-year research program and contains a whole section on why certain research methods were chosen over others. I have been reading and writing a fair bit about [agility](https://kislayverma.com/tag/agility/) and [moving fast](https://kislayverma.com/agile/being-fast-or-getting-faster-aka-build-momentum-not-velocity/), so most of the things in the book were not new, but the solid research backing it means that “CI/CD is a must-have” is not just an opinion anymore. The authors have shown it to be a demonstrable trait of successful teams in the wider industry. Accelerate is a short-ish read, but it is dense with information. I am sharing my highlights from it below so that you can get a taste of what the book is like. If you like these semi-organized snippets, you should definitely read the book. Preface improvements in software delivery are possible for every team and in every company, as long as leadership provides consistent support — including time, actions, and resources — demonstrating a true commitment to improvement, and as long as team members commit themselves to the work. Chapter 1 – Accelerate Small teams that work in short cycles and measure feedback from users to build products and services that delight their customers and rapidly deliver value to their organizations. [DevOps](https://kislayverma.com/tag/devops/) emerged from a small number of organizations facing a wicked problem: how to build secure, resilient, rapidly evolving distributed systems at scale. The Forrester report states that DevOps is accelerating technology, but that organizations often overestimate their progress (Klavens et al. 2017). Furthermore, the report points out that executives are especially prone to overestimating their progress when compared to those who are actually doing the work. The key to successful change is measuring and understanding the right things with a focus on capabilities — not on maturity. Maturity models are not the appropriate tool to use or mindset to have. Instead, shifting to a capabilities model of measurement is essential for organizations wanting to accelerate software delivery. Three reasons capability models are better than maturity models: Maturity models focus on helping an organization “ arrive ” at a mature state and then declare themselves done. Capability models focus on helping an organization continually improve and progress, realizing that the technological and business landscape is ever-changing. Maturity models are quite often a “lock-step” or linear formula, prescribing a similar set of technologies, tooling, or capabilities for every set of teams and organizations to progress through. Capability models are multidimensional and dynamic, allowing different parts of the organization to take a customized approach to improvement, and focus on capabilities that will give them the most benefit based on their current context. Capability models focus on key outcomes and how the capabilities, or levers, drive improvement in those outcomes — that is, they are outcome-based. Most maturity models simply measure the technical proficiency or tooling install base in an organization without tying it to outcomes. Maturity models define a static level of technological, process, and organizational abilities to achieve. In contrast, capability models allow for dynamically changing environments and allow teams and organizations to focus on developing the skills and capabilities needed to remain competitive. Chapter 2 – Measuring Performance Velocity, lines of code, and other typical technical measures focus on outputs rather than outcomes. Second, they focus on individual or local measures rather than [team or global ones](https://kislayverma.com/organizations/managing-developer-identities-in-autonomous-teams/). Ideally, we should reward developers for solving business problems with the minimum amount of code — and it’s even better if we can solve a problem without writing code at all or by deleting code (perhaps by a business process change). A successful measure of performance should have two key characteristics. First, it should focus on a global outcome to ensure teams aren’t pitted against each other. Second, our measure should focus on outcomes not output: it shouldn’t reward people for putting in large amounts of busywork that doesn’t actually help achieve organizational goals. In our search for measures of delivery performance that meet these criteria, we settled on four: Delivery lead time Deployment frequency Time to restore service Change fail rate. Lead time This is the time it takes to go from a customer making a request to the request being satisfied. There are two parts to lead time: the time it takes to design and validate a product or feature, and the time to deliver the feature to customers. In the design part of the lead time, it’s often unclear when to start the clock, and often there is high variability. However, the delivery part of the lead time — the time it takes for work to be implemented, tested, and delivered — is easier to measure and has a lower variability. Deployment Frequency Reducing batch size is another central element of the Lean paradigm. We settled on deployment frequency as a proxy for batch size since it is easy to measure and typically has low variability. By “ deployment ” we mean a software deployment to production or to an app store. Delivery lead times and deployment frequency are both measures of software delivery performance tempo. The key question becomes: How quickly can service be restored (if something goes wrong)? A key metric when making changes to systems is what percentage of changes to production ( including, for example, software releases and infrastructure configuration changes ) fail. In the context of Lean, this is the same as percent complete and accurate for the product delivery process, and is a key quality metric. The ability to take an experimental approach to product development is highly correlated with the technical practices that contribute to continuous delivery. Chapter 3 – Measuring and Changing Culture Organizational culture can exist at three levels in organizations: basic assumptions, values, and artifacts (Schein 1985). Basic assumptions are formed over time as members of a group or organization make sense of relationships, events, and activities. The second level of organizational culture are values, which are more “visible” to group members as these collective values and norms can be discussed and even debated by those who are aware of them. The third level of organizational culture is the most visible and can be observed in artifacts. These artifacts can include written mission statements or creeds, technology, formal procedures, or even heroes and rituals. Type of organization as defined by Ron Westrum- https://cloud.google.com/architecture/devops/devops-culture-westrum-organizational-culture Culture enables information processing through three mechanisms. First, in organizations with a generative culture, people collaborate more effectively and there is a higher level of trust both across the organization and up and down the hierarchy. Culture emphasizes the mission, an emphasis that allows people involved to put aside their personal issues and also the departmental issues that are so evident in bureaucratic organizations. The mission is primary. And third, generativity encourages a ‘level playing field’, in which hierarchy plays less of a role”. We should emphasize that bureaucracy is not necessarily bad. Westrum’s theory posits that organizations with better information flow function more effectively. A good culture requires trust and cooperation between people across the organization, so it reflects the level of collaboration and trust inside the organization. Better organizational culture can indicate higher-quality decision-making. In a team with this type of culture, not only is better information available for making decisions, but those decisions are more easily reversed. Finally, teams with these cultural norms are likely to do a better job with their people, since problems are more rapidly discovered and addressed. Failure in complex systems is, like other types of behavior in such systems, emergent (Perrow 2011). Following the theory developed by the Lean and Agile movements, implementing the practices of these movements can have an effect on culture. You can act your way to a better culture by implementing these practices in technology organizations, just as you can in manufacturing. Chapter 4 – Technical Practices [Continuous delivery](https://kislayverma.com/agile/how-to-speed-up-software-delivery/) is a set of capabilities that enable us to get changes of all kinds into production or into the hands of users safely, quickly, and sustainably. There are five key principles at the heart of continuous delivery: Build quality in: “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place ” (Deming 2000). Work in small batches: By splitting work up into much smaller chunks that deliver measurable business outcomes quickly for a small part of our target market, we get essential feedback on the work we are doing so that we can course correct. Computers perform repetitive tasks; people solve problems. One important strategy to reduce the cost of pushing out changes is to take repetitive work that takes a long time, such as regression testing and software deployments, and invest in simplifying and automating this work. Relentlessly pursue continuous improvement. Everyone is responsible In order to implement continuous delivery, we must create the following foundations: Comprehensive configuration management. It should be possible to provision our environments and build, test, and deploy our software in a fully automated fashion purely from information stored in version control. Continuous integration (CI): high – performing teams keep branches short-lived ( less than one day’s work ) and integrate them into trunk/master frequently. Continuous testing: Because testing is so essential, we should be doing it all the time as an integral part of the development process. Automated unit and acceptance tests should be run against every commit. No one should be saying they are “ done ” with any work until all relevant automated tests have been written and are passing. By giving developers the tools to detect problems when they occur, the time and resources to invest in their development, and the authority to fix problems straight away, we create an environment where developers accept responsibility for global outcomes such as quality and stability. We discovered nine key capabilities that drive continuous delivery. The comprehensive use of version control is relatively uncontroversial. Configuration is normally considered a secondary concern to application code in configuration management, but our research shows that this is a misconception. [Test automation](https://kislayverma.com/agile/testing-strategies-for-agile-teams/) is a key part of continuous delivery. Having automated tests that are reliable: when the automated tests pass, teams are confident that their software is releasable. Developers primarily create and maintain acceptance tests, and they can easily reproduce and fix them on their development workstations. It’s interesting to note that having automated tests primarily created and maintained either by QA or an outsourced party is not correlated with IT performance. Successful teams had adequate test data to run their fully automated test suites and could acquire test data for running automated tests on demand. Our research also found that developing off trunk/master rather than on long-lived feature branches was correlated with higher delivery performance. High-performing teams were more likely to incorporate information security into the delivery process. Their infosec personnel provided feedback at every step of the software delivery lifecycle, from design through demos to helping with test automation. A critical obstacle to implementing continuous delivery is enterprise and application architecture. Chapter 5 – Architecture The architecture of your software and the services it depends on can be a significant barrier to increasing both the tempo and stability of the release process and the systems delivered. We found that high performance is possible with all kinds of systems, provided that systems — and the teams that build and maintain them — are loosely coupled. This reinforces the importance of focusing on the architectural characteristics, discussed below, rather than the implementation details of your architecture. In teams that scored highly on architectural capabilities, little communication is required between delivery teams to get their work done, and the architecture of the system is designed to enable teams to test, deploy, and change their systems without dependencies on other teams. Organizations should evolve their team and organizational structure to achieve the desired architecture. The goal of a loosely coupled architecture is to ensure that the available communication bandwidth isn’t overwhelmed by fine-grained decision-making at the implementation level, so we can instead use that bandwidth for discussing higher-level shared goals and how to achieve them. If we achieve a loosely coupled, well-encapsulated architecture with an organizational structure to match, two important things happen. First, we can achieve better delivery performance, increasing both tempo and stability while reducing the burnout and the pain of deployment. Second, we can substantially grow the size of our engineering organization and increase productivity linearly — or better than linearly — as we do so. Architects should focus on engineers and outcomes, not tools or technologies. Chapter 6 – Integrating Infosec into the Delivery Lifecycle Infosec is a vitally important function in an era where threats are ubiquitous and ongoing. However, infosec teams are often poorly staffed and they are usually only involved at the end of the software delivery lifecycle when it is often painful and expensive to make changes necessary to improve security. We found that when teams “shift left” on information security — that is, when they build it into the software delivery process instead of making it a separate phase that happens downstream of the development process — this positively impacts their ability to practice continuous delivery. First, security reviews are conducted for all major features, and this review process is performed in such a way that it doesn’t slow down the development process. Chapter 7 – Management Practices for Software Limit work in progress (WIP), and use these limits to drive process improvement and increase throughput. Create and maintain visual displays showing key quality and productivity metrics and the current status of work. Use data from application performance and infrastructure monitoring tools to make business decisions on a daily basis. WIP limits on their own do not strongly predict delivery performance. It’s only when they’re combined with the use of visual displays and have a feedback loop from production monitoring tools back to delivery teams or the business that we see a strong effect. WIP limits are no good if they don’t lead to improvements that increase flow. Implement a lightweight change management process. We found that approval only for high-risk changes was not correlated with software delivery performance. Approval by an external body ( such as a manager or CAB ) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. Chapter 8 – Product Development The key to working in small batches is to have work decomposed into features that allow for rapid development, instead of complex features developed on branches and released infrequently. The ability of teams to try out new ideas and create and update specifications during the development process, without requiring the approval of people outside the team, is an important factor in predicting organizational performance as measured in terms of profitability, productivity, and market share. Chapter 9 – Making Work Sustainable The technical practices that improve our ability to deliver software with both speed and stability also reduce the stress and anxiety associated with pushing code to production. In order to reduce [deployment pain](https://kislayverma.com/programming/why-and-how-to-use-feature-toggles/), we should: Build systems that are designed to be deployed easily into multiple environments, can detect and tolerate failures in their environments, and can have various components of the system updated independently. Ensure that the state of production systems can be reproduced (with the exception of production data) in an automated fashion from information in version control. Build intelligence into the application and the platform so that the deployment process can be as simple as possible. Christina Maslach, a professor of psychology at the University of California at Berkeley and a pioneering researcher on job burnout, found six organizational risk factors that predict burnout (Leiter and Maslach 2008): Work overload Lack of control Insufficient rewards Breakdown of community Absence of fairness Value conflicts Chapter 10 – Employee Satisfaction, Identity, and Engagement Employees in high-performing teams were 2.2 times more likely to recommend their organization to a friend as a great place to work, and 1.8 times more likely to recommend their team to a friend. We found that the employee Net Promoter Score was significantly correlated with the following constructs: The extent to which the organization collects customer feedback and uses it to inform the design of products and features The ability of teams to visualize and understand the flow of products or features through development all the way to the customer The extent to which employees identify with their organization’s values investments in continuous delivery and Lean management practices, which contribute to a stronger sense of identity, may very well help reduce burnout. Being able to apply one’s judgment and experience to challenging problems is a big part of what makes people satisfied with their work. Chapter 11 – Leaders and Managers Being a leader doesn’t mean you have people reporting to you on an organizational chart — leadership is about inspiring and motivating those around you. According to this model (Rafferty and Griffin 2004), the five characteristics of a transformational leader are: Vision Inspirational communication Intellectual stimulation Supportive leadership Personal recognition Transformational leadership means leaders inspiring and motivating followers to achieve higher performance by appealing to their values and sense of purpose, facilitating wide-scale organizational change. A transformational leader’s influence is seen through their support of their teams ’ work, be that in technical practices or product management capabilities. …leaders cannot achieve goals on their own. As the real value of a leader or manager is manifest in how they amplify the work of their teams, perhaps the most valuable work they can do is growing and supporting a strong organizational culture among those they serve – their teams. Enable cross-functional collaboration by: Building trust with your counterparts on other teams. Encouraging practitioners to move between departments. Actively seeking, encouraging, and rewarding work that facilitates collaboration. From the internet No stinkin’ webhooks for [Anthony Accomazzo](https://twitter.com/accomazzo?ref=kislayverma.com), [he wants them events](https://blog.syncinc.so/events-not-webhooks?ref=kislayverma.com)! Paul Graham explains [how to write usefully](http://paulgraham.com/useful.html?ref=kislayverma.com). This Twitter thread gives a good introduction to how [Ethereum can be used for implementing decentralized single sign-on](https://twitter.com/BrantlyMillegan/status/1402388133086367751?ref=kislayverma.com). [Thierry Du Paw](https://twitter.com/tdpauw?ref=kislayverma.com) has collected [various interpretations of Conway’s Law](https://thinkinglabs.io/articles/2021/05/07/shades-of-conways-law.html?ref=kislayverma.com). All of these are fun to read, and neatly rabbit-hole into different aspects of socio-technical architecture. That’s all for this week folks. Happy Weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #42: The sense of purpose in system design URL: https://kislayverma.com/it-depends-42-the-sense-of-purpose-in-system-design/ Last updated: 1970-01-01T00:27:25.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #42: The sense of purpose in system design ![](https://kislayverma.com/content/images/2021/06/headphone-icon.jpeg) You can listen to this episode on [Spotify](https://open.spotify.com/episode/6chYoPPGCbRMv08JpvAiEh?si=32567f6cb3d04ca7&ref=kislayverma.com), [Google](https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81MTZiNzI0NC9wb2RjYXN0L3Jzcw/episode/NTBhMWZlOTItYTRiMC00MTk4LTg1N2MtMjhmNzczNmVkNTk0?ref=kislayverma.com), or [Apple](https://podcasts.apple.com/in/podcast/it-depends/id1571969334?ref=kislayverma.com#episodeGuid=50a1fe92-a4b0-4198-857c-28f7736ed594) Hello everyone! Welcome to the 42nd edition of It Depends. Hope you are all doing well. Today, I’m sharing a little more about teleology in designing systems (I learnt me some latin!) or what role “purpose” plays in the way we think about building systems. After that, it’s my hand-picked best of the internet as usual. A sense of purpose in system design ![](https://kislayverma.com/content/images/2021/06/System-boundary-environment-300x218.jpg) In [It Depends #39](https://kislayverma.com/?na=archive&email%5Fid=72), I wrote about how we should [design any system from one logical level up](https://kislayverma.com/software-architecture/system-design-from-one-level-up/), i.e. considering the environment of our system. A Redditor offered an interesting [comment](https://www.reddit.com/r/programming/comments/o6awmh/system%5Fdesign%5Fshould%5Fconsider%5Fthe%5Flarger%5Fsystem/h2rxjmo?ref=kislayverma.com) that this approach was contrary to how anything natural evolved. Natural evolution is always bottoms up and that seems a lot more flexible. This is certainly a valid observation and it got me thinking. Natural evolution happens bottom-up with everything co-evolving at the same time. Why should we not define and build the lowest levels first? The purpose of a system I have come to the conclusion that the difference is one of “purpose”. Purpose, in complex systems, is a decentralized thing, its shape differing from actor to actor. But the goal is always the same – an actor acts a certain way or pursues a certain goal because that, according to their limited mental model of the system, will allow them to perform better in their environment. The existence of a purpose implies the existence of one or more actors. For nature to have a purpose requires something to exist outside of nature. This supernatural entity would be contending with a supernatural environment. Without getting onto divine turf, a simpler conclusion is that nature has no definite purpose. Natural systems evolve without a sense of purpose or definite objective. Every element of the natural system tries to perpetuate itself in a changing environment by adapting. The system, on the whole, tries to attain a stable equilibrium regardless of what the equilibrium looks like. Any stable state will do – nature has no opinion on the quality of the outcome. It is acceptable in the natural world for entire evolutionary hierarchies to collapse if they [no longer fit in with the environment](https://pigontracks.substack.com/p/8-no-i-cant-give-you-certainty?ref=kislayverma.com). In man-made systems like large organizations or software architectures, we are not quite as generous. These systems exist to fulfill a certain role, and we intervene in them with clear intent. This is the [essence of strategy](https://kislayverma.com/books/book-review-wardley-mapping/) – defining goals and the action needed to achieve these goals. Evolution in man-made systems doesn’t run rampant, it is constrained to proceed in the directions which potentially lead to the outcomes we want – at least as far as we can tell at the moment. We want to minimize large failures that would result from completely uninformed trial and error. This is why system design should be done from one level up. First, we [visualize the effect we want to create](https://kislayverma.com/organizations/so-you-want-to-privatize-a-bank/) (our purpose), and then we take an action that is likely to attain that objective. In man ade systems, this is evolution. This is what makes [building shared context in teams](https://kislayverma.com/organizations/the-problem-is-not-the-problem/) so critical. A shared understanding leads to shared motivation and intent of action. The more actors share the same mental model, the more likely it is that an action can be pulled off successfully. Building the shared context is akin to extinction in the natural world, we are pruning those paths of evolution which might lead to an unsatisfactory equilibrium for us as a team. This is the “purpose” of the team. Hence the modern insistence on aligning engineering teams with top-level business objectives. If the teams are aligned to the organization’s global “purpose”, they are less likely to be f[ocussed on local maxima](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) when they operate. Everything is bottom-up This explanation, though practically useful, is philosophically misleading. Human and natural systems are not disconnected. All in all, the Redditor was right. Everything happens bottom-up. Even that is incorrect. Better to say that everything happens all at once. Every component of a system, whether above or below, reacts to changes around it. The confusion only exists because we subconsciously draw a logical boundary that defines up or down. It seems like we are building top-down because we are looking at only a part of the system to make a tactical decision. As an architect, I design from one level up because that allows me to impress my intent upon that specific neighborhood of the software system. I first define a boundary (consciously or subconsciously) inside which I want to take top-down action. Now I can be system-minded and look just outside the boundary to visualize what might be going on “outside”. The act of defining that scope gives the illusion of control, top-down action, and purpose. But seen from the outside, I am just another actor responding to my environment and motivations. This is the essence of Conway’s Law – regardless of my intentions, there are things I do not know and hence alternative actions that I cannot take. On the grand cosmic scale, we are all just doing the best we can to improve our situation in an un-opinionated universe. And that is good. From the internet [Laura Nolan](https://twitter.com/lauralifts?ref=kislayverma.com) explains what [essential complexity in software systems](https://www.infoq.com/presentations/complexity-distributed-behavior/?ref=kislayverma.com) is. If you‘ve read “[97 things every software architect should know](https://kislayverma.com/books/highlights-97-things-every-software-architect-should-know/)”, you would have come across repeated invocation of essential/accidental complexity in software. Laura’s talk sheds further light on this important subject. Here’s an [introduction to reactive systems](https://www.youtube.com/watch?v=Ysn6eInApYM&ref=kislayverma.com) by Dave Farley ([GOTO](https://twitter.com/GOTOcon?ref=kislayverma.com) ‘21). Simon Sarris explains how the world is a malleable place, and [the most precious resource to shape it is agency](https://simonsarris.substack.com/p/the-most-precious-resource-is-agency?ref=kislayverma.com). This is a powerful message for the managers and the managed. [Tristan Slominski](https://twitter.com/tristanls?ref=kislayverma.com) gives an [introduction to the Cynefin Framework](https://nomotherships.com/2021/03/10/cynefin-complexity/?ref=kislayverma.com). This is a great framework to know when making sense of complex, changing situations. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### It Depends #41: Systems Thinking with Dr. Russell Ackoff URL: https://kislayverma.com/it-depends-41-systems-thinking-with-dr-russell-ackoff/ Last updated: 1970-01-01T00:27:13.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #41: Systems thinking with Dr.Russell Ackoff Hello Everyone! Welcome to the 41st edition of It Depends, your weekly dose of awesome technical reading. Today, it’s about learning systems thinking with a legend (no podcast version sadly), and then the best of the internet as usual. Systems Thinking with Dr. Russell Ackoff I have recently started getting interested in [systems thinking](https://kislayverma.com/tag/systems-thinking/). I got started in this field by [Donella Meadows’ “Thinking in System”](https://kislayverma.com/books/book-review-thinking-in-systems-a-primer/) which was great. And a few weeks ago I discovered Dr. Russell Ackoff (Hat tip to [Trond Hjorteland](https://twitter.com/trondhjort?ref=kislayverma.com) for that). His genius for telling stories and correlating these funny stories back to a systemic view of the world is as good as anything I have ever come across. There are a lot of videos of him explaining systems and management etc on Youtube, but like a lot of old-time proponents of a specific topic, many of them carry similar explanations, examples, and deductions. So I thought I will summarize and condense a few of the most “academic” of these videos to bring out the common and prominent ideas. I absolutely recommend that you listen to the original lectures because these notes do not convey the richness of thought, expression, and experience that the lectures contain. I only hope to give you a quick boost in terms of ideas before you have to decide to commit \~4 hours of time. I have spent \~20 hours over the last two weeks listening to Russell Ackoff and I do not regret it one bit. The 3 I summarized are these: [A morning with Russell Ackoff](https://www.youtube.com/watch?v=4iomyRCjEHA&ref=kislayverma.com) [An afternoon with Russell Ackoff (Part 1)](https://www.youtube.com/watch?v=OaUT2bxGMNA&ref=kislayverma.com) [An afternoon with Russell Ackoff (Part 2)](https://www.youtube.com/watch?v=Rz6Ifs190no&ref=kislayverma.com) A system is a whole that is defined by its function(s) in a larger system of which it is a part and that consists of at least 2 parts without which it cannot fulfill its defining function Ways of solving a problem Absolution: Ignore the problem and hope it will go away Resolution: Solving the problem based on prior experience and qualitative judgment. This is “satisficing” – doing enough that is better than nothing. But this can cause further problems which are often more complicated than the original problem. Dissolution: Redesign the system to remove the problem On Systems A system is a whole that is defined by its function(s) in a larger system of which it is a part and that consists of at least 2 parts without which it cannot fulfill its defining function. The essential parts of a system must satisfy 3 conditions: Each essential part can affect the behaviour or properties of the whole No essential part has an independent effect on the whole Each subset of parts can have an effect on the whole but not an independent effect. The essential characteristics of a system depend on how its part interact, not on how they act taken separately. No part of a system taken independently can perform the function of the whole The performance of a system is not necessarily improved when the performance of its parts taken separately, is. We understand how the parts of a system interact by the process of design. Through idealized design, we understand how the system ought to behave. Why leadership courses are useless Leadership is an art and a talent – it can’t be taught. The difference in different roles when leadership is mentioned: Administration: Direct others in pursuit of goal using some means where goals and means are both selected by a third party. Management: Direct others in pursuit of goal using some means where goals and means are both selected by the manager. Leadership: Guiding and encouraging others in pursuit of goal using some means where goals and means are both selected by them Leadership requires the ability to bring the will of others into consonance with the will of the leader so that they follow voluntarily. It is inspiration, not persuasion. The vision is the idealized design produced by the leader. This whole conversation is tinted with the idea of a charismatic leader instead of bottom-up leadership. But it also somewhat aligns with what I wrote about [building shared context](https://kislayverma.com/organizations/the-problem-is-not-the-problem/) and maybe that’s best done by some people who are the leaders. Why transformations fail Transformation requires the intelligence to identify a problem and the courage to do something about it Two types of errors: errors of commission: doing something wring errors of omission: not doing something that should have been done. In most systems of accountability, only errors of commission are registered. Hence, rational people choose to not pursue change. On Panaceas The righter you do the wrong thing, the wronger you become! Some of the deficiencies of Panaceas: Management should be directed at what we want instead of what we don’t want. Focus on the quality of work-life for workers instead of focusing on the quality of output. Ignorance of consumer wants: Wants have to be discovered by the process of design – software architects need to know this. Continuous improvement cannot keep up with step jumps Process Re-engineering: Focusing on a different kind of slice of the system rather than the whole system – hence anti-systemic and ineffective. Downsizing: The purpose of an organization is to create and distribute wealth. Hence downsizing is an immoral act. De-bureaucratize and de-monopolize internal loss-making units. Benchmarking of parts: anti-systemic. Benchmarking of the whole is what competition is. Benchmarking against competition -> continuous improvements -> we give up the opportunity to ideally design what we want. We also set the competition as the gold standard by benchmarking. Creativity Every creative process has three steps: Identify an assumption that is limiting the choices that can be explored. Remove the assumption. Examine and utilize the new landscape of choice now revealed. Principles of creativity (These are more tricks to solving problems creatively in a corporate environment) Deny the “facts of the case” and find them out for yourself. Remove externally imposed constraints. Influence those who cannot be controlled Enlarge the system Role reversal by using the source of the problem as the solution. Other thoughts about organizations Eliminate job descriptions: They are limiting. Get in good people, put them in an area/department, and ask them to do what they think needs to be done. Provide guidance and keep discussing what and how they are going to do. Salary shouldn’t be linked to status Don’t create managers for status, increase compensation as needed. Pay what the employee is worth, status/role is incidental. Fun Fun = self determination. Let people find out what they want to do. Management is not a profession, it is a form of employment. Professions have standards to which professionals owe their highest obligation. E.g. Hippocratic oath for doctors. Employees owe their highest obligation to the good of the organization. The mission statement of a team/organization has to be a deliberately designed expression. If the inverse of the statement is not logically viable, then the mission statement is unlikely to be instructive. e.g. “we want to provide superior returns to our shareholders” is meaningless because the inverse of this doesn’t make sense as a goal. Hence this statement cannot inform action. From the internet this week The only DAOs I knew till recently were Data Acces Objects. But there is a new breed of DAOs in town, and they are claiming to change the way we think of corporations. Theodor Marcu explains [what these new DAOs are](https://1729.com/daos?ref=kislayverma.com). An awesome as usual discussion by Nick Tune on [risk-averse and risk-tolerant modes](https://medium.com/nick-tune-tech-strategy-blog/sequencing-architecture-modernization-risk-averse-vs-risk-tolerant-e74191e39c34?ref=kislayverma.com) of modernizing an organization’s architecture. Check out Tod Golding’s great talk at AWS re:invent ‘19 on [building serverless SaaS on AWS](https://www.youtube.com/watch?v=egskuX3YYO4&ref=kislayverma.com). Back in the day, the world picked up a thing or two from the Japanese managers. The [hottest new management model](https://corporate-rebels.com/next-influential-management-model-of-the-world/?ref=kislayverma.com) may be from China! That's it for this week folks! Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #40: So you want to privatize a bank? URL: https://kislayverma.com/it-depends-40-so-you-want-to-privatize-a-bank/ Last updated: 1970-01-01T00:27:02.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #40: So you want to privatize a bank? ![](https://kislayverma.com/content/images/2021/06/headphone-icon.jpeg) You can listen to this episode on [Spotify](https://open.spotify.com/episode/547qXp5SJPu1VEdi3mPP3w?si=4fbaa39098414429&ref=kislayverma.com), [Google](https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81MTZiNzI0NC9wb2RjYXN0L3Jzcw/episode/M2Q3MDljNTktNDk2NS00MzRjLTkyMDEtMjE3NjZhOWNiYTlh?ref=kislayverma.com), or [Apple](https://podcasts.apple.com/in/podcast/it-depends/id1571969334?ref=kislayverma.com) Hello everyone! Welcome to the 40th edition of It Depends. I hope all of you are doing good. As usual, here’s what I wrote this week, followed by the best of the internet hand-picked by me, for you! Let’s go. So you want to privatize a bank I was recently listening to my friends argue about whether to privatize govt banks or not. Some argued that this will improve efficiency, some wondered if private corporations can be trusted with all the banking in the country, some debated socialism and its pros and cons. This is a common discussion pattern in both formal and informal settings. A lot of public opinions are framed this way (e.g. rapists should be summarily hung, but what about humanitarian treatment or legal rights) so is a lot of tech hype (e.g. NoSQL should be used for high scalability, but is it too complicated or do we have the tools). One thing that really struck me is that how this discussion is actually upside down. It assumes that a course of action has been determined, and now all that is left to do is to debate whether it is acceptable to everyone or not. The conversation on privatizing national banks presents an opinion like a strategy to attain an undefined goal. The argument is not about the privatization of banks at all! People talk past each other because each of them is talking about a different thing like corruption, socialism, bad loans, etc. These things just cloud the issue and keep us from the actual discussion. To actually discuss whether banks should actually be privatized or not, we would have to define the effect we are trying to create. Why have the conversation at all? All the other words are about the side effects and symptoms of an action we have presumably already taken (privatize the banks), not about “why” we took the action. The real question is “what do we want to achieve”. Economics and banks are elements of a system that does something, i.e. has a purpose. We need to define this purpose before we can define if a proposed change serves this purpose. Once we understand the purpose, we can try to navigate the environment and achieve it. Without knowing the purpose, “privatize the bank” means nothing. This should be familiar from the way a lot of inter-team presentations are done. A decision is presented for discussion and critique, but with the understanding that fundamental questions are not to be raised and any feedback should be given in the context of the conclusion being presented. We look at the proposal, debate whether it is good or bad, and propose changes that may improve the proposal. But for this critique to be useful, the feedback has to consider the purpose of the system and the change first, and that sadly does not happen very often. Let’s consider some technical examples of this. Decoupling is one of my favourites. The statement “decouple your components” is unlikely to engender debate, or at best engender debate on the mechanism of decoupling. It is a great idea, but the bigger question lies in the effect that we want to create? What exactly is it that I want to get out of this? Do I really want to change one of my components? The principle of decoupling isn’t valuable by itself, it needs the context of motive to become so. Apple, for example, doesn’t decouple. It integrates deeply. Or take autonomy in teams. At this point, the Twitter-going crowd believes that [autonomous teams](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) (two-pizza, problem-driven, etc) are the best way to organize a workforce. But why do I want this? Someone told me that engineers like this because they like “freedom”. Freedom from what? Who knows. If the intended effect is to efficiently use a large, unmotivated workforce, maybe autonomy isn’t the way to go. Old-school command and control might be best. The words “decouple” and “autonomy” are trying to use the perceived authority of best practices without identifying the context of applying these ideas. The point is not whether they are good or bad, but that we haven’t thought about the objective, and therefore must work backward from solution to the problem and hope they fit each other. We don’t even know what the problem is in the larger context. Starting from the intended effect “I want to reduce the cost of change in my software” might lead us to consider that change is faster without decoupling if deep integrations are required and swapping out components is not necessary. Starting from the effect “I want my teams to create high impact” might leads us to consider that before teams can create impact, they should understand which direction the organization wants to go in and therefore what “impact” looks like. That alignment must come before autonomy can be unleashed. Applying a strategy by looking at one aspect reverses cause and effect. Decoupling doesn’t axiomatically improve software. Autonomy doesn’t automatically create impact. Without knowing the context and intent, a strategy can only succeed by chance. It is more likely to cause damage via second-order effects. So if you want to take action, first identify the intended effect and the environment in which you are working. All else will follow from that. From the internet Continuing with Gregory Hohpe’s theme of architects and architecture I shared in the [last week’s newsletter](https://kislayverma.com/?na=archive&email%5Fid=72), here’s [Eduardo Silva](https://twitter.com/emgsilva?ref=kislayverma.com) talking about [architecture topologies and architecture as an enabling team](https://esilva.net/tla%5Finsights/architecture-topologies?ref=kislayverma.com). The Capital One team has shared some useful guidelines on how to [decompose a monolith into microservices using event storming](https://medium.com/capital-one-tech/event-storming-decomposing-the-monolith-to-kick-start-your-microservice-architecture-acb8695a6e61?ref=kislayverma.com). [John Cutler](https://twitter.com/johncutlefish?ref=kislayverma.com) shared an awesome [visualization of work in progress](https://www.loom.com/share/5efceb288b634a449041918bdba08202?ref=kislayverma.com) for teams. Awesome as always from the man! [Tanya Reilly](https://twitter.com/whereistanya?ref=kislayverma.com) has a great tip for all technical “leaders” - [supporting other’s great ideas](https://leaddev.com/technical-decision-making/having-impact-engineering-supporting-other-peoples-ideas?ref=kislayverma.com) can be more powerful than generating great ideas. That’s all for this week folks. Have a great weekend! \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #39: System design from one level up URL: https://kislayverma.com/it-depends-39-system-design-from-one-level-up/ Last updated: 1970-01-01T00:26:52.000Z | [![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg)](https://kislayverma.com/) If you are already a subscriber, jump ahead to the good stuff! But if someone forwarded you this email, they probably love *It Depends* and think you would too! [Subscribe now!](https://kislayverma.com/newsletter-archive/) #39: System design from one level up ![](https://kislayverma.com/content/images/2021/06/headphone-icon.jpeg) You can listen to this episode on [Spotify](https://open.spotify.com/episode/3H1mZpqlwyYcybQF05sxAl?si=4d470320e7784f73&ref=kislayverma.com), [Google](https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy81MTZiNzI0NC9wb2RjYXN0L3Jzcw/episode/MjEyMTQ2MGUtYmE4MS00NjhiLWExMGItYjg2Y2UyZTM3MTBk?ref=kislayverma.com), or [Apple](https://podcasts.apple.com/us/podcast/it-depends/id1573155514?i=1000526599247&ref=kislayverma.com) Hello Everyone! Welcome to the 39th edition of *It Depends*. Hope all of you are doing good. I have tried to change the design to make discovering the podcast easier and to make it easier for you to share it with your friends. If you know someone else who says "it depends" as much as you or I, just forward them this email so they can join us. And if you have been liking what I have been writing, you can show some extra support via [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [BuyMeACoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com). Let's talk about system design. Given some requirements for a system, how should one start designing it? Before answering that question, let’s first think about how one understands an existing system. A typical approach is to first understand the boundary of the system. Inside the boundary is the system, outside the boundary, is the environment. The boundary separates the system from its environment and therefore, in a way, defines what the system is. Like a fence with a door, the boundary has gates that allow well-defined interactions between the environment and the system. Both the boundary and the internals need to be designed well. ![](https://kislayverma.com/content/images/2021/06/System-boundary-environment-300x218.jpg) In an existing system, we can look at the internals of the system and tell what each part does. We can say that MySQL has been chosen because transactionality is required (or the other way round – why would they use MySQL if they didn’t need transactionality) and so on. But we cannot look at the edges of the system and explain “why” they do what they do. Understanding the design of the boundary requires us to understand the environment in which it was built. This is the difference between what Russell Ackoff calls [know-how and information](https://www.youtube.com/watch?v=spm2HUxgI30&t=1s&ref=kislayverma.com). I can tell how the system does what it does by looking at it in isolation, but only by looking at its environment can I tell why it does what it does. Documentation, ADRs, tribal knowledge, etc. are all tools for creating this environmental context. Unit and integration tests are not because they only verify the internals of the system and not its objectives. Therefore, it only makes sense to establish the environment for a system and identify its role in that environment before we figure out how to make it work. In other words, start with why. When given a system to build, I treat it like a component of a larger system and go one level higher to ask “what role does my component play in the larger system” or “who will interact with this component”. Sometimes this answer is simple because it plays a minor role in the larger system. Sometimes, it plays a major role, and defining the environment (i.e. the larger system) initially lets me understand the role of my component a little better. The clearer the environment and the other subsystems are, the more confidence I can have that I have considered all the roles my component is required to play. In practical terms, documenting this process turns out like the[ narrative approach to software design](https://kislayverma.com/software-architecture/a-narrative-approach-to-software-design/) I have written about before. ![](https://kislayverma.com/content/images/2021/06/system-and-components-300x195.jpg) All this talk about edges and systems is not just philosophy. It helps in establishing the degree and dimensions of uncertainty, and this, in turn, has significant consequences on the design choices we make. If the role of the component in the environment is an experimental one, we should probably not harden the boundaries of our system too much just yet. We can go for simpler implementation and design internally. We should even be ready to throw away the component as the environment evolves. If we ignore the uncertainty in the environment, I might harden the interfaces of my component too early and get stuck with an inflexible component that is difficult to evolve. On the other hand, if the environment has well-defined expectations from my component (“Requirements are clear” in developer parlance), I choose to build well-defined interfaces from the get-go and go for a more robust internal implementation. In this case, leaving ambiguity in the interfaces would only cause confusion later. Maintainers might imagine uses for these flexible interfaces which are not needed or intended in practice. So we identify the shape of the system first before we start colouring between the lines. We can now start breaking down the walls of the component to design its internal parts. In fact, this is the same process we will follow for building the component. We identify the purpose of the component and then create sub-parts that collectively serve this overarching purpose. The component is the environment and each internal piece is a system to be built. We should apply the same principle to the original requirement specification. ![](https://kislayverma.com/content/images/2021/06/system-system-system-300x188.jpg) Requirements do not arise in isolation. The very fact there are “requirements” indicates that there is a purpose, and that purpose is created by something external to or “above” the requirement. Going one step higher to understand the environment will help us design a system that is much better suited to the context of the system. From the internet this week With microservices and autonomous teams being the flavour of the day, do we really need architects? [Gregor Hohpe](https://twitter.com/ghohpe?ref=kislayverma.com) explains [how to do the activity of software architecture with or without a person called software architect](https://architectelevator.com/architecture/organizing-architecture/?ref=kislayverma.com). Tim Urban has a deep yet funny exploration of [why we believe what we believe](https://waitbutwhy.com/2019/09/thinking-ladder.html?ref=kislayverma.com). This entire series of articles is worth reading. Alistair Cockburn has a great explanation of [Hexagonal Architecture](https://alistair.cockburn.us/hexagonal-architecture/?ref=kislayverma.com) (aka the ports and adapters pattern). Shishir Mehrotra offers an inside view of [how Youtube scaled its team](https://coda.io/@shishir/rituals-for-hypergrowth-an-inside-look-at-how-youtube-scaled??ref=kislayverma.com). That's it for this week folks! Have a great weekend. \-Kislay If you love reading *It Depends*, consider supporting it on [Patreon](https://www.patreon.com/kislay?fan%5Flanding=true&ref=kislayverma.com), [Gumroad](https://gumroad.com/l/it-depends?ref=kislayverma.com), or [Buymeacoffee](https://www.buymeacoffee.com/kislay?ref=kislayverma.com) Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #38: Combining rule systems and machine learning URL: https://kislayverma.com/it-depends-38-combining-rule-systems-and-machine-learning/ Last updated: 1970-01-01T00:26:40.000Z | ![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg) #38: Combining rule systems and machine learning Hello Everyone! Welcome to the 38th edition of It Depends. Hope y’all are safe. I know [NYC is back](https://www.instagram.com/reel/CQOjoQMJONq/?utm%5Fmedium=copy%5Flink&ref=kislayverma.com) and all that, but here in India we are just starting the long road to recovery. So if you are reading this in India or any other places where things are not back to normal yet, please get your vaccine, keep your mask on, and be kind to your teams. This newsletter just hit the 1600 subscriber mark ([the podcast](https://open.spotify.com/show/5gY1eGUE0RmNHj5t1HKpJI?si=n1-IyimsTmad-LH%5FQRqbWw&dl%5Fbranch=1&ref=kislayverma.com) launched last week has 24 followers)! My heartfelt thanks to each one of you. Please keep offering your critique and spreading the word. You can read this article directly on the [website](https://kislayverma.com/programming/combining-rule-systems-and-machine-learning/) or listen to it on [Spotify](https://open.spotify.com/episode/2jbETxH8y8hoQLHNhKX9Tj?si=n6EBjOtMStKNvw9-tVINyQ&dl%5Fbranch=1&ref=kislayverma.com). I was recently reading an [article by Neal Lathia](https://nlathia.github.io/2020/10/ML-and-rule-engines.html?ref=kislayverma.com) about how using machine learning is not always necessarily better than using rule-based systems. There are pros and cons to taking either approach, and you should take the one which suits the problem complexity, expected execution speed, and various other factors the best. I am working on a project which I and my team believe is a great fit for applying ML techniques. However, we have also achieved some measure of success in achieving our goals using a rule-based system. Without sharing the details, I can say that while we have a long way to go, one thing that is increasingly clear is that rule-based and AI/ML-based approaches to building systems are not mutually exclusive. There are various ways in which these can be applied together effectively. I want to discuss some of these ideas. The thing to remember is that every system is made up of many parts and each of them serves a different function. Most parts are simple and require little to no intelligence in the [machine learning](https://kislayverma.com/tag/machine-learning/) sense. So it only makes sense to slice the problem and the overall system into smaller parts and apply the technique most suited to it. The parts that require inferences made from lots of data can be built using machine learning techniques, and other parts can be built using rules or plain old application engineering. In fact, acknowledging that these two types of parts exist in a system can have a dramatic impact on the architecture and technology choices being made. Rule System output as ML feature Machine Learning models work on sets of inputs called features. Features might already exist as first-class artifacts in some data store, or as is often the case, they are attributes derived from multiple other data points of the system. These derivations are often built using simple rules or heuristics and then consumed by ML models. Both steps of the process operate independently but play an important role in the final outcome. Think of this as lower-level staff processing raw information into reports that can be consumed by upper management to make complex decisions. ML Model output as input to rule engine The inverse of the above process is also common. We can have ML models use various features to come to a conclusion which is then used as one of the inputs to a rule-based system. This again works by splicing a complicated into two parts – the more intelligent/complex part uses ML to process complex data patterns. Once the data is reduced to a simple conclusion, the rule-based system can jump in to make further, simpler decisions. Think of this as upper management passing down the results of complex strategic analysis where the lower layer can make comparatively simpler decisions about how to execute things. Rules as elements of AI This last pattern of combining rule and AI/ML is especially fascinating for me because it blurs the lines of what is simple and what is complex. ![](https://kislayverma.com/content/images/2021/06/Screenshot-2021-06-17-at-11.31.04-PM-300x137.png) A whole class of AI-based systems called [Learning Classifier Systems](https://en.wikipedia.org/wiki/Learning%5Fclassifier%5Fsystem?ref=kislayverma.com) (probably others too, I’m no expert) use rules as the building blocks of complex evolutionary hierarchies where rules mutate and evolve in such a way that we finally end up with rules that best fit the given application environment. This is a complex way of applying either model but it shows that there is sufficient common ground between them. Think of this as a brainstorming session with all levels of the company (autonomous team? 5 person startup?) where simple ideas are churned around till strategies and execution plans emerge together. Since this is a real-life problem for me at work, I want to hear what you think about it and if you have used such a combination with good results. Drop a note with your experiences. From the internet I discovered systems thinking gold last week in the form of lectures and interviews by Dr.Russell Ackoff. [This lecture](https://www.youtube.com/watch?v=C3j6IChvsBE&ref=kislayverma.com) talks about reductionist and expansionist modes of thinking in a way that I had never encountered. But please go ahead and listen to everything by this genius. [Evan Bottcher](https://twitter.com/evanbottcher?ref=kislayverma.com) on [what a platform and isn’t](https://martinfowler.com/articles/talk-about-platforms.html?ref=kislayverma.com). This vibes strongly with a lot of my thoughts on [platform systems](https://kislayverma.com/category/platform-thinking/). [Mark Greville](https://twitter.com/markgreville?ref=kislayverma.com) shares his perspective of [how technology architects make decisions](https://markgreville.ie/2021/02/17/how-architects-make-decisions/?ref=kislayverma.com). That’s all for this week folks. Happy Weekend! \-Kislay Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #37: The problem is not the problem URL: https://kislayverma.com/it-depends-37-the-problem-is-not-the-problem/ Last updated: 1970-01-01T00:26:14.000Z | ![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg) #37: The problem is not the problem Hello Everyone! Welcome to the 37th edition of *It Depends*. Hope you are all doing well and have received your shots of the vaccine. I have exciting news! As of yesterday, It Depends is also a podcast. You can check it out on [Spotify](https://open.spotify.com/show/5gY1eGUE0RmNHj5t1HKpJI?ref=kislayverma.com), [Pocket Cast](https://pca.st/pwor8upz?ref=kislayverma.com), and [Anchor](https://anchor.fm/kislayverma?ref=kislayverma.com). Apple Podcast is coming soon. Spread the good word to your friends and colleagues who would rather listen than read. What fun is software engineering if you can't discuss podcasts with friends :) Let's talk about making organizational change. *You see a problem at work and you think you have a solution. Some people you have spoken to in hallways agree with you. So you go ahead and try to solve it but suddenly start getting pushback from everywhere and eventually things don't work out as you had thought they would. You are left bitter and frustrated at "them".* How many of you have been in this situation? I know I have and today I want to talk about what I have learned from those failures. This is an idealized version of how I now think about orchestrating high-impact changes. There are many "real-world" details that you will have to fill in based on yourself, your team, and your organization. As an engineer, I have often had a somewhat reductive point of view of the organization's problems. My mind jumps straight to what system can I build to solve a problem. This tech-centric perspective told me that those who couldn’t see that building this type of system will solve the problem just “didn’t get it”. Sometimes, I believed that even if I couldn’t convince “them” (my manager, my PdM, other engineers in my team, etc), I would do it my way and they would thank me later. DON’T DO IT! This is the single worst thing you can do in a team. Working in a team successfully implies having the ability to convince your team and stakeholders about why you see a problem a certain way and why you think a certain solution will work. The kind of behaviour I just spoke about is a direct refusal to engage with the system around you. If your team saw you doing this, you will never regain their trust because not only did you fail to convince them, you overrode their disagreement and did something for which they now share accountability. I’ll share some systemic reasons why you are wrong to act like this in a minute. But there are practical reasons too for why the lone ranger act is unlikely to work. Software engineering is a team sport. Even if you go off and do something your own way, who will make your solution work on the ground? The people you did not convince? The manager you did not get buy-in from? Good intentions versus Systems Thinking Systems thinking tells us that all actors in a system have different mental models of the system. Donella Meadows calls this "Bounded Rationality" in her book "Thinking in Systems". Each actor has their own perspective, problems, and priorities. While the symptoms may be visible to everyone to some extent, their interpretation of underlying causes can vary wildly. This is one of the reasons "root cause analysis" is sometimes a fallacy - there may not be one root cause. Often, problems are caused by actors working on local information for local goals. Replace "actors" with departments or employees and you get a reasonable picture of how organizations work. ![](https://kislayverma.com/content/images/2021/03/blind-men-and-multiple-readmodels-300x156.jpeg) The solution, especially for large or complex problems, isn't often an objective truth that can be shown to be true or correct regardless of perspective. If a solution involves multiple actors, we are trying to change the way all of them view and interact with their world. We can do this in a command-and-control way but this takes away the agency of the actors. No one likes to be told to do something "because I said so". This can easily cause "Policy Resistance" - actors resisting or circumventing a central directive. Here's a compilation of some of the [most hilarious backfires](https://twitter.com/TrungTPhan/status/1396500898742824960?ref=kislayverma.com) of this model of working. This is what you are up against in trying to "get in and do it myself”. Let’s think about the problem. Is it even a problem? It may be, but my particular framing of it reflects only my interpretation of the aspect of the problem that is visible to me. Perhaps whatever-it-is is only a problem as perceived from a technology standpoint ("we are no longer a tech company, what should we do"?). Unless I have exposure to all viewpoints and information (practically impossible in an organization), my understanding of the problem could be biased or incomplete. Someone from the business teams could have a completely different point of view on the same visible symptom. What's ironic is that we could both be right! The other problem is that this POV assumes that all solutions are technical and I am the centre of the world. This is obviously not correct. In most organizations, technology is only one part of the landscape and often a small one. Until we surrender our vantage and ego, we are unlikely to see the full shape of the world. Build a shared context and worldview *If you want to build a ship, don't drum up people to collect wood and don't assign them tasks and work, but rather teach them to long for the endless immensity of the sea - Antoine de Saint-Exupery* A better way of creating change is to look at the system collaboratively from multiple actors’ perspectives and build a shared understanding/context of what it looks like. This goes beyond convincing or getting buy-in on my perspective. Invite collaboration by getting all involved actors to come together and build a common world. This new, shared worldview will likely be far richer than any single actor’s view, and the process of constructing it will start a dialog where problems and solutions are easier to discover and discuss. **The problem, therefore, is not the problem. Building a shared view of the system to understand it - that is the problem.** In this process, my perspective can serve as the basis out of which the discussion grows. This suggestion is based on Tim Casasola’s suggestion of building [containers for collaboration](https://theoverlap.substack.com/p/containers?ref=kislayverma.com). We want a collective framing of the environment and the problem, but having a starting point helps. Scoping the audience in these discussions is critical. The group should be as small as possible to cover all viewpoints needed. This process is about balance - not ivory tower, and not a committee. Every member of the group should be directly involved in the resulting decision or directly impacted by it. Anyone not directly impacted is a consultant who may inform the group but has no part to play in it. e.g. If the problem is "our systems have too many outages", including marketing in the conversation has no purpose - keeping it limited to technology and operations should suffice. From problem to solution This is where I have made the most mistakes in my career. Even in situations where I and my team agreed on the problem, I used this agreement as a vindication of my original perspective and reverted to acting on my solution. It was "ah everyone agreed, so now let's get to work now". The obvious problem here is that the shared context is abandoned and we go back to "Let me tell you what to do" mode. Given that a collective understanding exists, it is especially regressive for me to push my perspective all over again. A far better way is to simply continue the dialog now in the direction of solutions. Very often, the process of building the shared context will reveal solutions (often multiple) organically. Essentially rinse repeat till the team converges on a solution. Of course, it can be that there is no single solution acceptable to everyone. We can use compromises, or multiple limited solutions, or continue to disagree forever. Any way we choose, the key thing is that everything from now on happens with a collective agreement. The one thing we want to avoid is going rogue if the conclusion isn't to our liking. As I started out by saying, "against the tide" efforts are unlikely to succeed and erode the team's trust forever. What should we do next? Nothing stands still, ever. A shared perspective is not a static thing. Every action we take changes the environment, so to maintain our situational awareness, we need to keep the dialog going. With the dialog constant, the worldview is constantly updated and new possibilities are constantly explored. The next steps become organic increments rather than big bang “quarterly plan” type efforts. Since this shared understanding is the core of how a team functions, the context IS the team. If the worldview changes significantly, a whole new type of team may be required. In any case, the key is to keep talking and learning. Why doesn't it happen more often? This idealized process sounds good enough that you’d expect some variant of it to be attempted often enough. The process is not very efficient as a one-time activity to deal with some specific situation. As I said before, the dialog is a flywheel. If it isn’t kept alive, then you might as well not do it. It is extremely hard for intelligent people to surrender their POVs. Ironically, the more passionate each of them is about the problem, the harder it gets to contribute constructively to the collective. The idea of including "everyone involved" leads to involving so many people that building a shared context goes from being difficult to being impossible. Often, there are bad-faith actors or power-play situations. The participants are not even interested in the intent of the dialog - only in extracting some other types of outcome. From the internet this week [Apenwarr](https://twitter.com/apenwarr?ref=kislayverma.com) claims that [system design can explain the world](https://apenwarr.ca/log/20201227?ref=kislayverma.com). Can it? You decide. So you want to build a developer platform. Martin Fowler explains what [capabilities you need to build to execute this strategy](https://martinfowler.com/articles/platform-prerequisites.html?ref=kislayverma.com). Mind the gap! Pair programming is a powerful tool, but can it be taken too far? Nat Bennett explains [how](https://www.simplermachines.com/the-mortifying-ordeal-of-pairing-all-day/?ref=kislayverma.com). That's it for this week folks! Have a great weekend. \-Kislay Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #36: How to organize your code URL: https://kislayverma.com/it-depends-36-how-to-organize-your-code/ Last updated: 1970-01-01T00:25:48.000Z | It Depends #36 How to organize your code? ![](https://kislayverma.com/content/images/newsletter/thumbnails/2021/06/it-depends-1200x0.jpeg) Hello Everyone! Welcome to the 36th edition of *It Depends*. Hope you are all doing well and staying safe. I'm back this weekend with some technical thoughts and the best of what I read on the internet. What is the most popular style of arranging code you have come across in enterprise codebases? The one I have seen most often groups all classes (assuming Java-land) by the layer in the tech stack. So in an MVC style system, all controllers are together, all services are together, all repositories are together, all POJOs are together etc. Let’s call this convention the **“stack” style of organizing code**. ![](https://kislayverma.com/content/images/2021/06/organizing-code-stack-style-300x186.png) This is a terrible way of organizing code and I will explain why below. But first, allow me to offer the alternative. A much better way of organizing code to group it by the logical entities it represents. Let’s call this the **“entity” style of organizing code**. The idea is to make sure that all classes related to a single concept stay together. By putting the logical entities first, we are optimizing for human comprehension (compilers don’t care where you put which class). By virtue of how the code presents itself, developers are nudged to make smarter choices about where the actual system boundaries lie. Not between *SomethingRepository* and *SomethingElseRepository*, but between *Something* and *SomethingElse* as concepts. ![](https://kislayverma.com/content/images/2021/06/organizing-code-entity-style-300x164.png) Now let’s understand why I think the entity model is better than the stack model. Improper abstraction People don’t read code by layers of the stack. No one ever says “show me all the APIs of this system” or “give me all the queries being fired by this system”. People read code along domain boundaries. In a hotel management system, people think about rooms, and guests, and prices, and so on. Since “stack” style code is organized along technology layers, it is difficult to understand the logical model of the system from the way it lives in the repository. The boundaries the “stack” style exposes are technical layers. We cannot understand the “nouns” and the relationships between them from this code. You have to dig one level deeper for that. For a new person reading the code, this obfuscation of “logical” structure is a huge source of friction. In our hotel management example, the “entity style puts all code related to guests (regardless of the technical layer) into one package, all code related to rooms goes into another, and so on. Each of these packages can have its own internal organization in the “stack” style or just a few classes all at the same level. This makes it easy to find everything related to guest in one place. Poor cohesion Another common argument given for the “stack” style of arrangement is that it puts separate modules at different layers of the tech stack. e.g. Controllers are visibly separated from service, service from repositories etc. To find classes at different levels of the tech stack, you need to go to packages representing those levels. This encourages decoupling between the different layers. ![](https://kislayverma.com/content/images/2021/06/poor-cohesion-in-stack-style-210x300.png) The problem with this argument is that it focuses on coupling but disregards the other critical property – cohesion. Between what classes do we want to increase cohesion and which do we want to decrease coupling? Since all services are located together, can we say it is okay for them to be highly cohesive but decoupled from their model classes or repositories? Can we allow all repositories to become highly reliant on each other but decoupled from the business logic of the service layer? The obvious answer is NO! This kind of code would be textbook big-ball-of-mud. Refactoring such a system into smaller systems would be an absolute nightmare because you would have to decouple classes at every layer of the tech stack. It defeats the whole purpose of using an MVC style. The “entity” style, OTOH, promotes cohesion while still leaving room for tech stack style decoupling. It is okay if all hotel-related classes depend on each other (technically or conceptually) since they form a single unit of work anyway. It also makes future refactoring easier because the logical boundaries are clearer than in the “stack” style. ![](https://kislayverma.com/content/images/2021/06/high-cohesion-in-entity-style-229x300.png) Hard to Change To make any meaningful change in a codebase organized in “stack” style, a developer has to cut across multiple packages. e.g. to add a new field to an entity and its CRUD API, all packages will be modified. This creates cognitive load because the developer has to modify many “things” rather than a single logical thing. In the “entity”, if you change a thing, you make changes only in one logical boundary. This makes changes to them easier because we are working only in a small part of the codebase if working with a single entity. If you cut across top-level packages, you are cutting across logical constructs by definition and this will alert you to potential coupling-related considerations. Limits design choices Since code is organized by tech stack or functionality, it limits the way people think about system design. e.g. Since business logic should go into “services”, developers resist using proper design constructs and would rather shove everything inside services thereby creating nightmare classes thousands of lines long. Even when they use good design principles, the organization of the code resists them because every new “type” has to be in a unique package. If I want to use the factory pattern in different services, then I have to invent a whole new package hierarchy called *factory* and henceforth all factories should go there whether or not they have anything to do with each other. As I mentioned earlier, the “entity” makes no assumptions about how each logical package is grouped internally. It can be in the stack style, or have as many types of packages as required without influencing the choices made in another entity’s package. ![](https://kislayverma.com/content/images/2021/06/design-freedom-in-entity-style-221x300.png) One concern here can be about how to organize things that span across entities. e.g. workflows operating on multiple entities. Neither style has a neat answer to this, but IMO the “entity” style does a better job at it since it forces the creation of a new package outside all entity packages. This highlights that a workflow is a new concept and potentially a system boundary that should be developed independently. The idea is to group similar concepts together, but things not bound to a single concept can still have their own logical homes in the base. The modes of thinking that code organization promotes are something I feel we don’t think about enough. This is similar to Conway’s law at the codebase level. I’d love to hear more from you about how you organize your code and how you think it shapes developer behaviour, mental models, or efficiency. Drop a note in the comments! From the internet this week Vitalik Buterin explains why, in the blockchain context, the most precious thing is [legitimacy](https://vitalik.ca/general/2021/03/23/legitimacy.html?ref=kislayverma.com). I'm not convinced that a technology that poses a serious environmental hazard should be evaluated on legitimacy, but the blockchain story is still unfolding IMO, and we shall see. Luke Craven explains [why certainty isn't possible in complex environments](https://pigontracks.substack.com/p/8-no-i-cant-give-you-certainty?ref=kislayverma.com) (you know, like the real world we all live in), and embracing uncertainty is the real way to deal with problems. If you are interested in systems thinking, I strongly recommend signing up for his [substack](https://pigontracks.substack.com/?ref=kislayverma.com). If you are working on migrating your monolithic system to microservices, or just generally decomposing a system into smaller parts, [Matt Stine](https://twitter.com/mstine?ref=kislayverma.com)'s "[what's your decomposition strategy?](https://medium.com/built-to-adapt/whats-your-decomposition-strategy-e19b8e72ac8f?ref=kislayverma.com)" is worth a read. Tl;DR - several approaches to decomposition, and "it depends". [Eduardo Silva](https://twitter.com/emgsilva?ref=kislayverma.com) explains how to [evolve an organization using socio-technical architecture](https://esilva.net/articles/evolve%5Ftech%5Forgs%5Fusing%5Fsociotech?ref=kislayverma.com) concepts. That's it for this week folks! Have a great weekend. \-Kislay Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #35: How big should a method be? URL: https://kislayverma.com/it-depends-35-how-big-should-a-method-be/ Last updated: 1970-01-01T00:25:47.000Z | It Depends #35 How big should a method be? Hello Everyone! Welcome to the 35th edition of It Depends. Hope you are all doing well and staying safe. My apologies for not sending out an edition last weekend - I got caught up with a few things and couldn't make time. More than one person has reached out to me asking about how to structure code in methods. The question I hear most often is how much to break down a method. So let’s talk about how large a method should be. Let’s say there’s a method in the codebase that does 4-5 things one after the other. The code for doing all those things is well-written, but the result is a somewhat large method. At what point should you break it up into multiple methods? There is the old adage about every method being fully visible without scrolling which I have always found weird. Let’s try to do better. The good part about having all of the code inside a single method is that it is all in one place and in a sense, easy to absorb in one shot. The bad is part is that it makes the method large and if split, the reader will have to move in and out of multiple methods to understand what is going on. I prefer the second approach because smaller methods are easier to understand for me. But there has to be a balance in that direction too. So while the correct answer is obviously “it depends”, I’ll try to give a little more in the way of guiding principles. The key to writing or refactoring software is to reduce the cognitive load on the reader/maintainer of the code. All other considerations are subservient to that goal (of course, “it depends”). So the key to answering the question of granularity is to identify how people read and understand any piece of code. Or rather, how do we want them to read a piece of code. Typically people read code from a higher level of abstraction to a lower level of abstraction. The guiding principle of each layer of code, typically represented by methods, is that it should prevent the user from wanting to dig deeper into the next layer. Not in the sense of sending them away screaming as quickly as possible, but rather by making what must be happening underneath so obvious that the reader never feels the need to read the next layer. In the context of this article, cognitive load can manifest in two forms: engagement and curiosity. We don’t want them to feel like they have to think (engagement) to understand something, and we don’t want them to want to think (curiosity) about something. This is sometimes called the “[Principle of least astonishment](https://en.wikipedia.org/wiki/Principle%5Fof%5Fleast%5Fastonishment?ref=kislayverma.com)” and applying this can help us bound the upper and lower granularity of a method. At each level, a well-written method represents a declarative (what is to be done) unit to work to its higher layers (the ones who called it) and internally contains a set of imperatively defined steps (how is this to be done). In this sense, our method in question which does 4-5 things, is in some sense a workflow. As I wrote in [my previous article on workflows](https://kislayverma.com/software-architecture/architecture-pattern-orchestration-via-workflows/), the place where a workflow is defined should only define what steps are to be taken, not how each of them works internally. Let’s take the example of booking a doctor’s appointment at a hospital. A makeAppointment(hospitalId, doctorId, patientId, startTime, endTime) method can do several things: Check the working hours of the hospital Check if the doctor has no other appointment during that time Book the appointment Notify the doctor of the appointment Notify the patient of the appointment Let implement this by putting all the code in one place ([gist](https://gist.github.com/kislayverma/69d96b7b18497e7b6d376de6114fb9d2?ref=kislayverma.com#file-method-doing-multiple-things)). public Slot makeAppointment(Doctor d, Hospital h, Patient p, Date startTime, Date endTime) throws Exception { // Check if hospital is open during this time period String hospitalServiceBaseUrl = ConfigReader.readConfigName("hospitalServiceBaseUrl"); HospitalService hospitalService = new HospitalService(hospitalServiceBaseUrl); DateRange range = hospitalService.getWorkingHours(); if (!range.contains(startTime) \|| !range.contains(endTime)) { throw new Exception("Hospital isn't open in this time period"); } // Check if doctor has no other appointment in this time period String scheduleServiceBaseUrl = ConfigReader.readConfigName("scheduleServiceBaseUrl"); ScheduleSerivce scheduleService = new ScheduleService(scheduleServiceBaseUrl); Slot s = scheduleService.getAppointmentSlots(d).stream() .filter(slot -> !slot.isOccupied()) // Only consider free slots .filter(slot -> slot.contains(startTime) && slot.contains(endTime)) // Only consider slots which contain this time period .findFirst(); // Create the appointment s.setIsOccupied(true); Slot bookedSlot = scheduleService.bookSlot(d, p, s); // Notify doctor using custom email template DoctorEmailTemplate doctorEmailTemplate = new DoctorEmailTemplate(d, p, bookedSlot); notificationService.notify(d, doctorEmailTemplate); // Notify patient using custom email template PatientEmailTemplate patientEmailTemplate = new PatientEmailTemplate(d, p, bookedSlot); notificationService.notify(p, patientEmailTemplate); return bookedSlot; } This method is not so bad as some real-world code you may have seen. But the moment I reach this method, my brain has to wake up to the details of everything going on here. I wanted to understand how an appointment is booked but suddenly I’ve run into a bunch of API calls and other things. This doesn’t exactly match my [mental model](https://kislayverma.com/programming/how-to-write-self-documenting-code/) of how to book an appointment. This result is a high cognitive load. Therefore, this method should be broken down till it reaches the mental model “in English” as much as possible. Here’s a possibility ([gist](https://gist.github.com/kislayverma/daac16e77b4542455c82335accde9bfa?ref=kislayverma.com)). public Slot makeAppointment(Doctor d, Hospital h, Patient p, Date startTime, Date endTime) throws Exception { if (!isHospitalOpen(h, startTime, endTime)) { throw new Exception("Hospital isn't open in this time period"); } Slot s = getFreeSlotForDoctor(d, startTime, endTime); if (s == null) { throw new Exception("Doctor is not free in this time period"); } Slot bookedSlot = createAppointment(s, d, p); notifyDoctor(bookedSlot, d, p); notifyPatient(bookedSlot, d, p); return bookedSlot; } private boolean isHospitalOpen(Hospital h, Date startTime, Date endTime) { String hospitalServiceBaseUrl = ConfigReader.readConfigName("hospitalServiceBaseUrl"); HospitalService hospitalService = new HospitalService(hospitalServiceBaseUrl); DateRange range = hospitalService.getWorkingHours(); return range.contains(startTime) && range.contains(endTime); } private boolean getFreeSlotForDoctor(Doctor d, Date startTime, Date endTime) { String scheduleServiceBaseUrl = ConfigReader.readConfigName("scheduleServiceBaseUrl"); ScheduleSerivce scheduleService = new ScheduleService(scheduleServiceBaseUrl); return scheduleService.getAppointmentSlots(d).stream().filter(slot -> !slot.isOccupied) // Only consider free slots .filter(slot -> slot.contains(startTime) && slot.contains(endTime)) // Only consider slots which contain this time period .findFirst(); } private Slot createAppointment(Slot s, Doctor d, Patient p) { String appointmentServiceBaseUrl = ConfigReader.readConfigName("appointmentServiceBaseUrl"); AppointmentService appointmentService = new AppointmentService(appointmentServiceBaseUrl); s.setIsOccupied(true); return appointmentService.bookSlot(d, p, s); } private void notifyDoctor(Slot s, Doctor d, Patient p) { DoctorEmailTemplate doctorEmailTemplate = new DoctorEmailTemplate(d, p, bookedSlot); notificationService.notify(d, doctorEmailTemplate); } private void notifyPatient(Slot s, Doctor d, Patient p) { PatientEmailTemplate patientEmailTemplate = new PatientEmailTemplate(d, p, bookedSlot); notificationService.notify(p, patientEmailTemplate); } This looks a lot more like the workflow I was expecting. The details of each of the steps are now hidden, which means that they can change without us knowing about that. But the biggest thing is that I don’t feel like I have to go into each of the new methods to see what they do if all I want is a logical understanding of how appointments are booked. There are technical things like exceptions that still intrude upon the reading experience, but for the most part, astonishment has been eliminated by reducing the model-code gap. Now let’s go one step deeper and further break down the methods here ([gist](https://gist.github.com/kislayverma/f57d9f00efe913f99d369cabb49a18d8?ref=kislayverma.com)). public Slot makeAppointment(Doctor d, Hospital h, Patient p, Date startTime, Date endTime) throws Exception { if (!isHospitalOpen(h, startTime, endTime)) { throw new Exception("Hospital isn't open in this time period"); } Slot s = getFreeSlotForDoctor(d, startTime, endTime); if (s == null) { throw new Exception("Doctor is not free in this time period"); } Slot bookedSlot = createAppointment(s, d, p); notifyDoctor(bookedSlot, d, p); notifyPatient(bookedSlot, d, p); return bookedSlot; } private boolean isHospitalOpen(Hospital h, Date startTime, Date endTime) { String hospitalServiceBaseUrl = ConfigReader.readConfigName("hospitalServiceBaseUrl"); HospitalService hospitalService = new HospitalService(hospitalServiceBaseUrl); DateRange range = hospitalService.getWorkingHours(); return range.contains(startTime) && range.contains(endTime); } private boolean getFreeSlotForDoctor(Doctor d, Date startTime, Date endTime) { return buildScheduleService().getAppointmentSlots(d).stream().filter(slot -> !slot.isOccupied) // Only consider free slots .filter(slot -> slot.contains(startTime) && slot.contains(endTime)) // Only consider slots which contain this time period .findFirst(); } private ScheduleService **buildScheduleService**() { String scheduleServiceBaseUrl = ConfigReader.readConfigName("scheduleServiceBaseUrl"); return new ScheduleService(scheduleServiceBaseUrl); } private Slot createAppointment(Slot s, Doctor d, Patient p) { String appointmentServiceBaseUrl = ConfigReader.readConfigName("appointmentServiceBaseUrl"); AppointmentService appointmentService = new AppointmentService(appointmentServiceBaseUrl); s.setIsOccupied(true); return appointmentService.bookSlot(d, p, s); } private void notifyDoctor(Slot s, Doctor d, Patient p) { DoctorEmailTemplate doctorEmailTemplate = new DoctorEmailTemplate(d, p, bookedSlot); notificationService.notify(d, doctorEmailTemplate); } private void notifyPatient(Slot s, Doctor d, Patient p) { PatientEmailTemplate patientEmailTemplate = new PatientEmailTemplate(d, p, bookedSlot); notificationService.notify(p, patientEmailTemplate); } Note the *buildScheduleService* method. If I really want to understand how ScheduleService is invoked to get a doctor’s schedule, this still looks all right, although just about at this stage someone will start arguing that service creation should not be done in this class or that it can be more generically for all service. That’s fine, it can still stand alone as a method, if not in this class then elsewhere. But the question indicates the curiousity/astonishment quotient has started rising. Let’s take it one step further ([gist](https://gist.github.com/kislayverma/2a6c66a0602576545d74cb0a8e392ae4?ref=kislayverma.com)). public Slot makeAppointment(Doctor d, Hospital h, Patient p, Date startTime, Date endTime) throws Exception { if (!isHospitalOpen(h, startTime, endTime)) { throwException("Hospital isn't open in this time period"); } Slot s = getFreeSlotForDoctor(d, startTime, endTime); if (s == null) { throwException("Doctor is not free in this time period"); } Slot bookedSlot = createAppointment(s, d, p); notifyDoctor(bookedSlot, d, p); notifyPatient(bookedSlot, d, p); return bookedSlot; } private void **throwException**(String message) throws Exception{ throw new Exception(message); } private boolean isHospitalOpen(Hospital h, Date startTime, Date endTime) { String hospitalServiceBaseUrl = ConfigReader.readConfigName("hospitalServiceBaseUrl"); HospitalService hospitalService = new HospitalService(hospitalServiceBaseUrl); DateRange range = hospitalService.getWorkingHours(); return range.contains(startTime) && range.contains(endTime); } private boolean getFreeSlotForDoctor(Doctor d, Date startTime, Date endTime) { return buildScheduleService.getAppointmentSlots(d).stream().filter(slot -> !slot.isOccupied) // Only consider free slots .filter(slot -> slot.contains(startTime) && slot.contains(endTime)) // Only consider slots which contain this time period .findFirst(); } private ScheduleService buildScheduleService() { String scheduleServiceBaseUrl = ConfigReader.readConfigName("scheduleServiceBaseUrl"); return new ScheduleService(scheduleServiceBaseUrl); } private Slot createAppointment(Slot s, Doctor d, Patient p) { String appointmentServiceBaseUrl = ConfigReader.readConfigName("appointmentServiceBaseUrl"); AppointmentService appointmentService = new AppointmentService(appointmentServiceBaseUrl); s.setIsOccupied(true); return appointmentService.bookSlot(d, p, s); } private void notifyDoctor(Slot s, Doctor d, Patient p) { DoctorEmailTemplate doctorEmailTemplate = new DoctorEmailTemplate(d, p, bookedSlot); notificationService.notify(d, doctorEmailTemplate); } private void notifyPatient(Slot s, Doctor d, Patient p) { PatientEmailTemplate patientEmailTemplate = new PatientEmailTemplate(d, p, bookedSlot); notificationService.notify(p, patientEmailTemplate); } Any developer with any amount of experience in any codebase will get curious as to why we need a separate method just to throw an exception. They will try to go to the lower level to understand it, which is exactly what we set out to prevent. At this level of granularity, our method size is causing astonishment so we should abort this step. The point at which engagement or curiousity begins to rise depends on the business and technical context of a codebase. So while this is obviously a simple example, preventing cognitive load is widely applicable as a guiding principle in software engineering. When reading a piece of code, keep in mind your engagement and curiousity levels. If either increase, there may be the possibility of refactoring. From the internet this week Ingrid writes about [why decentralised applications don't work](https://ingrids.space/posts/why-distributed-systems-dont-work/?ref=kislayverma.com) in the real world. With crypto making news for all kinds of reasons, this is a thought-provoking take. Here's an old video of [Brian Kerninghan interviewing Ken Thompson](https://www.youtube.com/watch?v=EY6q5dv%5FB-o&t=1s&ref=kislayverma.com). The interaction between these two greats is worth listening to. [Jamie Brandon](https://twitter.com/sc13ts?ref=kislayverma.com) discusses [consistency in streaming systems](https://scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/?ref=kislayverma.com). Our words shape our perspectives. [Sarah Drasner](https://twitter.com/sarah%5Fedo?ref=kislayverma.com) reminds us that engineering managers should think of their team as ["us", not "them"](https://css-tricks.com/your-team-is-not-them/?ref=kislayverma.com). That's it for this week folks! Cheers! Kislay Modify your subscription | View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ### It Depends #34: Ditch the Urgency URL: https://kislayverma.com/it-depends-34-ditch-the-urgency/ Last updated: 1970-01-01T00:25:17.000Z | It Depends #34: Ditch the Urgency Hello everyone! Welcome to the 34th edition of It Depends. Hope you are all doing well and staying safe. Today is a special episode of It Depends. As of last week, more than 1500 readers are subscribed to this newsletter. I want to thanks every one of you for your notes of encouragement and appreciation, and for all the times you let me know that there was an “Error establishing database connection” :). Writing this newsletter has been a transformative experience for me, and I hope you enjoy reading it as much as I enjoy writing it. So. Onwards and upwards. Today, let’s talk about a trait often seen in tech-product teams - a constant sense of urgency to ship. A sense of urgency in shipping features is probably the worst result of the [agile](https://kislayverma.com/category/agile/) mindset. While it makes some sense in absolute early-stage startups where everything has to be built ground up, but in places that have a little bit of stability, this is a vestigial mindset which causes a lot of problems. ![](https://kislayverma.com/content/images/2021/05/Learn-Solve-Deliver-300x269.jpg) Delivering the right kind of product is a three-step process. *Learn*: This is where we try to identify the customer’s problem(s). *Solve*: We identify the best ways of solving the problem. *Deliver*: This is where the solution is built and delivered. While tools help, Learning and Solving can only be fast-tracked to a certain extent. These are the most valuable actions that a team can perform, so they should be given due importance. Unfortunately, the prevalent reading of move fast permeates both thinking and execution with the same level of urgency. While software engineering is an art and science by itself, seen from this level, it is operations, and operations work best when they are optimized to death. Best practices old and new create an environment where the execution pipeline can be made faster and faster. Each developer can exert a lot more technical leverage to produce a greater output. [Organizations](https://kislayverma.com/category/organizations/) have taken this to mean that more things can be shipped out with the same amount of resources. It actually can’t, as a simple application of the [Theory of Constraints](https://www.leanproduction.com/theory-of-constraints.html?ref=kislayverma.com) can show. Envisioning the process defined above, the true bottlenecks in the flow of value are learning and solving. So we have two choices. One is to widen the bottleneck by short-circuiting true ideation and pushing out all kinds of ideas down the line to be executed. The other is to not widen it at all and send only the most impactful things (as best as we can tell) out. This might mean that some execution capacity lies unused at some times, but the impact of delivery is not diminished. Guess which one gets chosen way more often than the other? I don’t think of moving fast in the sense of delivering many changes very quickly. This is a very operations-centric view of things. Looking at the entire team and the product holistically, iterating rapidly should be about buying the team as much time as possible for identifying and solving problems. It should focus on making the actual execution boring and efficient so that the time between identifying the solution and delivering the solution becomes minimal. ![](https://kislayverma.com/content/images/2021/05/Learn-Solve-easily-deliver-300x249.jpg) I think this is a better way of looking at the excellence of a team and engineering velocity because it put the most important steps of the value addition process in the spotlight. The tech-product team now has two very clear mandates: Identifying and solving the biggest problem(s). Eliminate everything causing friction in getting the envisioned solution in the hands of the customer. In management terms, the first is strategic thinking, and the latter is operational excellence. At the team level, strategic thinking should come first. We should spend a lot of time figuring out where we stand and what we want to do. This phase should be deliberate, intense, and the step where the team comes together behind a shared vision. But the ability to do this hinges on making the execution process “just work”. Great teams spend time and effort populating this phase with tools and processes that remove unpredictability and turn it into a well-oiled machine. Engineering bandwidth that is not fully occupied at all times is not a bad thing. A good team will use this time to make sure their execution phase stays smooth and boring. Paying down tech debt, adopting modern operating practices and tools, adding documentation, etc keep execution friction from rising and let small teams deliver big results. But this meta-work usually gets tagged as wasteful since it is not perceived as being beneficial to the business. In its place, teams put in unimpactful busywork in the name of agility. There is a balance between thinking too much and not at all. At the moment we seem to be leaning far, far towards the latter. Organizations should stop trying to push ideas down the pipe just for the sake of cranking the wheel. OTOH, engineering teams need to take learning from operations methodologies to identify what is slowing them down when it comes to delivering code and ruthlessly eliminate these bottlenecks. ![](https://kislayverma.com/content/images/2021/05/Mark_Zuckerberg_-_Move_Fast_and_Break_Things-300x225.jpeg) Ditch the urgency doesn’t sound like much of a mantra to motivate your employees, which is probably why the Zuckergerian adage has caught on much more. But ditch the urgency to move for the sake of moving. Don’t short-circuit the thinking process. Give it the luxury of time by making it super smooth to put good ideas into action. In smart teams, thinking should emerge when doing is taken out of the way. Think about learning to play a musical instrument. The ideal goal is not to learn songs and melodies very quickly and ad-infinitum. It is to develop the skill of playing the instrument to an “unconscious competence” so that the main focus can be set on musicality. This is where the “effortless” playing of the true masters comes from - they don’t even think about the physical act of playing the instrument. Delivering software is not so different. From the great interweb If you are an AWS user but have not yet come across this [“well-architected” document](https://docs.aws.amazon.com/wellarchitected/latest/framework/wellarchitected-framework.pdf?ref=kislayverma.com) on building systems on AWS, I strongly recommend you read it now. This is a fantastic set of guidelines from the AWS team itself on how to build great systems using their toolkit. This paper outlining the [history of socio-technical system design](https://research.tue.nl/en/publications/an-anthology-of-the-socio-technical-systems-design-stsd-paradigm-?ref=kislayverma.com) (STSD) by F.M. van Eijnatten explains the evolution of thought processes in the field from the 1950s onwards. It is a great read if you are interested in the intersection of software architecture and organization design. [Is event sourcing an anti-pattern](https://dev.to/olibutzki/why-event-sourcing-is-a-microservice-anti-pattern-3mcj?ref=kislayverma.com)? Oliver Libutzki shares his thoughts. Tom Sommer discusses [the art of self-organizing teams](https://leaddev.com/culture-engagement-motivation/art-self-organizing-engineering-teams?ref=kislayverma.com). This is a great article on what autonomous teams are, and how to identify/build one. That's it for this week folks. Have a great weekend! \-Kislay Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### It Depends #33: Flink, Start with why, Pace layer, Snowflake URL: https://kislayverma.com/it-depends-33-flink-start-with-why-pace-layer-snowflake/ Last updated: 1970-01-01T00:24:45.000Z | It Depends #33: Flink, Start with why, Pace layer, Snowflake Hello everyone! Welcome to the 33rd edition of It Depends. Hope you are all doing well and staying safe. If you haven’t got the vaccine yet, please do everything you can to get it. Best of luck to all of us! No original writing on the blog this week, but here’s the handpicked best of what I found on the internet this week. From the great interweb The Ververica team has a brief intro to [how Apache Flink handles backpressure](https://www.ververica.com/blog/how-flink-handles-backpressure?ref=kislayverma.com). This makes for a nice follow-up to what I wrote about [visualizing distributed systems as computational pipelines](https://kislayverma.com/software-architecture/distributed-systems-as-data-pipelines-throughput-capacity-and-backpressure/). Hat tip to Chris Patullo for sharing this [short video](https://www.youtube.com/watch?v=HjriwYrGL28&ref=kislayverma.com) of Simon Sinek explaining why “starting with why” is a key element of success. There’s a whole rabbit hole behind those \~2 minutes, so happy exploring :) Architectures of buildings and software keep running into each other far too often for it to be a coincidence. Here’s Stewart Brand and Paul Saffo explaining how their “[pace layer model](https://longnow.org/seminars/02015/jan/27/pace-layers-thinking/?ref=kislayverma.com)” for understanding the evolution of buildings has found widespread application in software engineering, complexity theory, and many other areas. For the more hardcore weekend reader, here’s a [paper](http://info.snowflake.net/rs/252-RFO-227/images/Snowflake%5FSIGMOD.pdf?ref=kislayverma.com) explaining the internal design of [Snowflake DB](https://www.snowflake.com/?ref=kislayverma.com). That's it for this week folks. Cheers! \-Kislay Modify your subscription \| View online | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #32: Why don't programmers write documentation? URL: https://kislayverma.com/it-depends-32-why-don-t-programmers-write-documentation/ Last updated: 1970-01-01T00:24:00.000Z | It Depends #32: Why don't programmers write documentation? Hello Everyone! Welcome to the 32nd edition of *It Depends*. Hope you are all doing well and staying safe. This week we are talking about that great bugbear of software engineering - documentation. You can [read the article on the blog](https://kislayverma.com/programming/why-programmers-dont-write-documentation/) directly if you prefer that layout. I have been writing about [documenting code](https://kislayverma.com/programming/how-to-write-self-documenting-code/) of late, so of course, my Medium recommendations threw out an article about “[the real reason why developers don’t write documentation](https://betterprogramming.pub/this-is-why-most-software-engineers-dont-write-documentation-670ceecb6a21?ref=kislayverma.com)”. The article claims that the lack of good tools for writing is the biggest culprit in discouraging software engineers from documenting their work and decisions. I usually don’t pick on specific articles, but this one triggered the hell out of me. The writer makes some okay points about diagramming tools, but the overall piece is so misleading that it obfuscates this important issue. If you are going to compare two drawing tools and claim that neither being good enough is the main reason for developers not writing docs, then either you are writing for clickbait or in bad faith. I believe that there are two main reasons software engineers don’t write documentation. Tools play their part but they are a hugely distant third. Writing is hard Software engineers, like everyone else, don’t write because writing clearly is very, VERY difficult. Writing is a tough, demanding task. It requires organizing our thoughts clearly, examining them critically, and expressing them clearly. While the expressing part can be simplified to some extent (depending on the quality of writing required), all three steps are taxing when done properly. In the world of programming, where “it depends” is often the best answer and everything is based on trade-offs, writing becomes that much harder. It needs to set the context, justify the decisions, and then power the low-level thinking leading into the code. This type of writing is only useful if done well, and since doing it well is tough, it often doesn’t get done at all. Bad code will still fly, bad documentation won’t. This is why a lot of people argue about the value of comments in code and the merits of self-documenting code (whatever that means). Kevlin Henney says that asking for comments around complicated code is futile because we expect the same people who could not express themselves clearly in code to now turn around and express themselves clearly in English. [![](https://kislayverma.com/content/images/2021/05/Screenshot-2021-05-01-at-11.55.59-AM-300x89.png)](https://twitter.com/KevlinHenney/status/381021802941906944?ref=kislayverma.com) Not documenting doesn’t block shipping If a developer doesn’t write documentation, their work still gets done. Not writing doesn’t block shipping (at least not right away). The damage done by not documenting technical decisions is cumulative. Much like tech debt, it doesn’t cause damage in the here and now. Like I said above, writing is primarily a matter of thinking and analyzing. In most places, coding can be done by the seat of your pants. A disorganized pile of classes and methods in code may work – a pile of work of words and paragraphs won’t work. Writing HAS to be clear if it is to be of any use. Code will be accepted (to some extent) as long as it does its job. And since most organizations focus only on getting the product shipped, that which doesn’t block shipping gets ignored. Unit tests face a similar problem in many teams. To test the code we need to understand it (that takes more effort than writing it), and not having tests doesn’t block shipping. Ergo, no unit tests in code. There is also the matter of obsolescence. Even good documents go obsolete, so engineers have to keep repeating the think-analyze-express over and over again as they build out systems. So dropping off the documentation wagon is easy. So even with best intentions, documentation often happens only in spurts of writing and cleanup. What about the tools There is no doubt that the commonly used set of tools used for documenting software today are woefully inadequate. We don’t think in terms of documents one at a time. We think in terms of ideas and goals by pulling together multiple concepts at once. The resultant document is just one manifestation of the thought process. We need tools that can help us collate ideas across time to solve the problem at hand. Google Docs, Confluence, Markdown are all poor tools for this type of writing. However, a new generation of tools like [Notion](https://www.notion.so/?ref=kislayverma.com) and [Roam](https://roamresearch.com/?ref=kislayverma.com) are attacking this problem of harnessing networked thought. Hopefully, these will work as intended and help in the thinking that goes into writing. However, the lack of a second brain cannot really be used as an excuse for not using the first one. Tools play their part, but the willingness to undertake the process is the real hurdle. So how to do documentation Writing software has taught me one thing. If you really want your users to do something, then doing it has to be a blocking step in their journey with your product. In the same way, tacking on documentation to written code is never going to work. Worse, it is useless. Writing is about critical thinking. It is meant to explain your thought process and intent to yourself and to your audience (e.g. your team). The thinking process is where documentation/writing adds value, not as a static record of already implemented code. Proponents of mob/pair programming and XP often disparage documentation. But barring the adoption of those techniques, the practice of writing and reviewing technical documents is the only way teams build a collective understanding of what they are trying to build. This shared world-building is what makes this process critical to the long-term health of the team and the codebase. I feel that the only way to make the process of writing documentation sustainable is to make it a blocker for software development. Make it lightweight but mandatory. It should become part of the process instead of being yet another thing to do. Some things that have worked for this in my experience. **Write before you code** \- Unless the change is trivial, every engineer writes a note about what they are going to do and runs it by the rest of the team. At the end of the discussion, the actual coding should become trivial. **Write simply -** Don’t complicate the writing, at least until it becomes second nature. Diagrams, fancy sections etc can wait. Write very simply about what you thought, what you are doing, and why. Even if the document can serve as a basic pointer to the rest of the team now and in the future, it is superbly valuable. **Document the decision with their alternatives** – Rather than documenting the actual implementation (which may change over time) in detail, focus on documenting the choices and why they were made. This is what the code cannot ever explain and hence writing it down adds the most valuable. Details can be documented based on the time you are willing to invest. **Make it searchable** – No amount of documentation will be of any use if people cannot find it. Use tools that support text searching out of the box. This is one of the reasons I like Google Docs for documentation. It is great for writing but just horrible for collaboration and discovery. **Track changes** \- Some organizations use version control to track changes to the system’s design over time. That’s great. But if you are not there yet, keep one document per feature and keep putting dated updates on it so that evolution can be tracked in one place with minimal hassle. The hope is that as the team seems the merits of having and reviewing some documents (e.g. new members need lesser hand-holding) and writing becomes muscle memory, the practice will become self-sustaining. Till then, it should be treated like working out or dieting – painful but necessary. From the internet this week If you have not yet tuned into the Linux-UMN drama, [this article](https://www.zdnet.com/article/university-of-minnesota-response-to-linux-security-patch-requests/?ref=kislayverma.com) will get you started. Apparently, a research team from the University of Minnesota started introducing random changes into the Linux kernel, go caught, got banned, and had all their changes very publicly reverted. Popcorn required! [Justin Jaffray](https://twitter.com/justinjaffray?ref=kislayverma.com) explains [push and pull-based query engines](http://justinjaffray.com/query-engines-push-vs.-pull/?ref=kislayverma.com) and the context in which they can be used. The folks over at Software mill have done a great visualization that explains [how Apache Kafka works](https://softwaremill.com/kafka-visualisation/?ref=kislayverma.com). Check it out. That's it for this week folks! Cheers! Kislay Modify your subscription \| View online | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #31: The second law of building software platforms URL: https://kislayverma.com/it-depends-31-the-second-law-of-building-software-platforms/ Last updated: 1970-01-01T00:23:34.000Z | It Depends #31: The second law of building software platforms Hello Everyone! Welcome to the 31st edition of *It Depends*. Hope you are all doing well and staying safe. Today let’s talk about the second most important rule for building platform systems (Read [this](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/) to learn what the most important rule is). You can [read the article on the blog](https://kislayverma.com/platform-thinking/external-programmability-the-second-law-of-building-platforms/) directly if you prefer that layout. TL;DR We should not have to modify central systems/platforms to achieve variant behaviours for different use cases. We should be able to plug in these behaviours from the outside to customize specific parts of the overall system behaviour. This will make our system more durable by offering a powerful mix of capability and customizability. The Problem ![](https://kislayverma.com/content/images/2021/04/external-prog-original-design-300x180.jpg) Imagine you are building a central system that is intended to be used by multiple other teams. Depending on the kind of complexity offered by the system, one or more clients may ask for variations of the original behaviour specific to their use-cases. We can readily imagine such situations arising in B2B software where every client needs some custom variant of the original feature. How do we accommodate these situations? ![](https://kislayverma.com/content/images/2021/04/external-prog-build-centrally-300x265.jpg) The most obvious way, of course, is to build it! The team that built the system also builds the customizations in the features as required by any client. This makes sense if these requests are rare (so the team can easily allocate time for it) or complex (this is the ONLY team that can do it). If this is not the case, however, the original team becomes a bottleneck for multiple teams because it cannot spare the time to take care of all the incoming customization requests. The second way is to ask the client teams to get into the code base and make the changes themselves. This removes the bandwidth bottleneck. Client developers can usually make the changes given sufficient enough tools/documentation and guidance (code review etc). But over time, this almost always leads to deterioration in code quality and blurry lines of ownership. It is difficult to hold any single team accountable for the quality of the system since everyone is making changes. Depending on the nature/complexity of the change, the oversight and communication required may well be a lot. Also, this model is practically impossible if the client team is external to the organization and hence cannot be given access to the codebase. System Boundaries are Team Boundaries Conway’s Law, Team Topologies, and various other schools of thought have made it abundantly clear that an organization’s software architecture mirrors its communication architecture. So the problem of building customizations can be generalized to a problem of defining how client teams interact and influence the team that owns a system, thereby influencing the design of the system. If multiple teams want to use and grow the same system, we need to define a model for coordination between them. To my mind, this model must minimally achieve two objectives: We should be able to evolve the software independently without getting bogged down in communication overhead. The first approach discussed above is ruled out on this ground because it puts the owning team on all change paths. Clients have to beg/bully/convince them into making the changes for them. We should be able to do this without degrading the quality of the codebase. The second approach discussed above is ruled out by this. Maintaining code quality and operational excellence is almost impossible if anyone can (and is expected to) make changes to your code. So we need a way to define a system boundary and change process such that others can make changes independently without impacting our code quality. We can do this if we can allow people to “hook in” to the internal decision points of our system and modify the behaviour for their use cases. This is what Steve Yegge calls External Programmability in his [legendary platforms rant](https://gist.github.com/kislayverma/d48b84db1ac5d737715e8319bd4dd368?ref=kislayverma.com) (you can read my redux [here](https://kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/)) and next to *[Eat your own Dogfood](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/)*, it is the second cardinal principle of building platforms. External Programmability The idea of External Programmability is to identify the parts of an application that we think should be customizable, turn them into hooks for variable functionality, and then expose these hooks externally. Clients can then plugin to these hooks and trigger custom behaviour or make decisions based on custom logic without having to go into the codebase of the system. As a result, the behaviour of the system is not completely determined by the logic implemented by the owning team, but by the collective impact of the core logic and customization hooks. ![](https://kislayverma.com/content/images/2021/04/external-prog-external-prog-289x300.jpg) This style is, in a way, OCP at a multi-system level, and has distinct advantages over the modify-from-within approach. Clients know exactly how to hook in custom behaviour because the design of the system makes it explicit. No risk of them going inside the system and breaking something by mistake. It also makes change faster for the client, because they do not have to learn how to work inside a new codebase. They integrate from the outside along well-defined interfaces, and the customizations themselves are implemented in a technical environment of their choosing. It’s like being able to tell another microservice about which of your APIs to call at which step without having to modify its code. External Programmability transforms an internal decision of a system into an open interface that users of the system can modify as per their needs. From a system design perspective, this means that the system interface is a lot less “closed” than you would normally expect. The internal parts which got turned into externally customizable hooks transform perhaps a suite of APIs into a collaborative interplay of decisions and actions. We are deliberately exposing a lot of system internals for customization so that we don’t have to expose the entire system to invasive change. If we look at the traditional layered architecture style, control always flows from higher layers to lower layers. However, in the platform architecture which external programmability creates, control flow back and forth between upstream and downstream systems (client systems being considered upstream and the platform system downstream). The emergent collaborative system architecture is better visualized as a 3-dimensional mesh of systems rather than a two-dimensional stack. There are still upstream and downstream pieces but the boundaries between them are a lot more fluid. How do we get there? One way of implementing this is to externalize all the business logic (even the original business logic) into workflows outside the core application. The core thus becomes very, very lightweight and all the logic moves out into the orchestration layer. In this way, clients have complete control of what they want to do. They do whatever they want and then call the simplistic APIs of the core system as they see fit. This gives the ultimate freedom and inversion of control – instead of modifying what exists, clients can compose whatever they want. The problem here is that the core that remains usually gets stripped of all business semantics and hardly remains a product at all! Clients have to build not just customizations of some existing behaviour but the entire functionality over and over again. The domain boundary completely breaks down. There is no way to know where the logic for processing a certain kind of order is implemented because that logic lives completely outside the core and we have no way of systemically finding out what is happening where. The other way is to implement a callback-based system. The original system identifies the parts which parts of the control flow it deems to be customizable (the other parts become core by definition since they cannot be modified by clients) and exposes them over APIs. The APIs allow clients to define the rules under which their specific customization should be triggered and exactly how they should be triggered (execute an API call back to the client system). ![](https://kislayverma.com/content/images/2021/04/external-prog-register-custom-300x122.jpg) Once these customizations are “registered” with the main system, whenever client A invokes the feature X, it executes all non-overridden points as per default behaviour but executes the registered override to achieve an end-to-end result customized for client A by client A. ![](https://kislayverma.com/content/images/2021/04/external-prog-execute-custom-300x122.jpg) I have written a [detailed explanation](https://kislayverma.com/platform-thinking/platform-nuts-bolts-flexible-decision-making-with-rule-engines/) of how we can use a combination of rule systems and workflow management systems to stitch the whole experience together. In this approach, all interactions for a certain problem come to the same central system, and we can identify from that place what we want to do. Either client uses the default behaviour of the system, or they will have registered specialized hooks to custom callbacks. In either case, it becomes easy to track down the flow of control because all branching out happens from well-known points of divergence. As a result, a porous technical domain boundary remains with much of the business logic running inside the boundary, but the occasional customization going back up the stack to client systems. Our core system is still the one place where all business logic can be traced from. Note that in this approach, [we need not distinguish between internal and external teams](https://kislayverma.com/platform-thinking/control-and-chaos-in-platform-systems/). All client teams communicate across a porous system boundary which defines a clear interface and protocol but otherwise, both teams operate independently. The team which owns the system and the teams that use the system are in effect co-building a much larger system by allowing each other to reach deep into each other’s systems to create business value. From the internet this week [Mikio Braun](https://twitter.com/mikiobraun?ref=kislayverma.com)’s take on why [we still don’t know how to create software at scale](https://margint.blog/2021/04/05/creating-software-at-scale/?ref=kislayverma.com) is definitely worth a read. This is a fascinating technical overview of [how Postman handles millions of concurrent connections](https://medium.com/better-practices/how-postman-engineering-handles-a-million-concurrent-connections-15c8807f6393?ref=kislayverma.com). As the Google AI Ethics fiasco continues to shake the industry and the company, here’s a scholarly take on [corporate research and academic integrity](https://arxiv.org/abs/2009.13676?ref=kislayverma.com). That's it for this week folks! Cheers! Kislay Modify your subscription \| View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #30: Eliminating the QA bottleneck URL: https://kislayverma.com/it-depends-30-eliminating-the-qa-bottleneck/ Last updated: 1970-01-01T00:23:26.000Z | It Depends #30: Eliminating the QA bottleneck Hello everyone! Welcome to the 30th edition of *It Depends*. Hope you are all doing well and staying safe - this is the most important thing that we can do right now. Today I talk about testing strategies for agile teams, and then share the best stuff I read this week. I tried to record this article as a podcast/audio thing but DAMN that stuff takes time! Perhaps I’ll do that for the next article, but it’s text as usual for now, and you can [read the article on the blog directly](https://kislayverma.com/agile/testing-strategies-for-agile-teams/), if you like that layout better. End-to-end testing refers to the approach of testing every step of every single user flow. e.g. In the e-commerce domain, this might mean testing every API call and database write/read from the moment an order is placed to the moment it is delivered. The end-to-end testing process will validate every notification sent to the customer, every tool used by the ground force, and every third-party interaction involved. This testing paradigm validates the end-user experience by validating behaviour across multiple domain boundaries and teams. ![](https://kislayverma.com/content/images/2021/04/end-to-end-testing-300x110.jpg) Back when all systems were monolithic, development batch-y, and releases infrequent, this was the de-facto approach for testing software before it was released. It made perfect sense because if the last release was some time ago and many features were being developed, it was quite possible that many parts of the codebase had changed simultaneously. Sadly, end-to-end testing has persisted in the modern microservice architectures as well where it adds very little value and creates a massive roadblock in attaining a [high speed of development and deployment of software](https://kislayverma.com/agile/how-to-speed-up-software-delivery/). Especially if an organization follows agile principles and ships code quickly using CI/CD etc, this approach of testing can bring the entire delivery pipeline to a standstill. Let's see how we can do better in the microservice world. There are two main technical advantages that microservice architecture gives us. The first is independent services which can be developed, deployed, and scaled independently. The second is that relationships between these services are now explicit and the dependencies trackable. This first means that at an organizational level, there is no longer a single artifact to release at a certain fixed time/cadence. All services change at their own pace. The second means that only a small subset of the overall system is impacted when any of the services deploy a change. It is easier to identify what impacts what in microservice architecture (if it is well designed). The end-to-end testing mindset focuses on the idea that since there are now many things, any of which can cause a problem, we must test all systems/features before we deploy anything. It therefore completely surrenders the second advantage of microservice architecture - identifiable dependencies. And by ceding this advantage, it becomes a blocker to agility rather than an enabler. It can be argued that this is not a microservice-monolith question but rather a slow/fast shipping question. In a slow shipping monolith/microservice, it is difficult to establish what the immediate neighbours are because many changes are being deployed together. If each deployment consists of a single change, then it is easy for devs and tester to establish what should be tested. I feel that microservices simplify this further by making the separation of concerns even more explicit or externalized. We don't need to test everything all the time. Looking at the service which changed, we can isolate all services that interact with it, and test only the interactions with them. If system boundaries are drawn well, only the immediate neighbours should be impacted by any change. What's the remedy? So how should agile organizations test a microservice-based architecture? I prefer a two-pronged approach, both of which assume automated integration testing ability. If you are still doing full manual, you need to get in the automation game ASAP. First, we need to speed up the testing of each service or component so that our independent teams can move fast. We need the ability to identify all its immediate neighbours and verify that all those interactions are working properly after the change. If service A is changing, then immediate neighbour means every service that service A calls and every service that calls service A (or consumes events from service A). So both upstream and downstream systems constitute a service's neighborhood. Identification of a service's neighborhood can be automated (eg. using a service mesh/request tracing to extract who calls who) or manual. Once identified, we can use techniques like CDC (Consumer-Driven Contract) Testing to verify that every interaction of the service under test is working fine. Many organizations also tag/group integration tests to be able to run subsets of integration tests for further optimization (e.g. if only API 1 is changing, only run tests related to that in the immediate neighborhood). The more widely used a service is, the larger its neighborhood will be, and hence the greater the amount of testing that needs to be done. While this means that deployments of this service won't be as fast, it also makes sense that the most used services should move slowly to prevent big outages. As long as the whole process after merging to main is automated, who cares anyway :) But the optimization in testing each service that we have done (we aren't testing everything all the time anymore) means that some unknown unknowns can cause bugs to slip through. So the second prong of our testing strategy is to continuously run integration tests on all critical customer-facing features in production. This can be automated tests against our public API, selenium style tests against the UI, or anything else which can flag any behaviour unexpected by the customer. If an anomaly is detected here, we raise a bug which the relevant dev team then takes over to investigate and fix. ![](https://kislayverma.com/content/images/2021/04/agile-testing-strategies-300x134.jpg) The latter set of tests should be put in place first - in fact, these should be considered part of a feature release. This is yet another argument for developers writing integration tests. A post facto QA team/process can't hope to keep up with a fast-moving developer team. The organization should focus on enabling dev-driven testing by giving them time to do it and setting up the tools to make writing tests easy. Dev teams can then focus on building and testing their features. Move as fast as you can, and break as little as you can help it! From the internet this week Martin Fowler weighs in on the ["outcomes over output" mindset](https://martinfowler.com/bliki/OutcomeOverOutput.html?ref=kislayverma.com). This phrase is popping up everywhere, but it remains to be seen if this will bring about the benefits its proponents think that it will. [Dude! Where's my flying car](https://rootsofprogress.org/where-is-my-flying-car?ref=kislayverma.com)? J. Storrs Hall explains why the aerial utopia hasn't come about yet, and what is standing in the way. The Wix Engineering team explains [how they reduced their service latencies](https://medium.com/wix-engineering/how-we-managed-to-reduce-our-latency-by-3-times-84d843dc84f2?ref=kislayverma.com). That's it for this week folks! Cheers! Kislay Modify your subscription \| View online | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### It Depends #29: On self documenting code URL: https://kislayverma.com/it-depends-29-on-self-documenting-code/ Last updated: 1970-01-01T00:23:16.000Z | It Depends #29: On self-documenting code Hello everyone! Apologies to my regular readers for the 2 weeks break. I had travelled to my parents’ place in Delhi and there got involved in a false-alarm COVID episode. All is well, and I welcome you to the 29th edition of It Depends. Today I will talk about self-documenting code (you can read the article on the website if you’d like), followed by the usual awesome from the internet. I love documenting code and systems. Many don't. A major argument against documents is that they get outdated as the system evolves. And the faster a system evolves, the faster its documentation gets outdated. Ironically, this is the very type of system which needs the most up-to-date documentation! An argument is often made, therefore, for self-documenting code. This is ostensibly the kind of code that doesn't need separate documentation because it is designed and implemented in such a way as to be self-explanatory to the reader. How does anyone reading a codebase understand it? First, they need to know what the code is "supposed to do". Then they can graduate to figuring out how it does that. And this is where the problem of self-documenting code lies. Because reading code is essentially reading the how. "Query some things from a database table and process them into a map, match them against some other things from some other table, and return everything that does not match as a list". [Well-written code makes it simple to understand how it is doing something](https://kislayverma.com/organizations/reduce-collaboration-by-good-design/). But it doesn't tell the reader why it is doing what it is doing. And hence the reader remains confused and documentation becomes necessary to understand the intent behind the system design. So what kind of code would reveal, even in some limited way, why it does things the way it does them? I want to talk about the only way (IMO) code can explain itself to readers and hence become self-documenting. Let's discuss the model underlying the code. Becoming self-documenting Before code comes the conceptual model that the code is the physical manifestation of. This is the mental model of the problem and the solution. This may be the domain model or another specific way of representing the programmer's thought process and how it will achieve a programmatic solution to the problem at hand. ![](https://kislayverma.com/content/images/2021/04/model-and-code-300x196.jpg) The model is the core of low-level system design. It defines the things that we are working with, what their nature is, and what role they play in solving the problem at hand. The model must be developed first before any other aspects of the low-level design like APIs, data stores, data flows can be determined. These are physical forms of the things conceptualized in the model – ways to make the model “run”. The model itself is the why, and the code is the how. **The only way to make the code self-documenting is to make the code reveal the model underlying it.** This is why the only way to make the code self-documenting is to make the code reveal the model underlying it. Code that highlights the core model instead of hiding it in implementation details builds [a narrative that is much more accessible](https://kislayverma.com/software-architecture/a-narrative-approach-to-software-design/) than trying to infer the meaning of some lines in code. The lines can always be interpreted, but the model made evident in well-written code sets the context under which those lines of code make sense. Let’s take an oversimplified version of Airbnb booking. In the object-oriented, REST-ful world, the code to update a booking might look something like this. (See [this gist](https://gist.github.com/kislayverma/a0575599d5f164091b9aa8b4f2ad5959?ref=kislayverma.com) if you have disabled JS). public class Booking { String uniqueId; User guest; User host; Date bookingTime; Date confirmationTime; Date cancellationTime; Status status; //PENDING, CONFIRMED, CANCELLED\_BY\_GUEST, CANCELLED\_BY\_HOST User lastUpdatedBy; } public class BookingUpdateRequest { Date updateTime; Booking updatedBooking; } // In Booking Service public void updateBooking(BookingUpdateRequest request) { Booking originalBooking = readFromDB(request.updatedBooking.uniqueId); if ((originalBooking.status == PENDING \|| originalBooking.status == CONFIRMED) && (request.updatedBooking.status == CANCELLED\_BY\_GUEST)) { originalBooking.lastUpdatedBy = originalBooking.guest; originalBooking.cancellationTime = request.updateTime // Trigger notification to host // Trigger refund if payment was taken } else if ((originalBooking.status == PENDING | | originalBooking.status == CONFIRMED) && (request.updatedBooking.status == CANCELLED\_BY\_HOST)) { originalBooking.lastUpdatedBy = originalBooking.host; // Trigger notification to guest // Trigger refund if payment was taken } // else if ........ more conditions to handle other combinations of new/old variables } This type of generic update code is fairly common. To me, this doesn’t explain why things are happening the way they are happening. Did we miss any cases? While this code can be refactored to a much cleaner form, but it does not reveal why things are happening in this way. Let’s consider an alternative (See [this gist](https://gist.github.com/kislayverma/dfdd81fd63d6772cac33d462c9fe3fed?ref=kislayverma.com) if you have disabled JS). public class Booking { String uniqueId; User guest; User host; Date bookingTime; Date confirmationTime; Date cancellationTime; Status status; //PENDING, CONFIRMED, CANCELLED\_BY\_GUEST, CANCELLED\_BY\_HOST User lastUpdatedBy; } // In Booking Service public void cancelBookingByGuest(String bookingId, Date cancellationTime) { Booking originalBooking = readFromDB(bookingId); originalBooking.lastUpdatedBy = originalBooking.guest; originalBooking.cancellationTime = cancelltaionTime; originalBooking.status = CANCELLED\_BY\_GUEST; // Trigger notification to host // Trigger refund if payment was taken updateInDB(originalBooking); } public void cancelBookingByHost(String bookingId, Date cancellationTime) { Booking originalBooking = readFromDB(bookingId); originalBooking.lastUpdatedBy = originalBooking.host; originalBooking.cancellationTime = cancellationTime; originalBooking.status = CANCELLED\_BY\_HOST; // Trigger notification to guest // Trigger refund if payment was taken updateInDB(originalBooking); } // Other APIs to handle combinations of new/old variables... Or perhaps this (See [this gist](https://gist.github.com/kislayverma/5497931c93250768518d2d62a47d5791?ref=kislayverma.com) if you have disabled JS). public class Booking { String uniqueId; User guest; User host; Date bookingTime; Date confirmationTime; Date cancellationTime; Status status; //PENDING, CONFIRMED, CANCELLED\_BY\_GUEST, CANCELLED\_BY\_HOST User lastUpdatedBy; } public class BookingUpdateRequest { AllowedActionOnBooking action; Date updateTime; Booking updatedBooking; } // Explicitly define the ways of modiying the Booking entity public enum AllowedActionOnBooking { GUEST\_CANCELLATION, HOST\_CANCELLATION, CONFIRM, DATE\_CHANGE } // In Booking Service public void updateBooking(BookingUpdateRequest request) { Booking originalBooking = readFromDB(request.updatedBooking.uniqueId); switch (request.action) { case GUEST\_CANCELLATION: handleGuestCancellationRequest(originalBooking, request); break; case HOST\_CANCELLATION: handleHostCancellationRequest(originalBooking, request); break; case CONFIRM: handleConfirmationRequest(originalBooking, request); break; case DATE\_CHANGE: handleDateChangeRequest(originalBooking, request); break; default: throw new Exception("Unhandled action on booking"); } } // Or move all these private methods to their own handler simplifying this class further private void handleGuestCancellationRequest(Booking originalBooking, BookingUpdateRequest request) { originalBooking.lastUpdatedBy = originalBooking.guest; originalBooking.cancellationTime = cancelltaionTime; originalBooking.status = CANCELLED\_BY\_GUEST; // Trigger notification to host // Trigger refund if payment was taken updateInDB(originalBooking); } private void handleHostCancellationRequest(Booking originalBooking, BookingUpdateRequest request) { originalBooking.lastUpdatedBy = originalBooking.host; originalBooking.cancellationTime = cancelltaionTime; originalBooking.status = CANCELLED\_BY\_HOST; // Trigger notification to guest // Trigger refund if payment was taken updateInDB(originalBooking); } private void handleConfirmationRequest(Booking originalBooking, BookingUpdateRequest request) { // Business logic } private void handleDateChangeRequest(Booking originalBooking, BookingUpdateRequest request) { // Business logic } What is the difference? Looking at these examples makes it obvious how we can write code that makes the conceptual model explicit and hence reduce the cognitive load on the reader. The way to do this is via abstractions. All code is likely to have some amount of abstractions, but not all abstractions surface the thought process behind the code. Often developers use the model abstractions only as data carriers without imbuing them with any behavioural or semantic significance. This is the case in the first example of the generic update API. The Booking abstraction merely carries the data, all meaning is encapsulated in the if-else conditions. Model-Driven code, on the other hand, uses model abstractions to represent the core elements and uses them as the building blocks of all other interactions in the system. They are the heart of the system and all other code only manipulates them in ways defined and controlled by the model itself. This is the case in both the second and the third examples. It is not the case that the code in the first example does not have a model underlying it. Much like it is not possible to have “no design” (there is always design, even if inadvertent and poor), it is not possible to have “no model”. The solution sitting in the developer’s head is the model. The first example just chooses to obscure it while the latter two make efforts to make it clear. I hope this has made clear the benefits of explicitly using a conceptual model, and building the system around it. This won’t make the system (especially a large system) automatically self-evident, but it does go a long way in that direction. From the internet this week [Manuel Pais](https://twitter.com/manupaisable?ref=kislayverma.com) and [Matthew Skelton](https://twitter.com/matthewpskelton?ref=kislayverma.com) reiterate their focus on “team cognitive load” in this [talk on monoliths and microservices](https://www.youtube.com/watch?v=haejb5rzKsM&ref=kislayverma.com). Their book [Team Topologies](https://www.amazon.in/gp/product/1942788819?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1942788819&ref=kislayverma.com) is a must-read - get it NOW if you haven’t already. Two book recommendations in one edition - that’s a first for this newsletter. [*Steps to an ecology of the Mind* by Gregory Bateson](https://ejcj.orfaleacenter.ucsb.edu/wp-content/uploads/2017/06/1972.-Gregory-Bateson-Steps-to-an-Ecology-of-Mind.pdf?ref=kislayverma.com) sheds fascinating light on how computing was expected to evolve. For audio fans, here’s [a sample in the author’s own voice](https://archive.org/details/css%5F000051/css%5F000051%5Ft01%5Faccess.mp3). [Javier Ramos](https://twitter.com/javierramosrod?ref=kislayverma.com) has written a good [comparison between Pulsar and Kafka](https://itnext.io/pulsar-advantages-over-kafka-7e0c2affe2d6?ref=kislayverma.com). Let the streaming wars begin! That's it for this week folks! Cheers! Kislay Modify your subscription | View online | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------- | ### It Depends #28 URL: https://kislayverma.com/it-depends-28/ Last updated: 1970-01-01T00:22:35.000Z | It depends, and thoughts on good/bad collaboration Hello everyone! Welcome to the 28th edition of this newsletter. As a fitness enthusiast software engineer working for a fitness company, I am focussed on software engineering and fitness pretty much all the time. In both these areas, there are very few definitive answers - context is king. “It depends” has pretty much become my mantra over the last few months. Ergo, I have decided to rename this formerly eponymous newsletter “It Depends” starting this week. Same awesome content under a more fun name. Let’s get on with it. The value of collaboration, talking to your team and your customers is in the brainstorming stage or in the learning phase. This is where we should put together our collective minds, bounce ideas, and figure out the next steps. Once this stage is crossed and we enter the execution phase, collaboration becomes a dangerous overhead. ![](https://kislayverma.com/content/images/2021/03/good-bad-collab-300x102.jpg) If multiple teams and people must build something new together, synchronization is inevitable to some extent. However, there is a specific type of low-impact collaboration where people talk to each other to find out what capabilities exist and how to use them. This is seen when a product team tries to onboard to a platform by first setting up a meeting on how to use it. This is seen in “integration war-rooms” where two teams try to set their systems up correctly to talk to each other. This is seen when our teammates ping us about the right way to use that API we wrote the other day. This conversation is not about building new stuff, it is about figuring out how to use what exists. In my opinion, this is utterly wasteful. Developers obsess about making all business interactions self serve. In operations heavy organizations, teams spend a significant amount of bandwidth building tools for “enabling” operations teams. The same engineers, however, are perfectly okay sitting with team after team in meetings to explain to them how to use their software. Working with non-engineering teams is considered overhead, working with other engineering teams is collaboration. The deeper a team’s systems are in the architecture stack, the more they are likely to be reused. The more people want to use them, the more this team needs to answer questions around what and how. The more time they spend answering these operational questions, the lesser time they are actually building cool new stuff. A perfect vicious cycle. Teams and individuals spend an immense amount of time simply explaining how things work to others. This situation is clearly less than optimal, so how can we do better? This is not so much a problem as a shift in perspective. As engineers, we often consider all engineering as internal, and capable of being able to navigate convoluted technical processes. The thing to keep in mind is that they shouldn’t have to. Just like operations and other “business” teams, if other teams and developers have well-defined SOPs for using your code, we will have gone a long way towards removing unnecessary human collaboration. In this world of operational excellence, there are two tools that stand heads and shoulders above all other techniques – documentation and design. What if we think of all software work as building a product? My customers, who happen to be other engineers in this context, should be able to explore and understand my product, play around with it, and then finally sign up for it with minimum human intervention. For them to be able to do this, documentation is a powerful tool. Imagine if your system’s documentation was up to date, had clear guidelines on how to do what, and explained the different scenarios. Wouldn’t that be great? The world of operations practically runs on this type of documentation – detailed, precise, and explicit. The problem is that this type of documentation is hard to maintain, especially in the world of agile software. Also, no one read docs. People are far more likely to ping the author of the document for a meeting rather than go through something very detailed. This is why I consider documentation a supporting actor in this play. The single most important tool that we can bring to bear on this problem, in my opinion, is good design. When the system is exposed in such a way that the user cannot make a mistake in using it, then the very core of the problem goes away. Documentation can further bolster this good design by herding stragglers to the right place, but the entire experience of using a system or tool or API should be explicitly designed to make the user do the right thing unambiguously. This is not an outlandish idea. This is what product thinking, as applied to most customer products, is. It’s just that engineers mostly don’t apply the same thinking to developer tools. Pretty much always, customer experience > internal tools experience > developer experience. This is kind of sad because developer experience is where the entire engineering organization spends its time. Imagine what your team could achieve if every experience of using a new software thing was as smooth as your company’s customer experience. “How should I use it” indicates a failure of design and requires focussed product thinking to fix it. It is worth the effort because answering this question is the lowest form of collaborative value-addition. I hear a lot of developer feedback on the lines of “X is a great dev because he really spent a lot of time helping me use her system”. While I do not disagree with the sentiment, X and X’s team have also failed in prioritizing their time. They could have been discussing and building the next set of things. Instead, they ended up doing some grunt work that could very likely have been automated. While writing a new class or a new method, ask yourself if your teammates are going to have to ask you how to use it? If your system is to be used by other teams, will they have to ask you how to get started? Will you have to make some manual DB entries or change some configurations? Is there any step where you must “talk” to them to explain things? If so, then regardless of how useful the thing you just built, you have increased the communication overhead in your organization which you will be paying, personally, many times over the next few weeks or months or worse. I’d love to hear more from you about how much of your time goes into what kind of collaboration, and what value do you think it brings to the table. This article only expresses my narrow experience, and it would be instructive for me to learn from others who have different kinds of organizations than mine. Are there other tools/processes that can make a team more effective? Drop an email with your ideas. From the great interweb [Dan North](https://twitter.com/tastapod?ref=kislayverma.com) argues that SOLID is outdated and suggests [CUPID](https://dannorth.net/2021/03/16/cupid-the-back-story/?ref=kislayverma.com) instead. What do you think? Nick Tune has written about how [misaligned incentives fuel organizational dysfunction](https://medium.com/nick-tune-tech-strategy-blog/misaligned-incentives-fuel-organizational-dysfunctions-a67a3ed03890?ref=kislayverma.com). Not exactly a hot take, but a well-written article nonetheless. I hate rewrites, but sometimes they are needed. It depends :). Ian Small explains how Evernote went about doing a [complete rewrite of their app](https://www.protocol.com/evernote-reboot-ian-small?ref=kislayverma.com). This Twitter thread questions the [efficacy of pull requests as a mechanism for building shared knowledge](https://twitter.com/searls/status/1370455315246952449?ref=kislayverma.com) in a team. I personally think pull requests and code review are a great mentoring mechanism, but the thread makes some interesting points. That's it for this week folks. Have a great weekend! \-Kislay Modify your subscription \| View online | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #27: CQRS, internal platforms, complexity, and AWS URL: https://kislayverma.com/kislay-s-newsletter-27-cqrs-internal-platforms-complexity-and-aws/ Last updated: 1970-01-01T00:22:24.000Z | Kislay's Newsletter #27 Hello Everyone! Hope you are doing well and staying safe. I'm trying out a new, cleaner (hopefully) look for this newsletter. Let me know what you think of it. Today we are going to look at the CQRS architecture pattern, followed by the usual awesome from the internet. Software systems serve a variety of purposes from their first day, and the requirements on them grow over time. Changing requirements may pertain to a change in business logic, scaling needs, or some other aspects of the system. To satisfy these often contradictory or overlapping requirements, engineers must make a variety of trade-offs in the design of the system. The problem in making trade-offs is that many of them are not required at the beginning and by the time the need arises, the system design has evolved in such a way that the trade-off cannot be made at all. In my opinion, the most pernicious incidences of the design getting locked in happen at the data layer. A typical application’s data model is designed by marrying domain knowledge with performance considerations. The domain knowledge dictates what the entities are and how they relate to each other logically. Performance considerations dictate how they are implemented physically (e.g. RDBMS-vs-NoSQL, primary keys, indexes, etc.). These two sets of choices together enable an application to serve its use-cases efficiently. ![](https://kislayverma.com/content/images/2021/03/blind-men-and-multiple-readmodels-300x156.jpeg) In large applications with a lot of data and complex entity models, some implementation details become “core” over time. This is sometimes explicitly done by engineers, but often it happens in an unstated or even inadvertent manner. In these situations, new requirements can be so far at odds with the existing implementation that they cannot be accommodated at all. This general class of problems is large with different solutions for different cases. In this article, I want to focus on problems that arise when the way data is read from an application is very different from how data is written to a system. The difference can be in terms of query patterns, output format expectations, or scale of operations. In an [earlier post](https://kislayverma.com/programming/asynchronous-programming-a-cautionary-tale/), I wrote about an encounter with this situation. The order management system I was working on at that time was optimized for working with entity ids (order id, item id, etc). But over time, complex read requirements emerged which the data model was unable to support. The problems were two-fold. New query patterns were emerging which were difficult to implement efficiently in the existing implementation. Far more worryingly, the readers of order data were beginning to expect a very different model of the data. E.g. sellers on the e-commerce platform wanted their slices of a larger customer to be represented a certain way, customer-facing apps wanted the data to look very similar to how it looked in the cart. This is not an uncommon occurrence, especially for systems that own the core entities of an organization. The data they encapsulate is so widely used that it is required to be available in many different formats. The system itself needs yet representation to work with its data. How can we bridge this gap? CQRS CQRS stands for **C**ommand **Q**uery **R**esponsibility **S**egregation. Systems built with the CQRS principle distinguish between data models used for Commands (write operations) and Queries (read operations). The command model is used to perform write/update operations efficiently while the query model is used for supporting the various read patterns effectively. The data between the two models is kept in sync by propagating the changes in the command model to the read model via domain events or any other mechanisms. ![](https://kislayverma.com/content/images/2021/03/cqrs-basics-300x242.jpg) If this sounds like two different microservices to you, let me point out a subtle difference. The physical implementation of these two data models can indeed be done as two separate microservices. A single command model can even be used to support multiple query models. However, a key construct of microservice architecture is that two microservices typically represent two independent domains. In CQRS, both the command and the query models are part of the same logical domain regardless of the runtime architecture. The query model cannot function without understanding the command model deeply. The coupling here is expected, unlike the decoupled behaviour we hope to create in two separate microservices. CQRS does not dictate how the two models are kept in sync. This may be done synchronously by updating both the models at the same time. It may also be done asynchronously by transmitting commands from the command model to the query model over a message broker like Kafka. The latter choice is the one made often because it creates a more scalable system, though it comes with the obvious tradeoff of eventual consistency between the write action and read action. ![](https://kislayverma.com/content/images/2021/03/cqrs-dual-or-async-write-300x117.jpg) Isn’t this just caching? A data mode dedicated only for reads sounds suspiciously like a cache. Indeed, the query model can be implemented using a caching technology like Redis. However, the purpose of applying CQRS is not just to separate the place where is written from the palace the data is read. The fundamental intent is to create multiply representations of the same data, each of which satisfies the needs of some users. A CQRS style may have many query schemas, each of which may use a different physical implementation. Some may use the same database, some may use Redis, etc. Why should I use CQRS? CQRS is a useful architecture pattern in a couple of different scenarios. The first one is that which I have pointed out earlier in this article. If the same data model is not able to satisfy the read and write patterns of a system effectively, then it makes sense to decouple the two schemas by applying CQRS. The resulting data models can then cater to their specific requirements. CQRS effectively unlocks the data from a single representation into any number of (read) representations all of which are kept consistent with the core representation which handles all updates made to it. The second scenario in which CQRS is helpful is in separating the read load from the write load. This may sound like cheating when I have explicitly distinguished between caching and CQRS just a couple of paragraphs above, but hear me out. CQRS doesn’t seek out caching as an objective. However, by separating the command and the query schemas, we can create the possibility of scaling one independent of the others. The query schema may live on a separate database and employ caching of its own. It may be implemented in a technology that best caters to the query patterns of a particular use case. In any of these cases, the command model is exempted from having to scale to the requirements of the query model. I would repeat here that despite all this, these are not independent systems. The coupling between them is deep and this is not a problem. Why should I not use CQRS? Using CQRS in a system introduces significant cognitive overhead and complexity. Instead of a single data model and technology choice, developers now have to contend with at least two data models and potentially multiple technology choices. All of this is an overhead that cannot be ignored. The next problem is keeping the command and the query data models in sync. If the choice is made to keep the updates asynchronous, the entire system is forced to deal with the fallout of eventual consistency. This can be extremely troublesome, especially if parts of the system are directly exposed to human users who expect their actions to reflect in the data immediately. Even a single requirement for consistency can imperil the whole design. On the other hand, if we choose to keep the model in a consistent state at all times, the CAP theorem and 2 phase commits come knocking around. If both the schemas are colocated on a single ACID-compliant database, we may still be able to use transactions to keep them consistent. However, this takes away much of the scaling benefit of CQRS. If more than one query model is to be supported, the write operations will continue to get slower and slower since they need to update all query models before they can succeed. Both these problems make the use of CQRS a proposition that should not be taken lightly. Judiciously applied, it can result in a highly scalable application. But supporting multiple data models is a tricky affair and should only be considered if there are no other means of satisfying the necessary query patterns. From the internet this week This is a great collection of resources about [building internal platforms](https://internalplatforms.com/resources.html?ref=kislayverma.com). A lot of learnings to be had. Once you know what to look out for, complexity and complex systems are everywhere. FOr those who do not know what to look out for, here's a [primer on complexity](https://complexityexplained.github.io/?ref=kislayverma.com). This interview with Werner Vogels touches upon a lot of the history of the [architectural evolution at AWS](https://cacm.acm.org/magazines/2021/3/250706-a-second-conversation-with-werner-vogels/fulltext?ref=kislayverma.com). That's it for this week folks! Cheers! Kislay Modify your subscription \| View online | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #26: Tech debt, moving to IC, regulation of innovation, and architecture principles URL: https://kislayverma.com/kislay-s-newsletter-26-tech-debt-moving-to-ic-regulation-of-innovation-and-architecture-principles/ Last updated: 1970-01-01T00:22:13.000Z | ![Sunset at Purple Martini (Vagator, Goa)](https://kislayverma.com/content/images/2021/03/Sunset-at-Purple-Martini-Goa.jpeg "Sunset at Purple Martini (Vagator, Goa)") Sunset at Purple Martini (Vagator, Goa) Hello everyone! Welcome to this week’s edition of my newsletter. Hope everyone is doing well and staying safe. I took a vacation to Goa, the first time out since the lockdown began, and have been recovering for a week now :). I’ve also been facing some writer’s block, if you have any suggestions/thoughts around what you would like to read about, shoot’ em at me and I’ll see what I can do. So no writing this week, but here some read-worthy stuff from the internet. From the great interweb Tech Debt is always a hot topic among software engineers. Here is [Alfonso De La Rocha with his take on it](https://adlrocha.substack.com/p/adlrocha-the-risks-of-technical-debt?ref=kislayverma.com). It’s an interesting, not uncommon opinion, though a little extreme for me. What do you think? [Jon Moore](https://twitter.com/jon%5Fmoore?ref=kislayverma.com) moved from the Chief Architect to an individual contributor and wrote an article dense with wisdom about [how to play senior roles at companies in different ways](https://blog.jonm.dev/posts/individual-contributor/?ref=kislayverma.com). Must tread for all senior engineers. This [Startup Engineering lecture at Stanford University by Balaji Srinivasan](http://d396qusza40orc.cloudfront.net/startup%2Flecture%5Fslides%2Flecture11-regulation-disruption-technologies-2013.pdf?ref=kislayverma.com) recaps technical innovations and companies from 2013 onwards and tries to evaluate the role of regulation and free market on the rate of innovation. A very interesting exposition of accumulation of power. [Ruth Malan](https://twitter.com/ruthmalan?ref=kislayverma.com) is one of the people whose writing I have revisited multiple times over the years. In this piece, she talks about [architecture principles](https://ruthmalan.com/ByTopic/architecture/202102ArchitecturePrinciples.pdf?ref=kislayverma.com) in her usual, insightful, pithy, beautiful style. That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #8 URL: https://kislayverma.com/kislay-s-newsletter-8/ Last updated: 1970-01-01T00:21:38.000Z | Kislay's Newsletter #8 ![](https://kislayverma.com/content/images/2020/07/kislay-profile-1-600x800.jpeg) View online Happy Friday! Hope all of you are doing good, staying safe, and wearing a mask when you go out. Here's your weekly round up from the blog and the best of the awesome tech internet. From the blog I published my [review of “The Great Mental Models (Vol. 1)"](https://kislayverma.com/books/book-review-the-great-mental-models-general-thinking-concepts/) by [Shane Parrish](https://twitter.com/ShaneAParrish?ref=kislayverma.com). This is the first volume in a series of book fundamental patterns in thinking in all the basic fields of learning and it covers general mental models like “The map is not the territory” and “Occam’s Razor”. If you have not heard of these concepts and/or trawled through every page of Shane’s [Farnham Street Blog](https://fs.blog/?ref=kislayverma.com) like I have - you should definitely go ahead and read this book. If you are, however, a little more seasoned traveller of the cross-functional learning path, you are unlikely to be enthused by the fairly basic treatment. I would suggest you skim through the highlights I have included in this article and move on. [Article Link](https://kislayverma.com/books/book-review-the-great-mental-models-general-thinking-concepts/) Reading now I FINALLY started reading [Domain Driven Design](https://www.amazon.in/gp/product/B00794TAUG?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00794TAUG&ref=kislayverma.com) this week and even with just one section done, it is looking very good. “Knowledge Crunching” is such an apt description for processing SME knowledge into semi-technical terms - easily my favourite part of the software engineering process because even if you don’t write any code, you get to learn so much! Expect a review by the next weekend. Here are some other [books on my radar](https://kislayverma.com/reading/books-to-read/) \- let’s see how much I can cover this year. From the great interweb A beautiful, beautiful [illustrated introduction to quantum computing](http://files.thoughtworks.com/pdfs/The%5FStory%5Fof%5FQuantum%5FComputing%5FAug%5F2020.pdf?ref=kislayverma.com) by the Thoughtworks folks. An absolute must-read if you care even a little bit about computers. A brutal [deconstruction of the “software craftsmanship” label](https://einarwh.wordpress.com/2020/04/05/death-of-a-craftsman/?ref=kislayverma.com) by Einar Host. I have called myself by this name without thinking much about what it might mean, and I cringed a few times while reading this. I stumbled upon different nuggets about developers getting into management this week, starting with this [Twitter thread](https://twitter.com/mipsytipsy/status/1303233899422605314?ref=kislayverma.com) (later moved to a [blog post](https://charity.wtf/2020/09/14/useful-things-to-know-about-engineering-levels/?ref=kislayverma.com)) by Charity Majors and pausing with [this article](https://aws.amazon.com/blogs/enterprise-strategy/the-management-trap-time-for-a-rethink/?ref=kislayverma.com) by Phil LeBrun. I still have mixed thoughts about this and I am beginning to believe it is because of the whole “Management as Promotion” angle. Here’s a very broad overview of the [developments in Augmented Reality](https://shodu.net/scan-everything-building-the-infrastructure-for-augmented-reality/?ref=kislayverma.com) and how everything will be scanned very soon. I found these [beautiful astronomy pictures](https://www.forbes.com/sites/jamiecartereurope/2020/09/12/we-all-need-space-11-sublime-images-from-astronomy-photographer-of-the-year-that-will-make-you-gasp/?ref=kislayverma.com#596f4af177ab) and having been staring at them intermittently all week - makes me feel a little less locked in. Hope you like them too. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #25: We are only 10% human URL: https://kislayverma.com/kislay-s-newsletter-25-we-are-only-10-human/ Last updated: 1970-01-01T00:21:14.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) View this email online Hello everyone! Welcome to this week’s edition of my newsletter. Hope everyone is doing well and staying safe. This week we are talking about the human microbiome, followed by the usual collection of awesome from the internet. In many of my circles, the topic of the microbiome and how it impacts our health has been gathering momentum for some time now. I had not gotten into this till now and finally decided to take the plunge by reading [10% Human: How your body’s microbes hold the key to health and happiness](https://www.amazon.in/gp/product/0007584059?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=0007584059&ref=kislayverma.com) by Alana Collen. This was my first introduction to the hidden world of the microbes that live in our gut. This book is a great introduction to many of the basic concepts and makes a great case for thinking about our body’s bacterial residents a little more holistically than we have done so far. The book is well written and engaging. There are a lot of surprising facts and new concepts that I could not help be intrigued by. However, as I find with more and more books these days, it is written in what I think of as “Malcolm Gladwell” style. A lot of the time and text is devoted to story-telling and making a captivating case for the microbiome instead of making the book more information-heavy. I think about half of the book could be edited out for a person who is looking for facts and not repeated anecdotes. This is, of course, my personal opinion. I know a lot of people whole liked reading about the many case studies and yet-unproven possibilities that the book explores. The author comes across as an evangelist of a brave new frontier, where despite scientific rigour, there is entirely too much speculation for my taste. 10% Human is full of zeal and enthusiasm. It is a powerful proponent of its core idea that our microbiome may be far more important than we have given it credit for so far. I definitely recommend reading the book if you don’t have any knowledge of this topic yet. It has definitely opened my eyes to a very different world and provided scientific backing to some things that people often say but don’t really understand. I have included my summary of the book below - you can [read it on the website](https://kislayverma.com/books/book-review-10-human-by-alanna-collen/) if you prefer. This is an actual summary, not a set of highlights as I have done in some other book reviews. I hope it conveys the essence of the arguments and knowledge of the book. If you find it interesting enough to make you read the book, or if you want to read the book anyway, I suggest you read Chapters 2,3, and 8 thoroughly since they have a lot of information. Chapter 7 also has some interesting insights. The rest of the chapters can be skimmed quickly. Introduction The human body doesn't have significantly higher genetic complexity than rats or pigs. Humans and microbes are symbiotes. The human microbiome project (similar to the Human Genome Project) maps the DNA of our microbiome. It is convenient from an evolutionary perspective to have microbes do some functions instead of having to evolve genes for everything. Darwin onwards, the Appendix was always thought to be useless, but it actually is a reserve of microbes and thus has evolutionary value. Our body is a tube with skin on the outside and the digestive track ALSO on the outside (the inner, exposed layer of the tube). Chapter 1: 21st-century sickness Pneumonia is actually a symptom of several difficult microbes working together. There is no single responsible microbe. 4 major medical breakthroughs have significantly reduced death and disease: Vaccination Germ Theory and hygienic medical practices Water sanitization Antibiotics There are diseases that we think are normal but are very recent in their scale of occurrence. They were extremely rare just two or more generations back. These are not NORMAL, nor is there anything inherently "human" about them. Autoimmune diseases Allergies Diabetes Obesity Autism Why are these diseases happening now : Genetics: Good explanation for individual cases, but does not explain the widespread increase in numbers. Our genes could not have changed that much. Environment: Two main themes tying disparate thing together Immune system common between Allergies and autoimmune Gut Dysfunction: Autistic people have chronic diarrhea 60% of the immune system's tissue is located in the intestine since the separation between and "outside" in "inside" is only a few cells deep. Chapter 2: All diseases begin in the gut Calories in - Calories out is not enough to explain obesity Warbler birds gain a disproportionate amount of weight before they migrate. They shed it equally fast post-migration. Even those in captivity do this without migrating at all. How? Irritable Bowel Syndrome is a microbial imbalance: No disease as such, hence no known cure Usually triggered by bouts of antibiotics or some other infection. Conjectured to be an imbalance in the microbiota. Microbial populations become unstable, thereby causing "irritable bowel" Obesity may be infectious: Usually blamed on lifestyle or genetics. Obese people have more of the Firmicute class of bacteria, lean people had more of the class of Bacteroidetes. Changing the proportion can cause weight loss/gain in mice. Since this is caused by microbes, it might mean that obesity is contagious. Some statistical data supports this. Calories in means not just what we eat, but what we absorb Small intestines absorb whatever we easily can. Leftovers go to large intestines, where microbes breakdown what they want further. What remains is simple enough to be absorbed again by us, hence increasing the calorie intake. A vegetarian suddenly eating meat will not get extra calories because they don't have the microbes to break down residual meat (which carnivores already have). Microbes can also switch specific genes on/off to control fat storage. The gut as a complex system: Leptin is released by fat cells to suppress appetite once we have sufficient energy stored in fat cells. In obese people, the brain becomes resistant to Leptin. This causes the feedback loop to breakdown and people just keep eating. Lean people make new fat cells to store sparsely with energy. In obese people, larger cells are formed with too much fat content. These fat cells are also surrounded by immune cells as if they are an infection. This indicates dysfunction in the energy storage mechanism. They also have high amounts of LPS (Lipopolysaccharide) in their blood. LPS causes this fat cell inflammation and also suppresses new fat cell creation, leading to existing cells being overstuffed with fat. LPS forces us to store rather than burn. LPS gets into the blood because there isn't any Akkermansia Muciniphila in the gut lining of fat people, which leads to a thinner layer of mucus over the lining, and hence LPS seeps in. Chapter 3: Mind Control We assume mental disorders are due to genetics or socio-environmental factors, but this assumption is baseless. Bacteria are known to modify animal behaviours for evolutionary advantage. e.g. Cordyceps forces ant to spread their spores Gastrointestinal symptoms are common in people with psychological disorders. Ellen Bolte's son became autistic after multiple rounds of antibiotics. She focussed on his gastrointestinal symptoms and came up with groundbreaking insights into how autism may be caused by microbiome damage/imbalance. Toxoplasma is a parasite that causes personality changes in humans. This is well documented. Disproportionate amounts of it are found in the bodies of people suffering from schizophrenia, OCD, and other mental disorders. The vagus nerve connects the gut to the brain and microbes can send small electrical spikes up it to make us "happy". e.g. If we eat the food they like, they can create chemicals like Serotonin, thereby "rewarding" us with happiness. Propionate has been known to cause autism-like behaviours among rats. It causes rats to lose the ability to "unlearn" which is essentially the process of unused synapses being cleaned up by the immune system. New connections can be formed, but the old ones never go away. It is possible that propionate (created by Short Chain Fatty Acids in our large intestine) may be a cause for autism. Ellen Bolte's theory is that autism is caused by the bacterium C.tetani after it enters the blood directly after the protective microbiome has been damaged by antibiotics (leaky gut). Chapter 4: The Selfish Microbe The most widespread microbes make us not-quite-sick-enough so that we can continue to move around and spread them. The most virulent diseases don't spread too much since they kill too fast. In most people, the problem is not how to boost immunity but how to dampen it. Our immune systems confuse harmless things with dangerous ones. e.g. Allergies to common things are treated by smothering the immune system using antihistamines. Hygiene hypothesis: Increase in allergies ties with an increase in hygiene and therefore too few infections at an early age. This could not be proved, and a strong counterargument is that if immune cells are lying idle in hygienic environments, why don't they just attack the whole microbiome? How does the immune system identify that which is external but acceptable (food, good microbes) or internal but to be attacked (unused synapses to unlearn/forget)? Where cooperation helps in spreading genes, groups are selected over individuals. Animals and their microbiota have always co-evolved (mitochondria are essentially very simple bacteria), and hence evolution selects for the best combination of human + microbial genes (called the holobiont). The immune system has different types of cells: macrophages consume threatening bacteria, memory B cells attack specific targets, T helper cells help in communication between other cells, T-regs calm down an immune response. Immune response is triggered by antigens, molecules coming off the surface of invading pathogens. But pathogens and our microbiota both have antigens coming off them. Evidence suggests that our microbes know how to increase the number of T-regs to prevent the immune system from attacking them. Each species has its own way of doing this. The cholera pathogen V. Cholerae uses diarrhea as a way of spreading, just as the immune system uses diarrhea as a way of flushing out germs. It has copied the immune system mechanism to its own advantage. Leaky Gut: When a pathogen is able to convince the body to open the protein walls of the gut lining and get in the bloodstream. When the microbiome is damaged, pathogens are able to reach the gut lining, triggering the immune response to open the cell wall, and hence cause a lot of diseases. Chapter 5 - Germ Warfare The use of antibiotics has been on a steep rise since 1945\. Farmers started giving antibiotics to chickens to get them to grow fat. They may be causing the same effect in humans by disturbing the microbiome. In the 1950s, antibiotics were successfully prescribed for premature or malnourished babies to get them to gain weight. The broader implications of this were ignored in medical research. In the western world (and generally everywhere), antibiotics are prescribed indiscriminately even when in a large majority of cases they are useless or overkill, resulting in antibiotic resistance. There is a statistical correlation that microbial imbalance caused by taking antibiotics can cause autism (or any other disease triggered by dysbiosis), but there is little hard evidence yet. Broad-spectrum antibiotics kill pathogens and benevolent microbes since they cannot distinguish between them. The diversity of microbiota reduces rapidly and can take weeks or sometimes years to recover. Antibacterial products (other than alcohol) like triclosan have no scientific basis, and can actually cause more infections by killing resident bacteria and allowing new opportunistic ones to take root. Streptococcus may be the cause of OCD. It normally just causes strep throat but occasionally triggers an autoimmune response that harms basal ganglia, thereby making us unable to choose between one of the multiple possible actions. This causes "twitching" like Tourette's Syndrome. OCD patients are often obsessed with washing hands. This may be because streptococcus can survive hand washing better than other species and hence might be triggering a mental reward cycle to perpetuate itself. Chapter 6: You are what they eat The Giant Panda is genetically carnivorous. It only manages a vegetarian diet with the help of microbes that break down cellulose. It is difficult to study nutrition in isolation since reducing one component must increase some other and vice-versa. We don't need to look at the diets of cavemen or other centuries-old people to know what we are "meant" to eat. We can look at remote tribes untouched by modernization. One of the biggest differences is fiber. Modern western diets are low in fiber and the decrease corresponds neatly with the obesity epidemic. Switching to a high fiber diet can promote Akkerminsia Municiphila as mentioned in chapter 2, thereby reducing obesity. A plant-rich diet makes for a "lean" set of gut microbes. Chapter 7: From the very first breath Babies are sterile in the uterus. Their microbial colonies are seeded by getting covered in vaginal and fecal matter in the process of being born. The vagina is covered in lactobacilli which can kill other harmful bacteria to protect the baby. They can also breakdown milk, hence the baby can maximize energy extracted from milk. C-section births have shot up in the last few decades. These babies are more prone to dysbiosis related diseases and other infections since they do not get the microbes from their mothers (as naturally born babies do). Over 130 types of Oligosaccharides are found in human breast milk. They nourish not the baby directly but nourish its microbiome instead. They also prevent pathogens from taking hold by occupying the specific attachment points pathogen use to latch on to the body - this is distinctly natural selection. The composition of breast milk changes as the baby grows older. It contains more oligosaccharides in the early and more lactose later on. The microbial composition of breast milk also changes. Gut microbes in the mother are transported to the breasts so that the baby can consume them for a more diverse bacterial colony. There are far more microbes in the early days than there are later (when the baby's microbiota is already stabilized). Bottle feeding takes away all of the advantages of breastfeeding. Bottle milk is mostly cow milk, and infant formula has nutrients but has no bacteria. Babies need very specific types of microbes at specific stages of growth. bottle feeding has none of the adaptations necessary for this. Chapter 8: Microbial Restoration Elie Metchnikoff suggested that eating bacteria (e.g. as yogurt) can cure autointoxication. These were the first Probiotics. The line between food supplements and drugs is blurring and there isn't a lot of legislation yet. Probiotics deliver a small amount of bacteria to the gut, but it difficult for such small numbers to set shop, or work harmoniously with existing bacteria, or not crowd out others which are also useful. The effects are therefore not very predictable. For complex illnesses like type 1 diabetes and multiple sclerosis, probiotics are too little too late. Take poop from a healthy person and put it in a sick person's gut to restore their microbiome. Other animals also eat feces. Cure rates are as high as 95% after two rounds of treatment Since stool is not medically regulated, there have been no formal clinical trials - a lot of doctors remain unconvinced. Open-Biome is a non-profit stool bank that works similar to a blood bank. They screen donors for health, maintain supplies, and ship samples. The "autointoxication" theory of diseases: Microbes rot the remains of our food in the colon and the organ produces all kinds of diseases. The widely adopted solution was to remove the colon. Probiotics Faecal Microbiota Trasplantation Prebiotics Probiotics need constant replenishment. Prebiotics promote the right kind of microbial growth by supplying the raw materials for it. Speculation: Can we tailor probiotics and fecal transplant like we personalize and choose at sperm banks? Now I'm moving on to reading [Gut: The Inside Story of Our Body's Most Underrated Organ](https://www.amazon.in/gp/product/1925228606?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1925228606&ref=kislayverma.com). Expect more notes on these lines. From the great interweb [Marc Booker](https://twitter.com/MarcJBrooker?ref=kislayverma.com) has a very brief intro to [how web scale systems are built](http://brooker.co.za/blog/2021/01/22/cloud-scale.html?ref=kislayverma.com). Hint: it’s not Paxos.Justin Etheredge explains [why it takes so long to build software](https://www.simplethread.com/why-does-it-take-so-long-to-build-software/?ref=kislayverma.com). All my product management friends, please give this a shot. There has been a lot of debate on the pros and cons of flat hierarchies, autonomous teams, etc ([including on this website](https://kislayverma.com/tag/autonomous-teams/)), Richard Bartlett feels that the problem is not hierarchy (or the lack thereof) but the [emergent power structures in an organization](http://richdecibels.com/stories/hierarchy-is-not-the-problem/hierarchy-is-not-the-problem.html?ref=kislayverma.com). That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-7/ Last updated: 1970-01-01T00:21:14.000Z | Book Review : The Great Mental Models Vol. 1 I just my published my review and essential highlights from "The Great Mental Models (General Thinking Concepts) by Shane Parrish. A good read for absolute beinners, but too shallow for the seasoned reader. I recommend reading the highlghts and then digging deeper into the models elsewhere on the internet [Check it out](https://kislayverma.com/books/book-review-the-great-mental-models-general-thinking-concepts/) To change your subscription, click here. | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #24: Semantic web, Kubernetes, Agile, and anti-network effects URL: https://kislayverma.com/kislay-s-newsletter-24-semantic-web-kubernetes-agile-and-anti-network-effects/ Last updated: 1970-01-01T00:21:07.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) Read this online Hi! Welcome to this week’s edition of my newsletter. Hope everyone is doing well and staying safe. This week got a little intense at work so I wasn’t able to write anything. I have a half-written piece on extensibility in shared systems and some ideas on the next episode of the “[For the layman](https://kislayverma.com/category/for-the-layman/)” series, so let’s what I can cook up this weekend. In other news, I started reading [10% human by Alanna Collen](https://www.amazon.in/gp/product/B00O0FY5TI?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00O0FY5TI&ref=kislayverma.com) last night, so expect a review/summary soon. Over to some great material I found on the web this week. From the great interweb With trust in existing social networks and media in free fall, interest in a decentralized, semantic web is growing. This article has a good review of the [basics of the semantic web](https://cacm.acm.org/magazines/2021/2/250085-a-review-of-the-semantic-web-field/fulltext?ref=kislayverma.com) and plenty of references to launch you into this weekend’s rabbit hole. The OpenAI team explains how they [scaled their Kubernetes cluster to 7500 nodes](https://openai.com/blog/scaling-kubernetes-to-7500-nodes/?ref=kislayverma.com). A quick aside for aspiring data scientists - don’t ignore the infrastructure/operations side of software engineering. This 2013 talk from [Oredev](https://oredev.org/?ref=kislayverma.com) outlines [the fundamental theorem of agile software development](https://www.youtube.com/watch?v=WSes%5FPexXcA&ref=kislayverma.com) in 7 minutes, 26 seconds. Superb and Insightful. Venkatesh Rao talks about [anti-network effects](https://breakingsmart.substack.com/p/anti-network-effects?ref=kislayverma.com) \- things that can damp the network effect from within or outside the network. That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #23 - On bypassing central systems (aka when good abstraction is bad) URL: https://kislayverma.com/kislay-s-newsletter-23-on-bypassing-central-systems-aka-when-good-abstraction-is-bad/ Last updated: 1970-01-01T00:21:06.000Z | Hello everyone! Welcome to the 23rd edition of this newsletter. This week we are talking about why [central platforms get bypassed by product teams](https://kislayverma.com/platform-thinking/preventing-go-around-with-platform-thinking/), followed by the usual selection of awesome from the internet. Consider this fairly common scenario. A team builds a system that is meant to be used by the entire company e.g. a video management platform (VMP). This system is supposed to take care of all needs like video storage, editing, bitrate optimizations, delivery/CDN, etc so that no one else has to deal with them. You just bring your video and all else is taken care of. As a result of this overarching goal, only one interface is exposed to the user of the system to consume its capabilities - the bring-your-own-video API/UI. This is great for the major use-cases and teams to start adopting this new Video-Management-Platform quickly. At the same time, new requirements are popping up which require only content distribution, or optimization, or only storage. However, none of these capabilities are available for use in the VMP. So to unblock themselves, teams start going around the entire system and building their own specific, small solutions that are just enough to meet their needs. ![](https://kislayverma.com/content/images/2021/01/go-around.jpg) End result? We are back, kind of, to square one. The overarching VMP was intended to solve all video-related needs in one place. But in the final accounting, the organization still has scattered (and duplicated) bits of video capabilities, each tailored to the needs of the team that built them. What went wrong The team that built the VMP took the product experience as defined at that time (just bring-your-own-video) and embedded it into the system architecture literally. They hid all system capabilities behind the opinion that there should only be one way to use them. As a result, when opinions changed, there was no way to leverage the existing capabilities because they can only be used in the context of bring-your-own-video. This tight coupling at the system architecture level meant that other teams that had to deal with a changing landscape (typically product-specific/vertical teams) had no choice but to bypass the entire stack and build their own things. This problem of go-around in systems that are expected to be central/platform/generic occurs often. The best designs often abstract the most, and while this is a great characteristic in small systems and end-user products, it turns out to be an expensive mistake in building large scale architectures. This is because most large software systems are composed of the ability to do multiple similar things. If abstracted behind the facade of the larger product, these capabilities become inaccessible in other scenarios. We lose agility in responding to change since we no longer have the building blocks to create new things. This has ramifications beyond just technical coupling and duplication. Products that do something well also often do it only in a certain manner. Strongly abstracted systems simplify many things, but they can lock the organization into patterns of behaviour. If there is only one way of using the system’s functionality, the organization often organizes along the same lines in behaviour (and vice-versa). Three outcomes are possible. Use cases get force-fit into the product. While this is manageable if the leadership is keeping an ear to the ground, the likely outcome of this is the accumulation of tech debt in a previously solid product and all the bad things this eventually leads to. Technical teams go-around the central product to reinvent the wheel in specialized ways. This is a waste of engineering resources. Business teams modify their processes into sub-par versions because that is the only version of the process that the technology can support. This is bad for business beyond just the technology team. Scope of the go-around decision One argument against all of this is that product teams should simply have modified the product to unlock the capabilities or the team that owns VMP could have done it for them. This is possible but complicated because it requires something very difficult - [coordination between two teams](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/). The first step is to determine what changes are needed in the existing system. Either team independently, or the two teams jointly need to figure out what is to be changed and what it will cost. This exercise might be time-consuming by itself in poorly documented systems. Now that we know what changes are to be done, who should do them? Should the VMP team do them by dropping some other things it was planning to do. Or should the team which wants the change do it in an unfamiliar codebase whose operational responsibility it does not have? Both are difficult decisions. In opposition to all this ambiguity, there is a simpler choice - just build something small and quick for the new requirement. It doesn’t have to be great as long as it serves a limited purpose. We can always talk about consolidation with VMP “later”. The option of Going-around limits the scope of the decisions that the team has to make and is one of the reasons it is the route taken so often. Platform Thinking to the rescue Applying platform thinking to the problem offers a simple (not easy) way out - separate the capabilities of the system from the specific uses of those capabilities. Also known as the [Golden Rule of Platforms](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/), this allows a team to identify core building blocks of a platform that can be used to build more than one product. Thought of in this manner, building a VMP means first identifying all the capabilities that need to be present to build it, building these capabilities independent of the requirement for a VMP. The more we can build these capabilities as standalone constructs, the better it is for the architecture in the long term because these are the blocks that will be difficult to bypass. An existing, atomic system that offers a single functionality and is easy to integrate with is our best insurance against changing needs. I’m not suggesting that we build feature complete versions of all these sub-systems. What we should build is a structure that identifies them as independent, self-contained constructs that have a definite boundary and purpose. Richer functionality can emerge over time within that boundary. A video encoding system should be identified and built which does only encoding and has interfaces only for that purpose. The types of supported encodings supported can grow over time, but we should first identify the scope of this system. The plan to build a VMP on top of this has no role to play at this time. Perspective: product-first to platform-first One trick that makes the platform perspective possible is the inversion of the design approach from product-first to platform-first. A typical system design approach would start from product requirements and then design a system that would fulfill these requirements. The resulting design might be modular, maintainable, etc, but its structure, like the mindset of its designer, is tied to the product requirements. It only evolves as the requirements of THIS product change. ![](https://kislayverma.com/content/images/2021/01/product-first-thinking.jpg) The image above is a typical component diagram that you might expect for our VMP. The problem here is that all those modules paint a misleading picture. While they represent capabilities that the system builders are thinking of as independent, they are seen as independent “only within the context of the larger product”. The product perspective embedded in the design is likely to create co-dependent sub-components rather than independent systems. ![](https://kislayverma.com/content/images/2021/01/implicit-product-boundary.jpg) This product perspective in system design is part of the reason why coupling emerges even in microservice architectures where different microservices are expected to be independent of each other. The real problem lies not in the architecture/design pattern but rather in the design mindset. Platform-First thinking approaches this problem statement approximately bottoms up. We analyze the product requirement to [identify the underlying capabilities required to build it](https://kislayverma.com/platform-thinking/platforms-and-dogfood-everywhere/). Once we have this list, we forget about building the bigger product and focus solely on the sub-parts and design/build them all by themselves. The resultant architecture, thus, grows outside in (all systems have their own requirements and boundaries) and bottom-up ([lower complexity systems are built first and then composed into higher complexity systems](https://kislayverma.com/software-architecture/layering-domains-and-microservices-using-api-gateways/)). ![](https://kislayverma.com/content/images/2021/01/platform-first-thinking.jpg) Once these systems are ready, we switch back to building the main product. The platform components continue to operate standalone and any team with divergent requirements can use them on their own, thereby removing the notion of go-around completely. ![](https://kislayverma.com/content/images/2021/01/the-platform-solution.jpg) While this inversion of design perspective in no way ensures that independent components will emerge (developers are human after all, and it is difficult to not think of the [main deliverable and the deadlines around it](https://kislayverma.com/agile/being-fast-or-getting-faster-aka-build-momentum-not-velocity/)), it makes it far more likely. Scope of the go-around decision Pre-existing platform components are a deterrent to the dreaded go-around. For one, the capability that is required for a new use case might already exist in a perfectly reusable manner. Even if there are enhancements required in the existing component, it is easier to approach them because the scope of the component will be lesser than the complete VMP. The cognitive load of understanding the system and the implementation overhead of making the changes are both likely to be much smaller – likely far lesser than building something ground up. The technical and the delivery incentives both align in favour of adopting and enriching the existing systems rather than going around them. Summing it up Designing a system with only the final product in mind has the flaw of fencing all aspects of the design with a boundary defined by the product experience. Any variation on the requirement becomes hard to accommodate and the system becomes less nimble. This problem can be mitigated by identifying the capabilities required for the product as standalone systems, building them independently, and then composing them into the necessary product experience. This creates versatile building blocks that can be combined in multiple ways to create new products as the need arises. From the great interweb Web 3.0 is something I think about a lot, and so, apparently does [Tom MacWright](https://twitter.com/tmcw?ref=kislayverma.com). Here’s his take on how [the internet can make a clean start](https://macwright.com/2020/08/22/clean-starts-for-the-web.html?ref=kislayverma.com). Zanzibar is Google global authorization system. This [paper laying out its design](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/41f08f03da59f5518802898f68730e247e23c331.pdf?ref=kislayverma.com) is a good insight into how to design a web-scale system with strictness requirements. [Stephanie](https://twitter.com/stephaniejyee?ref=kislayverma.com) and [Tony](https://blog.tonyhschu.ca/?ref=kislayverma.com) have built a [series that introduces Machine Learning](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/?from=@&ref=kislayverma.com) in a beautiful, visual style. [Mark Erikson](https://twitter.com/acemarke?ref=kislayverma.com) shares his thoughts and experience on [using Typescript](https://blog.isquaredsoftware.com/2019/11/blogged-answers-learning-and-using-typescript/?ref=kislayverma.com). That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New Post by Kisla Verma URL: https://kislayverma.com/new-post-by-kisla-verma/ Last updated: 1970-01-01T00:20:52.000Z | Publish Events, not Logs If you have ever wondered "What to log for Observability" in your system, you may be asking the wrong question. The correct question is not "what to log", but "What are my system's boundaries" and "What events happen on these boundaries"?. Here's my take on Observability mechanics. [Check it out!](https://kislayverma.com/programming/publish-events-not-logs/) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-8/ Last updated: 1970-01-01T00:20:49.000Z | For the Layman (Ep. 3) - What is Programming? Here finally is the answer to all the confusion. In this episode of the For the Layman series, I answer the ultimate question - what programming is and what does that "tech team" keep doing? [Check it out](https://kislayverma.com/for-the-layman/for-the-layman-ep-3-what-is-programming/) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #22 URL: https://kislayverma.com/kislay-s-newsletter-22/ Last updated: 1970-01-01T00:20:40.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) View online Hello everyone! Welcome to this week’s edition of my newsletter. Hope everyone is doing well and staying safe. I’m writing a piece on why so many internal platform systems get bypassed by product teams, but like all socio-technical discussions, it keeps getting more complicated the more I get into it. I hope to finish it this week, but till then, here’s some of the best stuff I found on the internet this week. From the great interweb Gleison Brito and Marco Valente have published the results of [a controlled experiment comparing REST and GraphQL](https://arxiv.org/abs/2003.04761?ref=kislayverma.com#:~:text=Our%20results%20show%20that%20GraphQL,complex%20endpoints%2C%20with%20several%20parameters.). While they claim GraphQL to be better in the more complicated scenarios, REST still occupies the center stage in building APIs. What’s holding back GraphQL from being the de-facto standard? Let me know what you think. [Indi Young](https://twitter.com/indiyoung?ref=kislayverma.com) discusses the situations when we need to [explore the problem space](https://medium.com/inclusive-software/when-why-to-explore-the-problem-space-16068f1a1dbc?ref=kislayverma.com) instead of looking for solutions. Here’s an [old chat (July 8, 1996) between Andy Grove and Bill Gates](https://archive.fortune.com/magazines/fortune/fortune%5Farchive/1996/07/08/214331/index.htm?utm%5Fsource=Benedict%27s+Newsletter&utm%5Fcampaign=aa30e7ee8e-Benedict%27s+newsletter%5FCOPY%5F02&utm%5Fmedium=email&utm%5Fterm=0%5F4999ca107f-aa30e7ee8e-70467949). It is a fascinating read with the benefits of hindsight, and it is worth thinking about what they got right/wrong and why. For those looking to level up as engineering leaders, this [leadership library](https://leadership-library.dev/The-Leadership-Library-for-Engineers-c3a6bf9482a74fffa5b8c0e85ea5014a?ref=kislayverma.com) by [Philip Paetz](https://twitter.com/chapati23?ref=kislayverma.com) is a great, curated resource of knowledge. That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### Kislay's Newsletter #9 URL: https://kislayverma.com/kislay-s-newsletter-9/ Last updated: 1970-01-01T00:20:34.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) Read online Happy Friday! How’s everyone doing? Great, I hope. The weekend is here, and so is your weekly round-up of what I wrote on the blog and some of the cool stuff I found on the internet. I managed to write two articles this week, which makes for a very productive week. But I am beginning to wonder if three mails a week is too much for you people. So for now I have decided that I will not send out email updates every time I publish a new article. The only email you receive from me will be this weekly missive. I will of course share the article on my social media accounts ([Twitter](https://twitter.com/kislayverma?ref=kislayverma.com), [LinkedIn](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com), [HN](https://news.ycombinator.com/user?id=kislayverma&ref=kislayverma.com), [Reddit](https://www.reddit.com/user/kislayverma?ref=kislayverma.com), [Patreon](https://www.patreon.com/kislay?ref=kislayverma.com)). If you prefer the email route though, just drop a reply to this mail and I will continue sending the latest post to your inbox. Onwards and upwards! From the blog I published the third episode of the “For the Layman” series, and this one is a biggie. I tackle the greatest question of them all - “What is programming?”. Writing this one was a lot of one - got reminded of a lot of the basic digital electronics I studied all those years ago in college. [Article Link](https://kislayverma.com/for-the-layman/for-the-layman-ep-3-what-is-programming/) Logs are a great way to debug programs, but I have found them iffy as a means to Observability. And yet, logging is one of the primary ways we monitor our software in production. I propose a paradigm for effective logging where instead of thinking “what to log for observability”, we ask “what are my system’s inner boundaries, and what events happen there”. Check it out - [Article Link](https://kislayverma.com/programming/publish-events-not-logs/) From the great interweb Will Larson argues that[ migrations are only way to fix tech debt at scale](https://lethain.com/migrations/?ref=kislayverma.com). I thoroughly disagree with the assessment (having experienced the migration treadmill at Uber personally), but he does make some very solid points about building large software systems and teams. Let me know what you think on this. I came across this fantastic [Twitter thread](https://twitter.com/drose%5F999/status/1296610279170564096?ref=kislayverma.com) on how leveling the playing field by adopting a platform centric approach super-charged third party revenues on Amazon. The [Golden rule of Platforms](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/) is eat your own dogfood, and it’s good to hear and learn from these platformization war stories. Simon Wardley (of the Wardley Maps fame) has this [great article](https://blog.gardeviance.org/2016/11/amazon-is-eating-software-which-is.html?ref=kislayverma.com) about how Amazon is eating software which is eating the world. For Naval Ravikant fans, the [Navalmanack](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2Fkislay%2F9VJx7OZgRU.pdf?alt=media&token=daf3820f-c1d5-453f-9d9b-275e975f5ecc&ref=kislayverma.com) is (yet another) compilation of a lot of what he has said and written over the years. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #21 URL: https://kislayverma.com/kislay-s-newsletter-21/ Last updated: 1970-01-01T00:20:33.000Z | Happy Saturday! Welcome to this week’s edition of this newsletter. Hope everyone is doing well and staying safe. Here’s your weekly dose of great technical reading for the weekend. From the blog You can [read this article on the website](https://kislayverma.com/uncategorized/perhaps-we-shouldnt-be-so-well-connected/) if you prefer. I’ve been watching the events unfolding in the US political scene with a sort of dread fascination and thinking about the role technology has played in this. While it is good to see technology companies responding in some way to the madness of Donald Trump, the timing is so suspect that it creates more mistrust than faith in my mind. FB et al are jumping in the convenient direction - all this canceling should have been done 2, if not 4 years ago. But then, is a for-profit company obliged to have a moral imperative, high-minded mission statements notwithstanding? The far-right and far-left have always been at the forefront of anti-intellectualism in all countries. In an increasingly polarized world, since FB, Twitter is identified as technology companies rather than media companies, these shenanigans further erode the trust in technology to build a better world for everyone and in fact further the “fake news” narrative instead of setting tech as a custodian of empowerment. I’ve read a few histories of the first world war and one of the things that most commentators observe is that much of the carnage resulted (at least partly) because the scale and destructive power of the weapons far outstripped the communication capabilities at the disposal of the commanders. Huge attacks would be launched without the ability to properly manage the information. To me, our current scenario feels like the exact reverse of this. The internet has allowed us to communicate and spread information/propaganda on a global scale, but I do not think that human beings have the mental/emotional capability to deal with it. Nor do we have the structural frameworks to navigate this mess of data. Our collective fictions like society, country, etc that have helped us grow as a species so far are being pulled in so many directions that they are beginning to mean completely different things for different people, effectively making them meaningless as a unifying force. Perhaps the solution is to not connect everyone on the planet. A smaller world, connected by more individual choices and technology which supports individuals rather than scale. And at the root of it all, the basic idea of loving a person for themselves, not because they are part of our favourite collective. From the great interweb Jose Valim has written a look back at [10 years of Elixir](https://dashbit.co/blog/ten-years-ish-of-elixir?ref=kislayverma.com) which sheds light on the goals of the language and how it has evolved over a decade. In yet another attack on the microservice hype, [Uwe Friedrichsen](https://twitter.com/ufried?ref=kislayverma.com) calls out 7 widely accepted properties of microservices as myths and explains why in a [seven part series](https://www.ufried.com/blog/microservices%5Ffallacy%5F1/?ref=kislayverma.com#the-fallacies). This is an all-time classic read on [how complex systems fail](https://how.complexsystems.fail/?ref=kislayverma.com). I keep reading this every 6-8 months or so and it never fails to give new insights on every reading. This is a brilliant article on the [ethics of the attention economy](https://www.cambridge.org/core/journals/business-ethics-quarterly/article/ethics-of-the-attention-economy-the-problem-of-social-media-addiction/1CC67609A12E9A912BB8A291FDFFE799?ref=kislayverma.com). Extremely relevant to this edition of the newsletter and the world today. That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #20 URL: https://kislayverma.com/kislay-s-newsletter-20/ Last updated: 1970-01-01T00:20:07.000Z | View online A very, very happy new year to all of you folks! Welcome to 2021, and the twentieth edition of this newsletter. I had promised in #19 that I will share some of my thoughts about 2020\. I realized while writing them that 2020 was humbling and devastating for people all over the world, and anything I wanted to say about the bigger picture and life, in general, was inadequate. In the end, I decided to keep my head down, focus on my personal highlights, and write about [what I had learnt from them](https://kislayverma.com/uncategorized/hindsight-in-2020/). Since this article is more personal than my usual tech stuff, I have included it in this email directly. Of course, you can read it [on the blog](https://kislayverma.com/uncategorized/hindsight-in-2020/) if you prefer that. 2020 was a milestone year in all our lives. From acknowledging that such a plague had visited us, to pretty much living indoors for 9 months (and counting), to hearing horror stories of death and loss of livelihood, political turmoil the world over, getting used to a continuous stream of Zoom/Team/Hangout meetings while never actually meeting your colleagues – it has all been a handful. At the end of the year, I wanted to look at some of the things I managed to achieve, and some of the things I managed to learn. These are the highlights of my 2020. Personal Moved on from my Uber gig and got a new job at [Curefit](https://www.cure.fit/?ref=kislayverma.com). Got the strongest ever in my life. Went from \~20% body fat to the 8-10% range. From a previous max of 60 kg deadlift (8 years ago) to \~150 kg in April. Lost some ground during the lockdown but not all the way. Learned how to drive a car. Learned how to cook a very basic meal. Writing Started my [personal website](https://kislayverma.com/) instead of blogging on other platforms. Blogged consistently – 51 articles this year (target was 1 per week), up from 9 last year (target was 1 per month). Started this weekly newsletter about technology and teams that build technology. It now has \~1200 subscribers. I published the first edition on July 24 and a total of 19 editions in 2020. Built and released [Rulette Server](http://demo.rulette.org/?ref=kislayverma.com) as an easier interface for [Rulette](https://kislayverma.com/rulette/). Just for kicks, I self-published Rulette documentation and case studies as [a book on Amazon](https://www.amazon.in/Working-Rulette-Mastering-business-management-ebook/dp/B089S7NWS6/ref=sr%5F1%5F1?dchild=1&keywords=rulette&qid=1610172147&sr=8-1&ref=kislayverma.com). A lot of people have asked me about how I get the time to stick to writing, or working out or any of the other things I mentioned above. So I thought I will put some of my learnings and realizations of the year in writing. None of this is new, but some of them have registered deeply for me this year, and others I find valuable enough to merit repeating. Be Consistent Having a schedule and sticking to it is the single most powerful I did – writing when I just wanted to chill with Netflix, working out despite a hectic day of work, cooking when ordering-in felt so much more convenient. Every instance of doing these things added up. A lot of research suggests that our thinking patterns change after we repeat any activity over some time. We start liking those activities. Till you reach the “liking it” stage, just show up, day after day. Trust the process. Learn the basics I hate it when developers say that they don’t need to know how a piece of technology works internally as long as they can use it to good effect. To me, peering inside the hood of my tech has always been important. However, I realized that I was doing the same to other aspects of life e.g. fitness. At the beginning of 2020, I had been working out irregularly for about 12 years. I had done whatever workouts a bunch of trainers told me without understanding why. And at the end of all this time, I didn’t have any remarkable physical gains to show for it. In January ‘20 I decided to learn about the fundamentals of muscle building and fitness. Following Youtube channels like AthleanX, Jeff Nippard, Shredded Sports Science, and others, I finally got into the fundamentals principles of fitness and bodybuilding. While this world is as much of a rabbit hole as the tech world, there are a few basic principles of biomechanics and biochemistry that underpin everything. Understanding the theory behind exercises has helped me understand what I am doing, find replacements when I can’t do my usual gym training (this was a blessing when I had to cut over to home workouts), and made the pain of training more palatable, at least intellectually. It helped me understand how much misinformation surrounds us, and how to steer clear of it. Learn the basics. It can completely change the way you operate. Spend time consciously Not spending time consciously is the biggest reason for not having any time to spend. To extract more out of a day, we have to deliberately choose to do some things over others. Without this, it is difficult to be consistent in the long run. I advised creating a schedule for consistency earlier. That’s great, but where most of us spend our time is in the nothings between two tasks. Time just goes by over one more Youtube video, 10 extra minutes over a coffee, and so on. And at the end of the day, I often feel guilty for doing all these normal things that I like doing. I felt that I had wasted my time. However, I realized that there the difference between wasting and spending is one of being conscious of things. Taking an active, conscious decision to do one thing over the other forces me to evaluate the decision at hand. I can choose to do what I planned to do, or I can choose to watch another episode of Star Trek. Choosing the latter, however, forces me to create an alternative plan for when I am going to write in a conscious self-dialogue. For me, this active budgeting made the trade-off clear and removed the guilt which I had felt earlier. I now try to have an active awareness of how I am spending my time, hence there is no waste, only choices that I make. I can change them if I do not like the outcome. Choose to do the things that you are doing, even if the choice is to do nothing. Build a personal brand 2020 was the year of the passion economy. A few articles I wrote this year went viral on HN/Reddit and helped me connect with some of the best folks in the tech industry. For me, this really drove home the importance of having a personal identity in a niche, how underrated this idea still is, or how shallowly this is done by “hustlers”. A personal brand is about creating good odds for yourself. Extremely unlikely things are possible at scale. There is immense power in people recognizing your name and abilities beyond the specifics of your job. One viral tweet/article/anything can lead to very interesting results. And it takes effort to create a brand. We need to identify what we want to be known for, find our unique voice to express our ideas, and dive deep into the community to connect with others like us. It goes beyond writing a blog post or two. But it is absolutely worth it. Just do it I put this thought at the last because above all the earlier musings, this is the one I want you to leave with. If you want to do something, anything, just doing it in any way is infinitely better than planning to do it in the perfect way. Many people have told me that they have many ideas for writing and want to start a blog. However, some just keep polishing that one article, or keeping looking for the perfect platform, or making a long list of topics so that they can keep writing for some time, or keep thinking about what to write. All this planning ensures that the writing never starts. Same thing with exercise. People first want to figure out the best diet on the internet, or find a good gym, or home workout is boring, or any of the thousands of reasons for why workouts will start the coming Monday. They never do. Write a short article. Write that documentation for your team at work. Write an email to your friend tweet. It doesn’t matter if it’s been said before. Just start writing. There is no perfect diet. There is no perfect gym. Your workout shoes are fine. Just start exercising. Just do it. From the great interweb A collection of [best 2020 reads on platform design and architecture](https://stories.platformdesigntoolkit.com/best-of-2020-c1cacb43177e?ref=kislayverma.com) by Simon Cicero’s team at The Platform Design Toolkit. Simon’s was the first outfit I discovered pursuing the platform design approach rigorously. I strongly recommend their blog and podcast. I had never heard of [The Open Group Architecture Framework](https://en.wikipedia.org/wiki/The%5FOpen%5FGroup%5FArchitecture%5FFramework?ref=kislayverma.com#:~:text=The%20Open%20Group%20Architecture%20Framework%20%28TOGAF%29%20is%20the%20most%20used,high%2Dlevel%20approach%20to%20design.) for enterprise architecture till very recently. While there are many overlapping principles in this area, this formal structure is something that I believe more architects should know about. I want to tell you what Kevlin Henney’s [Out of Control](https://kevlinhenney.medium.com/out-of-control-97ed6efa2818?ref=kislayverma.com) essay is about, but it is about so many things that you better go see for yourself :) Ruth Malan has a brilliant article/presentation on the [importance of visual design in engineering](https://ruthmalan.com/Journal/2019/201902OReillySAConPresentationAll.htm?ref=kislayverma.com). That's it for this week folks. Have a lovely weekend and a great 2021! \-Kislay To change your subscription, click here. | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #7 URL: https://kislayverma.com/kislay-s-newsletter-7/ Last updated: 1970-01-01T00:19:55.000Z | Kislay's Newsletter #7 ![](https://kislayverma.com/content/images/2020/07/kislay-profile-1-600x800.jpeg) View online Happy Friday! I hope everyone had a cracker of a week. My publishing streak broke in just six weeks :( If any of you were looking out for this newsletter last Friday and couldn’t find it, I apologise - I was shifting houses and the boxing and unboxing didn’t leave any time for writing. But here we are this week, and here’s what happened on the blog and on the internet. From the blog This week I published some [thoughts on unit testing](https://kislayverma.com/programming/more-than-testing-writing-unit-tests-for-better-design/). My feelings towards unit testing have changed a lot from “who-cares” to “this is awesome”, and not just because tests are good. Increasingly I find myself writing a test just to see if the code is as modular as I could make it. This post gives some code examples of doing this and talks about discovering the hidden boundaries in code using unit tests. [Article Link](https://kislayverma.com/programming/more-than-testing-writing-unit-tests-for-better-design/) From the great interweb I found this [short but brutally incisive piece](https://janbosch.com/blog/index.php/2017/11/25/structure-eats-strategy/?ref=kislayverma.com) by Jan Bosch on how structure eats strategy in an organization. No matter your plans, if the organization isn’t designed for those plans, they will fail. Echoes of [James Clear](https://kislayverma.com/books/book-review-atomic-habits/) (“We don’t rise to the level of our goals, we fall to the level of our processes”). Alvaro Videla presents a humane, [design centric approach to API design](https://increment.com/apis/consider-the-interface-api-redesign/?ref=kislayverma.com). I came across this superb [guide to becoming a first -principles thinker](https://www.notion.so/A-Beginner-s-Guide-to-becoming-a-First-Principles-Thinker-637ab7236c8f49978f37457fa565edba?ref=kislayverma.com). I’m generally cautious when approaching this kind of advice but this is a genuinely well written and structured approach towards drilling down to the heart of a concept. How do you set the baseline when looking for a new house? Here’s a fun little [twitter thread](https://twitter.com/AGARvalaAgarwal/status/1300040376061837313?ref=kislayverma.com) that resonated with me recently. Apparently you should devote 37% of your total budget (time/money/whatever) on finding the baseline - that’s how the mathematicians do it! That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #19 URL: https://kislayverma.com/kislay-s-newsletter-19/ Last updated: 1970-01-01T00:19:40.000Z | View online Happy Saturday! Belated Christmas wishes to all you lovely people. Welcome to the nineteenth edition of this newsletter, the last for this year. I have been blogging off and on for over 15 years now, but never took it very seriously. I would get motivated, churn out 4-5 articles, then go back into hibernation for many months. Rinse. Repeat. Making a commitment to writing one article a week and publishing this weekly (for the most part) newsletter has been a revelatory lesson in discipline and consistency. 2020 has had a lot to look back and reflect upon in any case, so I will share some more coherent thoughts in the next week’s missive. For all the ideas, comments, warnings of “Error establishing database connection” (ugh!), and especially for the notes telling me that what I was writing was useful and to keep going. I am deeply indebted to all of you. Thank you very much! I wasn’t able to write anything new this week, but as usual, here’s some awesome stuff that better people wrote and I found interesting. From the great interweb Michelle Bu of the Stripe engineering team has published a record of the [first 10 years of the evolution of the Stripe Payments API](https://stripe.com/blog/payment-api-design?ref=kislayverma.com). If you are interested in payments, APIs, or programming in any way, this is a strongly recommended read. Continuing on the API theme, Packy McCormick writes about strategic API play and the power it unlocks in his “Not Boring” newsletter. [“APIs all the way down”](https://notboring.substack.com/p/apis-all-the-way-down?ref=kislayverma.com) is reminiscent of [Steve Yegge’s API rant](https://kislayverma.com/platform-thinking/distilled-steve-yegge-s-platform-rant/), both being strong reminders of the difference being API-first can make. John Cutler’s article on the pitfalls of [giving everyone their own project](https://cutlefish.substack.com/p/tbm-5253-real-teams-not-groups-of?ref=kislayverma.com) really hit home. My article on [autonomy and too many small teams](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) came from the same kind of non-team experience at Uber, but John expresses it better and with fewer words (of course). If you are on a break and tinkering around with code, here’s a [guide to writing good command-line tools](https://stripe.com/blog/payment-api-design?ref=kislayverma.com). That's it for this week folks. Have a great weekend and a very, very happy new year! \-Kislay To change your subscription, click here. | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #18 URL: https://kislayverma.com/kislay-s-newsletter-18/ Last updated: 1970-01-01T00:19:36.000Z | View online Happy Saturday! Hope everyone is doing well and staying safe. Welcome to the eighteenth edition of this newsletter. Here’s some of the best of what I read and wrote about this week. From the blog This week I about [what Feature Toggles are and how you can use them](https://kislayverma.com/programming/why-and-how-to-use-feature-toggles/) for faster and safer code deployments. This should be read in conjunction with my article on [removing bottlenecks (technical and mental) to faster software delivery](https://kislayverma.com/agile/how-to-speed-up-software-delivery/). Feature Toggles are an extremely powerful and accessible tool for developers for decoupling software deployment from feature release. They, alongside other forms of runtime configuration, can also play an important role in mitigating incidents in production. Let me know if you have memorable use cases of feature toggles saving your systems in production - I love hearing war stories! [Article Link](https://kislayverma.com/programming/why-and-how-to-use-feature-toggles/) From the great interweb Remember the Google outage which saved us all a few hours of Meets meetings and caused tons of other grief around the world? Google has published an [RCA for the incident](https://status.cloud.google.com/incident/zall/20013?ref=kislayverma.com), and while many like Cindy continue to get [triggered by the use of the word “RCA”](https://twitter.com/copyconstruct/status/1340140498418208770?ref=kislayverma.com), I think the document contains some important things to learn for everyone. While this has been said many, many times over by now in many forms, I take this opportunity to say it one more time in Kyle Evans’ words. [Products over Project](https://productcoalition.com/product-thinking-vs-project-thinking-380692a2d4e?ref=kislayverma.com). Kousik Nath has written a good series of articles on understanding distributed consensus in general and the Raft protocol specifically. Here are parts [one](https://kousiknath.medium.com/making-sense-of-the-raft-distributed-consensus-algorithm-part-1-3ecf90b0b361?ref=kislayverma.com) and [two](https://kousiknath.medium.com/making-sense-of-the-raft-distributed-consensus-algorithm-part-2-4f12057b019a?ref=kislayverma.com). This reminded me of another take on this that I had seen some time ago and found very intuitive to understand - here [that one](http://thesecretlivesofdata.com/raft/?ref=kislayverma.com) too! Matt Bornstein, Martin Casado, and Jennifer Li outline the [latest architectural trends in data engineering](https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/?ref=kislayverma.com) in this article for Andreesen-Horowitz. That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### Kislay's Newsletter #17 URL: https://kislayverma.com/kislay-s-newsletter-17/ Last updated: 1970-01-01T00:19:28.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) View online Happy Sunday! Welcome to the seventeenth edition of this newsletter. Hope everyone is doing well and staying safe. Let’s jump right into the good stuff! From the blog This week I wrote about the workflow architecture pattern. I find that using workflows as a first-class construct really cleans up system architecture in a very nice way, making business processes very explicit and decoupling technical components. However, I have noticed a lot of confusion around what workflow is, especially in the choreography-happy world of microservices. Leveraged in the right places, workflows can be direct triggers for platform architectures. Done poorly, they can create a worse mess than what they were meant to solve. Here’s what you need to know - [Article Link](https://kislayverma.com/software-architecture/architecture-pattern-orchestration-via-workflows/) From the great interweb Alexander Wang talks about [information compression and communication overheads](https://alexw.substack.com/p/information-compression?ref=kislayverma.com) in this memo he sent to his team. Orkhan Gasimov lays out the [basic principles of solution architecture](https://medium.com/@ogasimov/solution-architecture-for-beginners-22753be02adf?ref=kislayverma.com) for beginners. Bruce Wang (two Wangs in one newsletter - what are the odds!) discusses his [pursuit of creating impact](https://www.linkedin.com/pulse/pursuit-impact-bruce-wang/?trackingId=52Npge1tNWmQK17k392NTQ%3D%3D&ref=kislayverma.com) and his journey of building high-performance teams at Netflix. Kilian Weinberger on the [importance of deconstructing complex topics](https://slideslive.com/38938218/the-importance-of-deconstruction?ref=kislayverma.com) into the underlying principles, in life as in Machine Learning. That's it for this week folks. Have a great weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### Kislay's Newsletter #16 URL: https://kislayverma.com/kislay-s-newsletter-16/ Last updated: 1970-01-01T00:19:16.000Z | Read online Happy Saturday! It's good to be back with all you again. I apologise for the four week break. I spent this time stuffing my face with some awesome festival food and reading Rick Riordan books - nothing technical or work like at all. Now that the indian festive season is winding down, I hope to be back to the usual weekly cadence. So welcome to the sixteenth edition of this newsletter - here’s your weekend dose of great technical reading! From the blog I wrote about developer self identities and how managing them effectively is a critical capability for managers in autonomous teams. In most organizations, developers are invited to solve technical challenges rather than being called upon to identify and solve business problems. This means that depending on the maturity of the team, not all kinds of developers will be excited to work in it. Managers can address this by coaching developers to love the problem and not just the solutions they build. If done right, this can get developers engaged with the “why” rather than just the “how”. [Article Link](https://kislayverma.com/organizations/managing-developer-identities-in-autonomous-teams/) I experimented with this article a little by publishing it in the bullet-point format I usually use to write. I hoped that readers might find this more concise and direct to consume. Some of you did, but an overwhelming majority thought that the format made the article a blob of points and difficult to grasp. So I will go back to normal prose writing for now - but definitely want to try some more experiments on this format. Perhaps sections of points? Let me know if you have recommendations. From the great interweb [Vasco Figuera](https://twitter.com/vlfig?ref=kislayverma.com) talks about why he believes [microservices are architectural nihilism disguised as minimalism](https://vlfig.me/posts/microservices?ref=kislayverma.com). There are some good points in this article about logical-versus-runtime architecture and drawing system boundaries. [Arnaud Porterie](https://twitter.com/arnaudporterie?ref=kislayverma.com) discusses his journey as an engineering leader in his blog. This is a peep into the mind of how a good senior leader operates. Maven 4 is on its way! [Maarten Mulders](https://twitter.com/mthmulders?ref=kislayverma.com) talks about [what he is looking forward to](https://maarten.mulders.it/2020/11/whats-new-in-maven-4/?ref=kislayverma.com) in the new and much awaited version. [Benji Weber](https://twitter.com/benjiweber?ref=kislayverma.com) wants to [make code worse](https://benjiweber.co.uk/blog/2020/09/12/the-benefits-of-making-code-worse/?ref=kislayverma.com)! Madness, I know, but one with a strong reason behind it. Beneficial tech debt is something that I have always agreed with, and the ability to take on reasonable amounts of it often indicates that the current code is in good shape. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #15 URL: https://kislayverma.com/kislay-s-newsletter-15/ Last updated: 1970-01-01T00:19:02.000Z | View online Happy Sunday! Welcome to the fifteenth edition of this newsletter. I spent Saturday travelling to my parent’s place in Delhi, which is why this letter is coming out a day late this time. The smell of early winter is already in the air here, and I’m sure many of you are looking forward to Diwali festivities. Here’s you weekend dose of great technical reading! From the blog This week I wrote about [speeding up software delivery](https://kislayverma.com/agile/how-to-speed-up-software-delivery/) by deploying as frequently as possible. IMO we attach way more importance to deploying software than we should - it is (and should be) a mundane activity. I tried to model and analyze the entire software delivery pipeline in reverse and applied a bit of constraint theory style thinking to identify the bottlenecks at each step. TL;DR - Think in terms of deploying changes (not complete features), use feature gates, and automate deployments. Check out the article for the full scoop! [Article Link](https://kislayverma.com/agile/how-to-speed-up-software-delivery/) From the great interweb Edgar Dijkstra was one of the most influential and controversial leaders in the computing industry. Here is a great (not too long) [history of his life and work](https://inference-review.com/article/the-man-who-carried-computer-science-on-his-shoulders?ref=kislayverma.com) which is entirely worth reading. I did not do a lot of the behind the scenes action that this article covers. I'm still on the theme of writing advice on this newsletter since this is one of the things I am learning as I share. [David Perell](https://twitter.com/david%5Fperell?ref=kislayverma.com) has this [AWESOME guide](https://www.perell.com/blog/the-ultimate-guide-to-writing-online?ref=kislayverma.com) on writing online effectively. Erik Dietrich has a great article on [how software developers stop learning](https://daedtech.com/how-developers-stop-learning-rise-of-the-expert-beginner/?ref=kislayverma.com) by getting labelled experts too early in their careers. This “familiarity disguised as expertise” pattern is something that I have fallen into myself, and seen others suffer it too. If the description seems to fit you, it might be time to consider mixing things up again. The Netflix engineering team has published an article on the way they [identify priority traffic to shed load](https://netflixtechblog.com/keeping-netflix-reliable-using-prioritized-load-shedding-6cc827b02f94?ref=kislayverma.com) effectively. For those who have encountered it before, the Netflix tech blog is full of articles sharing real data and techniques like this. I definitely recommend spending time on it. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #10 URL: https://kislayverma.com/kislay-s-newsletter-10/ Last updated: 1970-01-01T00:18:48.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) View online Happy Saturday! Welcome to the tenth edition of this newsletter! I hope this weekend finds you well and you are ready to learn some new stuff. Publishing this newsletter by Friday evening was beginning to become challenging with the day job heating up. So instead of trying to rush through it under pressure, I have decided to send it out by Saturday afternoon/evening to buy myself some more time. You still get plenty of time to go through it and let me know how you like it. Better late than unpublished :) From the blog I didn’t write anything myself this week, but my good friend [Bharath Reddy](https://www.bharathkreddy.com/?ref=kislayverma.com) chipped in with this interesting piece about why he feels the next big leaps in artificial intelligence will come from unsupervised learning. With GPT-3 making waves the last few weeks, this topic is on many people’s minds. While we have made a lot of progress using ML and supervised learning, is unsupervised the way forward? [Article Link](https://kislayverma.com/technology/the-revolution-will-be-unsupervised/) This was the first guest post on this blog, and I am looking to make this a more regular feature. So if you are a hands on software practitioner and passionate about anything technical, I’d love to host your thoughts on this blog. Just drop a mail and we can discuss it. From the great interweb Martin Kleppman discusses if writing a book is worth it after his “[Designing Data Intensive Applications](https://www.amazon.in/gp/product/B06XPJML5D?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B06XPJML5D&ref=kislayverma.com)” sold more than 100,000 copies. That book is absolutely brilliant, and having [self-published a book](https://www.amazon.in/Working-Rulette-Mastering-business-management-ebook/dp/B089S7NWS6/ref=sr%5F1%5F1?dchild=1&keywords=rulette&qid=1601706856&s=digital-text&sr=1-1&ref=kislayverma.com) myself (10 copies sold - BOOYAH!), I absolutely agree with this article too. Struggling to understand your organization’s structure and dynamics? This long but in-depth article explains the [Organization Sensemaking paradigm](https://oxfordre.com/psychology/view/10.1093/acrefore/9780190236557.001.0001/acrefore-9780190236557-e-78?ref=kislayverma.com) and how to use it to navigate and design companies. James Stanier writes about why [you shouldn’t make yourself redundant](https://www.theengineeringmanager.com/?ref=kislayverma.com) as an engineering manager. This is contrary to the popular idea that to move up you need to be redundant in your current role, and an interesting read, as are many of the other articles on his website. This is a great [Twitter thread](https://twitter.com/jackbutcher/status/1304858588238745601?ref=kislayverma.com) on how we can “productionize” ourselves and our work. Make once, sell repeatedly! That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #14 URL: https://kislayverma.com/kislay-s-newsletter-14/ Last updated: 1970-01-01T00:18:45.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) View online Happy Saturday! Welcome to the fourteenth edition of this newsletter. This is the first time in 14 weeks when I have no original writing to share with you. I ran into a bit of a blogger’s block all through this week - I spent several hours writing 4 incomplete pieces but nothing came together completely. So this week, I’m only sharing what those better than me have shared on the internet. From the great interweb The finest thing I discovered this week was this article by [Tanya Reilly](https://twitter.com/whereistanya?ref=kislayverma.com) on the [“Glue” role that some engineers end up playing](https://noidea.dog/glue?ref=kislayverma.com) and the impact this has on their careers. [Coupled with this collection of articles on the role of staff engineer](https://staffeng.com/?ref=kislayverma.com) \- I have been doing a lot of introspection about what I have been doing and what I should have been doing (or not doing) instead. I strongly recommend reading everything Tanya has written on her blog - solid food for thought. This [talk from James Lewis](https://www.youtube.com/watch?v=tYHJgvJzbAk&ref=kislayverma.com) sits right at that edge of technical and organization design which I find absolutely fascinating. Let’s talk microservices and flow of work! This Harvard Business School working paper on [“The Architecture of Platforms : A unified view” by Carliss Bladwin and C. Woodard](https://www.hbs.edu/faculty/Publication%20Files/09-034%5F149607b7-2b95-4316-b4b6-1df66dd34e83.pdf?ref=kislayverma.com) has some great insights into platforms, architectures and the combination of the two. Pairs very well with some of what I have written on [building technical platforms](https://kislayverma.com/category/platform-thinking/) on this blog. I found this [interesting twitter thread](https://twitter.com/vishwanath95/status/1319675801030193153?ref=kislayverma.com) on managing our tech diet and the responsibility designers bear for it. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #13 URL: https://kislayverma.com/kislay-s-newsletter-13/ Last updated: 1970-01-01T00:18:34.000Z | View online Happy Saturday! Welcome to the thirteenth edition of this newsletter - here’s what I wrote this week on the blog and the absolute best of what I read on the internet. Hope you are all doing well. As the lockdown continues to relax in Bangalore, I am beginning to feel a little more comfortable with stepping out. Just had my first dinner outside after 7 months and not sure if it was the hygiene theater or what, but did not feel as tense as I thought I would be. Still following social distancing rigorously and wearing a mask at all times - so hopefully nothing goes wrong. From the blog This week I published my review and highlights of the first section of [Simon Wardley’s book on Wardley mapping](https://kislayverma.com/books/book-review-wardley-mapping/). Simon developed and used Wardley maps as a CEO, but even as a novice I’m finding this technique very instructive in determining what software systems my team has and what we need to build. The technique has a wide following now and the book (or other resources on the mapping technique) are an absolute must read for anyone who needs to make a decision about getting something done. [Article Link](https://kislayverma.com/books/book-review-wardley-mapping/) From the great interweb Kishore Gopalakrishnan [walks us through the origin, design principles and architecture of Apache Pinot](https://www.youtube.com/watch?v=B3AK-eIPL8E&ref=kislayverma.com) \- a real time distributed OLAP database originally created a LinkedIn. Chris Ball has a great article about [GitTorrent - a completely distributed Git hosting system](https://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-decentralized-github/?ref=kislayverma.com) to break the near monopoly (as a system - not just a business) of Github. Preetam Nath has some good advice on [why everyone should write](https://www.preetamnath.com/blog/why-you-should-write?ref=kislayverma.com). Tl;dr - writing is not about writing, it is about clear thinking. I wrote a review of [The Great Mental Models (Vol. 1) by Shane Parrish](https://kislayverma.com/books/book-review-the-great-mental-models-general-thinking-concepts/) some time ago. This book only covers the general thinking concepts, but here’s a [list of all mental models](https://fs.blog/mental-models/?ref=kislayverma.com) Shane considers fundamental and that will be covered in the other volumes of the book series. Enjoy the rabbit hole :) That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #12 URL: https://kislayverma.com/kislay-s-newsletter-12/ Last updated: 1970-01-01T00:18:10.000Z | ![Kislay Verma](https://kislayverma.com/content/images/newsletter/thumbnails/2020/08/kislay-profile-600x200.jpeg) View online Happy Saturday! How’s everyone doing? Welcome to the twelfth edition of this newsletter - here’s what I wrote this week on the blog and the absolute best of what I read on the internet. From the blog This week I published a [summary and the highlights of my reading](https://kislayverma.com/summary/working-around-the-cap-theorem/) of this [great article](https://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed/?ref=kislayverma.com) by Eric Brewer on how engineers have started working around the constraints enforced by CAP theorem by understanding that consistency and availability are not binary but rather a broad spectrum of user experience decisions. Application designers can use this insight to build more flexible applications by making the consistency-vs-availability choice only when a partition occurs, instead of choosing one of the two paradigms up front and hobbling all further architectural choices. One mind-blowing but obvious-in-hindsight takeaway for me was that for all practical purposes, a remote invocation timeout is as good as a network partition. So in effect, we are dealing with partitions everytime we call a remote service. I had never really thought timeouts like that, and it further underscores the idea that CAP related trade-offs should properly be made locally in the codebase instead of being made in an all-in, upfront decision. [Article Link](https://kislayverma.com/summary/working-around-the-cap-theorem/) Reading Now I started reading [Domain Driven Design by Eric Evans](https://www.amazon.in/gp/product/B00794TAUG?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00794TAUG&ref=kislayverma.com) a few weeks ago and I’m still working on it (it is denser than a casual skim suggests). I have also started reading [Simon Wardley’s book on Wardley Maps](https://medium.com/wardleymaps/on-being-lost-2ef5f05eb1ec?ref=kislayverma.com) and shared some [highlights from the first chapter](https://twitter.com/kislayverma/status/1310146611670953985?ref=kislayverma.com) here. These two books complement each other very nicely. Wardley Maps paints a larger picture of how strategy can be informed by identifying the key parts of the system and understanding how they evolve. Domain Driven design goes a little lower and highlights how engineers and domain experts can work together to create systems that capture and build on top of real world complexities. Between the two books, a very coherent spectrum of strategy-modelling-execution emerges which I am really enjoying. From the great interweb John Salvatier has this great article on [the amount of detail reality has](http://johnsalvatier.org/blog/2017/reality-has-a-surprising-amount-of-detail?ref=kislayverma.com) and how we are almost trained to skim over it. This ties in with so many different things, but since I’m struggling a little bit with my deadlines of late, I’ve been thinking of the cray amount of detail even small software features have and how utterly insane it is that try to estimate them without digging into the mess. Andrew Chen in his usual good form discussing [what is next for marketplaces](https://andrewchen.co/how-marketplaces-will-reinvent-the-service-economy/?ref=kislayverma.com) and marketplace economy. Dropbox goes virtual, and the team there explains their approach to the paradigm in [this document](https://blog.dropbox.com/topics/company/dropbox-goes-virtual-first?ref=kislayverma.com). Takeaway : Remote is not Distributed, and it matters a lot. For space geeks, here’s a [short intro to Mega Drive](https://medium.com/predict/mega-drive-the-tech-that-promises-near-light-speed-travel-d9d9ded1ca44?ref=kislayverma.com), a technology that promises near light speed travel. That Medium publication has a lot of cool space stuff - kind of superficial, but enough to start you off into the rabbit hole. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #11 URL: https://kislayverma.com/kislay-s-newsletter-11/ Last updated: 1970-01-01T00:17:56.000Z | View online Happy Saturday! Hope everyone is doing well and staying safe. Welcome to the eleventh edition of this newsletter - here’s what I wrote this week and the absolute best of what I read on the internet. From the blog A lot of people responded to my article on [using events instead of log for Observability](https://kislayverma.com/programming/publish-events-not-logs/) by saying that logs are critical for debugging issues, or that logs needed metrics to complement them, or that the mechanics of logging are different from publishing events. I felt that a follow up was needed for more context and clarifying different terms being used in the discussion. [So I wrote one](https://kislayverma.com/software-architecture/observing-is-not-debugging-and-other-misnomers/) which distinguishes between observability from debugging, logs from metrics, and logging paradigm from logging implementation. I stand by what I said - events with lots of metadata are the best, most consistent mechanism we can use to build Observability tools. Regardless of the mechanism you use for these events (“log” files, event buses, whatever else - all that is implementation detail), or the aggregates you build them into (e.g. metrics), events are our best option at defining a single source of truth for understanding system behaviour. Even [some vendors seem to be coming around](https://newrelic.com/resources/ebooks/observability-2020-manifesto?ref=kislayverma.com), in their own sideways, slippery, vague manner. What do you think? [Article Link](https://kislayverma.com/software-architecture/observing-is-not-debugging-and-other-misnomers/) From the great interweb Michael Chadwick writes about how his team is [using Architecture Decision Records (ADRs) to move fast and not break things](https://tech.ao.com/post/how-were-using-architectural-decision-records-to-move-fast-and-not-break-things-as-much/?ref=kislayverma.com) as often. Having just joined a startup which unfortunately has little internal documentation, I cannot agree more with the need to document the “why” of technical decisions. Engineers can mostly go through the code to figure out how, but the why is often missed. Use MIchael’s method or any other method, but write down why things are being done a certain way. I came across this superb [Architecture Playbook](https://nocomplexity.com/documents/arplaybook/index.html?ref=kislayverma.com) discussing various aspects of building a great software system. I’m still going through it so can’t summarize it yet but it is well thought and better written. A definitely recommended read for any software architects/senior engineers. And it’s an open-source book, so you can contribute too! Kevin Kwok breaks down [Mike Speiser’s style of adding value to his companies](https://kwokchain.com/2020/09/22/the-mike-speiser-incubation-playbook/?ref=kislayverma.com). This is a great strategy breakdown for anyone interested in startups and VC thinking. Kevin writes one of the best strategy newsletter I have read on the internet - subscribe if you haven't already. I came across [this book](http://www.iwritewordsgood.com/apl/set.htm?ref=kislayverma.com) on town planning and architecture somewhere in Twitterland and for some reason I have been very fascinated with it. A lot of people have compared software engineering and city planning - I hope to learn a bit myself. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-6/ Last updated: 1970-01-01T00:17:34.000Z | More than testing - writing unit tests for better design For the longest time I thought unit tests were a good tool for testing (Duh!). But now I think they are greater than that - they are direct feedback on the design of the code. Writing tests helps us write better code. I give some examples and walkthroughs of unit tests informing the design in this article. [Check it out!](https://kislayverma.com/programming/more-than-testing-writing-unit-tests-for-better-design/) To change your subscription, click here. | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #6 URL: https://kislayverma.com/kislay-s-newsletter-6/ Last updated: 1970-01-01T00:14:08.000Z | Kislay's Newsletter #6 View online Happy Friday! Hope all of you are keeping safe and staying sane. I hope another productive week has just gone by, and here’s this week’s assortment of the good stuff to make the weekend a lot of fun. If you are feeling a little stressed out, try looping [Una Mattina](https://www.youtube.com/watch?v=j1Ck42-%5FbtY&ref=kislayverma.com) on headphones. Works every time for me! From the blog This week I published [some arguments against building a “central” platform team](https://kislayverma.com/organizations/a-case-against-platform-teams/). I have written in favour of platformization and platform thinking for about two years now. And yet I am not convinced the “central platform team” model is the best way to go for a company that is trying to adopt platform strategy. An internal platforms team is often so focussed around reuse that they don’t have time to understand the domains they are in. Too often, they are in too many domains building reusable components. What do you think? What has/hasn’t worked out for you? [Article Link](https://kislayverma.com/organizations/a-case-against-platform-teams/) Reading now Fantasy series are an absolute sinkhole. I had planned to get started with the widely recommended [Domain Driven Design (Eric Evans)](https://www.amazon.in/gp/product/B00794TAUG?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00794TAUG&ref=kislayverma.com), but here I am reading the [1634 : The Bavarian Crisis](https://www.amazon.in/gp/product/B00AP91ONI?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00AP91ONI&ref=kislayverma.com), another book in the Ring of Fire series. Some stuff is about to go down in medieval Bavaria, and I need to find out what. From the great interweb Phil Calcado has written a breakdown of how [Meetup is building a layered service architecture](https://philcalcado.com/2018/09/24/services%5Flayers.html?ref=kislayverma.com) as they break down their monolith into microservices. Fascinating and in-depth, whether or not you agree with the choices. Parts of this definitely resonated with [my thinking on layering domains](https://kislayverma.com/software-architecture/layering-domains-and-microservices-using-api-gateways/). Here’s a [basic introduction to concurrency in Erlang](https://www.skcript.com/svr/concurrency-in-the-erlang-vm/?ref=kislayverma.com). It is very basic, but if you are unfamiliar with the Actor model, this is a good place to start. And since Akka and others have brought Actors to other languages, this is useful for more than just Erlang. I revisited this awesome document on [expectation from an Uber tech lead](https://docs.google.com/document/d/10ZK3WTOyO0ywKP6Cv%5Fsz2tPR5tNUf9FgV%5Fzgomu6TxE/edit?ref=kislayverma.com) by Gergely Orosz. This is a great read for everyone playing or aspiring to play the tech lead role. Everything on [Gergely’s blog](https://blog.pragmaticengineer.com/?ref=kislayverma.com) is worth reading at least once, if not twice. That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #5 URL: https://kislayverma.com/kislay-s-newsletter-5/ Last updated: 1970-01-01T00:10:46.000Z | Kislay's Newsletter #5 ![](https://kislayverma.com/content/images/2020/07/kislay-profile-1-600x800.jpeg) View online Happy Friday! Hope all of you are keeping safe and staying sane. I am settling into my new workplace, beginning to understand my project, and almost ready to start bugging everyone about thinking and building platforms :) There’s a lot to do, and some sharp folks to do it with, so life is good. Here’s this week’s roundup of my blog and things worth reading from the internet. From the blog This week I published the second episode of my “[For the layman](https://kislayverma.com/category/for-the-layman/)” series. This time I talk about “[The Cloud](https://kislayverma.com/for-the-layman/for-the-layman-ep-2-what-is-the-cloud/)” in dead simple terms. If you have been hearing about “using the cloud” or “cloud computing” or other similar things and did not know what it meant exactly - you should definitely read this fun article. [Check it out](https://kislayverma.com/for-the-layman/for-the-layman-ep-2-what-is-the-cloud/). Reading now I wasn’t feeling up to reading any heavy duty stuff, so I ended up finishing [1634 : The Galileo Affair](https://www.amazon.in/gp/product/B00AP9426E?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=B00AP9426E&ref=kislayverma.com), the fifth book in the Ring of Fire series. Though not as awesome as [1632](https://www.amazon.in/gp/product/0671319728?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=0671319728&ref=kislayverma.com) and [1633](https://www.amazon.in/gp/product/0743471555?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=0743471555&ref=kislayverma.com), this book breathed some life back into the series for me. The residents of the West Virginian town Grantsville take their mayhem to Venice this time, trying to rescue Galileo and setting up trade partnerships at the same time. Well worth spending a lazy evening or two From the great interweb If you are interested in statistics, [Information Theory for Intelligent People](http://tuvalu.santafe.edu/~simon/it.pdf?ref=kislayverma.com) and [Bayesian Reasoning for Intelligent People](https://wiki.santafe.edu/images/2/2e/Bayesian-Reasoning-for-Intelligent-People-DeDeo.pdf?ref=kislayverma.com) are two excellent articles from the Santa Fe Institute. Each covers the basics of its respective topic briefly but effectively, and forms a good introduction to it. HN has a great discussion around the [information theory article](https://news.ycombinator.com/item?id=20611270&ref=kislayverma.com). I’m a great fan of [asynchronous programming](https://kislayverma.com/tag/asynchronous-programming/), but callback hell has been a real problem in Java land (and in every other language). EA has perhaps solved that problem with their [ea-async](https://github.com/electronicarts/ea-async?ref=kislayverma.com) library which brings async-await to Java. I’m looking forward to doing something with this. Camille Fourier [shares her experiences](https://www.infoq.com/news/2020/08/fournier-internal-platform/?ref=kislayverma.com) in building platform teams and keeping them product/customer centric instead of technology centric. The distinction is critically important for “internal” platform teams which aren’t always exposed to the eventual customers. For some weekend inspiration, here’s a fun guide to [how to teach yourself new things](https://www.jackkinsella.ie/articles/autodidactism?ref=kislayverma.com). That's it for this week folks. Happy weekend! \-Kislay To change your subscription, click here. | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #4 URL: https://kislayverma.com/kislay-s-newsletter-4/ Last updated: 1970-01-01T00:09:31.000Z | Kislay's Newletter #4 View online Happy Friday! Welcome to this week's edition of my newsletter. There's a couple of exciting announcements, the blog roundup, and fun things I found on the internet. **Got a new job!** As [some of you might know](https://twitter.com/kislayverma/status/1292732915797725184?ref=kislayverma.com), I started my new job as a software engineer at [Cure.fit](https://www.cure.fit/?ref=kislayverma.com) this week. The first few days have been a whirlwind of intros and catchups, and I'm loving the feeling of being back in startup mode and ready to hustle alongside some smart team-mates. As a fitness enthusiast myself, I'm excited to be working in the online/offline fitness space. I admit I wasted some hours going over the fitness database to discover some new workouts :) **Got my first patron!** A huge shoutout to [Chrys](https://www.linkedin.com/in/krishna-%E2%80%9Cchrys%E2%80%9D-kattirisetti-b1237476/?ref=kislayverma.com) for becoming my first supporter on Patreon! This really has had me over the moon for the entire week.Chrys has been a strong supporter of my writing through the last few months and this is really the icing on the cake. Thank you Chrys! If you like what I've been writing, you can join her in [showing some love](https://www.patreon.com/bePatron?u=29760145&ref=kislayverma.com). **From the blog** Blogging time was scarce this week because of all the stuff that goes on in joining a new workplace, but I did manage to [publish a little piece](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/) on what Steve Yegge calls the "Golden rule of Platform" (aka Eat your own dogfood). I keep on stressing the importance of this and the concepts on "external programmability" because between these two, they form the essential and necessary criteria for a [platform architecture](https://kislayverma.com/category/platform-thinking/) to arise. The post explicitly lays out what the golden rule is, why it is so critical, and how we can and should apply it. [Check it out](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/). **Reading now** I set up a brand new section on the website called "[Reading now](https://kislayverma.com/reading/)" which I intend to keep updated with what I'm reading, what I have read, and [what I intend to read next](https://kislayverma.com/reading/books-to-read/). This replaces the [google sheet I was maintaining](https://docs.google.com/spreadsheets/d/1z3Kp1Gvg721qEVomPnBqImIEdMOM9LwvV3rDVFWQEac/edit?ref=kislayverma.com#gid=0) earlier. It's a little more effort, but worth it I think. **From the great interweb** [Deterministic Aperture method for client side load balancing](https://blog.twitter.com/engineering/en%5Fus/topics/infrastructure/2019/daperture-load-balancer.html?ref=kislayverma.com) \- A ton of gold in this Twitter article on the specific algorithm and general problems of doing client side load balancing fairly and efficiently. [Managing microservice complexity](https://blog.doit-intl.com/untangling-microservices-or-balancing-complexity-in-distributed-systems-7759987d44b1?ref=kislayverma.com) : This is a deep and interesting take on managing complexity in microservice architectures. This theme continues to become more and more commonplace and popular. [The theory of the firm](https://en.wikipedia.org/wiki/Theory%5Fof%5Fthe%5Ffirm?ref=kislayverma.com) : I was recently introduced to Ronald Coase and his theory of the firm and I must say that rarely have I read so much power packed into so fw words. I especially recommend his essays [*The nature of the Firm*](https://onlinelibrary.wiley.com/doi/epdf/10.1111/j.1468-0335.1937.tb00002.x?ref=kislayverma.com) and [*The Problem of Social Cost*](https://www.law.uchicago.edu/files/file/coase-problem.pdf?ref=kislayverma.com). Trust me that it is worth it spending a weekend reading them. [Smruti Patel](https://twitter.com/smrutirp?ref=kislayverma.com) writes a [superb essay](https://leaddev.com/debugging-engineering-velocity-and-leading-high-performing-teams?ref=kislayverma.com) on measuring and understanding the velocity of an engineering team. Can big social media companies move away from the ad based model? Wired seems to think that [they can](https://www.wired.com/story/big-platforms-could-change-business-models/?ref=kislayverma.com). But Dare Obasanjo [disagrees](https://medium.com/@dareobasanjo/wired-is-wrong-the-problem-with-social-media-isnt-ads-the-real-problem-is-right-there-in-the-5f949ffecd01?ref=kislayverma.com). That's it for this week folks. Happy weekend! \-Kislay [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-5/ Last updated: 1970-01-01T00:09:27.000Z | The Golden Rule of Platforms In the new post, I explain why "Eat your own dogfood" is the Golden Rule of building technical platforms, and how it has far reaching implications when combined with "external programmability". [Check it out!](https://kislayverma.com/platform-thinking/the-golden-rule-of-platforms/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### Kislay's Newsletter #3 URL: https://kislayverma.com/kislay-s-newsletter-3/ Last updated: 1970-01-01T00:09:15.000Z | Kislay's Newsletter #3 View it online Happy Friday! Hope everyone is doing good. Welcome to the third edition of this newsletter with a roundup of the blog and some very interesting things from across the internet. **From the Blog** With the huge uptick in anti-microservice articles all over the internet, I believe we are finally coming out of the hype cycle to understand microservices for what they are - a complicated solution to the complicated problem of autonomy at scale. In [my essay this week](https://kislayverma.com/software-architecture/layering-domains-and-microservices-using-api-gateways/), I explore the use of API Gateways (or other edge technologies) as a means of defining business/technical domains which can encapsulate multiple microservices within them. This type of business aware use of API gateways is not traditional, but it is a very powerful way to evolve the overall architecture as a cooperating set of macro-components. I published my review of "[Thinking in Systems : A Primer](https://kislayverma.com/books/book-review-thinking-in-systems-a-primer/)" by Donella H. Meadows. This is one of the best "first principles" books I have ever read. Not only does it give a basic but solid introduction to the world of systems thinking, but it also provides the philosophical and moral context for reining in our intellectual arrogance because the world is very, very complex by design. If you have ever felt the curiousity to understand how such things as economies and evolution come to be, [this book](https://www.amazon.in/gp/product/1603580557?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1603580557&ref=kislayverma.com) is a great place to start. **From the great Interweb** Uber published a [blog post](https://eng.uber.com/microservice-architecture/?ref=kislayverma.com) about how they are using domain modelling as a way to rein in microservice explosion. The ideas mirror my essay fairly closely (though Uber has a fancy name for it - DOMA), and some parts speak closely to my articles on [extendable data models](https://kislayverma.com/platform-thinking/platform-nuts-bolts-extendable-data-models/) and the [use of rule engines to build exensible platforms](https://kislayverma.com/platform-thinking/platform-nuts-bolts-flexible-decision-making-with-rule-engines/). I finally got around to reading Farnham Street's [guide to reading better](https://fs.blog/reading/?ref=kislayverma.com), and I have to tell you that it is an absolute rabbit hole. They have a very systematic breakdown of how and what to read, perspectives on reading and understanding what is read by many leading thinkers and writers, and tons of book recommendations. Eugene Wei [wrote superbly](https://twitter.com/eugenewei/status/1290629787073908736?ref=kislayverma.com) on how Tiktok won the algorithmic engagement battle. If you weren't following the "TikTok ban in US" drama over the last weeks, you should start now. Not only does it involve some very interesting legal wrangling over how international tech should be regulated, it also features Microsoft's open bribe to the US government. [A peek into the Maproom project](https://medium.com/nightingale/indelible-impressions-why-you-wont-forget-the-map-room-project-d4d4c3158f3c?ref=kislayverma.com) and how better data visualization and collaboration can help us understand our communities and cities better. That's it for this week folks. Happy weekend! \-Kislay [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-4/ Last updated: 1970-01-01T00:09:04.000Z | Book Review : Thinking in Systems - A Primer Here's my review of the absolutely MUST READ introduction to systems and systems thinking by Donella Meadows. If you want to understand how simple actions can have unpredictable results, and why empathy might be a better course of action than limited rationality in a complex world - get your copy now! I have also included the most important excerpts from my reading of the book to give you a taste of what to expect. [Check it out!](https://kislayverma.com/books/book-review-thinking-in-systems-a-primer/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-3-2/ Last updated: 1970-01-01T00:08:19.000Z | Layering domains and microservices using API Gateways We apply some interesting systems thinking principles to "Microservice Hell" and demonstrate how API gateways can be used to group microservices into hierarchical bounded contexts. [Check it out!](https://kislayverma.com/software-architecture/layering-domains-and-microservices-using-api-gateways/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #2 URL: https://kislayverma.com/kislay-s-newsletter-2/ Last updated: 1970-01-01T00:07:55.000Z | Kislay's Newsletter #2 View this email online Hello folks, Welcome to the second edition of this newsletter. Here's a roundup of this verrrry eventful week and some other intersting things I cam eacross. **From the Blog** I published my [review and essential highlights](https://kislayverma.com/books/book-review-atomic-habits/) from ["Atomic Habits" by James Clear](https://www.amazon.in/gp/product/1847941834?ie=UTF8&tag=kislayverma-21&camp=3638&linkCode=xm2&creativeASIN=1847941834&ref=kislayverma.com). I had heard a lot about this book and got the wrong impression that it was a deep dive into the psychological mechanics of habit formation. It is that, to some extent. What it is a lot more is a solid practioner's guide to making and breaking habits. Its simple 4-point framework and plethora of techniques will be really useful to anyone who has struggled to pick up new habits. The message at the heart of it all - "Love the process instead of the outcome". The so-called "two-pizza team" has occupied a central place in technology organizations for some time now. But I argued in [my article this week](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) that we had misinterpreted the original premise of making this kind of team by confusing independence with autonomy. The result - lots and lots of small teams which get nowehere as fast as we would like them to. This article attracted a lot of attention and discussion on [Hacker News](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) and [Twitter](https://twitter.com/search?q=independence%2C%20autonomy%2C%20and%20too%20many%20small%20teams&src=recent%5Fsearch%5Fclick&ref=kislayverma.com), with some big names echoing the sentiment ([Michael Feathers](https://twitter.com/mfeathers/status/1288506277270663168?ref=kislayverma.com), [Shreyas Doshi](https://twitter.com/shreyas/status/1288628852806135808?ref=kislayverma.com), [Chris Richardson](https://twitter.com/crichardson/status/1288898950163038213?ref=kislayverma.com)). The article was on HN front page for about 24 hours and the traffic took down my website twice. More on that next week. **New on the website** A select-and-share option now allows you to select you favourite part of a blog post and share it directly on social media. Go and give it a shot! Social logins now supported for signing in/up to post comments. **From the Great Interweb** If you are feeling a little cooped up and frustrated, what with having to stay locked inside your house all the time, the lovely people of Iceland will [let you scream into their beautiful wilderness](https://lookslikeyouneediceland.com/?ref=kislayverma.com) for catharsis. I went there last year and would recommend it for everyone but for, you know, the 'C' word :( Everyone is going digital full-tilt in this pandemic, including governments. Here's a [peek into the API strategy](https://www.publictechnology.net/articles/features/inside-dwp%E2%80%99s-digital-coronavirus-response-%E2%80%93-apis-reuse-and-micro-services?ref=kislayverma.com) of UK's Department for Work and Pensions. Alex Danco [writes about SPAC](https://danco.substack.com/p/spac-man-begins??ref=kislayverma.com), a kind of reverse IPO where already public money raised by a "promoter" is invested into private company, thereby making it public. Chamath Palihapitiya is among those leading this counter-intuitive new style of tech investments. Thanks it for this week folks. Stay safe! \-Kislay Verma [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-3/ Last updated: 1970-01-01T00:07:05.000Z | Book Review : Atomic Habits I just pubished a review of the bestselling book "Atomic Habits" writtten by famous blogger and Habit-Coach James Clear. The article also includes the most impactful highlights fromt the book. [Check it out on the blog](#) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma/ Last updated: 1970-01-01T00:07:05.000Z | Book Review : Atomic Habits I just pubished a review of the bestselling book "Atomic Habits" writtten by famous blogger and Habit-Coach James Clear. The article also includes the most impactful highlights fromt the book. [Check it out on the blog](#) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### Kislay's Newsletter #1 URL: https://kislayverma.com/kislay-s-newsletter-1-2/ Last updated: 1970-01-01T00:06:51.000Z | Kislay's Newsletter #1 View online Heylow! Hope everyone is doing well and staying safe. I'm very excited to welcome you to the first edition of what I hope to make a weekly newsletter. I'll summarize what I published on the blog, shares interesting things I found on the great interweb, and generally talk about some behind-the-scenes stuff. Say a prayer for regular publication, would you? This was a hectic week. I started a new series of articles called "[For the Layman](https://kislayverma.com/category/for-the-layman/)" which focusses on explaining frequently encountered and often complicated software engineering concepts in as simple words as I can manage. The first episode tries to explain "distributed systems" by imgining what a "distributed" car might look like. It also features hand drawn diagrams by your truly for the first time. [Check it out](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/)! In other big news, I have moved [kislayverma.com](https://kislayverma.com/) from Wix hosting to self-hosted Wordpress. Creating the site on Wix was convenient, but the lock-in this created had been grating on me for a while. Wix has not option for me to export my data, and not having full control on my data is not something I want to live with now. I thought about using Substack, Ghost, and some other platforms, but "owning" everything, for me, eventually meant owning it ground up. So I got a server on Digital Ocean and set up setting up Wordpress. Those of you who follow me on Twitter have probably seen [some tweets around this](https://twitter.com/kislayverma/status/1285291730460241925?ref=kislayverma.com). It has taken 4 days (and counting) to shift all the content, fix all the links, and verify that everything looks fine. I'm still working on fixing email related issues, and setting up Paypal. If you find any broken links or any other problems, just give a holler and I'll fix things right up. In other interesting stuff on the internet, I have been digging into on the technology behind [decentralized identity and data management](https://kislayverma.com/uncategorized/adios-uber-and-ideas-in-the-afterglow/). But more than the tech, the deep questions [this article](https://www.moxytongue.com/2012/02/what-is-sovereign-source-authority.html?ref=kislayverma.com) raises around the very concept of identity gave me pause. *"The act of 'registration' implies that an administration process controlled by Society is required for "identity" to exist. This approach contrives Society as the owner of "identity", and the Individual as the outcome of socio-economic administration."* You should also read this [super interesting article](https://www.lifewithalacrity.com/2016/04/the-path-to-self-soverereign-identity.html?ref=kislayverma.com) which covers a lot of historical ground on how identity management on the internet has evolved and what "self-sovereign identity" may look like. Until next week. Cheers, Kislay [![image](https://kislayverma.com/content/images/2020/07/car-as-a-system-300x221.jpg)](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/) [For the Layman (Ep. 1) – What is a Distributed System?](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/) [![image](https://kislayverma.com/content/images/2020/07/decentralized-web-300x150.png)](https://kislayverma.com/uncategorized/adios-uber-and-ideas-in-the-afterglow/) [Adios Uber, and ideas in the afterglow](https://kislayverma.com/uncategorized/adios-uber-and-ideas-in-the-afterglow/) [![image](https://kislayverma.com/content/images/2020/07/reco-high-level-300x191.jpg)](https://kislayverma.com/software-architecture/a-narrative-approach-to-software-design/) [A Narrative Approach to Software Design](https://kislayverma.com/software-architecture/a-narrative-approach-to-software-design/) [![image](https://kislayverma.com/content/images/2020/07/97-things-every-software-architect-should-know-200x300.jpg)](https://kislayverma.com/books/highlights-97-things-every-software-architect-should-know/) [Book Highlights: 97 things every software architect should know](https://kislayverma.com/books/highlights-97-things-every-software-architect-should-know/) [![image](https://kislayverma.com/content/images/2020/07/systems-thinking-300x155.png)](https://kislayverma.com/programming/design-review-checklist-for-distributed-systems/) [Design review checklist for Distributed Systems](https://kislayverma.com/programming/design-review-checklist-for-distributed-systems/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### Kislay's Newsletter #1 URL: https://kislayverma.com/kislay-s-newsletter-1/ Last updated: 1970-01-01T00:06:51.000Z | Kislay's Newsletter #1 View online Heylow! Hope everyone is doing well and staying safe. I'm very excited to welcome you to the first edition of what I hope to make a weekly newsletter. I'll summarize what I published on the blog, shares interesting things I found on the great interweb, and generally talk about some behind-the-scenes stuff. Say a prayer for regular publication, would you? This was a hectic week. I started a new series of articles called "[For the Layman](https://kislayverma.com/category/for-the-layman/)" which focusses on explaining frequently encountered and often complicated software engineering concepts in as simple words as I can manage. The first episode tries to explain "distributed systems" by imgining what a "distributed" car might look like. It also features hand drawn diagrams by your truly for the first time. [Check it out](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/)! In other big news, I have moved [kislayverma.com](https://kislayverma.com/) from Wix hosting to self-hosted Wordpress. Creating the site on Wix was convenient, but the lock-in this created had been grating on me for a while. Wix has not option for me to export my data, and not having full control on my data is not something I want to live with now. I thought about using Substack, Ghost, and some other platforms, but "owning" everything, for me, eventually meant owning it ground up. So I got a server on Digital Ocean and set up setting up Wordpress. Those of you who follow me on Twitter have probably seen [some tweets around this](https://twitter.com/kislayverma/status/1285291730460241925?ref=kislayverma.com). It has taken 4 days (and counting) to shift all the content, fix all the links, and verify that everything looks fine. I'm still working on fixing email related issues, and setting up Paypal. If you find any broken links or any other problems, just give a holler and I'll fix things right up. In other interesting stuff on the internet, I have been digging into on the technology behind [decentralized identity and data management](https://kislayverma.com/uncategorized/adios-uber-and-ideas-in-the-afterglow/). But more than the tech, the deep questions [this article](https://www.moxytongue.com/2012/02/what-is-sovereign-source-authority.html?ref=kislayverma.com) raises around the very concept of identity gave me pause. *"The act of 'registration' implies that an administration process controlled by Society is required for "identity" to exist. This approach contrives Society as the owner of "identity", and the Individual as the outcome of socio-economic administration."* You should also read this [super interesting article](https://www.lifewithalacrity.com/2016/04/the-path-to-self-soverereign-identity.html?ref=kislayverma.com) which covers a lot of historical ground on how identity management on the internet has evolved and what "self-sovereign identity" may look like. Until next week. Cheers, Kislay [![image](https://kislayverma.com/content/images/2020/07/car-as-a-system-300x221.jpg)](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/) [For the Layman (Ep. 1) – What is a Distributed System?](https://kislayverma.com/for-the-layman/for-the-layman-ep-1-what-is-a-distributed-system/) [![image](https://kislayverma.com/content/images/2020/07/decentralized-web-300x150.png)](https://kislayverma.com/uncategorized/adios-uber-and-ideas-in-the-afterglow/) [Adios Uber, and ideas in the afterglow](https://kislayverma.com/uncategorized/adios-uber-and-ideas-in-the-afterglow/) [![image](https://kislayverma.com/content/images/2020/07/reco-high-level-300x191.jpg)](https://kislayverma.com/software-architecture/a-narrative-approach-to-software-design/) [A Narrative Approach to Software Design](https://kislayverma.com/software-architecture/a-narrative-approach-to-software-design/) [![image](https://kislayverma.com/content/images/2020/07/97-things-every-software-architect-should-know-200x300.jpg)](https://kislayverma.com/books/highlights-97-things-every-software-architect-should-know/) [Book Highlights: 97 things every software architect should know](https://kislayverma.com/books/highlights-97-things-every-software-architect-should-know/) [![image](https://kislayverma.com/content/images/2020/07/systems-thinking-300x155.png)](https://kislayverma.com/programming/design-review-checklist-for-distributed-systems/) [Design review checklist for Distributed Systems](https://kislayverma.com/programming/design-review-checklist-for-distributed-systems/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-2-2/ Last updated: 1970-01-01T00:06:35.000Z | Too many "two-pizza" teams The new post talks about how we pushed the idea of small, independent "two-pizza" teams into its very antithesis by creating too many of them working in too much colaboration. [Check it out on the blog](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ### New post by Kislay Verma URL: https://kislayverma.com/new-post-by-kislay-verma-2/ Last updated: 1970-01-01T00:06:35.000Z | Too many "two-pizza" teams The new post talks about how we pushed the idea of small, independent "two-pizza" teams into its very antithesis by creating too many of them working in too much colaboration. [Check it out on the blog](https://kislayverma.com/organizations/independence-autonomy-and-too-many-small-teams/) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/twitter.png)](https://twitter.com/kislayverma?ref=kislayverma.com) [![](https://kislayverma.com/wp-content/plugins/newsletter/emails/themes/default/images/linkedin.png)](https://www.linkedin.com/in/kislayverma/?ref=kislayverma.com) To change your subscription, click here. | | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |