Connect with me
4
Kochi
There are photos and videos of me from before I can remember. The family archive has been growing along with me. It was pretty easy back then, whenever someone’s phone storage got filled, just move them to the only PC at home. Simple. Later, I switched to Google Photos for almost 2 years. But I did not feel comfortable knowing that all this stuff is sitting there somewhere where someone could view it and lately, train AI on it or whatnot. Which led to me setting up a homelab and self-hosting Ente Photos a year back. Recently on a fine day, checking on why it stopped working, reading through the logs, finding a kernel panic about a dying disk — I found myself in…source: InternetBackups were necessary and I needed a proper set-once-and-forget system. Bought a disk, was going to try ddrescue on the dying disk when I found out that the disk wasn’t actually dead, but it was a SATA cable issue, fixed just by reseating and running fsck for corrupted files. But now I had tasted a bitter disk-dying event, the risk of losing the 230 GB of data that had grown over the years. I had to do something. To trust my own setup I had to ensure two things — reliable uptime and reliable backup. I had to secure myself from the box dying on any given day — given that it is more than a decade old.Taking on the backup problem, after brief research on the topic — data had to have 3 copies. Now that I have a second disk, I could just have a periodic backup to it — like every other day, with a retention of 14 days. Ente backup means object storage backup, two config files and a trivially sized pg_dump. Doable. This deals with 2 out of the 3 copies and I already have a manual SSD archive, so there are three. Easy peasy.But wait, how do I keep periodic copies of a growing 230 gigs? Doesn’t that mean I’d have to copy 230 GB+ every other day?source: meOkay, incremental backups of the MinIO objects. Cool. Great. I can also take a corresponding pg_dump every other day. With Ente, pg_dump is of trivial size, so taking it every other day makes sense and the restore pair is good.But how do I make sure that a photo I deleted today won’t corrupt my past backups? It’s incremental, right? So I need to preserve states too. But wait, how do I do incremental backups again?I could just use hard links! They bypass all these — when I take an incremental backup, I only make new hard links to the same objects in the previous backup. On deletions, I just do not make hard links to them. Perfect. State preserved, incremental covered.But what happens when the first backup gets deleted after the retention period? Won’t that break every other backup?Well, that’s not how hard links work. The issue happens when we think of files as their file names. It clears out when we see them as inodes which can have multiple hard links. Just like the same person can be known by multiple names/nicknames. An inode is only deleted when all the hard links to it get deleted, so the retention just deletes one link, but it does not delete the file, since all the other backups still hold hard links to it.Write a script and set up a cron. But the script should run only when the second disk is mounted. I also need to get logs about the backup. How do I make sure cron does not skip a timer because the box was off during the time? Solution is to set up running the script as a systemd service with a timer trigger — journalctl for the logs, and the timer (Persistent=true) handles the catchup backups too. Perfect. I wrote the script with rsync --link-dest for the hard linked backup system and set up the timer. Manually triggered it once. Worked perfectly. Done.But how do I verify that at the time of need, I do not get any surprises and that I can actually restore from my backup?A backup that can’t be restored defeats the whole idea.Also I need something simple enough to run under the stress of a failed drive. Well, testing this backup with actual restoration steps would mean I’d have to copy the data back, bring prod down and all that slow mechanics which are hard to test and undo. If I can directly mount the object storage from the backup, that works, pg_dump is restore anyway. But MinIO booting up means it will write to the backup, which corrupts it. Which means I do not want to make a duplicate copy nor corrupt the backup while testing it.Here comes the interesting part. Linux has this file system called overlayfs. It can cleverly map out the data from the backup and present it to MinIO, allowing it to read, write and delete from it, without copying or modifying the original. Sounds crazy, right? How does that work without data duplication? I found this analogy on the internet that makes it click. Think of it as a glass sheet on a paper. The backup is the paper — the lower dir, mounted read-only. The glass sheet is the upper dir. MinIO reads through the glass and sees the paper. Writing is just writing on top of the glass, deletion is just whiteout — blocking the underlying content with something opaque on the glass. You lift the glass, your original paper stays pristine.Perfect for our use case, and when it’s done working, I can just tear down the temporary containers and the upper dir. Tested against the overlay — the stack came up, the photos loaded, and deleting/updating photos did not affect the lower dir, verified with checksums. The script also includes an actual recover mode. So if your main disk dies, you can just attach a new one, run it in recover mode and it brings your whole setup back in one single command. Tested with docker compose down -v wiping the live stack and its volumes and brought everything back from the backup with just running the script. Photos loaded, library intact. Read more about it from notes in the repo.This brings us to the count — two hot copies, one cold archive on the SSD. Well, this does not squarely obey the 3–2–1 rule because the SSD lives under the same roof. But being offline and away from the box, it’s safe from power surge, ransomware or an accidental rm. The cold archive is manual, so its protection is only as good as its staleness. The fix is boring: a calendar reminder to refresh the SSD. Also works as a reminder to monitor the backup system, once in every retention period.Now about the second half, uptime. First, mapping out the issues.Auto reboot after power outages — this was my main issue, because when I started out self-hosting, it was in the office and I had to be safe against disk theft, which meant encryption with the tradeoff being no auto reboot. Now that I moved the homelab home, I got LUKS removed. Then, the system rebooted to a “BIOS was reset” message every time — classic CMOS battery issue, so I got it replaced. Set the BIOS power-loss behaviour to always-on, fixing the auto reboot issues.Network — the system had an external dongle attached, which was unreliable at times. Sometimes, I’d get kernel logs in the terminal for dropped packets from the dongle, so I switched to ethernet with failover to the wifi dongle.Disk mount — Auto reboot should mount the two disks correctly. A reboot should not flip sda and sdb. Edited the fstab to be UUID-based so that the mounts stay put and the backups keep happening. Well, all of these, along with the dedicated UPS on the way, solve the uptime issues for the setup.That brings the homelab revamp to an end — you can use the full scripts from here if you are self-hosting Ente.BonusNow if you’re curious, why not RAID 1 instead of this setup? RAID means if one disk dies, I can just keep it up and replace it. Saves you from a re-setup. Well, the tradeoff being: anything that corrupts the data, like a badly timed power outage and both my disks die together, leaving me nowhere to go. But here, I always have a previous backup to fall back to.Now, for the careful readers thinking — the stack stays up during the backup and pg_dump and MinIO rsync happen over a window, so if an object gets uploaded after the pg_dump and during the MinIO rsync, there are chances of mismatch, right? That is why the ordering is deliberate — pg_dump runs before MinIO rsync. Now that upload will just be an orphan object which will be taken up in the next backup cycle. But deletion is a real corruption because the DB will be pointing to an object MinIO does not hold. But my entire user base is for sure asleep at 3 am when this happens, so I kept the script simple rather than taking the stack down during the backup for a highly unlikely race condition.For those who have space on their daily driver laptops and want to skip buying a second drive, Ente supports ente-cli for export. Their desktop app also has a setting to keep a synced offline copy of your library. I was using this feature for the past year. But this may not be sustainable, as space might be a real constraint on daily drivers.Edit: expanded the race-condition, restore-test, and 3–2–1 sections based on reader feedback.How I verify my Ente Photos backups without copying 230 GB was originally published in WriteaByte on Medium, where people are continuing the conversation by highlighting and responding to this story.
When I joined my first company, we were at a stage where the MVP was successful and we were trying to aggressively scale. The existing codebase was messy and highly coupled, but it had served its function of verifying product-market fit. Now the system had to handle scale, and that’s where I started my work at. The part I owned was the latency of a CPU-intensive API that took 9 long seconds to calculate and respond, measured with Locust. Cutting that down was necessary if we wanted to serve more customers with ease.Diving into the codebase, we wanted to start optimizing everything on the go as we read it, but we had to resist the urge and start measuring instead. Rather than fixing random things we noticed while going through the code, the better approach was to profile the functions, sort them by time taken, and fix the longest ones first. And that’s exactly what we did. The first step was to identify the main functions and profile them for computation time. I wrote a small decorator for it (a lifesaver, instead of having to edit every function definition). The timings were pasted into an Excel sheet to compare across different input cases.import timeimport functoolsdef time_logger(func): @functools.wraps(func) def wrapper(*args, **kwargs): start = time.perf_counter() result = func(*args, **kwargs) elapsed_ms = (time.perf_counter() - start) * 1000 logger.info("[TIMING] %s: %.1f ms", func.__name__, elapsed_ms) return result return wrapper@time_loggerdef calculate_something_function(...): ...A few things became clear. Some function calls could be parallelized. Many functions had external dependency calls to other microservices to fetch data required for the price calculations. Some of those external calls had high chances of being called repeatedly.Now that the numbers lay in front of us, we picked the highest on the list — the CPU-intensive tasks were being done sequentially. This could be parallelized. Looking into the functions, data dependencies from external microservices were intertwined, and each API call was implemented with a different library in a slightly different way. Changes in one place kept interfering with another. So we refactored the codebase to clean architecture: interfaces for service functions, repository classes for DB operations, proper dependency injection, and an interface for the external client. With clear separation of concerns in place, we started fixing things one by one.The algorithm calculated top-K sets and re-ranked them to choose the best according to the calculation results. This meant the CPU-intensive calculation could be split into independent parallel chunks. Threads would not be the solution, because CPython’s Global Interpreter Lock (GIL) lets only one thread execute Python bytecode at a time. All threads give us is concurrency and the feeling of parallelism, not true parallelism for CPU-bound work. This was solved with ProcessPoolExecutor. It meant serialization overhead and more memory usage, but on the flip side, it cut the time down to under 4 seconds!Next, we moved on to the external API calls. On analysing them, a few observations stood out. The same microservices were being called multiple times with different request bodies. Some of the data wasn’t real-time critical and could safely lag by a few hours, which made it a caching candidate. Some calls made via the requests library needed to be migrated to an async client like httpx. And all of these calls were being made sequentially. There was a lot that could be done. The first step was to standardise the API calls: a base class with a make_request method that held the global settings for all external calls. During the refactoring, we had already grouped the data aggregation outside the main calculation block. So instead of calling them one after another, we fired them all together asynchronously and waited with asyncio.gather. Time went from the sum of all calls to the time taken by the slowest call. We saved over a second here.Now, about the same microservice getting hit multiple times? Every call paid the full TCP + TLS handshake cost. This was an overhead that could be eliminated with HTTP connection pooling. By instantiating a single global httpx.AsyncClient at the application level rather than per-call, multiple API calls could now be made over the same connection, and the handshake was skipped for subsequent calls. This saved us around 100ms.We introduced caching too, with Redis, for DB lookup tables and external API responses. Calls that took 300–400ms were now resolved from cache in a few milliseconds, and repeated calls often didn’t fire at all. This meant analysing which data could be cached and which couldn’t, and a proper cache invalidation strategy — TTL and active invalidation where applicable was made and documented. It was worth the effort: on requests with a warm cache, this alone brought the response time down by close to a second.While we were at it, we hardened the API as well. It depended on a lot of external calls, and it would fail if any of them failed. This wasn’t part of speeding things up but of making them reliable, and since the refactoring had given us a single external client base class, it was a matter of editing one place to add retries with exponential backoff. We also added jitter to the retries to avoid the thundering herd problem, where synchronized retries stampede a recovering service. Refactoring the code earlier paid off here too.The API now reliably performs at around 3 seconds on cold calls and under 2 seconds with the cache populated. And since the refactoring improved the structure of the entire microservice, every other API in it benefited with lower latency too. Had we gone ahead and fixed things as we noticed them, we wouldn’t have ended up with the reliable and faster service we have now. Profiling and analysis upfront is what made the path clear — where to focus, what to rewrite, where to apply the fixes. It saved us from an endless loop of re-refactoring and rewarded us with huge savings in latency.So the next time you feel the urge to rewrite something, wait. Measure. Profile. The numbers will tell you exactly what to prioritise.How we took a CPU-heavy API from 9 seconds to under 2. was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.
The Ottamind trend went viral because everyone hates the trip planning process. While it’s a real pain to get your friends aligned on a trip, it’s even more painful to plan one properly — half the search results will be closed or outside your trip timings, and GPT will suggest visiting Anamudi Peak, which was never even open to the public.chatGPT paranj Gmaps vazhi kaanicha anamudi closed gatil nokki nilkunna kootukaranThe trip was about the bike ride anyway, so we set off to ride the Munnar Gap Road to Idukki Dam. The bike ride was heavenly! We reached the dam at 5:15 pm, because Google Maps said it closes at 6 pm. The security chettan welcomed us with a smile and said entry is blocked after 5 pm. The site is open till 6, but that’s for the people who got in before 5 to get out. Getting the itinerary planned was not fun, and when validations fail like these, you decide to do something about it.Suggesting validated destinations that fit the vibe of the travelling group, the mode of travel, budget, and duration, then arranging them in an optimized order considering opening and closing times and the vibe of each destination became the problem statement.The first approach was a ranking algorithm with weighted scores for the suggestions. This did not end well, because weights can’t reason about context. A waterfall scores the same on a rainy day. I then started treating it as a CSP (Constraint Satisfaction Problem). This didn’t go well either, because the constraints were open-ended and dynamic. You cannot formulate bike-friendliness, holiday crowd risk, and trip vibes into a clean set of constraints.Well, if you keep knocking on doors, someone’s gonna open up! If I could make a workflow that follows the same steps I would take while planning a trip, with a validation step after each one, it might work. Since the problem is open-ended, explicitly programming each validation would not work. I needed something that could generically “think” through each step. Yeah, LLMs could do it. I tinkered around the idea and found the LangGraph library, which lets you build FSMs (Finite State Machines) with LLM calls in between to do the thinking per node. Perfect for my use case. I went through the docs, noted down the steps required, and planned out the graph and states. Now it was time to touch code.Picnix’s LangGraph state graphI felt comfortable using AI to code since there was a crystal-clear plan for what to build and little room to hallucinate. I chose Streamlit for the UI to test the application and built the 8-node graph in a couple of days. A few things were off, and they pointed to a major flaw in my build process: I was looking at just the logs and running blind on the LLM call requests and responses. Observability became the bottleneck for debugging. I integrated Arize Phoenix tracing and fixed the issues — dwell times, route order optimization, suggestion relevance, and so on. For curious users, I added a Trip Auditor meta-agent they can chat with to walk through the steps the LLM took to arrive at the plan.Now it was time to ship. I added a user management module, dockerised it, and put it into prod on GCP. It’s available at picnix.aswinpradeepc.com and will stay up until the free credits run out.Now comes the interesting part — user feedback came in (friends and fam ofc). Most of them had a slightly different use case. I carefully noted down all the feedback, analysed it, and sat down to make the fixes. But generalising for each piece of feedback meant breaking some other flow. The 8-node graph was too rigid. I remembered the opening of Anthropic’s blog post on building effective agents, which differentiates workflows from agents. This pointed to a flaw in the initial thought process, which was to build the graph for my current case and add more branches until it generalised. The workflow choice works well for the current trip scenario, but to generalise it, adding branches is not the best solution. An agentic node with autonomous tool-calling would be. In the meantime, Google shipped a feature — Gemini integrated Google Maps, and trip planning showed up in the Gmaps app itself. Well, this meant two things. First, I can take up a different challenge instead of reinventing the wheel. Second, it comes as validation for the problem space, as well as for some of my learnings from the Picnix journey.So Picnix is riding into the sunset after doing what it was build for, it planned our trips, it taught me LangGraph, tracing, and engraved the difference between a workflow and an agent. The app stays up until the credits run out, and the code stays public on GitHub. If you’re building anything with LLMs, take the two lessons: design your states before you touch code, and never trust an LLM call you haven’t traced.And if a planner ever tells you the damn dam is open till 6, ask it who gets to enter after 5.Why rule-based trip planners fail (and what I built with LangGraph instead) was originally published in WriteaByte on Medium, where people are continuing the conversation by highlighting and responding to this story.
