<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet href="/scripts/pretty-feed-v3.xsl" type="text/xsl"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Blog - Dat Le</title><description>Just a simple blog.</description><link>https://lethanhdat.com/</link><language>en</language><item><title>Pentomino and Sudoku: generate your own puzzles, print on A4 and play</title><link>https://lethanhdat.com/blog/pentomino-sudoku-a4-puzzles/</link><guid isPermaLink="true">https://lethanhdat.com/blog/pentomino-sudoku-a4-puzzles/</guid><description>Two small tools to generate your own tiling puzzles and Sudoku, then print them on A4 paper to play offline.</description><pubDate>Mon, 26 Oct 2026 17:00:00 GMT</pubDate><content:encoded>&lt;p&gt;I have just made two more small games: &lt;strong&gt;Pentomino&lt;/strong&gt; and &lt;strong&gt;Sudoku&lt;/strong&gt;. This time, the part I most want to show off is the &lt;strong&gt;generator: a tool that creates puzzles for you&lt;/strong&gt;, so every time you feel like playing there is a new challenge, and you do not have to hunt for ready-made puzzles.&lt;/p&gt;
&lt;h3 id=&quot;pentomino-create-your-own-tiling-puzzle&quot;&gt;Pentomino: create your own tiling puzzle&lt;/h3&gt;
&lt;p&gt;With the &lt;strong&gt;&lt;a href=&quot;https://tdat00.github.io/pentomino/generator/&quot; rel=&quot;nofollow noopener noreferrer&quot; target=&quot;_blank&quot;&gt;Pentomino Generator&lt;/a&gt;&lt;/strong&gt;, you place pieces so that they cover the whole board without overlapping. You can pick a board shape or block out cells yourself, generate a suitable set of pieces, and then create puzzles at the difficulty you want. There is even an option to require that a puzzle has only one solution.&lt;/p&gt;
&lt;p&gt;The A4 printout has the board and the pieces to cut out: print it, grab a pair of scissors, and you have a paper puzzle set. You can also print the solutions to check against later.&lt;/p&gt;
&lt;img src=&quot;https://lethanhdat.com/images/pentomino-generator-en.gif&quot; alt=&quot;Creating a board, generating puzzles and print options in the Pentomino Generator&quot; class=&quot;zoomable&quot;&gt;
&lt;h3 id=&quot;sudoku-not-just-the-99-grid&quot;&gt;Sudoku: not just the 9×9 grid&lt;/h3&gt;
&lt;p&gt;The &lt;strong&gt;&lt;a href=&quot;https://tdat00.github.io/sudoku/generator/&quot; rel=&quot;nofollow noopener noreferrer&quot; target=&quot;_blank&quot;&gt;Sudoku Generator&lt;/a&gt;&lt;/strong&gt; has 15 variants, from classic Sudoku to Killer, Jigsaw or Samurai, along with many difficulty levels. Another fun detail is that you can replay, step by step, how the machine created a puzzle.&lt;/p&gt;
&lt;p&gt;You can create many puzzles, choose to print 1, 2 or 4 per page, and keep the solutions on a separate page. Print on A4 paper or save as PDF and you instantly have a stack of puzzles to keep.&lt;/p&gt;
&lt;img src=&quot;https://lethanhdat.com/images/sudoku-generator-en.gif&quot; alt=&quot;Generating puzzles, exploring variants and print options in the Sudoku Generator&quot; class=&quot;zoomable&quot;&gt;
&lt;h3 id=&quot;create-on-the-computer-play-on-paper&quot;&gt;Create on the computer, play on paper&lt;/h3&gt;
&lt;p&gt;Both come with English and Vietnamese interfaces and need no account or installation. Each game also has a &lt;strong&gt;player&lt;/strong&gt; so you can play right in the browser, but that is only an extra: &lt;strong&gt;generate, print on A4 and step away from the screen&lt;/strong&gt; is the way I want to recommend this time.&lt;/p&gt;
&lt;p&gt;Try creating a few puzzles, get paper and a pen ready (and a pair of scissors if you pick Pentomino), and play whenever you have a free moment.&lt;/p&gt;</content:encoded></item><item><title>Captain Tsubasa II Password Generator: A Childhood Classic, Rebuilt with AI</title><link>https://lethanhdat.com/blog/captain-tsubasa-ii-password-generator/</link><guid isPermaLink="true">https://lethanhdat.com/blog/captain-tsubasa-ii-password-generator/</guid><description>A modern, browser-based password generator for the NES/Famicom game Captain Tsubasa Vol. II - Super Striker, rebuilt with the help of AI from an existing algorithm.</description><pubDate>Sun, 11 Oct 2026 02:00:00 GMT</pubDate><content:encoded>&lt;img src=&quot;https://lethanhdat.com/images/captain-tsubasa.jpg&quot; alt=&quot;Captain Tsubasa Vol. II - Super Striker title screen and in-game scenes&quot; class=&quot;zoomable&quot;&gt;
&lt;p&gt;If you grew up in the 90s, there is a good chance that &lt;strong&gt;Captain Tsubasa Vol. II: Super Striker&lt;/strong&gt; (キャプテン翼II スーパーストライカー) was part of your childhood. Released for the Famicom/NES by Tecmo in 1990 and based on Yoichi Takahashi’s famous manga, it turned football into a tense, command-driven game: pick a move, spend your guts, and hope your hero lands the Drive Shot or the Cyclone before the goalkeeper stops it.&lt;/p&gt;
&lt;p&gt;For the millennial generation, it was more than a game. It was a cartridge on a borrowed console, a story told in Japanese we could not read, and a friend sitting next to us figuring it out one button at a time.&lt;/p&gt;
&lt;h2 id=&quot;the-password-screen&quot;&gt;The password screen&lt;/h2&gt;
&lt;p&gt;The game has no battery save. To continue later, you write down an &lt;strong&gt;18-character password&lt;/strong&gt; made of Japanese kana and enter it on the input screen. That password stores everything: the round you reached, the level and experience of your players, your special shots, and even Misugi’s guts.&lt;/p&gt;
&lt;p&gt;Back then, a password was either copied from a magazine or lost forever. Today we can generate our own.&lt;/p&gt;
&lt;h2 id=&quot;the-tool&quot;&gt;The tool&lt;/h2&gt;
&lt;p&gt;I built a password generator so you can start from &lt;strong&gt;any round&lt;/strong&gt;, with &lt;strong&gt;any player levels&lt;/strong&gt;:&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://lethanhdat.com/toys/captain-tsubasa-ii-super-striker.html&quot;&gt;&lt;strong&gt;Open the Captain Tsubasa II Password Generator&lt;/strong&gt;&lt;/a&gt;&lt;/p&gt;
&lt;img src=&quot;https://lethanhdat.com/images/captain-tsubasa-password-generator.png&quot; alt=&quot;Captain Tsubasa II password generator&quot; class=&quot;size-75 zoomable&quot;&gt;
&lt;p&gt;What you can do with it:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Pick the round.&lt;/strong&gt; All 33 official rounds are listed with their opponents, from Fluminense in the opening round to Brazil in the World Cup final stage.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tune every player group.&lt;/strong&gt; Set the Level and Exp of each of the ten groups. The players shown follow the current round: São Paulo, Nankatsu or Japan.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Unlock abilities.&lt;/strong&gt; Toggle the Cyclone Shot, the Drive Overhead Shot, and whether Misugi’s guts are lower than 100.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Roll the dice.&lt;/strong&gt; &lt;em&gt;Random match&lt;/em&gt; randomizes the whole setup, with player levels suited to the chosen round. Two alternatives widen the range: unbalanced with each group getting its own random level between the round’s hard and easy levels, plus 3 extra levels on each end, or true random with every level picked anywhere from the round’s lowest to highest sensible level.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Follow along in the game.&lt;/strong&gt; The password updates as you type, and the &lt;em&gt;Game input screen&lt;/em&gt; highlights the exact key to press next. Use &lt;code&gt;&amp;#x3C;&lt;/code&gt; and &lt;code&gt;&gt;&lt;/code&gt; to step through all 18 characters.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Switch language.&lt;/strong&gt; The input screen can be shown in Japanese kana or in the English version’s character set.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Everything runs in your browser, with nothing to install and nothing sent anywhere.&lt;/p&gt;
&lt;h2 id=&quot;rebuilt-with-ai&quot;&gt;Rebuilt with AI&lt;/h2&gt;
&lt;p&gt;I did not invent the algorithm. It comes from the &lt;a href=&quot;https://cah4e3.shedevr.org.ru/cheatsbase_passg.php#249&quot; rel=&quot;nofollow noopener noreferrer&quot; target=&quot;_blank&quot;&gt;password generator by cah4e3&lt;/a&gt;, with the algorithm details credited to Gennadiy_Master. That tool has served the community well, but its interface shows its age.&lt;/p&gt;
&lt;p&gt;With the help of AI, I rebuilt the same algorithm in plain HTML, CSS and JavaScript with a cleaner, more modern interface: steppers and sliders instead of raw fields, a highlighted input keyboard, tooltips that explain each option, and a layout that works on a phone.&lt;/p&gt;
&lt;p&gt;Under the hood, the password packs the levels, experience, round, flags and a 6-bit seed into 18 six-bit values. It then adds a checksum, scrambles the values with a substitution table, and shifts each one by a running offset. The seed has no effect in the game, but changing it gives you a different and equally valid password for the same setup.&lt;/p&gt;
&lt;p&gt;Now grab your emulator (or that old Famiclone), pick your round, and score a few goals for old times’ sake.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Note:&lt;/strong&gt; You can play the game online at &lt;a href=&quot;https://www.retrogames.cc/nes-games/c/index.html&quot; rel=&quot;nofollow noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.retrogames.cc/nes-games/c/index.html&lt;/a&gt;.&lt;/p&gt;
&lt;/blockquote&gt;</content:encoded><category>captain-tsubasa</category><category>retro-games</category><category>ai</category></item><item><title>Three small games: word search, number search and animal search</title><link>https://lethanhdat.com/blog/three-small-games/</link><guid isPermaLink="true">https://lethanhdat.com/blog/three-small-games/</guid><description>Three simple games to play while sharpening your eye, which you can also print on A4 paper.</description><pubDate>Fri, 11 Sep 2026 17:00:00 GMT</pubDate><content:encoded>&lt;p&gt;I made three small games for those moments when you want to play and train your powers of observation at the same time. All three can be printed on A4 paper to play together.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://lethanhdat.com/toys/word-search-puzzle.html?lang=en&quot;&gt;Word Maze&lt;/a&gt;:&lt;/strong&gt; Find the hidden words in a letter grid that you create yourself.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://lethanhdat.com/toys/animal-search-puzzle.html&quot;&gt;Animal Maze&lt;/a&gt;:&lt;/strong&gt; Find sequences of animals in a picture grid, which also suits children who cannot read yet.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href=&quot;https://lethanhdat.com/toys/find-number-puzzle.html&quot;&gt;Number Search&lt;/a&gt;:&lt;/strong&gt; Test your sharp eyes with a shuffled grid of numbers from a range you choose.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Each game has a language switcher (Vietnamese and English) in its page header.&lt;/p&gt;
&lt;img src=&quot;https://lethanhdat.com/images/animal-search-puzzle.gif&quot; alt=&quot;Illustration of the Animal Maze game&quot; class=&quot;zoomable&quot;&gt;</content:encoded></item><item><title>Adaptive Sampling: Cutting Distributed Tracing Data by 99% Without Losing Visibility</title><link>https://lethanhdat.com/blog/adaptive-sampling-distributed-tracing/</link><guid isPermaLink="true">https://lethanhdat.com/blog/adaptive-sampling-distributed-tracing/</guid><description>How we scaled Distributed Tracing across thousands of Microsoft 365 servers without drowning in data, and without sacrificing reliability for Active Monitoring.</description><pubDate>Sat, 02 May 2026 02:00:00 GMT</pubDate><content:encoded>&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;Disclaimer: System names in this post have been changed to avoid disclosing confidential information. All numbers are accurate.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;For the past two years, my team at Microsoft has been integrating Distributed Tracing (DT) into most of the critical microservices in &lt;strong&gt;Substrate&lt;/strong&gt;, the platform that powers Microsoft 365. DT is the tool we reach for when an API misbehaves: it lets us pinpoint the exact node that produced the error, and surface the components that are dragging latency up.&lt;/p&gt;
&lt;p&gt;DT pairs naturally with a system run by a partner team — &lt;strong&gt;Active Monitoring&lt;/strong&gt; — a fleet of synthetic probes that continuously hit internal services and physical servers to make sure everything stays healthy. When a probe fails, a trace is the difference between &lt;em&gt;“something is broken somewhere”&lt;/em&gt; and &lt;em&gt;“this specific downstream call timed out on this specific machine.”&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-scaling-problem&quot;&gt;The scaling problem&lt;/h2&gt;
&lt;p&gt;Microsoft 365 runs on &lt;strong&gt;hundred thousands of physical servers&lt;/strong&gt; spread across the globe. Active Monitoring alone generates roughly &lt;strong&gt;78 million requests per hour&lt;/strong&gt; at that scale. Every request that gets sampled produces a trace, and every trace carries dozens of spans across half a dozen services. The total volume was simply too large for our DT backend to ingest, store, and query at an acceptable cost.&lt;/p&gt;
&lt;p&gt;So we needed to sample. The catch was the constraint Active Monitoring put on us:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;If a synthetic request fails, we &lt;em&gt;must&lt;/em&gt; have its trace. No exceptions.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Lose the trace and the alert is useless — on-call gets paged with no path to root cause. That’s worse than no alert at all.&lt;/p&gt;
&lt;p&gt;The challenge, then, was not “reduce data.” It was: &lt;strong&gt;reduce data aggressively, but never drop an unhealthy request.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&quot;why-the-off-the-shelf-samplers-didnt-fit&quot;&gt;Why the off-the-shelf samplers didn’t fit&lt;/h2&gt;
&lt;p&gt;OpenTelemetry ships with two families of sampling strategies. We looked hard at both.&lt;/p&gt;
&lt;h3 id=&quot;head-based-ratio-sampler&quot;&gt;Head-based: Ratio Sampler&lt;/h3&gt;
&lt;p&gt;Head-based sampling decides whether to keep a trace at the moment it starts, before any spans have been recorded. The most common variant is the &lt;strong&gt;ratio sampler&lt;/strong&gt;: keep N% of traces, drop the rest.&lt;/p&gt;
&lt;p&gt;Pros:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Cheap. The decision is local and instant.&lt;/li&gt;
&lt;li&gt;Predictable data volume. Pick 1%, you get 1%.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Cons (for our use case):&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It’s blind to outcome. An unhealthy request has the same chance of being kept as a healthy one.&lt;/li&gt;
&lt;li&gt;At global scale, the failure rate is &lt;em&gt;tiny&lt;/em&gt; — a fraction of a percent. Any ratio low enough to fit our data budget would silently throw away most of the failures we exist to investigate.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id=&quot;tail-based-sampler&quot;&gt;Tail-based Sampler&lt;/h3&gt;
&lt;p&gt;Tail-based sampling makes the decision &lt;em&gt;after&lt;/em&gt; the trace finishes, when the outcome is known. In theory it’s perfect: keep 100% of failed traces, drop 100% of successful ones.&lt;/p&gt;
&lt;p&gt;In practice, it has two problems that turned out to be deal-breakers for us:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;You have to buffer every span of every trace until the decision is made.&lt;/strong&gt; That buffer is itself an enormous storage and memory cost — you’ve moved the problem rather than solved it. At our volume, the “temporary” buffer would be larger than the backend we were trying to protect.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;You have to know where the trace ends.&lt;/strong&gt; Tail-based sampling needs a “tail” — the last span — to trigger the decision. Microsoft 365 is a sprawling, constantly-evolving graph of services. There is no single canonical tail. Services get added, call graphs shift, async fan-outs make “the end” ambiguous. Any heuristic we picked would rot within a quarter.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id=&quot;the-shape-of-a-better-answer-adaptive-sampling&quot;&gt;The shape of a better answer: adaptive sampling&lt;/h2&gt;
&lt;p&gt;While digging through OpenTelemetry discussions and conference talks, I kept running into the term &lt;strong&gt;adaptive sampling&lt;/strong&gt; — the idea that the sampler’s &lt;em&gt;configuration&lt;/em&gt; itself should change over time in response to what the system is doing. There were hints, sketches, and one-paragraph descriptions, but nothing close to a battle-tested design we could lift.&lt;/p&gt;
&lt;p&gt;So we designed our own. The core insight is small but powerful:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;A head-based ratio sampler is fast and cheap. A tail-based sampler knows about outcomes. What if we kept the head-based sampler, but let &lt;em&gt;outcomes from a few minutes ago&lt;/em&gt; drive its ratio?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;We split the work between two new components:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;AdaptiveSampler&lt;/code&gt;&lt;/strong&gt; is our customized sampler, applied to Active Monitoring. It’s still head-based — the decision happens at the start of the trace, with no buffering. But instead of a single hard-coded ratio, it keeps a live per-server ratio map (&lt;code&gt;[{serverA: 1.0}, {serverB: 0.1}, ...]&lt;/code&gt;) pushed in by the Config System. When a request lands on server X, it looks up X’s ratio and rolls a random number: &lt;strong&gt;SAMPLED&lt;/strong&gt; with that probability, &lt;strong&gt;DROP&lt;/strong&gt; otherwise.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;AdaptiveSamplerWorker&lt;/code&gt;&lt;/strong&gt; is the controller. Every 5 minutes it reads the last 5 minutes of Active Monitoring results from MDM, picks out any server that produced &lt;strong&gt;at least one unhealthy request&lt;/strong&gt; (the &lt;em&gt;hot&lt;/em&gt; servers), and writes a new ratio map to the Config System: &lt;strong&gt;100%&lt;/strong&gt; for hot servers, &lt;strong&gt;10%&lt;/strong&gt; for everyone else.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The Config System then propagates the map to every Active Monitoring server, where each &lt;code&gt;AdaptiveSampler&lt;/code&gt; picks up the change and adjusts behavior on the next request.&lt;/p&gt;
&lt;p&gt;The trade-off is a &lt;strong&gt;time lag&lt;/strong&gt;. If a server fails for the first time at time T, we won’t sample it at 100% until the next worker tick computes the map and the Config System pushes it. The question becomes: &lt;em&gt;how long is that lag, and is it acceptable?&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;why-this-works-at-microsoft-365-existing-infrastructure&quot;&gt;Why this works at Microsoft 365: existing infrastructure&lt;/h2&gt;
&lt;p&gt;The lag question is where the design got lucky. Microsoft 365 already had two pieces of plumbing we could lean on:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Active Monitoring → MDM&lt;/strong&gt; (our metrics store). Probe results land in MDM with a maximum latency of &lt;strong&gt;~2 minutes&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Config System → every server.&lt;/strong&gt; Configuration changes propagate to every server worldwide within &lt;strong&gt;~5 minutes&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Stitching those together:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;An unhealthy request happens at time &lt;strong&gt;T&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Its result is queryable in MDM by &lt;strong&gt;T + 2 min&lt;/strong&gt; at the latest.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;AdaptiveSamplerWorker&lt;/code&gt; reads MDM, computes new ratios, and writes the updated map into the Config System.&lt;/li&gt;
&lt;li&gt;The new config arrives at the affected server by &lt;strong&gt;T + 7 min&lt;/strong&gt; at the latest.&lt;/li&gt;
&lt;li&gt;That server stays at 100% sampling through the next decision window — until &lt;strong&gt;T + 12 min&lt;/strong&gt; at the latest.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So in the worst case, a single failing request gives us a &lt;strong&gt;5–10 minute window of full-fidelity tracing&lt;/strong&gt; on the offending server, starting roughly 7 minutes after the failure. Long enough to capture the &lt;em&gt;next&lt;/em&gt; failure if the issue is ongoing, which is the case that matters for on-call.&lt;/p&gt;
&lt;h2 id=&quot;high-level-design&quot;&gt;High-level design&lt;/h2&gt;
&lt;p&gt;Four components, one loop:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://lethanhdat.com/images/2026-05-02-adaptive-sampling-diagram.svg&quot; alt=&quot;Adaptive sampler control loop: Active Monitoring writes results to MDM, the AdaptiveSamplerWorker reads MDM every 5 minutes, and the Config System pushes new sampling ratios back to Active Monitoring — closing the loop.&quot; class=&quot;zoomable&quot;&gt;&lt;/p&gt;
&lt;p&gt;The only new components are the &lt;strong&gt;&lt;code&gt;AdaptiveSampler&lt;/code&gt;&lt;/strong&gt; (customized sampler, extending the OpenTelemetry base) and the &lt;strong&gt;&lt;code&gt;AdaptiveSamplerWorker&lt;/code&gt;&lt;/strong&gt; (Azure Function, triggered every 5 minutes). MDM, the Config System, and Active Monitoring itself were already in place and maintained by other teams. We didn’t build a sampling platform — we wrote a small custom sampler and a reactive controller that drives it.&lt;/p&gt;
&lt;p&gt;A few notes on the controller that aren’t obvious from the diagram:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Idempotent decisions.&lt;/strong&gt; Each tick recomputes the full set of hot servers from scratch. There’s no incremental state to corrupt; if the worker crashes and restarts, the next tick is correct.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Hysteresis-free, on purpose.&lt;/strong&gt; A server that was hot at T and quiet at T+5 drops back to 10%. We considered a cooldown (“stay hot for two intervals after recovery”) and decided against it — the extra data wasn’t worth the added complexity, and 5 minutes of buffer at the next failure was already enough.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two ratios, not a curve.&lt;/strong&gt; We tried a sliding scale (more failures → higher ratio) and abandoned it. Binary 100/10 is easier to reason about, easier to debug, and the data savings are dominated by the size of the &lt;em&gt;quiet&lt;/em&gt; fleet, not by fine-tuning the noisy one.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;results&quot;&gt;Results&lt;/h2&gt;
&lt;p&gt;Once it was rolled out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;~99% reduction&lt;/strong&gt; in healthy traces ingested by the DT backend.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;0% reduction&lt;/strong&gt; in unhealthy traces captured by Active Monitoring’s window.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In the &lt;strong&gt;Test ring&lt;/strong&gt;, where unhealthy traffic is elevated by design, the savings were modest — there isn’t much healthy traffic for the sampler to drop:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://lethanhdat.com/images/2026-01-21-active-monitoring-tdf.png&quot; alt=&quot;Active Monitoring traffic in the Test ring across a 7-day window. Healthy (blue) and unhealthy (orange) bars stay within the same band; healthy is reduced slightly but most of the volume is preserved.&quot; class=&quot;zoomable&quot;&gt;
&lt;em&gt;Test ring: healthy traces drop only modestly — most of the traffic is unhealthy by design, so there isn’t much for the sampler to throw away.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;In the &lt;strong&gt;Global ring&lt;/strong&gt;, where the fleet is overwhelmingly healthy, the drop is unmistakable. Look at where the rollout lands:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://lethanhdat.com/images/2026-01-21-active-monitoring-ww.png&quot; alt=&quot;Active Monitoring traffic in the Global ring across a 7-day window. Healthy traces fall off a cliff once the AdaptiveSampler is enabled — from roughly 400M down to a small fraction per bar — while unhealthy traces stay roughly constant.&quot; class=&quot;zoomable&quot;&gt;
&lt;em&gt;Global ring: healthy traces fall off a cliff the moment the AdaptiveSampler kicks in, while unhealthy traffic is almost unchanged.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id=&quot;takeaways&quot;&gt;Takeaways&lt;/h2&gt;
&lt;p&gt;A few things I’d pass along to anyone facing a similar problem:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Don’t pick a sampler. Pick a control loop.&lt;/strong&gt; Head-based vs. tail-based is a false dichotomy if you can move sampling configuration in minutes. The interesting design space is &lt;em&gt;what signal drives the configuration change&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Lean on the latency budget you already have.&lt;/strong&gt; Our design only worked because MDM and the Config System were already fast. If they’d been hourly batch systems, this approach falls apart and you’re back to tail-based with all of its problems. Know your existing infrastructure’s clock before designing on top of it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;“Lossy but in the right direction” beats “lossless but unaffordable.”&lt;/strong&gt; We accept losing a single first-occurrence transient failure. We do not accept losing a sustained one. Defining the &lt;em&gt;acceptable&lt;/em&gt; loss precisely was more useful than chasing zero loss.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Adaptive sampling is underdocumented.&lt;/strong&gt; The idea floats around in conference talks and OTel issues, but I couldn’t find a real-world write-up when we started. If you’re designing one, the answer is probably simpler than the literature makes it look — ours is fewer than a few hundred lines of worker code plus a SQL query against MDM.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;If you’re staring at a DT bill that won’t fit your budget and a reliability requirement that won’t bend, this pattern is worth a look.&lt;/p&gt;</content:encoded><category>distributed-tracing</category><category>opentelemetry</category><category>microsoft-365</category></item><item><title>Cheer up.</title><link>https://lethanhdat.com/blog/cheer-up/</link><guid isPermaLink="true">https://lethanhdat.com/blog/cheer-up/</guid><description>He feeds and he curses...</description><pubDate>Tue, 21 Nov 2017 11:45:00 GMT</pubDate><content:encoded>&lt;img src=&quot;https://lethanhdat.com/assets/images/20171121/dota2_feed.jpg&quot; alt=&quot;&quot; class=&quot;size-75 zoomable&quot;&gt;
&lt;blockquote&gt;
&lt;p&gt;He feeds and he curses. It’s always the same: as soon as he has fed, he starts cursing. First he curses mid. What’s the big deal? Two guys went mid anyway. Then he curses the team. Fine too: the team is everyone, yet nobody. Fed up, he curses this whole SEA server. But everyone on the server thinks, “he must mean everyone except me”, and nobody says a word. How infuriating! Yes! This really is infuriating! Infuriating enough to die! So he has to curse whoever isn’t feeding together with him. But nobody reacts either. Damn it! Doesn’t that waste all his try-harding? Doesn’t that make him suffer? Which bastard came up with the low priority queue just to make him suffer like this? Aha. That’s right: he keeps cursing, gritting his teeth as he spits out “bobo putang ina mo IceFrog”. But who is IceFrog? Only Bruno would know. He doesn’t know; not even Steam knows…&lt;/p&gt;
&lt;footer&gt;Anonymous&lt;/footer&gt;
&lt;/blockquote&gt;</content:encoded><category>funny</category><category>dota2</category></item><item><title>Jekyll is impressive.</title><link>https://lethanhdat.com/blog/jekyll-is-impressive/</link><guid isPermaLink="true">https://lethanhdat.com/blog/jekyll-is-impressive/</guid><description>Host a blog on GitHub for free with Jekyll.</description><pubDate>Mon, 20 Nov 2017 11:20:00 GMT</pubDate><content:encoded>&lt;p&gt;With Jekyll, you can host a blog on GitHub for free.&lt;/p&gt;
&lt;p&gt;You can add code block…&lt;/p&gt;
&lt;div class=&quot;astro-code astro-code-themes github-light github-dark&quot; style=&quot;background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;js&quot;&gt;&lt;pre&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;const&lt;/span&gt;&lt;span style=&quot;color:#005CC5;--shiki-dark:#79B8FF&quot;&gt; express&lt;/span&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt; require&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#032F62;--shiki-dark:#9ECBFF&quot;&gt;&apos;express&apos;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;const&lt;/span&gt;&lt;span style=&quot;color:#005CC5;--shiki-dark:#79B8FF&quot;&gt; app&lt;/span&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt; express&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;app.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;get&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#032F62;--shiki-dark:#9ECBFF&quot;&gt;&apos;/&apos;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;, (&lt;/span&gt;&lt;span style=&quot;color:#E36209;--shiki-dark:#FFAB70&quot;&gt;req&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#E36209;--shiki-dark:#FFAB70&quot;&gt;res&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;=&gt;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt; res.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;send&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#032F62;--shiki-dark:#9ECBFF&quot;&gt;&apos;Hello World!&apos;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;app.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;listen&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#005CC5;--shiki-dark:#79B8FF&quot;&gt;3000&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;, () &lt;/span&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;=&gt;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt; console.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;log&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#032F62;--shiki-dark:#9ECBFF&quot;&gt;&apos;Example app listening on port 3000!&apos;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;span class=&quot;language ps-1 pe-3 text-sm bg-muted text-muted-foreground&quot;&gt;js&lt;/span&gt;&lt;button class=&quot;copy text-muted-foreground p-1 box-content border rounded-lg bg-card&quot; aria-label=&quot;Copy code&quot; data-code=&quot;const express = require(&amp;#x27;express&amp;#x27;)
const app = express()

app.get(&amp;#x27;/&amp;#x27;, (req, res) =&gt; res.send(&amp;#x27;Hello World!&amp;#x27;))

app.listen(3000, () =&gt; console.log(&amp;#x27;Example app listening on port 3000!&amp;#x27;))&quot; onclick=&quot;
          navigator.clipboard.writeText(this.dataset.code);
          this.classList.add(&amp;#x27;copied&amp;#x27;);
          setTimeout(() =&gt; this.classList.remove(&amp;#x27;copied&amp;#x27;), 2000)
        &quot;&gt;&lt;div class=&quot;ready&quot;&gt;&lt;svg class=&quot;size-5&quot;&gt;&lt;use href=&quot;https://lethanhdat.com/icons/code.svg#mingcute-clipboard-line&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/div&gt;&lt;div class=&quot;success hidden&quot;&gt;&lt;svg class=&quot;size-5&quot;&gt;&lt;use href=&quot;https://lethanhdat.com/icons/code.svg#mingcute-file-check-line&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/div&gt;&lt;/button&gt;&lt;/div&gt;
&lt;p&gt;You can also attach an image…&lt;/p&gt;
&lt;p&gt;&lt;a href=&quot;https://jekyllrb.com/&quot;&gt;&lt;img src=&quot;https://lethanhdat.com/images/2017-11-20-jekyll.png&quot; alt=&quot;jekyll&quot; class=&quot;thumb&quot;&gt;&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;or embed a clip on Youtube…&lt;/p&gt;
&lt;iframe width=&quot;560&quot; height=&quot;315&quot; src=&quot;https://www.youtube.com/embed/2MsN8gpT6jY&quot; title=&quot;YouTube video&quot; frameborder=&quot;0&quot; allowfullscreen&gt;&lt;/iframe&gt;
&lt;p&gt;Please give it a try :)&lt;/p&gt;
&lt;p&gt;Demo: &lt;a href=&quot;https://github.com/tdat00/blog&quot; rel=&quot;nofollow noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://github.com/tdat00/blog&lt;/a&gt;&lt;/p&gt;</content:encoded><category>jekyll</category><category>github</category><category>free</category></item><item><title>Hello World</title><link>https://lethanhdat.com/blog/hello-world/</link><guid isPermaLink="true">https://lethanhdat.com/blog/hello-world/</guid><description>Just another Hello World.</description><pubDate>Fri, 14 Jul 1989 12:05:00 GMT</pubDate><content:encoded>&lt;p&gt;The boy who cried…&lt;/p&gt;
&lt;img src=&quot;https://lethanhdat.com/assets/images/avatar.png&quot; alt=&quot;the boy who cried&quot; class=&quot;zoomable&quot;&gt;
&lt;p&gt;with code…&lt;/p&gt;
&lt;div class=&quot;astro-code astro-code-themes github-light github-dark collapsed&quot; style=&quot;background-color:#fff;--shiki-dark-bg:#24292e;color:#24292e;--shiki-dark:#e1e4e8; overflow-x: auto;&quot; tabindex=&quot;0&quot; data-language=&quot;csharp&quot;&gt;&lt;pre&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#6A737D;--shiki-dark:#6A737D&quot;&gt;// A Hello World! program in C#.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;using&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt; System&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;namespace&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt; HelloWorld&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;{&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;    class&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt; Hello&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;    {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt;        static&lt;/span&gt;&lt;span style=&quot;color:#D73A49;--shiki-dark:#F97583&quot;&gt; void&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt; Main&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;        {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;            Console.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;WriteLine&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#032F62;--shiki-dark:#9ECBFF&quot;&gt;&quot;Hello World!&quot;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#6A737D;--shiki-dark:#6A737D&quot;&gt;            // Keep the console window open in debug mode.&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;            Console.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;WriteLine&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#032F62;--shiki-dark:#9ECBFF&quot;&gt;&quot;Press any key to exit.&quot;&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;            Console.&lt;/span&gt;&lt;span style=&quot;color:#6F42C1;--shiki-dark:#B392F0&quot;&gt;ReadKey&lt;/span&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;();&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#24292E;--shiki-dark:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;span class=&quot;language ps-1 pe-3 text-sm bg-muted text-muted-foreground&quot;&gt;csharp&lt;/span&gt;&lt;button class=&quot;copy text-muted-foreground p-1 box-content border rounded-lg bg-card&quot; aria-label=&quot;Copy code&quot; data-code=&quot;// A Hello World! program in C#.
using System;
namespace HelloWorld
{
    class Hello
    {
        static void Main()
        {
            Console.WriteLine(&amp;#x22;Hello World!&amp;#x22;);

            // Keep the console window open in debug mode.
            Console.WriteLine(&amp;#x22;Press any key to exit.&amp;#x22;);
            Console.ReadKey();
        }
    }
}&quot; onclick=&quot;
          navigator.clipboard.writeText(this.dataset.code);
          this.classList.add(&amp;#x27;copied&amp;#x27;);
          setTimeout(() =&gt; this.classList.remove(&amp;#x27;copied&amp;#x27;), 2000)
        &quot;&gt;&lt;div class=&quot;ready&quot;&gt;&lt;svg class=&quot;size-5&quot;&gt;&lt;use href=&quot;https://lethanhdat.com/icons/code.svg#mingcute-clipboard-line&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/div&gt;&lt;div class=&quot;success hidden&quot;&gt;&lt;svg class=&quot;size-5&quot;&gt;&lt;use href=&quot;https://lethanhdat.com/icons/code.svg#mingcute-file-check-line&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;/div&gt;&lt;/button&gt;&lt;button class=&quot;collapse-toggle bg-card text-muted-foreground rounded-lg m-2&quot; aria-label=&quot;Toggle collapse code block&quot; onclick=&quot;this.parentElement.classList.toggle(&amp;#x27;collapsed&amp;#x27;)&quot;&gt;&lt;svg class=&quot;size-5&quot;&gt;&lt;use href=&quot;https://lethanhdat.com/icons/code.svg#mingcute-arrow-down-line&quot;&gt;&lt;/use&gt;&lt;/svg&gt;&lt;span class=&quot;desc&quot;&gt; code&lt;/span&gt;&lt;/button&gt;&lt;div class=&quot;collapse-fade&quot;&gt;&lt;/div&gt;&lt;/div&gt;</content:encoded><category>hello-world</category></item></channel></rss>