Quick background

I’m a French dev who finished my studies, and I build random projects to keep sharpening my skills while I look for a job. As usual, my gf had some kind of issue and as a dev, I can cook something up to help. I already did something similar with Better Maps, a webapp that uses the Google Maps API to display custom pinpoints on top of the map. I created it because we were moving in together to a city neither of us knew well.

What I built (and how)

This time, she wanted a way to manage her finances. So far, she’d been doing it in a Google Sheet, which isn’t a bad idea (I mean, accounting is one of the main reason spreadsheets exist, right ?) but isn’t very practical, and for a few reasons :

• Overkill : Google Sheets has a load of features that she’ll never use
• Not mobile-friendly : Although the Sheets app on Android is sleek and works well (props to the Google engineers for that 🙌), it isn’t as good as on desktop. And like with Better Maps, she was also going to use it on her phone, so making the webapp responsive was a top priority
• No stats, no analysis : Sure, you can create formulas to compute what you need and display graphs, you have to set it all up manually, update them when needed, and duplicate all that work for every month’s sheet

So naturally, she asked me to do the same kind of thing I did with Better Maps. Fortunately, that meant that I could reuse the codebase ☺️ (which I’d already partly reused back then from my website and a school project), do a few minor tweaks to host the data, and be done with it.
At least, that’s what I thought would happen…

Wanna see a demo ?

If you want to skip the ramble, you can check the app at spendly.edm115.dev. There’s a link to a fully fledged demo with sample data so you can see what’s possible, along with a complete landing page (Google OAuth validation team said it wasn’t complete enough… so I gave them all the details !). The app is public and ready to use, feel free to create an account, try it and give me feedback :)
The source code is available at EDM115/Spendly.

What I initially planned to do

The webapp is built with Nuxt, a Vue.js full-stack framework that solves a lot of pain points with a batteries-included approach (SSR, API, routing, crazy good DX, …).
Personal preference, but I prefer Vue over React, I feel like I can ship faster with it, and AI models give much more consistent results thanks to the way the ecosystem is structured (wanna do X ? here’s the Y official solution that everyone uses and that’s well documented !).
The app is built and bundled in a Docker container, and it’s linked to a volume containing an SQLite DB, because I couldn’t be bothered to use either a cloud DB or run a separate MySQL server.
So 3 months ago, I ported the code from Better Maps over to Spendly… and then didn’t do anything for a month because I had other projects to work on.
On December 2nd, 2025, I finally decided to sketch how the app would look, what the features would be (and to calm down my gf’s unrealistic expectations lol), and… once again, other projects took priority.
December 29th was the day where I finally locked in and created the first “working-ish” version of Spendly.
Over January, I improved the app a bit, added a landing page (Better Maps didn’t had one and threw you straight into the login page because it was only meant for private usage, the Google Maps API ain’t free 😭 but here I wanted to make it public so landing page it is !), refined the visual style (I struggled quite some time with this 😅), some perf improvements, …

What does the app even do anyway ?

With Spendly, you can create “budgets”. They can represent an actual bank account where you track your expenses, a trip you want to plan, a project with strict budgeting, …
Each budget has its own “categories”, with a color and an icon, so you can (you guessed it) categorize expenses.
Then you simply add the transactions (expense or income), their amount, associated category and date.
You can then freely sort and search for them, filter by any date range, and most importantly get 4 pre-made charts to analyze the data (with a fully customizable time range as well) :

• Evolution : The balance over time. See how expenses/income/balance evolves over time
• Repartition by category: Check which category represents most of your expenses/income
• Comparison : Review whether you spent more money than you earned
• Distribution : The cashflow overview. Check the percentage of savings/deficit you have compared to income/expenses

You can also export your data at any time :

• Transactions : CSV/JSON
• Charts : SVG/PNG/PDF

Each budget can be shared with any number of other users, with roles (kinda like Google Drive) : viewer, editor, admin, and owner.
Finally, the app has desktop and mobile layouts, a light & dark theme, and is available in both English and French.

How the GitHub Copilot CLI helped massively (and got me carried away)

So far, Spendly was developed in VS Code like usual, with the help of Copilot (I’ve been a very early adopter, part of the Technical Preview on April 2nd 2022, Copilot in the CLI on March 22nd 2023 and Copilot Chat private beta on May 26th 2023).
On January 22nd, a friend told me about this challenge. I figured it was a good time to finally learn how to use the Copilot CLI, since I hadn’t really tried any TUI-based AI tool. Time to catch up, I guess !
And boy, did I ship.
Before that date, the project had +20 126 LoC since inception, tho to be real it’s closer to +7 468 / -2 062 LoC if you remove the Better Maps base, and that in 2 months.
In comparison, in just 24 days, there were +29 595 / -5 143 LoC, and that’s not just lockfile changes 😉 A lot of major features landed that I wouldn’t have been able to implement in time without AI.
To give you an overview of what shipped since I started using the Copilot CLI :

• Better on mobile : Virtualization, cards instead of a table, simplified charts, …
• Demo : Add a complete set of demo data in 2 languages, covering 17 believable categories and 150 transactions over the course of 10 months
• Icon search : Rather than expecting users to know what MDI icons are, they can just search for what they need (with support for categories and aliases), yes, even baguette 🥖
• Proper charts exports : This was a pain point. Exports were low-res on image, and the SVG just embedded the PNG… Now it generates a proper SVG, and renders ultra high-quality images from that same SVG !
• Database change (Drizzle ORM) : Instead of raw SQL queries everywhere, we now have an ORM, migrations, easier queries, and more. This finally lets me do core changes to the app without resetting the DB lol. This change was massive and instrumental for the next change :
• Auth change (Better Auth) : The auth system used to be hand-rolled (bad idea, I know) : username/password login, JWT sessions and some requests validation. That was it. Now, thanks to this (massive too) change, we have proper username/email auth, OAuth (Google/GitHub), proper session management (no more random disconnects), proper admin management, captcha support (Cloudflare Turnstile), … and most importantly : the ability to signup !
• Emails (Resend) : We can now send emails, for example on forgotten password, or even for admin actions (an user requests their data or want to delete their account)
• Account page : Since we have a proper auth, we can allow users to manage their account in a centralized place. It’s also the place where they can donate if they want to 🫶
• Admin rework : After the DB & Auth changes it was broken, but Copilot restored it and improved it with a centralized place to handle user requests
• PWA : Finally, my gf wanted to “install” the app on her phone, so a PWA was the best choice. Copilot helped tremendously to fix some pesky install quirks and cache invalidation issues !

Models used

I love tinkering with models to find out what they’re good at (I might even have a blog post coming that compares all of them against a deceptively hard prompt… 🫣). Overall I used 3 models for Spendly :

• 🥇 OpenAI’s GPT 5.2-Codex : The best one overall. Versatile, eager to tackle hard tasks, cheap, large context window, loves reading docs like me, reasons well and follows instructions very closely.
• 🥈 Anthropic’s Claude Opus 4.5 : Although I got hit with the x3 multiplier 🥲 this model is really powerful. I mostly used it as a scaffolder for Codex : generate plans, implementation guidelines, the AI guidance, potential perf improvements, … then handed off to Codex for the actual code changes.
• 🥉 Google’s Gemini 3 Pro : I didn’t used it a lot, only for UI changes (pick a style, refactor all components to match it, unify everything and make it look good on mobile). I found it to be better at UI/UX overall.

What helped along the way

• AGENTS.md : I added this pretty late to the project, but having a centralized file that guides AI agents is a must. You can define your own rules, document the codebase so the LLM doesn’t waste tokens exploring blindly, note gotchas or unique bits of the project, … I already had some experience writing one for my website, you can find Spendly’s one here (they all had been largely inspired by Oxc’s AI guidelines).
• Skills : Same story, I added them quite late. At first I didn’t bought into the hype, because I thought (like MCPs) they’d eventually get used to fix problems they weren’t meant for. But HOLY was I wrong. With a few simple skills (libraries, tool calling, search, brainstorming, …) I got crazy good results with much less prompting ! Also thanks to the Copilot CLI team for supporting the ~/.agents standard, so I don’t have duplicated files everywhere 🙏

An example : wide event logs

This is the perfect example of a feature that I didn’t need, but got carried away because Copilot is too good 🐐
So the app basically got no logs from the start (as Better Maps didn’t either, since it was targeted at just me, my gf and my mom). But if I wanted the app to be public, I needed some insight, especially for failures or slow aah requests.
Weeks prior, I read the excellent Logging Sucks article by Boris Tane, and it described exactly what I needed : wide event logs.
Fortunately, he also provided a skill that made implementing it much easier.
Here’s how I proceeded :

1. Prepare. I used ChatGPT’s website with 5.2 Thinking (extended) & web search to review the website/skill, then explained my webapp and asked it to generate a complete prompt to hand off to an AI agent. I could’ve used the Plan mode inside the Copilot CLI (and I did for other complex tasks), but here I wanted sources checked first—and ChatGPT’s web search is great for that.
2. Implement. GPT 5.2-Codex handled that. Thanks to the complex but detailed prompt + skills + tools, it was able to one-shot it (1 750 LoC changes across 32 files), all while staying under the context window !
3. Refine. Although it worked perfectly, I asked later to add guidance in the AGENTS.md, add implementation notes for features that weren’t ready yet (like emails), … Small touches on a working product.
4. Over-obsess. This is the part where I got carried away… 😅 After seeing how well it went, I asked for completely out-of-scope and unnecessary tools to accompany it. First a CLI tool to parse the logs and extract insights. And when it did make it in only 2 back-and-forth prompts, I turned the difficulty up and asked it to make a whole-ass TUI to dynamically browse logs and view metrics. Will I ever use it ? Probably not, but boy was it fun !

My overall thoughts about AI usage in this project

I’ve used Copilot since the start of this project, just like with Better Maps before it, which let me move fast, iterate quickly, ship often, and explore ideas I wouldn’t bother to consider otherwise.
Switching to Copilot CLI was a breath of fresh air. Even though Copilot is very well integrated into VS Code, I had 2 big issues :

• Performance : VS Code is an Electron-based app. On my potato laptop, running Chrome + Discord + Spotify + VS Code + a dev server with HMR reloading on every keystroke doesn’t help. Dropping one Electron app can make a real difference, and Copilot CLI is very well optimized, low on resources and quite snappy !
• Code-first approach : When I’m in VS Code, I’m a lot tempted to watch changes live and obsess over them. In a TUI, I only checked the diff and intervened when necessary. That more laid-back approach helped me relax a bit and stress less (though I wouldn’t necessarily do this on other projects, here the result matters more than code quality).

Now, was Copilot (or, the 3 models I used) perfect ? No.
I had some pain points that even prompting, reference files, documentation links, skills and AGENTS.md couldn’t solve. For instance, the PNG generated from charts was around a megabyte. The PDF generated from the same chart (which just embeds that PNG) was… 77 Mb ! Copilot tried all sorts of techniques to reduce it, while all it needed was to enable 2 config flags in the library’s config (that I found by reading the docs myself).
So I still had to do some cleanup passes after AI edits, and I also enjoyed building some parts entirely by hand, but Copilot handled the heavy lifting and boilerplate.

App screenshots

The homepage

Budget selector

Transactions list

Categories

Evolution chart

Repartition chart

Comparison chart

Distribution chart

Login page

Account page

Light theme

Transactions on mobile

The overkill Logs viewer TUI

This blog post will detail the 5 stages of grief levels I went through when trying to optimize a function for a CLI tool I built (monorepo-hash).

Is the title clickbait ? If you think so, it means that it worked anyway :)

Yet another CLI tool ?

Yeah, oopsies… 😅
I made this tool during my last internship, more details can be found in its README.
All that you should know is :

1. It generates hashes for the different workspaces of your monorepo, with support for internal transitive dependencies
2. It was just a quick script written to solve an issue but I wanted to turn it into its own thing
3. It was more of an excuse for me to make a CLI tool, mess with some interesting new tech (rolldown, bun, …) and try to see how far I could optimize it (with benchmarks backing it up)

As such, it needs to be able to process potentially lots of files, fast enough so it’s not visible to the user (ex when used as a pre-commit hook).
Implementing it in TypeScript is… debatable if performance is the whole point, but that’s another fight 🤭.

Our focus today

Since the code spans nearly 1k LoC (excluding comments), we won’t cover everything here. We will only focus on the code that computes per-file hashes and returns them.
There are some subtleties tho, that won’t change between each implementation :

1. The returned paths should be POSIX-style. We use a custom function for this that’s been simplified here (we have a cache in the actual script)
2. It’s async and exported for programmatic usage
3. We initialize the returned record to Object.create(null) instead of {} to skip proto initialization
4. We exit early if the list is empty

The shell of that function looks like this :

import { sep } from "node:path"

/**
 * Normalize a path for display purposes (always POSIX-style separators)
 * @param p The path to normalize
 * @returns The normalized path
 */
export function displayPath(p: string): string {
  return sep === "/"
    ? p
    : p.replace(/\\/g, "/")
}

/**
 * For a given `dir` and list of relative file paths (`fileList`), compute per-file SHA-256 on (normalizedPath + rawContent)
 * Always returns a map : { "posix/rel/path": "hex" }
 * @param dir The absolute path to the directory containing the files
 * @param fileList An array of relative file paths within the directory
 * @returns A promise that resolves to a record mapping POSIX relative paths to their SHA-256 hex hashes
 */
export async function computePerFileHashes(
  dir: string,
  fileList: string[],
): Promise<Record<string, string>> {
  const result: Record<string, string> = Object.create(null)

  if (fileList.length === 0) {
    return result
  }

  // process...

  return result
}

Level 1 : Sequential processing (new grad)

Pretty straightforward, just iterate over the files and compute their hashes bro :

import { createHash } from "node:crypto"
import { readFile } from "node:fs/promises"
import { join } from "node:path"

export async function computePerFileHashes(
  dir: string,
  fileList: string[],
): Promise<Record<string, string>> {
  const result: Record<string, string> = Object.create(null)

  if (fileList.length === 0) {
    return result
  }

  for (const file of fileList) {
    const norm = displayPath(file)
    const fullPath = join(dir, file)
    const content = await readFile(fullPath)
    const fileHash = createHash("sha256")
      .update(norm)
      .update(content)
      .digest("hex")

    result[norm] = fileHash
  }

  return result
}

Indeed, very simple. But you can already smell the issue : it’s slow. Very slow 🐌.
The more bytes you have to read & hash, the longer it takes (roughly $\mathcal{O}(totalBytesRead)$ + some overhead).

Level 2 : Streaming like we’re Netflix (the vibe coder)

A new foe has entered the chat !
The vibe coder ran the program and it was slow as shit, so he asked his buddy Claude Code to “make it faster, no mistakes plz ! 🥺”.
And this is where the false good idea comes in : streaming the files.
You see, one bottleneck here could be that we’re waiting for the file to be entirely loaded into memory before we can compute its hash. And if there’s a massive file to process, loading it from disk into memory could take some time, slowing the program down. In that case, streaming can absolutely help with memory and responsiveness, Node.js streams are literally built for chunked processing.
So here’s what the vibe coder Claudy came up with :

import { createHash } from "node:crypto"
import { createReadStream } from "node:fs"
import { join } from "node:path"

export async function computePerFileHashes(
  dir: string,
  fileList: string[],
): Promise<Record<string, string>> {
  const result: Record<string, string> = Object.create(null)

  if (fileList.length === 0) {
    return result
  }

  for (const file of fileList) {
    const norm = displayPath(file)
    const fullPath = join(dir, file)
    const h = createHash("sha256")

    h.update(norm)

    await new Promise<void>((resolve, reject) => {
      const stream = createReadStream(fullPath)

      stream.on("data", (chunk) => h.update(chunk))
      stream.on("error", reject)
      stream.on("end", () => resolve())
    })

    result[norm] = h.digest("hex")
  }

  return result
}

When presented with the opportunity to make the code faster, nearly all AI models will suggest file streaming (GPT models more aggressively than others btw).
But why is this a bad idea ? Well it would be better if we were processing massive files. However the majority of them are code files, so text, and at best we have some fonts, images, maybe 1 or 2 demo videos laying around in the repo… So the event/chunk overhead isn’t “free”, and definitely not enough to be beneficial.

Level 3 : Parallelism ftw ! (junior dev)

The junior dev is tasked with making this function faster, and he thinks (rightfully) that it’s the perfect time to implement something he learned recently : parallelism.
Instead of processing files sequentially, we will process all of them at the same time !

import { createHash } from "node:crypto"
import { readFile } from "node:fs/promises"
import { join } from "node:path"

export async function computePerFileHashes(
  dir: string,
  fileList: string[],
): Promise<Record<string, string>> {
  if (fileList.length === 0) {
    return Object.create(null)
  }

  const entries = await Promise.all(
    fileList.map(async (file) => {
      const norm = displayPath(file)
      const fullPath = join(dir, file)
      const content = await readFile(fullPath)

      const fileHash = createHash("sha256")
        .update(norm)
        .update(content)
        .digest("hex")

      return [norm, fileHash] as const
    }),
  )

  return Object.fromEntries(entries) as Record<string, string>
}

And indeed, this is way faster !
However, it comes with some issues :

• We might completely saturate the I/O queue of Node.js. You see, you can queue up a ridiculous amount of filesystem work. In Node.js, filesystem operations are backed by libuv’s threadpool (with a default size of 4), so you can easily create pressure without actually getting infinite throughput
• The disk’s cache could be quickly exhausted, drastically affecting fishing season read performance
• With enough files being processed/things running outside of this script, we can just fill all the available memory, causing the program to crash, let alone running into file descriptor limits (hello EMFILE 👋)

Level 4 : Batching our way in (mid-level dev)

The project grew and the mid-level dev noticed the script crashing in the pipeline, so he decided to take matters into his own hands.
Instead of raw parallelism, we will process batches of files :

import { createHash } from "node:crypto"
import { readFile } from "node:fs/promises"
import { join } from "node:path"

export async function computePerFileHashes(
  dir: string,
  fileList: string[],
): Promise<Record<string, string>> {
  const result: Record<string, string> = Object.create(null)
  const CONCURRENCY = 100

  if (fileList.length === 0) {
    return result
  }

  // Pre-normalize paths to avoid repeated split/join
  const normalized = fileList.map((rel) => [
    rel,
    displayPath(rel),
  ])

  for (let i = 0; i < normalized.length; i += CONCURRENCY) {
    const batch = normalized.slice(i, i + CONCURRENCY)

    // oxlint-disable-next-line no-await-in-loop : Needed to not blow up memory with too many concurrent reads
    const partial = await Promise.all(batch.map(async ([ rel, norm ]) => {
      const fullPath = join(dir, rel)
      const content = await readFile(fullPath)
      const fileHash = createHash("sha256")
        .update(norm)
        .update(content)
        .digest("hex")

      return [ norm, fileHash ] as [string, string]
    }))

    for (const [ norm, partialHash ] of partial) {
      result[norm] = partialHash
    }
  }

  return result
}

Here we have the best of both worlds : process X files at once, but don’t load way too many files from that damn repo into memory !
Now where does that magic number 100 come from ? Saying that I pulled it out of my ass wouldn’t be that far-fetched, but it comes down mostly to tests. Too low and you hurt perf, too high and you blow up memory. It felt like a decent compromise after multiple test runs.
Does pre-normalizing the paths actually help ? Who knows 🤷‍♂️.
Is there a cleaner way to write this ? Probably 🫣.

Level 5 : Proper concurrency (senior dev)

During runs, the senior dev notices some inconsistencies in execution times and decides to take a look at the script. And he has an idea to make the process even faster : a worker pool.
Basically, instead of taking 100 files out of the list, processing them all, then taking the next 100 files and repeating over and over again, we create 100 queues of files (workers) to be processed with only 1 spot available. Then, each file on the list gets assigned to the first free spot.
In other words, instead of “100 at a time, then wait for the slowest one”, we keep 100 workers busy. When one finishes, it grabs the next file :

import { createHash } from "node:crypto"
import { readFile } from "node:fs/promises"
import { join } from "node:path"

/**
 * Map over an array with a concurrency limit
 * @param items The array of items to process
 * @param limit The maximum number of concurrent operations
 * @param fn The async function to apply to each item
 * @returns A promise that resolves to an array of results
 */
export async function mapLimit<T, R>(
  items: T[],
  limit: number,
  fn: (item: T) => Promise<R>,
): Promise<R[]> {
  const results: R[] = Array.from({ length: items.length })
  let idx = 0

  async function worker() {
    while (idx < items.length) {
      const current = idx++

      // oxlint-disable-next-line no-await-in-loop
      results[current] = await fn(items[current])
    }
  }

  await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker))

  return results
}

export async function computePerFileHashes(
  dir: string,
  fileList: string[],
): Promise<Record<string, string>> {
  const result: Record<string, string> = Object.create(null)
  const CONCURRENCY = 100

  if (fileList.length === 0) {
    return result
  }

  const entries = await mapLimit(fileList, CONCURRENCY, async (file) => {
    const norm = displayPath(file)
    const fullPath = join(dir, file)
    const content = await readFile(fullPath)
    const fileHash = createHash("sha256")
      .update(norm)
      .update(content)
      .digest("hex")

    return [ norm, fileHash ] as const
  })

	for (const [ norm, partialHash ] of entries) {
		result[norm] = partialHash
	}

  return result
}

This is, so far, the best I could do.
In level 4, if one file in the batch takes forever to be processed, the other 99 are done and the next batch is just… waiting for that one guy 😤.
If that happens here, that file will clog one worker, but the rest keep going through the queue as if nothing ever happened.

Conclusion

So those “5 levels” were basically :

• Level 1 : I did a for loop and I’m proud of it ^^ (good boy 🫳)
• Level 2 : STREAMS !!! (great for memory/huge files, not automatically faster for lots of small ones)
• Level 3 : Promise.all() go brrrr (until your machine starts sweating 😰)
• Level 4 : Oh shit, let’s not spawn thousands of reads at once (batching to do damage control)
• Level 5 : Keep X workers busy and stop waiting for the slowest guy in the batch (worker pool/concurrency-limited queue)

The main lesson for me : optimization is mostly about picking the right bottleneck to bully.
Sometimes the bottleneck is “you’re doing things one by one”, sometimes it’s “you’re doing too much at once”, and sometimes it’s “you added complexity because it felt fast”, which is apparently what I’m really good at doing 🥹.
Also : streaming is not a magic “go faster” button. If your files are mostly small text blobs, you might just be paying overhead to feel productive 🤡.
If you take one thing from this post : measure first, meme later (ok fine, measure and meme, but in that order 😉).
Anyway, the current version (level 5) is where I landed, and it’s the best combo I’ve found so far between “fast”, “doesn’t explode in CI”, and “I can still read this code without crying” (sorta kinda 🥲).
If you want the actual numbers/setup, the repo has benchmarks and the rest of the context. And if you’ve got a better trick… send it, I’m ready to over-optimize this again for no reason whatsoever 😎.

We all know Zod, an awesome library to validate data by type-checking, pattern-matching and more.
And recently the long-awaited version 4 has been released, and as the repo I worked on during my internship used it, I thought I’ll take care to do this migration before the end of my internship, and learn Zod in the process !
You know, I believe that the companies I work at could easily call me the migrator, as during my 2 internships, my first task was to migrate all dependencies to their latest versions (often migrating whole frameworks and dealing with lots of deprecations).
I don’t mind doing this, and as I’m very methodic with the upgrades (always checking the release notes/changelogs/diffs if there’s nothing) and I thoughtfully test changes, so everything goes buttery smooth. I also leave notes and easy tips for migration in the other dev’s branches.
Anyway, the monorepo I’m talking about uses Zod in 2 main ways :

• types definition
• schema validation

For the first one, they simply define types in a shared types package, and these types can later be reused in the backends, frontend and more. For example :

const deviceDetails = z.object({
  id: z.string(),
  name: z.string(),
  createdAt: z.date(),
  lastMessage: z.date().nullable().optional(),
  siteId: z.number(),
  enable: z.boolean(),
  serialNumber: z.string().nullable().optional(),
  deviceType: z
    .object({
      id: z.number(),
      name: z.string(),
    })
    .nullable()
    .optional(),
  deviceStatus: z
    .object({
      id: z.number(),
      name: z.string(),
    })
    .nullable()
    .optional(),
  health: z
    .object({
      network: z.string().optional(),
      status: z.string().optional(),
      config: z.string().optional(),
    })
    .optional(),
})

export type DeviceDetailsType = z.infer<typeof deviceDetails>

This way, we have a programmatic way to define a type, and we have a handy Zod object that goes along with it.
The second way basically references the Zod object (deviceDetails) as a way to validate an API route (either input or output) with fastify-zod.

Part 0 : The upgrades

The release of Zod v4 came weeks after I already did a massive pass of upgrades so I didn’t had much else to look out for (and I regularly kept dependencies up to date each week).
Upon reading the release notes and migration guide, I instantly saw that Zod now supports generating JSON Schemas ! This is a great news, because it allows us to drop the unmaintained fastify-zod library (and we would finally drop the override on fastify 😅).

Here’s how API validation worked before :

1. Define your schema

// packages/types/src/thing.ts
import { z } from "zod"

const somethingSchema = z.object({
  value: z.string(),
  other_value: z.string().optional(),
})

export const thingSchemas = {
  // ...
  somethingSchema,
}

2. Grab your schemas for your individual route

// services/backend/src/modules/thing/thing.schema.ts
import { buildJsonSchemas } from "fastify-zod"

import { thingSchemas } from "@company/types/thing"

export const { schemas: refThingSchemas, $ref } = buildJsonSchemas(
  {
    ...thingSchemas,
  },
  { $id: "thingSchemas" },
)

3. Register your schemas for the server

// services/backend/src/server.ts
import { refThingSchemas } from "./modules/thing/thing.schema"

// Init Fastify and all...
for (const schema of [
  // ...
  refThingSchemas,
]) {
  fastify.addSchema(schema)
}

4. Use in your route

// services/backend/src/modules/thing/thing.route.ts
import { $ref as refThingSchemas } from "./thing.schema"

const thingRoutes: FastifyPluginCallback<{ prefix: string }> = (
  fastify,
  _options,
  done,
) => {
  fastify.get("/", {
    schema: {
      response: {
        200: refThingSchemas("somethingSchema"),
      },
    },
    handler: getThingHandler,
  })
}

It works, but we can do better.

Part 1 : A new package

Well, not exactly.
To use Zod v4, all you have to do is to use the latest version, and change your imports :

- import { z } from "zod"
+ import { z } from "zod/v4"

This will yield all deprecation warnings and errors so fix them first :)

Now that we’re done with this, it’s time to use the new JSON Schema converter !

Part 2 : Building the schemas

Surprisingly, the amount of code to change for it to work isn’t that big.
But first off, we have to talk about another new feature of Zod v4 : Metadata and registries.
When Fastify wants to validate a schema, it needs a way to identify it. It is done through the $id property in the schema itself. To add this information, adding metadata seems the best way to go about it.

Here’s how you do it :

// packages/types/src/thing.ts
import { z } from "zod"

const somethingSchema = z.object({
  value: z.string(),
  other_value: z.string().optional(),
}).meta({ $id: "something" }) // I preferred to drop the "schema" part of the name to remove redundancy in the code

export thingSchemas = [
  // ...
  somethingSchema,
] // Note how we switch from an object to an array, this will allow for easier parsing down the line, but if you needed to access thingSchemas.somethingSchema, just export somethingSchema directly from now on

Easy, right ?
Well, no.
The .meta() method is a shorthand for .register(z.globalRegistry, { ... }), which means that the metadata you pass in is registered globally, and as such, must be unique.
You cannot have 2 schemas with the same $id, even if they’re in different files and used in different backends.

const backendRegistry = z.registry<{ $id: string }>()

backendRegistry.add(somethingSchema, { $id: "something" })

const somethingSchemaNoMeta = z.object({
  value: z.string(),
  other_value: z.string().optional(),
})
const somethingSchema = somethingSchemaNoMeta.meta({ $id: "something" })

const nestedSchema = z.object({
  id: z.number(),
  something: somethingSchemaNoMeta,
})

Once your schemas are ready to be used, you have to, well, use them.

Part 3 : Using the schemas

Remember when we had to use fastify-zod ?
Well no more, here’s how simple it gets (I recommend to create a helper function like I did) :

// packages/types/src/schemaHelper.ts
import { toJSONSchema, type ZodType } from "zod/v4"

export function zodSchemasToJSONSchema(schemas: ZodType[]) {
  const jsonSchemas = schemas.map((schema) => {
    return toJSONSchema(schema, {
      target: "draft-7", // Fastify acccepts this format only, and it isn't the default for Zod
      unrepresentable: "any", // Accepts some types impossible to represent, check the docs for more info
    })
  })

  return jsonSchemas
}

// services/backend/src/modules/thing/thing.schema.ts
import { zodSchemasToJSONSchema } from "@company/types/schemaHelper"

import { thingSchemas } from "@company/types/thing"

export const refThingSchemas = zodSchemasToJSONSchema(thingSchemas)

// services/backend/src/server.ts
import { refThingSchemas } from "./modules/thing/thing.schema"

// Init Fastify and all...
const schemasList = [
  // ...
  ...Object.values(refThingSchemas),
]

for (const schema of schemasList) {
  fastify.addSchema(schema)
}

// services/backend/src/modules/thing/thing.route.ts
const thingRoutes: FastifyPluginCallback<{ prefix: string }> = (
  fastify,
  _options,
  done,
) => {
  fastify.get("/", {
    schema: {
      response: {
        200: { $ref: "something" }, // no more imports !
      },
    },
    handler: getThingHandler,
  })
}

Part 4 : Final thoughts

As I was doing this migration, several things happened in Zod issues (when you have a package that’s this much used, obviously changes will cause issues somewhere), so here’s stuff I had to deal with :

• For some time, the $schema identifier was wrong, so I had to work around it :

  for (const schema of schemasList) {
    fastify.addSchema({
      ...schema,
      // hack until https://github.com/colinhacks/zod/issues/4412 is fixed
      $schema: "http://json-schema.org/draft-07/schema#",
    })
  }
  

• A change to how additional properties are handled caused an issue in one of the services of the monorepo, as Zod was used there merely to check some “required” properties in order to sort the processing depending on which “provider” sent us the data, but obviously there’s a ton of extra props that might be sent as well in the object, and they change regularly.
  Here’s how to fix it :

  import { core, toJSONSchema, type ZodType } from "zod/v4"
  
  export function zodSchemasToJSONSchema(schemas: ZodType[]) {
    const jsonSchemas = schemas.map((schema) => {
      return toJSONSchema(schema, {
        // Allow to accept additional properties in objects
        override(ctx) {
          const def = (ctx.zodSchema as core.$ZodTypes)._zod.def
          if (def.type === "object" && !def.catchall) {
            ;(
              ctx.jsonSchema as core.JSONSchema.ObjectSchema
            ).additionalProperties = true
          }
        },
        // another way is to set io: "input", tho I'm unsure about this
        target: "draft-7",
        unrepresentable: "any",
      })
    })
  
    return jsonSchemas
  }
  

• I encountered an interesting issue with Fastify : apparently it can’t get right tuples with “limits”. Here’s an example :

  const baseObject = z.object({
    surface: z
      .object({ value: z.number(), unitSource: z.string() })
      .nullable()
      .optional(),
    timezone: z.string().nullable().optional(),
    presenceHours: z
      .object({
        timezone: z.string(),
        hours: z.record(
          z.enum(["0", "1", "2", "3", "4", "5", "6"]),
          z
            .array(
              z.object({
                start: z.tuple([
                  z.number().min(0).max(23),
                  z.number().min(0).max(59),
                ]),
                stop: z.tuple([
                  z.number().min(0).max(23),
                  z.number().min(0).max(59),
                ]),
              })
            )
            .min(1)
        ),
      })
      .optional(),
    reducedHours: z.array(z.tuple([z.string(), z.string()])).optional(),
  })
  
  const extendedObject = baseObject
    .extend({
      siteId: z.string(),
      organizationId: z.string(),
    })
    .meta({ $id: "extendedObject" })
  

  This will yield the following error if your Fastify server is in strict TypeScript mode :

  strict mode: "items" is 2-tuple, but minItems or maxItems/additionalItems are not specified or different at path "extendedObject/properties/presenceHours/properties/hours/additionalProperties/items/properties/start"
  strict mode: "items" is 2-tuple, but minItems or maxItems/additionalItems are not specified or different at path "extendedObject/properties/presenceHours/properties/hours/additionalProperties/items/properties/stop"
  strict mode: "items" is 2-tuple, but minItems or maxItems/additionalItems are not specified or different at path "extendedObject/properties/reducedHours/items"
  

  If this happens to you, add this in your Schema generator function before the return :

  // oxlint-disable-next-line no-explicit-any (or eslint)
  function enforceTuples(obj: any) {
    if (obj && typeof obj === "object") {
      if ("items" in obj && Array.isArray(obj.items)) {
        const len = obj.items.length
        obj.minItems = len
        obj.maxItems = len
      }
  
      for (const key of Object.keys(obj)) {
        enforceTuples(obj[key])
      }
    }
  }
  
  for (const schema of jsonSchemas) {
    enforceTuples(schema)
  }
  

• This is not related to Zod but for posterity I wanted to mention it : Fastify will drop any schema in oneOf and allOf validations when they are too similar.
  You can find more info about it here : https://github.com/fastify/fastify/issues/6133
• Migrating to Zod v4 for JSON Schema creation and validation wasn’t hard and actually eases a lot of things !

What ?

Today, I will share my solution to a problem I recently encountered : sending DTMF inputs during a call.
Despite it seeming trivial, there’s actually no built-in solution for my use case, and I had to write something from scratch to make it work.
But, what is the use case ?

A bit of context, sir

Recently, I worked on LifeCompanion, an open-source, free, and highly customizable digital assistant. It supports individuals with motor, sensory, or cognitive disabilities by offering features like speech synthesis, virtual keyboards, pictographic communication, and compatibility with assistive devices. Developed since 2015, it promotes autonomy and social participation.
With some colleagues at my school, we were tasked to add a plugin to handle communication with an Android phone (send and read SMS, and take calls). This is a great force of LifeCompanion : being able to extend itself through plugins.

To clarify, 2 teams of students made the original plugins a year prior (messages and calls parts), and we were tasked to merge them into a singular plugin.
Still, many features were missing, and DTMF was one of them.
Also, the code of the Android app is written in Kotlin but the implementation can be transferred to Java code.

But what is DTMF ?

DTMF, aka Dual-tone multi-frequency signaling is a system that uses specific frequencies to convey a character. I won’t go over the Wikipedia page (y’all can read), but all you should know is that this system is still in use to this day when composing on the Keypad during a call (ex : when you’re prompted to enter your client number or to re listen your voice message).
And while this is an essential part of calling apps, the team that worked on the call part of the plugin didn’t implement it. So naturally, I decided to do it. What could possibly go wrong ?

Turns out, there’s no API for it

Or, is it ?
Well, the class android.telecom.Call have a method playDtmfTone() (and obviously stopDtmfTone()). But to use these functions, you need to have access to the current Call object, and to do so you need to be the default phone app. Although feasible, it would require to implement all features that users require from a regular call app (dialer, contacts list, call history, voicemail, …), and honestly we didn’t had the time nor the motivation (and imagine all the edge cases to handle !).
Obviously, I didn’t catched that so I started to write code that would not work…

At this moment, it would be great for you to understand quickly how the app we created works.
Basically, the phone is connected with the PC through a cable, and we communicate via ADB.
When we need to request or send data to the phone, we start an intent with extra data, which is a Base64-ed JSON with fields explaining what we want to get/send.
The app then proceeds to pass the data to the relevant controller, which will do its magic, and if needed, output JSON to a specific folder on the phone that LifeCompanion will poll regularly to know if the response is ready to obtain.
So for example, if we want to send the DTMF “8”, we will send from the pc adb shell am start-foreground-service -a org.lifecompanion.phonecontrolapp.services.JSONProcessingService --es extra_data eyJzZW5kZXIiOiAicGMiLCJ0eXBlIjogImNhbGwiLCJzdWJ0eXBlIjoibnVtcGFkX2lucHV0IiwiZGF0YSI6eyJkdG1mIjogIjgifX0=, the extra data symbolizing this :

{
  "sender": "pc",
  "type": "call",
  "subtype": "numpad_input",
  "data": {
    "dtmf": "8"
  }
}

The app will then process this, and pass the data to the services.CallController, which will call the right function. Also at this point we don’t care if the DTMF has been sent correctly, so there’s no request_id in the JSON, we don’t check if it has worked or not.
Finally, file paths for Kotlin files are relative to src/main/java/org/lifecompanion/phonecontrolapp unless specified otherwise (ressource files for example).

So, in the following code examples I will only focus on the DTMF-related pieces, leaving the rest of the Call and JSON processing logic aside. You can find more about the app itself on the repo.

Oh, the misery

// services/CallService.kt
package org.lifecompanion.phonecontrolapp.services

import android.telecom.Call
import org.lifecompanion.phonecontrolapp.services.CallStateListener
// ...

class CallService : Service(), CallStateListener {
    private var currentCall: Call? = null
    // ...

    override fun onStartCommand(intent: Intent, flags: Int, startId: Int): Int {
        // ...
        CallWatcher.callStateListener = this
    }

    override fun onCallStateChanged(call: Call?, isIncoming: Boolean, isActive: Boolean, phoneNumber: String?) {
        currentCall = call
        // ...
    }

    private fun sendDtmf(dtmf: String) {
        if (currentCall == null) {
            return
        }

        try {
            currentCall?.playDtmfTone(dtmf)
            // Pause between tones to ensure proper transmission
            Thread.sleep(300)
            currentCall?.stopDtmfTone()
        } catch (e: Exception) {
            // womp womp
        }
    }

    // ...

// services/CallStateListener.kt
package org.lifecompanion.phonecontrolapp.services
import android.telecom.Call

interface CallStateListener {
    fun onCallStateChanged(
        call: Call?,
        isIncoming: Boolean,
        isActive: Boolean,
        phoneNumber: String?
    )
}

// services/CallWatcher.kt
package org.lifecompanion.phonecontrolapp.services

import android.telecom.Call
import android.telecom.InCallService
import org.lifecompanion.phonecontrolapp.services.CallStateListener

class CallWatcher : InCallService() {
    companion object {
        private const val TAG = "CallWatcher"
        var callStateListener: CallStateListener? = null
    }

    override fun onCallAdded(call: Call) {
        super.onCallAdded(call)
        call.registerCallback(callStateCallback)
        notifyStateChange(call)
    }

    override fun onCallRemoved(call: Call) {
        super.onCallRemoved(call)
        call.unregisterCallback(callStateCallback)
        notifyStateChange(null)
    }

    private val callStateCallback = object : Call.Callback() {
        override fun onStateChanged(call: Call, state: Int) {
            super.onStateChanged(call, state)
            notifyStateChange(call)
        }
    }

    private fun getPhoneNumber(call: Call?): String? {
        return call?.details?.handle?.schemeSpecificPart
    }

    private fun notifyStateChange(call: Call?) {
        val isIncoming = call?.details?.state == Call.STATE_RINGING
        val isActive = call?.details?.state == Call.STATE_ACTIVE
        val phoneNumber = getPhoneNumber(call)
        callStateListener?.onCallStateChanged(call, isIncoming, isActive, phoneNumber)
    }
}

While this code should work (probably not), as you can see we extend InCallService(), and that would require us to be the default phone app.

Everybody wants to be my enemy

So because it didn’t worked, I decided to do everyone’s favorite activity : searching for documentation :)
What I found is crazy : devs asked for this feature all the way back in 2008 ! It was marked as Won’t Fix (Obsolete) in 2014, and even the patches submitted by some devs were rejected in 2021.
From what I was able to read, it apparently is possible for an app to send DTMF tones but only over VoIP. But guess what ? It is deprecated by now !

So, time for more searches ! And I browsed everyone’s second favorite website : StackOverflow (I come from a time where LLMs weren’t the meta).
And across all the related questions, I haven’t found any single answer :(

• This one suggests that we can send it at call time, but not during a call (and also that one).
• This one was about another technology (that one is the same question but with other answers that confirms what I said previously).
• This one tells us that we can “fake” it by playing the frequency ourselves (but that work only when the speakerphone is enabled).
• This is the post that confirmed the playDtmfTone() method.
• And finally this one is a very interesting approach to the problem, although not working.

Okay, so I guess it’s time for more broad searches, right ?

• A random library that I found but for a completely separate project.
• An API that someone tried to create but that doesn’t work sadly.
• I know that we would need to be the default phone app !
• A lot of tutorials have suggested to use a separate app to emit the according frequencies (or for us, to embed them in the app).

Hmm… What does LLMs have to say about this ?
Well all of them had either the brilliant idea to suggest android.telecom.Call.startDtmfTone() but without telling me that we need to be the default phone app, or they would straight up hallucinate methods. Even GPT o1 (SOTA at the time) had trouble helping me in this task or provide alternate ways to handle it.

The revelation

At this point I was convinced that it wasn’t feasible. If it wasn’t for Andriy Antonov’s solution !
This guy had a brilliant idea : Android phones have accessibility services, which allows to emulate clicks on the screen. Why not using them to click on the keypad buttons directly !
However, his solution had some flaws, notably the fact that we needed the screen coordinates of the buttons, and our app needed to work on all kind of Android devices (including tablets), which made this impossible. But hey, that’s nothing that code couldn’t solve… :)

Time to lock in

Here’s the very first implementation that I made :

// services/DTMFAccessibilityService.kt
package org.lifecompanion.phonecontrolapp.services

import android.accessibilityservice.AccessibilityService
import android.util.Log
import android.view.accessibility.AccessibilityEvent
import android.view.accessibility.AccessibilityNodeInfo

class DTMFAccessibilityService : AccessibilityService() {
    companion object {
        private const val TAG = "DTMFAccessibilityService"
    }

    override fun onAccessibilityEvent(event: AccessibilityEvent?) {
        if (event == null || event.source == null) return

        Log.i(TAG, "Accessibility event received: ${event.eventType}")
        val rootNode = rootInActiveWindow ?: return

        if (isInCallScreen(rootNode)) {
            Log.i(TAG, "In call screen detected")

            if (!isKeypadOpen(rootNode)) {
                openKeypad(rootNode)
            }
        }
    }

    private fun isInCallScreen(node: AccessibilityNodeInfo): Boolean {
        // Check for specific elements that indicate the call screen
        return node.packageName?.contains("dialer", ignoreCase = true) == true
    }

    private fun isKeypadOpen(node: AccessibilityNodeInfo): Boolean {
        // Check for the presence of specific keypad elements that are unique to the keypad
        val keypadElements = listOf("1", "2", "3", "4", "5", "6", "7", "8", "9", "0", "*", "#")

        return keypadElements.any { element ->
            node.findAccessibilityNodeInfosByText(element).isNotEmpty()
        }
    }

    private fun openKeypad(node: AccessibilityNodeInfo) {
        // Find and click the button to open the keypad
        val keypadButtonTexts = listOf("Keypad", "Clavier")

        for (text in keypadButtonTexts) {
            val keypadButton = node.findAccessibilityNodeInfosByText(text).firstOrNull()

            if (keypadButton != null) {
                keypadButton.performAction(AccessibilityNodeInfo.ACTION_CLICK)
                Log.i(TAG, "Keypad opened with text: $text")

                return
            }
        }

        Log.i(TAG, "Keypad button not found")
    }

    fun pressKeypadButton(buttonText: String) {
        val rootNode = rootInActiveWindow ?: return

        if (isInCallScreen(rootNode)) {
            if (!isKeypadOpen(rootNode)) {
                openKeypad(rootNode)
            }

            searchAndClick(rootNode, buttonText)
        }
    }

    private fun searchAndClick(node: AccessibilityNodeInfo, buttonText: String) {
        if (node.text?.toString() == buttonText && node.isClickable) {
            node.performAction(AccessibilityNodeInfo.ACTION_CLICK)
            Log.i(TAG, "Clicked button: $buttonText")

            return
        }

        for (i in 0 until node.childCount) {
            node.getChild(i)?.let { searchAndClick(it, buttonText) }
        }
    }

    override fun onInterrupt() {
        Log.i(TAG, "Accessibility Service Interrupted")
    }
}

// services/CallService.kt
package org.lifecompanion.phonecontrolapp.services

import android.content.Intent
// ...

class CallService : Service() {
    // ...

    private fun sendDtmf(dtmf: String) {
        val intent = Intent(this, DTMFAccessibilityService::class.java)
        startService(intent)

        val dtmfService = DTMFAccessibilityService()
        dtmfService.pressKeypadButton(dtmf)
    }
}

Obviously we need to ask for accessibility service privilege :

// MainActivity.kt
package org.lifecompanion.phonecontrolapp

import android.app.Activity
import android.app.AlertDialog
import android.content.Intent
import android.os.Bundle
import android.provider.Settings
import org.lifecompanion.phonecontrolapp.services.DTMFAccessibilityService
// ...

class MainActivity : Activity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        // ...

        promptEnableAccessibilityService()
    }

    // ...

    private fun isAccessibilityServiceEnabled(service: Class<*>): Boolean {
        val enabledServices = Settings.Secure.getString(
            contentResolver,
            Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES
        ) ?: return false
        val colonSplitter = enabledServices.split(":")
        val serviceName = componentName.flattenToString().replace(packageName, service.name)

        return colonSplitter.any { it.equals(serviceName, ignoreCase = true) }
    }

    private fun promptEnableAccessibilityService() {
        if (!isAccessibilityServiceEnabled(DTMFAccessibilityService::class.java)) {
            AlertDialog.Builder(this)
                .setTitle("Enable Accessibility service")
                .setMessage("This app requires the Accessibility service to emulate keyboard inputs during calls. Please enable it in the Accessibility settings. You may need to do this on every restart.")
                .setPositiveButton("Open settings") { _, _ ->
                    val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS)
                    startActivity(intent)
                }
                .setNegativeButton("Already done", null)
                .show()
        }
    }
}

And finally, to declare the accessibility service itself :

<!-- src/main/res/values/strings.xml -->
<resources>
    <!-- ... -->
    <string name="accessibility_service_description">Accessibility service for the LifeCompanion app</string>
</resources>

<!-- src/main/res/xml/accessibility_service_config.xml -->
<?xml version="1.0" encoding="utf-8"?>
<accessibility-service
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:accessibilityEventTypes="typeWindowContentChanged|typeViewClicked"
    android:accessibilityFeedbackType="feedbackGeneric"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true"
    android:accessibilityFlags="flagReportViewIds" />

So, what the hell are we actually doing here ?

1. We declare an accessibility service that is triggered everytime the content of a window changes or when a view is clicked, and ask to get the view IDs.
2. On the CallService, we create the DTMFAccessibilityService and call its function.
3. On the Accessibility service, whenever we receive an event we check if it’s from the active window. If it is, we check if we’re in the call screen and then open the keypad if it isn’t.
4. isInCallScreen() is very primitive and only checks the presence of dialer in the window’s package name.
5. isKeypadOpen() checks if any of the keypad elements (0-9, * and #) is present on the node (which represents the active window in a tree). At first it returned true only if all of them were present but it didn’t worked so I used any instead.
6. openKeypad() searches for a button in the node that contains in its text “Keypad” or its french version, “Clavier” (the app was made mainly for french users so I had to include it), and clicks on it.
7. The public pressKeypadButton() does all of the above and calls searchAndClick() which clicks on any node that matches exactly the input string and is clickable.

Obviously, it goes without saying that this is not optimal, and far from working :

• Due to the incessant calls to openKeypad() on the accessibility events, and the barely working check functions, the keypad was constantly opening and closing (which was funny to stare at).
• The logic to check if we’re in a call screen and if the keypad is open are very barebones and not flexible.
• The call screen possesses the phone number of the callee, which obviously will always match our any clause.
• Overall it wasn’t even able to click on the keypad buttons…

So, it is time for fixes.

Make it workey pretty please 🙏

The first edit I did was to comment the code in onAccessibilityEvent() to avoid the keypad from flickering in my screen.
Then, I edited the logic in the isKeypadOpen() method :

private fun isKeypadOpen(node: AccessibilityNodeInfo): Boolean {
    // Count the number of keypad elements on the screen
    val initialCount = countKeypadElements(node)
    // Trigger openKeypad and count again
    openKeypad(node)
    val afterOpenCount = countKeypadElements(node)
    // Determine if the keypad is open based on the counts
    val isOpen = afterOpenCount > initialCount
    // Revert the state by triggering openKeypad again
    openKeypad(node)

    return isOpen
}

private fun countKeypadElements(node: AccessibilityNodeInfo): Int {
    val keypadElements = listOf("1", "2", "3", "4", "5", "6", "7", "8", "9", "0", "*", "#")
    return keypadElements.sumBy { element ->
        node.findAccessibilityNodeInfosByText(element).size
    }
}

The logic is simple : we count the number of each element that should be present on the keypad. Then, we click on its button and count again.
If there’s more elements, it means it is open. If not, we closed it, so we click on it again.
I thought I was clever writing this, but honestly it was still bad : because we call this function anyway at each pressKeypadButton() call, we would close and reopen the keypad at every click.
Or we open only once per phone call, but if the user closed the keypad by accident, then we couldn’t reopen it and any button presses would fail.
So it is time for…

The working solution

After detailing the goddamn implementation I did and our motive, GPT o1 came in clutch and was finally able to help in this task. Here’s the code changes that we did based off some of its recommendations :

<!-- src/main/res/xml/accessibility_service_config.xml -->
<?xml version="1.0" encoding="utf-8"?>
<accessibility-service
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/accessibility_service_description"
    android:accessibilityEventTypes="typeWindowContentChanged|typeViewClicked|typeWindowStateChanged"
    android:accessibilityFlags="flagReportViewIds|flagIncludeNotImportantViews"
    android:accessibilityFeedbackType="feedbackGeneric"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true" />

// services/DTMFAccessibilityService.kt
package org.lifecompanion.phonecontrolapp.services

import android.accessibilityservice.AccessibilityService
import android.content.Context
import android.content.Intent
import android.content.pm.ResolveInfo
import android.util.Log
import android.view.accessibility.AccessibilityEvent
import android.view.accessibility.AccessibilityNodeInfo

object DTMFAccessibilityServiceSingleton {
    var instance: DTMFAccessibilityService? = null
}

class DTMFAccessibilityService : AccessibilityService() {
    companion object {
        private const val TAG = "LC-DTMFAccessibilityService"

        // Known dialer package names, more may be needed depending of the default call app
        private val KNOWN_DIALER_PACKAGES: List<String> by lazy {
            getPackagesOfDialerApps().apply {
                val additionalPackages = listOf(
                    "com.google.android.dialer",
                    "com.android.dialer",
                    "com.samsung.android.incallui"
                )
                additionalPackages.forEach { pkg ->
                    if (!this.contains(pkg)) {
                        this.add(pkg)
                    }
                }
            }
        }

        // Common text or contentDescriptions for opening the dialpad
        private val DIALPAD_TOGGLE_KEYWORDS = listOf("Keypad", "Clavier", "Dial pad", "Show dial pad")

        private fun getPackagesOfDialerApps(): MutableList<String> {
            val packageNames = mutableListOf<String>()
            val context = DTMFAccessibilityServiceSingleton.instance?.applicationContext

            if (context != null) {
                val intent = Intent(Intent.ACTION_DIAL)
                val resolveInfos: List<ResolveInfo> = context.packageManager.queryIntentActivities(intent, 0)

                for (resolveInfo in resolveInfos) {
                    val activityInfo = resolveInfo.activityInfo
                    packageNames.add(activityInfo.applicationInfo.packageName)
                }
            } else {
                Log.e(TAG, "Context is null, cannot get dialer packages")
            }

            return packageNames
        }
    }

    // Keep a reference to the latest root node
    private var _rootNode: AccessibilityNodeInfo? = null

    override fun onCreate() {
        super.onCreate()
        DTMFAccessibilityServiceSingleton.instance = this
    }

    override fun onDestroy() {
        super.onDestroy()
        DTMFAccessibilityServiceSingleton.instance = null
    }

    override fun onAccessibilityEvent(event: AccessibilityEvent?) {
        if (event == null) {
            return
        }

        // Whenever there's a window content/state change, update our root node reference
        if (event.eventType == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED || event.eventType == AccessibilityEvent.TYPE_WINDOW_CONTENT_CHANGED) {
            _rootNode = rootInActiveWindow
        }
    }

    override fun onServiceConnected() {
        super.onServiceConnected()
        DTMFAccessibilityServiceSingleton.instance = this
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        intent?.let {
            val action = it.getStringExtra("action")
            val buttonText = it.getStringExtra("button_text")

            if (action == "press_keypad_button" && buttonText != null) {
                pressKeypadButton(buttonText)
            }
        }

        return START_NOT_STICKY
    }

    /**
     * Heuristically checks if the current screen belongs to a known dialer or in-call UI.
     */
    private fun isInCallScreen(rootNode: AccessibilityNodeInfo): Boolean {
        val pkg = rootNode.packageName?.toString() ?: return false

        return KNOWN_DIALER_PACKAGES.any { pkg.contains(it, ignoreCase = true) }
    }

    /**
     * Check if the dial pad is open by searching for all digits 0-9 plus * and # in the hierarchy.
     */
    private fun areAllDigitsVisible(rootNode: AccessibilityNodeInfo): Boolean {
        val required = listOf("1", "2", "3", "4", "5", "6", "7", "8", "9", "0", "*", "#")

        for (digit in required) {
            val nodesWithDigit = rootNode.findAccessibilityNodeInfosByText(digit)

            if (nodesWithDigit.isNullOrEmpty()) {
                return false
            }
        }

        return true
    }

    /**
     * Attempt to find a toggle button (by text or contentDescription) to open the dialpad.
     * If we already see digits on the screen, we skip toggling.
     */
    private fun openDialPadIfNeeded(rootNode: AccessibilityNodeInfo) {
        // If we already see the digits 0-9, #, * in the node tree, the dial pad is probably open
        val allDigitsPresent = areAllDigitsVisible(rootNode)

        if (allDigitsPresent) {
            return
        }

        // Otherwise, BFS to find a dialpad toggle
        val queue = ArrayDeque<AccessibilityNodeInfo>()
        queue.add(rootNode)

        while (queue.isNotEmpty()) {
            val node = queue.removeFirst()
            // Check text & contentDescription
            val textStr = node.text?.toString() ?: ""
            val descStr = node.contentDescription?.toString() ?: ""

            // If either text or contentDescription matches known keywords
            if (DIALPAD_TOGGLE_KEYWORDS.any { keyword ->
                    textStr.contains(keyword, ignoreCase = true) ||
                    descStr.contains(keyword, ignoreCase = true)
                }
            ) {
                if (node.isClickable) {
                    node.performAction(AccessibilityNodeInfo.ACTION_CLICK)
                    Thread.sleep(300)

                    return
                }
            }

            for (i in 0 until node.childCount) {
                node.getChild(i)?.let { queue.add(it) }
            }
        }

        Log.w(TAG, "Could not find any dialpad toggle button in the current UI.")
    }

    /**
     * Checks if a dialer button's text or contentDescription is relevant for this digit.
     * e.g. digit = "2" matches "2", "2 ABC", "2,ABC", "2." ...
     */
    private fun isDialerButtonMatch(nodeText: String?, digit: String): Boolean {
        if (nodeText == null || nodeText.isEmpty()) {
            return false
        }

        val escapedDigit = Regex.escape(digit)
        val pattern = Regex("^$escapedDigit[\\s,]*(.*)?$", RegexOption.IGNORE_CASE)

        return nodeText.matches(pattern)
    }

    /**
     * Find and click a single digit (0-9, *, #).
     * Returns true if the digit was successfully clicked, false otherwise.
     */
    private fun pressKeyDigit(rootNode: AccessibilityNodeInfo, digit: String): Boolean {
        // BFS
        val queue = ArrayDeque<AccessibilityNodeInfo>()
        queue.add(rootNode)

        while (queue.isNotEmpty()) {
            val node = queue.removeFirst()

            // Skip system UI or non-dialer packages
            val nodePackage = node.packageName?.toString() ?: ""
            if (!KNOWN_DIALER_PACKAGES.any { nodePackage.contains(it, ignoreCase = true) }) {
                // This node isn't from a recognized dialer package. Skip its subtree.
                continue
            }

            val textStr = node.text?.toString()
            val descStr = node.contentDescription?.toString()

            // If either text or contentDescription is a partial match
            if (isDialerButtonMatch(textStr, digit) || isDialerButtonMatch(descStr, digit)) {
                if (node.isClickable) {
                    node.performAction(AccessibilityNodeInfo.ACTION_CLICK)
                    Thread.sleep(300)

                    return true
                }
            }

            for (i in 0 until node.childCount) {
                node.getChild(i)?.let { queue.add(it) }
            }
        }

        return false
    }

    /**
     * External entry point:
     * 1) Check if we are in a known dialer UI
     * 2) Open the dial pad if needed
     * 3) Press the requested button
     */
    fun pressKeypadButton(dtmfString: String) {
        val rootNode = _rootNode ?: run {
            Log.w(TAG, "No root node available. Are we sure the dialer is in the foreground ?")

            return
        }

        if (!isInCallScreen(rootNode)) {
            Log.w(TAG, "We are not in a recognized in-call/dialer screen !")

            return
        }

        openDialPadIfNeeded(rootNode)

        if (dtmfString.matches(Regex("[0-9*#]"))) {
            var success = pressKeyDigit(rootNode, dtmfString)

            if (!success) {
                // We probably just opened the dial pad
                Thread.sleep(300)
                success = pressKeyDigit(rootNode, dtmfString)

                if (!success) {
                    Log.e(TAG, "Failed to press digit : $dtmfString")
                }
            }
        } else {
            Log.e(TAG, "Invalid DTMF char : $dtmfString")
        }
    }

    override fun onInterrupt() {
        DTMFAccessibilityServiceSingleton.instance = null
    }
}

// services/CallService.kt
// ...

    private fun sendDtmf(dtmf: String) {
        val intent = Intent(this, DTMFAccessibilityService::class.java)
        startService(intent)

        DTMFAccessibilityServiceSingleton.instance?.pressKeypadButton(dtmf)
    }

Time for some explanations once again :)

1. We created (quickly) a singleton to avoid creating another DTMFAccessibilityService object at each call.
2. We added a bit more possible names for the keypad button. I’m sorry, but if you want to use this code, you might need to i18n it depending on the countries you’re targeting.
3. The detection of the call app name is way much improved ! We first get the list of package names from all installed apps that can handle calls on the phone, and add 3 common packages just in case.
4. Everytime we get an accessibility event (we added when the state of a window changes, and asked also for the “non important views”), we would edit a global root node that we process. This ensures reactivity to content changes (ex the keypad just opened or closed) and also helps us working on an always up to date node tree.
5. We also improved the detection of “visible” keypad elements (we require all of them to be apparent on the screen).
6. If the keypad isn’t opened, we do a quick BFS on the root node to find the button and click on it.
7. When calling pressKeypadButton(), we perform all required actions (as before), and check if the DTMF symbol is an accepted one (technically A-D exists but nobody uses them) with Regex("[0-9*#]").
8. We then call pressKeyDigit() to perform a BFS on the root node tree again, but this time to search for the DTMF’s button. Note that we sleep 300ms after performing any click to let the UI (and root node object) update. We also take care of skipping anything where the buttons will not be present.
9. To be really sure, we check in the button’s text and description for the content we search for with a good ol’ regex : "^$dtmf[\\s,]*(.*)?$".

Obviously, the DTMF button name is in the description, not the text ! For example, the button 2 have a null text, and 2,ABC as a description.
But we still had to do a quick change. You see, every button press worked, except *, which hanged up the call. Well it was because we had to escape it, as it matched any text on the regex !

The end

It was really fun to make the code and write this blog post, but seriously frustrating that there is no standard API to do it in a non convoluted way, 17 years after the release of Android !
I hope that this helped you to implement such a basic feature in your app (although working with accessibility services is a pain in the rear end), and don’t hesitate to follow me on random places (feel free to start a convo).
This blog will contain random writeups like this one (technical) and more random standard things too, thoughts, …

Quick backstory

My gf have been given a laptop by her school. When she finished her studies, she was able to keep it. But it was still linked to an education account, managed by Intune/Azure ActiveDirectory and it was Windows Education. Here’s how i turned it into a normal Windows machine :)

Prerequisites

The PC has to have an admin session. If you don’t, try to find how to make yourself admin :)

Step 1 : The migration

1. Create a new local account and make it admin. Go on Settings => Accounts => Other users and add one. Make sure it’s a local account, decline as much as possible any Microsoft related info.
2. Write somewhere all the apps that are installed. Some of them are system-wide but some may be just for the current user.
3. Zip every important folder. Documents, Downloads, Images, … I also like to zip the Appdata folder just in case.
4. Open a cmd and type echo %COMPUTERNAME%. Write this somewhere ! Also your current account is very probably a cloud account, so open the settings and write down the email shown at the top under your username. If it doesn’t show up, go on Settings => Accounts => Professional or School access and it should be there. If you close the session by mistake, this email will be the username to enter.
5. Lock the current session but DON’T CLOSE IT ! Connect to the other user, username will be COMPUTERNAME\username (backslash is important). Replace with your computer name and the username you have chosen.
6. Copy-paste all the previously zipped files, install any software that isn’t present, re-log in if needed.
7. Do Win + R and type systempropertiesadvanced. Here, click on the “PC Name” tab, Edit, and change the PC name to something more fitting to you (mine is for example Lenovo-EDM115).
8. Go back on the previous user and log out after you made sure you backed up anything important (ex browser passwords, apps settings, …)
9. If you are on Windows Pro, open the Control Panel => BitLocker Drive Encryption => Turn off BitLocker. This will take some time but your drive will be much faster afterwards

Step 2 : The unlink

1. Disable Wifi (anything that connects to internet) and restart the PC
2. Once restarted, go on your new user (UpdatedPcName\ChosenUsername), and go to Settings => Accounts => Family & other users, and remove it. Go on Settings => Accounts => Professional or School access, click on the account, and disconnect. Now for some weird reason this should remove the existing user account but it doesn’t work fully, so go on C:\Users. Try to go on the old account and confirm so it grants you the rights. Then go back and delete it.
3. Go on Settings => System => Activation => Change product key option => Change, and enter VK7JG-NPHTM-C97JM-9MPGT-3V66T (make sure you’re offline !). This is a generic Windows 11 Pro key, so I guess it would be working only if you were on Windows 11 Pro Education. If your system is Windows 11 Education (check in Settings => System => System informations), use YTMG3-N6DKC-DKB77-7M9GH-8HVX7 instead. Full list can be found at https://www.elevenforum.com/t/generic-product-keys-to-install-or-upgrade-windows-11-editions.3713/
4. Reconnect to the internet, open a PowerShell as admin, and enter irm https://get.activated.win | iex, then hit 1, wait for it to be complete, Enter, 0. You now have a genuine Windows licence.
5. Open the Task Manager (Ctrl + Shift + Esc) and close anything related to Intune. Now install Revo Uninstaller and remove softwares that you didn’t installed yourself + some Intune related stuff (some are system stuff, if you see anything related to your PC manufacturer/Realtek/AMD/Intel, …, keep it !). Take also a look at Windows apps, some are hidden there. Make sure you do an advanced scan and check the “Check on all windows accounts” checkbox. Then clean some remaining trash, probably at C:\Intune, C:\Windows\Intune, and similar folders.
6. Now do Win + R, gpedit.msc. Go to Computer Configuration => Administrative Templates => All settings, and click on the State column. Double click on anything that is marked as Enabled or Disabled, and select “Not configured”.
7. Open a PowerShell as admin, and run the following lines one by one :

reg delete "HKCU\Software\Microsoft\Windows\CurrentVersion\Policies" /f
reg delete "HKCU\Software\Microsoft\WindowsSelfHost" /f
reg delete "HKCU\Software\Policies" /f
reg delete "HKLM\Software\Microsoft\Policies" /f
reg delete "HKLM\Software\Microsoft\Windows\CurrentVersion\Policies" /f
reg delete "HKLM\Software\Microsoft\Windows\CurrentVersion\WindowsStore\WindowsUpdate" /f
reg delete "HKLM\Software\Microsoft\WindowsSelfHost" /f
reg delete "HKLM\Software\Policies" /f
reg delete "HKLM\Software\WOW6432Node\Microsoft\Policies" /f
reg delete "HKLM\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\Policies" /f
reg delete "HKLM\Software\WOW6432Node\Microsoft\Windows\CurrentVersion\WindowsStore\WindowsUpdate" /f
reg delete "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Enrollments" /f

8. Restart the PC. You should now be able to login with just the username and without specifying the PC name. Open a cmd as admin and run DSREGCMD /debug /leave and then DSREGCMD /debug /cleanupaccounts. This will do its best to leave linked organizations.
9. Do Win + R, regedit, and click on HKEY_CURRENT_USER (so the search will start from here. click again on this when doing another search). You will now have to search for anything related to “Intune”, “Enrollment” or the domain part of your school email that you noted prior (ex if it was pc123456@school.country.com, search for school.country.com). Delete the keys found (use common sense tho) to remove the last bits of organization links. You can backup the registry before if you think you will mess this up.

Step 3 : Final touches and quick remarks

1. Open Windows Update an check for any updates available. Install them and restart as necessary. If some fail to install, it’s probably because when on Intune, they restricted which versions could be installed (ex for my gf she could only install cumulative updates of every 3 month). To avoid further issues, open a cmd as admin and run the following (then check for updates again) :

net stop wuauserv
net stop cryptSvc
net stop bits
net stop msiserver
rd /SQ "C:\Windows\SoftwareDistribution"
rd /SQ "C:\Windows\System32\catroot2"
net start wuauserv
net start cryptSvc
net start bits
net start msiserver
DISM /Online /Cleanup-Image /CheckHealth
DISM /Online /Cleanup-Image /ScanHealth
DISM /Online /Cleanup-Image /RestoreHealth
sfc /scannow

2. If some cumulative updates still fail, or if when doing Win + R, winver the version is lower than the current actual one (23H2 as I write this, will soon be 24H2), let’s do an in-place upgrade. Go at https://www.microsoft.com/en-us/software-download/windows11 => Download Windows 11 Disk Image (ISO) for x64 devices, select the only choice then your language and click on 64-bit download. Once downloaded, right click the .iso => Properties => Check Unblock => Ok. Then do right click => Mount, and it should appear on the left of the file explorer as a DVD. Run Setup.exe => Next => Accept => Check that it says “Install Windows 11 (Pro)” and “Keep personal files and apps” (by default) => Install. The PC will be inaccessible and may restart several times. But it’s finally over !
3. By now it should be fully unlocked and all settings should be accessible. Maybe go on Settings => Network and Internet => Wi-Fi => Manage known networks and remove anything linked to your school. The only setting that was locked for my gf was to change her lock screen image. To change this, copy-paste the image you want into C:\Users\Public\Pictures (avoid any spaces), then Win + R, regedit, paste on the top bar HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Personalization and make the key AllowPersonalization to 1 (if it doesn’t exist right click on Personalization folder => New => DWORD, it’s hexadecimal). Then, a folder just under should be called PersonalizationDSP (in CurrentVersion). Paste the full path of your wanted lock screen background in the String keys LockScreenImagePath and LockScreenImageUrl (ex : C:\Users\Public\Pictures\awesome_bg.png), plus set the DWORD LockScreenImageStatus to 1

As you may know, Google is ending their Edu shared drive plan.
On this article, we are going to see why, what would be the consequences for you and which solutions you may use.

1) Is it serious ?

Yes it is, and to be honest it didn’t surprise me…
Google have announced it, but it was said nearly nowhere else (that is why I’m writing this article)
This follow their new storage policy, which is going to take effect in July (theorically on 1st July, but it may take some days to be effective). Also, some Shared Drives may be deleted from June (Google Admin already provide a solution to delete them)

2) A bit of chronology…

• 24th April 2014 : Creation of Google Drive, formerly known as project Platypus (original blog post)
• September 2016 : Creation of Team Drives
• 29th April 2019 : Renaming them to Shared Drives
• In the end of 2019, proof here : Creation of Team Drives generators
• 18th February 2022 : Google announcing changing their storage policy

3) What are Team/Shared Drives ?

In September 2016, Google announced Team Drives, later renamed Shared Drives, as a new way for Google Workspace teams to collaborate on documents and store files. In Shared Drives, file/folder sharing and ownership are assigned to a team rather than to an individual user. Since 2020, Shared Drives had an ability to assign different access levels to files and folders to different users and teams, and an ability to share a folder publicly. Unlike individual Google Drive, Shared Drives offer unlimited storage.

They provided unlimited cloud storage for schools and enterprises that needed it

4) What caused this ending ?

People were seriously abusing it…
While some people needed it, using it as a backup of their data, or because they have some files that were above the 15 Gb limit ; others overused it. Among them, people that runs Telegram mirror groups. Because a lot of users mirror a lot of files to Google Drive, and because making such groups is easier than ever thanks to Heroku, the number of files uploaded to Shared Drives exponentially raised.
As Google saw the storage consumption growing up, and as people were only abusing Edu Drives, they decided to “end” this offer.
Plus, let me tell you something from my experience. For the context, I’m in over 250 shared drives that host around 5 Pb of data. If I search for one file (let’s say After Effects 2022), I have hundred of results, most of the time the exact same file, just on several drives…

5) Which solutions remains for schools ?

In facts, they still can use their Edu plan. But this time, the storage is pooled with a baseline of 100 TB for all users, no longer unlimited. Meaning that administrators of schools will be more attentives to what happen on their drives, and the creation of Shared Drives will seriously be resricted.
The real problem may come for schools like Harvard or the MIT, them needing to store huge amount of data. The solution for them could be either buying an upper Drive plan, or having ther own servers.

6) What are the consequences for the average user ?

If like me, you’re in a lot of Shared Drives, they will nearly all disappear from your drive.
If you’re running one, it would be better to inform your users that the storage may end very soon, and telling them to backup their files as soon as possible.
If you still need to have a cloud storage, keep reading.
All the links that leads to files/folders on those drives will no longer work, and mirror bots will probably no longer work.
TeamDrives generator such as the one of MSGsuite will need to shut down their services.

7) You need to store your files on the cloud ?

Here are the best free solutions for you :

Sources

• Google
• Universities that uses Drive :
  • NCKU
  • Minnesota
  • Northern Iowa
  • Stanford
  • Alabama
  • a lot of others…
• HashHackers

Keep in touch with me 🥰

https://github.com/EDM115
https://t.me/EDM115