Wong Jack Yih Full-Stack Software Engineer

Why a rewrite was needed

The original approach loaded orders into memory and applied complex array manipulation. Both order volume and the number of templates added pressure. For some high-volume merchants, processing reached the three-minute request timeout.

The central engineering decision

I moved rule evaluation into database queries so the engine could select and group the relevant orders without loading the entire order set into application memory. The rewrite needed to preserve the meaning of user-defined rules while changing how they were executed.

What changed

The result was faster, more reliable wave processing for users and a more scalable basis for developers to extend the rules. I combined the new engine with saved workflow state, scheduled background processing, and a staged rollout to support the transition.

After release

I continued diagnosing live issues involving order visibility, grouping behaviour, and queries at merchant data volumes. That support work formed part of my ownership of the feature, alongside its initial delivery.

What merchants get

A merchant blacklists a buyer with one of eight fixed reasons, such as refused cash on delivery, repeated returns or card fraud, plus an optional note, and can do it straight from an order. From then on a red Blacklisted tag appears on that buyer's orders in the order list, order detail, order processing, picking waves and returns, and the order list, order processing and returns can filter by it. Opening a blacklisted buyer shows how many other merchants reported the same buyer, with a breakdown by reason, and the same figures appear when a merchant is about to blacklist someone from an order. The tag is a warning only. It never holds, blocks or cancels an order, and a merchant's tags come only from their own list, never from other merchants' reports.

Who the buyer is

A buyer is identified by marketplace storefront, such as Shopee Malaysia, and buyer ID together, because buyer IDs are only unique within a marketplace. Orders did not keep that ID, so I added it to the order write that every Shopee, TikTok and Lazada sync already performs. Older Shopee and TikTok orders are filled in from stored data. For an order that still has none, the system fetches it from the marketplace again and saves only the buyer ID, with none of a full re-sync's side effects on status, shipments or totals.

Tags that cannot go stale

I did not store a blacklisted flag on orders. The tag is matched live when orders are read, against the store's own list. Adding a buyer flags all of their past and future orders at once, and removing one clears them, with no background job to update orders and no flag to drift out of date. The cost is that every screen returning orders must work the tag out for itself. The order list does it with a join, which the blacklist's unique key limits to one match per order, and the package and returns screens look up every buyer on the page in one query per storefront instead of one per order.

The central engineering decision: where the count lives

The count needs one place that sees every merchant, and the central auth service is the only service every database server can reach. I put the registry there as a set of rows, one per reporting store and buyer, rather than as a counter. That one choice makes reporting safe to repeat: a retried or duplicate report updates the same row, and editing a reason refreshes it without moving the count. Removing a buyer deletes a row, removing a store deletes its rows, and a count can leave out the merchant who is asking, which a bare counter could not do. A count request takes up to 100 buyers and returns totals and a per-reason breakdown, never the stores behind them. The main API checks reasons against its fixed list, while the auth service checks only their format, so a new reason ships without redeploying a service that runs on a different release cycle.

Keeping two services in step

Reports travel to the registry in background jobs, so a merchant's action always succeeds locally, even when the auth service is down. A failed call is retried every five minutes until five failures or two hours have passed, and then reported to Sentry. The jobs re-read the store's own data when they run, so a delayed retry always settles on the current truth: a report retried after the buyer was removed does nothing, a retry after an edit sends the newest reason, and a removal is skipped if the buyer was blacklisted again in the meantime. That depends on one ordering rule: the local row is deleted before the removal job is queued. The other way round, a fast worker could still see the row, conclude the buyer had been blacklisted again and skip, leaving a stale report that inflates every other merchant's count. The auth service allows 100 calls a minute per endpoint from each IP address, and every merchant on a database server shares that server's address, as could every server if they all leave through one gateway. So the jobs on every server draw from one shared budget of 90 calls a minute per endpoint, held in the Redis that all servers share. A job over budget waits and tries again, and nothing is lost. After an outage, a command replays a server's reports, with a dry-run option, and draws on the same budget.

Why a cache, not a Redis registry

Other merchants' counts appear in two places: a blacklisted buyer's details, and the screen for blacklisting a buyer from an order. To stop those views calling the auth service every time, I first moved the registry itself into the Redis that every server shares. I replaced that within a day. Redis cannot be the only copy: an evicted or flushed key leaves nothing to rebuild that buyer from, and a registry of every buyer ever reported only grows. So the database stayed the source of truth and Redis became a cache in front of it. Each buyer has one hash with a field per merchant who has viewed them, because each merchant's count leaves that merchant out. Each field carries its own timestamp and is trusted for 24 hours, and the hash expires a day after its last write, so only recently viewed buyers are held. When the registry accepts a change, the job clears that buyer's hash, so the figures are normally seconds old. A flushed Redis just means a few misses. If the auth service stops answering, a 30-second marker stops every server from calling it, and the response says the figures are unavailable rather than reporting zero.

Signed calls, bound to their endpoint

The platform already had a signed server-to-server link, built by a colleague, with timestamps, single-use nonces and a 60-second window. I added the registry's four endpoints to it: report, remove, count, and purge a store. Each one signs every field it sends, and the auth service checks the signature before it validates any input, so a caller without the secret only ever sees "unauthorised" and cannot probe the rules through validation errors. In review, a colleague pointed out that the signed text did not name the endpoint, so in some cases a signature made for one endpoint could be accepted by another. I put the endpoint name into every signed string, for the registry's endpoints and for the link's older endpoint too. The two services deploy separately, so the auth service accepts both forms during the rollout and logs every old-form call. The old form goes once those logs stop.

Details that mattered

When a store is deleted or reset, its reports must leave the registry too, or they keep counting towards other merchants' totals. The purge job deletes the store's reports and then re-reports every entry the store has at that moment. A reset store can start blacklisting buyers again before a delayed purge runs, and a plain purge, or a plain retry of a failed one, would delete those new reports as well.

What it adds up to

Merchants get a platform-wide warning about a buyer before they ship, shared only as numbers. Tags are always current and never depend on another service. Reports stay correct through retries, edits, outages and store resets, and stay under the auth service's limits however many database servers send them.

How the capture works

The camera feed is drawn onto a canvas, and the order's details are drawn into the same pixels: date and time with the time zone, tracking number, order number, package number, marketplace order ID, and product name, SKU and quantity, each one switchable per store. Because the details are part of the image, they cannot be separated from the footage. The canvas is streamed into the browser's MediaRecorder, and the video is saved to the packing PC. Photos come from the same canvas, up to three per order. They are uploaded and listed in the Parcel Proof Log, and can be sent to the buyer's Shopee, Lazada or TikTok chat. Recording is hands-free: scanning an order starts it, and scanning the next order saves it and starts the next one. The overlay values are frozen at the moment of saving, so a fast re-scan cannot stamp the next order's details onto the previous order's photos. One component serves all three stations, each with its own settings.

Why CCTV footage stuttered

Some merchants already have CCTV over the packing table and bring it into the browser with OBS Virtual Camera, which the browser treats as an ordinary webcam. Only those stations reported choppy video. The cause was in the recorder. The canvas was repainted on the browser's display loop, and the capture stream sampled it on its own fixed timer of 30 frames a second. A USB webcam delivers frames on a steady hardware clock, so its timing stays close to that timer. A virtual camera pushes frames from a software render loop that stutters under load. Against a fixed timer, some of its frames were recorded twice and others were never recorded at all, and that plays back as judder.

The central engineering decision

I made the camera the clock. In CCTV mode the capture stream is created with a frame rate of zero, so it has no timer of its own and only emits a frame when it is told to. The paint loop is driven by requestVideoFrameCallback, which fires once for each frame the camera actually delivers. Each callback draws that frame and the overlay onto the canvas, then calls requestFrame() on the capture track. One camera frame goes in and one recorded frame comes out, however uneven the camera's timing is.

Keeping it to the stations that need it

The new path runs only when two things are true: the store has turned on CCTV mode, and the selected camera's name matches one of 15 known virtual-camera names, such as OBS, ManyCam and DroidCam. Stations with USB webcams keep exactly the path they had. Camera names stay blank until the browser has camera permission, so when CCTV mode is on the component opens the camera once to read them; if that fails, CCTV mode stays off for the session instead of guessing. Browsers without requestVideoFrameCallback fall back to the display loop. Virtual cameras report their canvas size, usually 1080p, rather than what the CCTV actually produces, so the browser would have encoded 1080p from a 720p feed; CCTV mode caps them at 720p. When a re-render swaps the camera's video element, the loop is stopped and restarted so that no callback stays attached to the old element.

Earlier: the recording-lag fix

In February 2026 packing stations reported lag while recording. The render loop never stopped, and every recording session created a new canvas capture stream, so streams piled up and dragged the frame rate down with each session. I rewrote it as an explicit start and stop loop, reused one capture stream per camera and released it when the camera changed or the screen closed. I also kept the recorder in a ref so that stopping a recording never reads a stale value. That work is the base CCTV mode builds on.

What changed

CCTV-fed stations now record every frame the camera delivers, once, instead of repeating and skipping frames. The recorder follows the camera's clock instead of its own timer. Virtual cameras are capped at the resolution the CCTV really produces. Stations with USB webcams are untouched, and browsers without frame callbacks fall back safely.

The wider feature

CCTV mode is the latest of several increments I delivered on the same feature: product name, SKU and quantity on the overlay (April 2026), automatic sending of packing photos to buyers' marketplace chats with a resendable log and 90-day media clean-up (June 2026), and a save-only mode that keeps evidence even when a store does not send it to buyers (August 2026). It detects which resolutions each camera really supports, is covered by Playwright tests that run against simulated cameras, and is translated into five languages.