← Writing

Feedback Loop Field Guide

Here I’ve gathered as many feedback loops as I could think of and lay my hands on, with their latency in application. For increasing the quality of AI output, it helps to feed back deterministic signals. Starting from the basics, you can reduce errors with strong, pervasive typing, linting and automated testing at multiple levels, but that only covers building a thing initially, not all the other questions: is it the right thing, is it deployed correctly, and so on. There are a few basic guidelines and conclusions at the bottom of the page.

How to use feedback loops, and how they relate ↓What loops to use for what decisions →

high fidelitymedium fidelitylow fidelitylatency range (min to max)◆ human-mediated⚡ AI can shorten this loophover a pill to trace its proxy chain & balancing metrics
machine timebuild timeproduct timestrategy time100ms1s1min1hr1day1wk1mo1qtr1yr3yrBuild· Can we make the thing?CorrectnessIDE inline feedbackSyntax / parserHot reload / REPLType systemLint / static analysis◆ Pair / mob programming · XPRuntime schema validationUnit testsCoverage / mutation testingVisual regression testing ⚡AI code reviewAccessibility checksIntegration testsE2E tests ⚡Error monitoring◆ Code review ⚡◆ Bug reports ⚡ReliabilitySynthetic monitoring / uptime checksAlerts / on-call pagesCanary / progressive rolloutAPM / distributed tracing◆ Chaos engineering / game days◆ Incident post-mortems ⚡SLO / error budget burnPerformanceBundle size / asset weightLighthouse / synthetic performanceMemory / resource utilizationBenchmark tests◆ Load / stress testingCore Web Vitals (field / RUM)Capacity / scaling signalsSecuritySecrets detectionSAST / code security scan ⚡Runtime security detectionSCA / dependency vulnerabilities ⚡Cloud misconfiguration / CSPM◆ Threat modeling ⚡DAST / automated web scanningOpen vulnerability age / patch SLA ⚡◆ Bug bounty / VDP ⚡◆ Pen test / red teamMaintainabilityUnused code / dead exportsCode complexity & coupling metricsBuild & CI durationRework rate / code churnHotspot analysis (churn × complexity)Dependency freshness (libyear)◆ Tech debt accumulation ⚡Delivery flow◆ Daily standup · Scrum◆ App store review gateWIP · KanbanWork item age · KanbanSprint predictability (say:do ratio) · ScrumThroughput · KanbanLead time for changes · DORADeployment frequency · DORAMean time to restore · DORAChange failure rate · DORATask cycle timeFlow efficiency · KanbanPR cycle timeFeasibility◆ Feasibility spike / throwaway prototype · Cagan product risks◆ Technology / model evaluation · Cagan product risks◆ Design / RFC review ⚡DataData quality testsPipeline freshness / data SLAsData anomaly detectionAI & modelEval suite / golden-set regressionModel latency / time-to-first-tokenToken / cost per requestLLM-as-judge scoring◆ AI red-teaming / adversarial evals ⚡Model & output driftValue· Is it worth making?Usability testingFirst-click / findabilityClicks / interaction cost◆ Task ease (SEQ / SUS)◆ Dogfooding◆ Sprint review / stakeholder demo · Scrum◆ Beta / early-access program◆ Usability tests ⚡◆ Task success rate◆ Time on taskUsability signalsAI synthesis of qualitative streamsSession recordings / rage clicks ⚡Thumbs up/down on AI output◆ Public reviews (G2 / app stores / Reddit) ⚡Funnel / drop-off analysis◆ Support tickets ⚡Support opsTime to first response ⚡Resolution time ⚡Contact rate / tickets per customerFirst contact resolutionTicket CSATDesirability◆ Opportunity scoring (importance vs satisfaction) · ODI◆ Willingness-to-pay research · Van Westendorp◆ Feature request board / upvotes ⚡◆ Weekly discovery interviews ⚡Feature adoption / activationFake door / prototype testsSean Ellis PMF survey · Sean Ellis◆ Kano survey · KanoA/B experimentsNPS / CSATSolves the problem◆ Churn interviews / exit surveys ⚡Trial / freemium conversionStickiness (DAU/MAU)Expansion / referral revenueRetention curvesBetter than alternatives◆ Win/loss analysis ⚡Reach· Can we get it to people?PaidCPM / auction pressureCost per click (CPC)Paid ads CTR / CPALanding conversion rate (CVR)◆ Influencer / sponsorship performanceQuality / relevance scoreAd-spend liquidationFrequency / creative fatigueIncrementality / lift testsROAS / channel CACAttributionLast-click attributionMulti-touch / data-driven attribution◆ Self-reported attributionMarketing mix modelingEmail deliverabilitySpam / reject rateDelivered / inbox placementBounce rateClick-through rateOpen rateUnsubscribe / list churnSender reputationOutbound◆ Cold outreach reply rates◆ Meetings booked / pipelinePipeline stage conversionSales cycle lengthContentContent engagementSEO◆ Keyword research (difficulty vs. potential) ⚡Technical SEO / indexationKeyword rankings / SERP positionAI search visibility (AEO / GEO)Domain authority / backlink profileOrganic traffic & CTRContent decay / refresh signalApp storesApp store ratings & ASOShort-form video◆ Comments / audience conversation ⚡Views / impressionsAudience retention / watch timeCompletion / loop rateHook rate / swipe-awaysFollows-per-viewShares & savesOrganic social◆ Comments & DMs ⚡Engagement rateFollower / subscriber growthViral / referralReferral / K-factorBrand◆ PR / press coverage◆ Brand awareness / recallPartnerships◆ Partnership pipelineEvents◆ Events / webinar performanceCommunity◆ Community activity & sentimentTeam· Can we sustain the people doing the thing?HiringOnboarding ramp / time-to-first-commit◆ Sourcing response rate◆ Interview signal◆ Offer acceptance rate◆ Quality of hire / ramp time◆ Time-to-hire◆ Compensation benchmarkingEngagement◆ 1:1 sentimentPulse surveys ⚡◆ Team retrospectives◆ Developer experience surveyeNPS / engagement surveys ⚡CapacityOn-call load / pages per personMeeting load / focus timeBus factor / knowledge concentration◆ Attrition / regretted departuresGrowth◆ Performance reviews / 360 feedback◆ Growth / promotion readinessAlignment◆ OKR check-ins / goal scoring · OKRRisk & Moat· Do the economics allow success to continue?CapitalCloud spend / burn alerts◆ 13-week cash flow forecastAI / inference spend◆ Budget vs actuals (monthly close)◆ Investor / fundraising feedbackPricing realization / discount rateCash collection / DSO◆ Revenue forecast accuracyRunway / burn multipleRevenue concentrationRevenue churn / GRRCAC payback periodMRR / ARR growthGross marginBlended CACLTV:CAC ratioLTVPlatform◆ Platform / API dependency changes◆ Vendor / provider concentrationCompliance◆ Compliance audits (SOC2 / ISO) ⚡◆ Regulatory & legal signals ⚡Trust & safetyFraud / abuse / chargeback rateMoatChurn by tenure / lock-in strength◆ Competitor launches ⚡◆ Pricing powerAnticipatory◆ Pre-mortems ⚡