Between 2017 and 2019 I kept a technical blog. It had 33 posts, half of them syndicated articles I had reposted from other sites, the rest my own: a few career essays and a handful of analyses in R and Python. The site died years ago. Last week I pulled the posts back out of the Wayback Machine to give the good ones a second life on this site.
The first pass was a facelift. An agent rewrote the prose in my voice, stripped the names of employers and friends, and added a “looking back” section to each post. It read well. Then I asked a question I should have asked first: was the code redone to match today’s tools? It was not. The R code from 2018 was tidied, not replaced, and the figures were the original screenshots. The posts described modern practice without doing any of it.
So I had the agent do it. This is what happened, and what I would tell anyone thinking of pointing an agent at their own old work.
The setup
Two analyses were worth a real re-run. One was a five-part series on building a credit risk scorecard on Lending Club’s 2015 loan book, the kind of model I built for a living at the time. The other was a short investigation of employee turnover on a Kaggle HR dataset. Both had code, both had plots, and both had conclusions I had been quietly proud of.
The data turned out to be the easy part. Lending Club still serves its 2015 loan and rejected-application files from its own site, and the HR dataset lives on a dozen GitHub mirrors. The hard part was writing the brief.
For the scorecard I wrote the brief the way I would write a spec for a junior analyst I trusted with the method but not the judgement: keep the 2018 structure, use only features a lender knows at application time, list every excluded column with a reason, split by issue date rather than at random, bin with optbinning, fit an L1 logistic regression tuned by cross-validation, add a monotone gradient boosted challenger with SHAP, run reject inference on the real rejected applications, grade on equal-frequency bins, scale with points to double the odds, and write every number to a JSON file. For the turnover post the brief was shorter, because I wrote that one myself.
The agent took the scorecard brief and came back about ten minutes later with a 750-line script, ten figures, a results file and a report. The report is where the story is.
What it found
The 2018 series reported a cross-validated AUC of 0.8755 and a test AUC of 0.8792 for the scorecard. The re-run, on the same loans with the same labels, gets 0.673 out of time and 0.670 on a random split. The gradient boosted challenger gets 0.700.
I knew the number was coming down. I did not expect it to come down by 0.2. The agent’s report put the cause in one line: the 2018 feature list kept a few columns that only exist after a loan is issued. The collection recovery fee was one. It is zero for every loan that was repaid and positive for most that were charged off. A model that can see it does not need to know anything else. My 2018 test set was a random quarter of the same loans, which is exactly the kind of test that cannot catch this.
It found three more things in the same pass, each of which I had published:
- Five coefficients in the refitted model had the wrong sign, which the 2018 version never checked. The points for those features would have run backwards on the printed scorecard.
- The 2018 reject inference called R’s
predictwithdata=instead ofnewdata=. R ignores the wrong argument silently, so the “rejected applicants” histogram I had written a paragraph about was the accepted loans again. - The score scaling used
log(p)instead of the log-odds, so the claim that twice the risk cost 165 points was arithmetic on the wrong quantity.
On the turnover post, which I re-ran myself in pandas, one of my 2017 conclusions reversed. I had written that workplace accidents added to the risk of leaving for one group of employees. The numbers say employees with an accident on record left at 7.8% against 26.5% for everyone else. I had compared a filtered matrix against my memory of an unfiltered one and seen what I was looking for.
None of this was subtle. All of it had sat on the public internet for eight years.
What it got wrong
The agent was not right about everything, and the mistakes are as instructive as the finds.
The first text extraction missed 19 of the 33 posts, because the old site had changed themes in 2019 and the second theme wrapped the post body in a different container. The parser reported those posts as empty and moved on. I only noticed because a post I remembered as long was showing zero words.
The Lending Club export it downloaded no longer carries the FICO range, which the 2018 file had. The agent flagged this correctly and worked without it, but a less careful run would have silently produced a scorecard with no bureau score and said nothing.
LightGBM crashed twice before it ran. The agent changed two settings at once and could not say which one fixed it. That is an honest report, and also a reminder that “it works now” is not an explanation.
While writing up the home ownership result I read three bad rates off a chart by eye instead of out of the results file, and got all three slightly wrong. The results file caught me. The rule I had set for the agent, that every number in the prose must come from the JSON, applied to me too.
And in the first prose-only pass, one of the writing agents added the phrase “early in my time as an analyst” to a post, then removed it in its own review because it had invented a detail of my biography. It said so in its report. I would not have caught it.
What I would do again
Write the brief as a spec, and put the discipline in the brief. The feature exclusion list with reasons, the time-based split and the sign check were all instructions, not things the agent chose to do. An agent asked to “modernize this” would have produced a cleaner version of the 2018 mistake, with a better AUC and the same leakage. The judgement about what a scorecard is allowed to see was mine, written down before any code existed.
Demand a results file, then check the prose against it. Every number in the two rewritten posts traces to a key in a JSON file the code wrote. This is what let me catch my own chart-reading error, and it is what made the agent’s report trustworthy: it could not claim a number the code had not produced.
Separate the writer from the checker. The prose for the first pass came from agents that never ran code, and it read well while describing nothing real. The second pass had the code run first, and the writing came from the output. Reading the figures myself before anyone wrote a sentence was the single most useful hour of the whole exercise.
Treat the agent as a reviewer of your past self. This is the use I had not expected. A model from 2018 with a number that looks too good is a cheap thing to re-run now: hand an agent the data, a written feature policy and a test set drawn from after the training window, and see what survives. Mine did not, and I would rather know.
Keep the admission. The rewritten scorecard post says the 0.88 was leakage, in those words, with the reason. The turnover post says the accident claim was wrong and why. An agent makes the correction cheap. Publishing it is still a choice.
What it cost
About an afternoon, most of it reading. The scorecard pipeline ran in under a minute on 421,000 loans and a 300,000-row sample of rejects. The agents that wrote the code and the first drafts used a few hours of model time between them. The code for both re-runs is public, on GitHub for the scorecard and in a new repository for the turnover analysis, with every figure generated by the code rather than pasted from a screenshot.
I used Claude Code for all of it, with the site’s own conventions written into a skill file so each agent knew the frontmatter, the voice and the privacy rules without being told twice. That file is in the repository too. Writing it down once is most of why the second pass went better than the first.