Converting 600k lines to TypeScript in 72 hours

Background

Users have been using Lucidchart for a long time to make their diagrams, since 2010 and IE6. And for nearly that long, Lucid has used Google’s Closure Compiler to manage its increasingly sophisticated code base. Closure Compiler is a typechecker and minifier that uses JSDoc annotations in comments to understand type information. As Lucid grew from 5 to 15 to 50 engineers, the tool proved enormously helpful for quality and productivity.

Then in 2016, Lucid began experimenting with TypeScript. TypeScript offers more flexible and comprehensive type checking, cleaner syntax, and better IDE support than we could get with Closure-annotated Javascript.

// Closure JavaScript

/** @const {!goog.events.EventId<lucid.model.DocumentEvents.PagesContentChanged>} */
lucid.model.DocumentEvents.PAGES_CONTENT_CHANGED =
    new goog.events.EventId(goog.events.getUniqueId('PAGES_CONTENT_CHANGED'));

/**
 * @constructor
 * @extends {goog.events.Event}
 * @final
 * @param {!Object<boolean>} pages
 * @struct
 */
lucid.model.DocumentEvents.PagesContentChanged = function(pages) {
    lucid.model.DocumentEvents.PagesContentChanged.base(
        this, 'constructor', lucid.model.DocumentEvents.PAGES_CONTENT_CHANGED
    );
    /** @const {!Object<boolean>} */
    this.pages = pages;
};
goog.inherits(lucid.model.DocumentEvents.PagesContentChanged, goog.events.Event);

is equivalent to

// TypeScript

export const PAGES_CONTENT_CHANGED: EventId<PagesContentChanged> =
    new EventId(events.getUniqueId('PAGES_CONTENT_CHANGED'));

export class PagesContentChanged extends Event {
    constructor(public readonly pages: {[key: string]: boolean}) {
        super(PAGES_CONTENT_CHANGED);
    }
}

TypeScript was well received among our engineers, and by summer 2017, Lucid had 100k lines of TS and 600k lines of Closure-typed JS. To bridge various parts of the codebase, we used clutz (converts Closure-annotated JS to TypeScript declarations) and tsickle (compiles TypeScript to Closure-annotated JS).

While the benefits of TypeScript were clear, it looked like it would be years before we could entirely replace our JavaScript with TypeScript.

Idea

Every summer, Lucid has a two-day hackathon during which Lucid employees work on an interesting project of their choice. The hackathon presented a unique opportunity to migrate to TypeScript while no production-ready code was being authored. Compared to an active migration, a stop-the-world migration could be much faster and less overall effort.

Developing on a fast moving code base is like changing the wheels on a car while it's in motion
Stopping active development for a couple of days significantly speeds up a major migration, it's like pit stop in racing.

Google has an open-source tool called gents (generate TS) which converts Closure-annotated JavaScript to TypeScript. The tool has many caveats and limitations — bare-bones, idiomatic code translates well, but more unusual code and sophisticated language features (e.g. generics) do not. Initially, gents crashed with runtime errors on Lucid’s codebase. A few patches got gents to about 80% of what we needed. Ad-hoc pre- and post-processing scripts added an additional 10%.

Even then, the remaining 10% was still enormous in absolute terms and would possibly be too large to complete in a couple days. We asked our CTO to join in the migration. He supported the idea but preferred working on a project more within the realm of possibility.

Ben's response

Six of us engineers decided to try anyway.

Plans

A very challenging aspect of the migration would be the lack of feedback. Computers are picky things; 70% of code working looks a lot like 0% working. While it would be difficult to get incremental runtime feedback, we decided to at least get incremental compile-time feedback. We would create a dependency graph of the 2000+ files and start working from the leaves, moving each file to a new source root, fixing the type errors, and committing it. At any given time, we would have a set of successfully compiling TS files.

To parallelize this effort across the six engineers, we constructed a 2,840-line spreadsheet with each files, its status, and its dependencies. Once a file’s dependencies were marked completed, the file would be color coded as ready for work. One of us would self-assign it, move it, fix it up after automatic translation, and mark it as done. Then that process would repeat on the remaining files.

Google Sheet Project Status

If you would like to try using this example, make a copy of the Google Sheet.

Hackathon

To process all 2,840 files, we needed to complete one file per minute for 48 hours straight. The six of us worked around the clock, sleeping only a few hours during the two-day hackathon. Some files had cyclical dependencies (allowed, as long as some of the dependencies are type-only rather than value dependencies), and we frequently had to tackle large groups of files all at once. Velocity varied. Twenty hours in, prospects looked bleak. This was disappointing, as partial success for this hackathon project was the same as no success—there would be too many conflicts if this project had to be extended into a normal working day.

There were several recurring challenges:
1. Base constructor requirements are different between Closure JS and TS. Complex, overgrown inheritance trees with tricky initialization semantics were challenging to re-architect.

2. TypeScript is more aggressive than Closure Compiler at identifying errors. For example, tsc detected the assignment-vs-comparison mistake that Closure Compiler did not:

var usernameOnTeam = (alreadyOnTeamFilter['usernameOnTeam'] || []).map(onTeam =>
    goog.array.find(me.loadedUsers, user => user.username = onTeam.username);
);

Naturally, in a legacy system, it’s hard to say what is the bug and what is the unexpected feature. Therefore, we generally left the code as is and added a workaround for the typechecker. No one is proud of this, but we had a VS Code macro to insert `as any` for highlighted expressions.

3. Closure JavaScript predates the ES2016 module system by many years. Traditionally, it uses `goog.provide` and `goog.require` to identify dependencies between files, and all functions are added to a global namespace. Idiomatic TypeScript typically uses modules and imports, which was a substantial paradigm shift. For example, Closure JS permits odd constructs like multi-file classes and implicit circular dependencies, but these do not work with TS imports.

4. The autogenerated imports from #3 meant we often had shadowing conflicts with existing local identifiers. And thanks to #2, we often missed this and mistakenly forced tsc to accept broken code.

Despite these challenges, after thirty-six hours, it looked more likely that we might finish, though it was impossible to be sure. Perhaps it would all compile, but we’d encounter a hopeless wall of errors at runtime. At forty-four hours, we met with engineering leadership to discuss the project’s potential. The stakes were significant. On the one hand, a wholesale rewrite of the entire codebase could harm the company’s success. On the other hand, if it wasn’t completed now, it would take years to reach this point incrementally.

At forty-six hours, we partially loaded the main Lucidchart document list and editor. They were definitely broken, but we had proof of life within the two-day hackathon. That was enough. The hackathon ended on a Friday, and with the bulk of the work completed, we worked through the weekend on rounding out the build process and getting the primary parts of the product working well enough that it would not impede the 60 engineers returning to work on Monday.

Monday at 9 a.m., we pushed. 600k lines of typed JS became 500k lines of TS.

Release

Of course, the story was far from over. We still needed to ship it.

Lucid releases every two weeks and has not missed a release for over four years. The next two weeks were spent getting secondary parts of the product functioning, unit tests passing, tree-shaking and minification working (still using the Closure Compiler), and CI running. QA made multiple passes, finding dozens of bugs. When release day came, every team had a member on call in case of problems. Though we had worked hard to ensure everything would run smoothly, we anticipated something within the half a million lines of a seven-year-old code base to go wrong.

We waited and waited. And the issues never came. The combination of unit tests, manual testing, and a new, very robust type system bought us a TypeScript migration with zero customer-facing issues.


22 Comments

  1. […] Learn how the Lucid engineers successfully converted 600k lines to TypeScript in 72 hours with zero customer-facing issues. Read more […]

  2. Incredible feat. Thanks for sharing this.

  3. […] Learn how the Lucid engineers successfully converted 600k lines to TypeScript in 72 hours with zero customer-facing issues. Read more […]

  4. […] Source: Converting 600k lines to TypeScript in 72 hours – Lucidchart […]

  5. Wow. I’m so impressed with what you all did! Congrats. Really inspiring seeing work like this.

  6. you are known for using tsickle. it has many issues in a non angular env. do you have example of usage ?

  7. Awesome post, and great work!
    Before the migration, did you have 100% type safety using the Closure Compiler?

  8. you write about tsickle and typescript but there is no technical content. tsickle is riddled with bugs and not ready for production. i have encountered many issues with it.
    i don’t see any info in your repos – so can you show us all what is you tsickle configuration ? tsconfig + tsickle config. share your knowledge.

  9. @Fave We did not. Not even the Closure Library has 100%.

    IIRC we had 90-93% type coverage, as reported by the Closure Compiler. Of course, some of those types are weak, e.g. Object. TypeScript makes typing convenient and flexible enough that devs write stronger types.

  10. @Jamie @jake We haven’t had many issues with tsickle.

    We do use tsickle with and without Angular code.

    You can read our blog post about tsickle and Angular 2 and see an example repo.

  11. How did you handle the global namespace problem? Did you convert everything to use module scoping, or did you add shims to export TypeScript-generated types to the global scope?

  12. Idiomatic Closure JS uses global namespacing, and our code was no exception. Our team briefly considered using global namespaces in the new TS as well. However, (1) gents produced modules, not namespaces (2) our existing TS used modules and (3) modules are idiomatic in the TS community. So we bit the bullet and made the modules output from gents work. It took some work up front (before our hackathon) to identify implicit circular dependencies or namespaces split across files or other oddities. Those were removed before the migration happened, and it wasn’t as bad as I initially feared. We didn’t use exporting to namespaces as a general mechanism for the conversion.

  13. Paul & Ryan – I’m now doing the same at neiybor.com. I’ve already caught several bugs and I’m 10% done. Super excited.

  14. Dmitry PashkevichFebruary 13, 2018 at 2:52 pm

    Good luck Derrick! 🙂

  15. […] Converting 600k Lines to TypeScript in 72 Hours Specifically, from Google Closure-annotated JS. javascript typescript […]

  16. I think the CTO knew what he was doing when he said it couldn’t be done in time. He gotcha!

  17. […] import Google-module-system modules and the reverse, with (much of) the type information preserved. One company successfully used the tools we published to automatically translate their entire code base while preserving their minified […]

  18. […] I had been at Lucid for a few months and was at least somewhat familiar with our code base, all the JavaScript was converted to TypeScript in the space of a few days. Angular 1 also changed drastically and became Angular 2. The languages, […]

  19. […] can choose to convert an existing project to TypeScript gradually, or perform a “big bang” conversion, or simply use it on a new project. There are also community projects like TypeWiz that […]

  20. […] 임포트 할 수 있다는 걸, 그리고 그 반대도 가능하다는 걸 의미한다. 한 회사는 우리가 만든 도구를 사용하여 그들의 전체 코드 기반을 미니파이된 결과물을 보존하면서도 […]

  21. BTW – I’ve had a lot of luck lately with TypeScript and React (create-react-app). The Redux, React, and create-react-app systems all support it as a first class citizen now.

  22. […] can choose to convert an existing project to TypeScript gradually, or perform a “big bang” conversion, or simply use it on a new project. There are also community projects like TypeWiz that […]

Your email address will not be published.