Intro
Adam: Hi, this is CoRecursive and I’m Adam Gordon Bell. Each episode is the story of a piece of software being built.
In 2010, my girlfriend, now wife and I got a TiVo, the DVR, off eBay from the States. They weren’t available in Canada yet and it blew our minds. The first night we plugged it in, it started taping Ghostbusters right away. We never told it to, it just did. Apparently everybody loves Ghostbusters and you could just thumbs up shows you liked and thumbs down ones you didn’t and it would find similar shows and tape them for you.
It was amazing.
Now I have a DVR that my cable company licenses from who knows where and it sucks. I ask it to tape jeopardy and it struggles with that and I don’t blame the people who built that software.
I feel like they have the wrong incentives. They’re selling to my cable provider, not me, and I’m stuck with whatever the cable company chooses and I choose the cable company because the only one that brings fiber to my home. Incentives change everything and I’m not the customer of the DVR maker, their customer is my cable company.
And today’s about that, how bad incentives can make a mess of things. My guest is Shai Almog, and Shai was working at Sun’s Offices in Tel Aviv when the world started to change.
Mobile Market
Shai: Mobile started as just those devices that you would use to make phone calls and turned into something else. And I had the benefit of being at the company that sort of enabled it first and I worked with the operator that was the most innovative in that regard.
Adam: Shai loved working at Sun.
Shai: There was this huge mural of Duke, the Sun mascot on the wall, and I’m a geek that bought Solaris X86 in high school.
Adam: And Shai says nobody knows the real story of the transformation from feature phones to smartphones and the reason that iOS and Android One is not so straightforward.
Shai: I think some of the narrative there is a bit misleading. The popular narrative of, oh, they just made good products that were better than the competition and things like that. That’s not an accurate narrative and the narrative’s a bit more complex.
Adam: You see, pre-iPhone, Sun was a powerhouse of mobile development, Sun made and licensed JVMs for all kinds of devices under the Java two platform, micro edition, J2ME, it was Sun’s most popular mobile software. Second most popular though was something Shai’s friend handmade, something Shai told him that he shouldn’t make.
Building a Light-Weight Mobile UI
Shai: One evening, I remember distinctly, I was walking my dog back then and I get a call from Chen and he’s at Sun at that time.
Adam: Chen was having trouble making a modern looking UI for an app project he was working on.
Shai: So they decided they’ll build their own user interface library. He obviously called me immediately to consult if it’s a good idea to do that, he wants to do this, he wants to do that, and I spent about an hour telling him how wrong he is trying to do that and it’s never going to work because it’s too much work and it will take years, which by the way, I wasn’t wrong about, but it was far more successful than I’d given him credit. And he did decide to do that despite my strong advice to the contrary. And he started building that, after six months or so, they had something.
Adam: So they were able to use that UI toolkit to start building apps. This was around 2007. The iPhone had just come out and only certain operators could get it. The rest were eager for a sleek user interface that could sell phones. They wanted to compete. And so Chen and Shai, two friends from Tel Aviv, were at the forefront of a shifting market, a market worth billions and billions and rapidly growing. And they had literally grown up right next to each other.
Sinclair Spectrum
Shai: We’re literally a couple of months apart in age and we’re like brothers when we grew up. Still are to a large degree. I mean, probably better, fighting all the time. I mean, when I say brothers, but really great friends. And during those years, both of us got Sinclair Spectrum because every responsible parent bought that for his kids back then. Back then the computer would connect to your TV and we had this old black and white TV. It was literally black and white period. And we connected to that huge TV and worked on that, just on the floor in front of the living room TV, and that’s how we did stuff back then.
Adam: It came with basic and there were shortcuts for commands like print and beep and go-to, so you were programming right away.
Shai: I didn’t really know how to read yet, but somehow we figured it out because that was such a simple computer. We were able to build games you would literally change the ASTI characters to different shapes, shapes of cars and sort of make the cars drop, which kind of looked like a race game and things like that. It sort of exposed both of us to programming at that point and really made a huge difference later on when I continued computer classes, 68K assembly at [inaudible 00:05:40], stuff like that, and I started hanging out in computer stores and everything. Chen took a bit of a different route and only came back to programming later on in life. But I got deeper and deeper into that.
90s Craziness and ICQ
Adam: Shai ran a BBS out of his parents’ home and he picked up programming jobs in high school. And then after military service he started working as a consultant for startups. And in Israel at that time, everybody was talking about one startup in particular, ICQ, which was founded by four local guys.
Shai: And when they got sold, it was an explosion around, everyone went completely insane about startup culture and everything, but the whole idea of you can sell a startup for 300,000,000 and four kids who no one heard about suddenly become multi-millionaires, it sort of ignited the collective imagination. A few years later there was a show on television here about a bunch of guys that did that and it became a thing. There is a word for it in Israel, it’s getting organized, but it isn’t getting organized, set for life is the word in English I guess. And it sort of became a thing and everyone was looking, back then in the nineties, everyone was trying to build the instant messaging app. Literally when I started my consulting company, I had I think seven customers that were doing IM apps.
Adam: One of these was something like Intercom, chat with customer support right on a company’s website, and it ended up doing pretty well, but as the need for Shai’s experience grew, he needed to hire more people. And so, one of his first hires was Chen.
Shai: And he didn’t take the same path I did, so I spent lots of time mentoring him and trying to get him up to speed with all of the stuff that we were doing. And he worked for me as a consultant for years.
Adam: This consultant company, it gives Shai lots of real world experience on what works and what doesn’t, what startups succeed and what fail.
Shai: There is two types of startups you run into, the smart ones and the ones that look like the last days of Caligula. And I don’t like those startups that kind of need to send a message of success and they overspend and you come into their offices and their offices are so lavish and amazing and it seems like they’re… And then you go to another startup and it’s in the bad side of the town and they have these sort of wall to wall carpeting instead of parquet and stuff like that. That’s kind of not an ideal environment, and those are the startups are like. You don’t get baked goods in the office kitchen, which is fine by me, I get too much of those, but the money is well spent because sometimes startups reach their situation where they need to fire. And if you fire people and you have these amazing lavish offices, I think maybe one person could have saved his job.
Sun Israel
Adam: So Shai, he keeps things scrappy at his firm, no baked goods, but as they switch between projects, there will be downtime with no billable work.
Shai: One of the things I wanted to do since childhood, my father was a wholesaler and I went to lots of restaurants. I’m still obsessed with the restaurant business. And I was in the back kitchens there of lots of those businesses and I love the whole business of it, and I thought an app to run a restaurant is such a complex thing. They had these touchscreen apps, pretty terrible stuff back then I thought, back then PalmPilots were the big thing and I loved PalmPilots. It was before cell phones became as prevalent as they are. And I thought, why not give a PalmPilot to every waiter with a wireless connection, do the ordering that way. So I started working on that and I used the PalmPilot development kit, which was a nightmare. It was such a terrible environment to work with.
Adam: But Sun had just come out with Java for PalmPilots and Shai was a big Java fan, so if he could get Java for PalmPilots working, he could do all this in Java. There was a problem though.
Shai: All the computers in my company were Linux, no Windows machines, nothing. And that environment from Sun was only for Windows and Solaris. Luckily Sun publishes source code. So I just took it and poured it into Linux, which didn’t require too much work, just changing stuff here and there and getting it to run on Linux. And I posted about it to a mailing list and said, “Hey, I got this working on X, if anyone’s interested, I’m not sure if it’s legal for me to post it, but I’ll be happy to assist,” or whatever. Like an hour later I get a call from a guy called Dov Zaman. And he’s like, “Hey, I’m Dov, I’m from Sun Israel Development Center. Did you know we have an Israel development center?” And I don’t recall the exact conversation, but the way he describes it, which is very, very reliable, is that I didn’t shut up for the following hour, which yeah, sounds right. Anyway, we became friends.
Joining Sun
Adam: So Dov invited Shai to join his team as a contractor. For a Java fan, working at Sun’s offices with its mural of Duke, the Java mascot on the wall, that was mind-blowing.
Shai: I’m a big geek and coming to a company where you’ve got these, every desktop there had a Solaris on it, a spark machine and a KVM to Windows machine. So you’d kind of be able to switch back and forth. And was like, wow, working for a company that has that sort of hardware that you can play with.
Adam: Sun was an amazing place for a person like Shai to work.
Shai: Eric Schmidt was the CTO of Sun for a while before he joined Google. And a lot of the culture of Google or the stuff people like about Google, that’s Sun. Sun was an amazing engineering driven company, for better and for worse, the innovations at Sun, I mean, these guys literally invented distributed computing. They invented RPC, NFS, Git is actually inspired by a project that started at Sun, distributed version control. A guy from Sun quit Sun, built a startup for version control, which later on Leister Voz wanted to use but they didn’t allow him because it wasn’t open source.
Adam: That project was BitKeeper and Linux would create Git to replace it, but Shai’s job at Sun wasn’t about source control or distributed computing. It involved getting JVMs working on mobile phones. Sometimes the phones weren’t even made yet, so Shai was working with breadboards.
Shai: So a breadboard is like imagine if you ever saw a computer open with a motherboard, sort of like that, and you’re supposed to imagine a phone, you can sort of boot it and run it kind of like a phone. So you’ve got these when you’re in a really early stages of development and that’s based on the components that you’ll eventually get. And because these things, the whole packaging and everything, hasn’t happened yet, you are sort of left to work with that. Sometimes you also work with an emulator, which means you’ve got nothing. Sometimes, you do actually have a device, sometimes an older version of a device or a device that’s already on the market and you’re building for the next version, which is compatible.
Adam: The development feedback loop can be a bit slow.
Shai: So, you’ve had a process that’s sort of documented by the manufacturer, and every phone’s a bit different, of how to flush a new version of the firmware to upload your new version and test it on the hardware. Sometimes you were able to do on device debugging through these commands, sort of like remote debugging, but obviously there’s no TCP connection, so it’s remote debugging over these serial connectors.
Adam: So sometimes you couldn’t get much information off the phones and had to use some tricks.
Shai: You’d want to know the values of the variables and that’s where we used colors instead of printouts because we’d have a physical phone and printing out on a physical phone isn’t really very practical. So we’d use colors to signify the value of a variable. If something behaved one way or the other, then update it on the device, look at the pixels to see if it’s green, then it’s that way. And if it’s red, then it’s that way.
Adam: They’d watch those pixels with literal magnifying glasses.
Shai: My eyes are worse today obviously, but they weren’t all that great all the way back then, so magnifying glass is essential.
Operators
Adam: This wasn’t ideal, but in a way it was fine because Sun was getting paid, Sun charged the manufacturers or the operators to port the JVM to new hardware and then Sun also got a licensing fee for the use of the JVM.
Shai: They made money over every device sold with a Java logo. So that was a lot of money obviously because they got money from doing nothing, just licensing money for placing the Java logo and running Java on the device. That was a billion dollar market there just annually. But beyond that, we specifically built VMs for manufacturers and operators. So we worked with companies like Samsung, LG, and companies like that, and built the virtual machines that ran on their phones. Then we’d give them the source code and they’d keep going with their engineers sometimes, sometimes they just kept rehiring us to build and adapt, but still it made a lot of money.
Adam: They made money because operators could fight it out, manufacturers could fight it out, but they were all paying Sun, which was good because Sun’s server business was not doing well. But the J2ME fragmentation was confusing. Some phones with the Java logo were underpowered and others were much more full-featured. The most impressive phones were found in Japan with the operator DoCoMo.
DoCoMo
Shai: They literally invented mobile internet, they invented mobile applications. They had phones with chips on them that you can pay with back ages ago. They had phones with motion sensors, which they used to play golf and fishing games and things like that, all of that way before the iPhone existed. Cameras, double cameras, stuff like that. All of that existed in early 2000s. So they had these amazing phones. The system that they built this on was called Doja, which is the DoCoMo Java environment. I was tasked with porting Doja to a mobile phone by a French company called [foreign language 00:16:50], which I haven’t heard of before, but it’s a big company over there. I worked on that. It was a team of me and another woman I worked with. We were faced with two options. One of them is to take the reference implementation from DoCoMo and use that as the basis and then port that to the phone, which is what every single implementation of Doja ever has done. Or take an existing implementation of the VM that already works on the phone and port Doja on top of that.
Adam: They did the second approach. And so the DoCoMo teams were super methodical at checking this. They compared the phone to binders of specifications.
Shai: And the thing is you can’t truly automate mobile phone tests to that much of a degree. So they’d literally lock a few Japanese guys in an office for a couple of weeks and they would just sit and run a test on the devices one by one, on the physical devices, one after the other, for 12 hours a day. It’s insane and obviously very expensive. So if you failed more than X cycles of tests, you have to start paying fines. And it was a very big deal getting everything working and passing.
Adam: The problem with these specifications, with these tests, is what if they have bugs?
Shai: So for instance, rectangles in Java, really weird. They define them based on coordinates that are one pixel less than what it’s supposed to be based on the width and height because it’s supposed to be drawn on a sort of virtual space between the pixels. This is a very common mistake when you implement this sort of rectangle, is that developers sort of look at the width and just do X plus width and then raw. And you need to do X plus width minus one, or X plus height minus one. And you won’t see that because you’re drawing the rectangle, it will just appear as a rectangle one additional pixel, no difference. But if someone does a clipping area, which means that you don’t draw outside of this particular area, then the part of the rectangle won’t appear. But if you draw it correctly, it will appear.
Adam: Shai knew this behavior, he had been porting JVMs for a while, but the DoCoMo test didn’t include this one pixel offset and so they failed. The contact at DoCoMo got pretty upset.
Shai: He demanded that we’d passed the test because every other device passed it. And I had to prove to him that the bug was in their side and not in ours, and initially he refused to accept it because the lots of phones passed and I proved that all the phones were buggy and ours was working.
Wireless Toolkit
Adam: And as Shai’s team got more projects like this under their belt, they started building tools to help them do their job.
Shai: And one of the tools for Doja was to package a mobile application, a Java mobile application. It’s not just compilation, there’s a process of pre-verifying in Java and there’s a process of synchronizing your manifest to a Judd, which is sort of an external manifest file that describes how the app is built. So the guys from DoCoMo loved that and was a huge success. And Dov said, “I like that. That worked really well. I want to do that for the standard version of Java on mobile devices,” which was the MIPS standard. And we did that. We created what became the wireless tool kit. It wasn’t just a packaging system, it was also a simulator and everything. So when you have a phone simulator, you have device skins that enable you to change the current device to different devices. So you’ll be able to see how the app looks in a different device form factor. So there’s lots of things going under the hood, but you’re also running on a simulator.
Adam: This was a dev tool, a high level dev tool, that made mobile development easier. And if mobile is the future and Java can be the easiest way to write mobile apps for future phones, then that JVM will win and Sun will get more licenses and everybody will be happy. But before they ship it, there’s a problem.
Shai: A person called R, won’t mention his name, just R, was very adamantly against us releasing this product. And the reasoning is that Warland, the company that built Tubro Pascal and stuff like that, is the company that builds IDs, back then they were the big ID vendor. And they build the leading Java ID at the time. And he said if we’ll build a tool to package mobile applications, it’s sort of stealing their business. We’re giving away something for free that Warland wants to charge for and then they won’t like us and they’ll boycott us.
Adam: The Sun way, R said, is to leave the actual high level dev tools to the vendors. Sun builds lower level stuff and also codifies standards and maybe does a reference implementation, but that’s it.
Shai: Now that guy R, when Dov was supposed to show off the product to get review and approval from management, he promised that he’ll come and show a competing presentation where he explains why this product is terrible and shouldn’t launch and Dov kind of panicked. And he sat down with the CEO of Sun Israel, amazing woman, Nancy, and she sort of formed the company and ran the local branch of Sun, and her spirit helped make it such a wonderful place to work at. She sat down with him and she explained to him, “You ignore that guy. You don’t mention him in your presentation, you just do a presentation, you talk the best about what you’re doing. You don’t even mention anything that he says or that he’s going to say because you don’t know what he’ll say and you don’t try to predict him, you do you. Nothing else.” That turned out really well because the guy just stood down and didn’t show up.
Adam: And so the project was released and guess what? Borland loved it. It became a big project and then Chen joined Sun to help him with it. But this was a repeated problem. Sun had smart people and it was at the center of a growing mobile world and they knew that would be important, but internally things were chaotic and full of infighting. Sun knew mobile was going to be huge, but they also knew who was in control.
Enter Apple
Shai: This field is dominated by operators completely, even today, most iPhones sold in the States, iPhones not something else, are bought from operators and not from Apple, which is to me insane. The thing is these guys control the entire thing. They control the manufacturers themselves, they can get them to do whatever they want. And in fact, Nokia back then was built in such a way to satisfy operators because when an operator said, okay, we want the device to be like this or like that, then Nokia had a dedicated team that adapted a device for that operator. Now the thing is if another operator asks for the exact same thing, then Nokia had a separate team for that operator so that the engineers won’t mix with one another and IP would somehow leak between the teams. Now this is super redundant, but that’s essential when you are working with operators because they’re very competitive with one another and they control the market.
Adam: And so the cell phone companies, the operators, they held all the cards, they’re the ones who got the money from the customers. And then, one company shook things up.
Shai: People might have forgotten that by now, but Apple produced a phone with Motorola before they produced the iPhone. This is a common trick, this is a trick that Sony did when they wanted to get into the console game. They worked with Nintendo and then after they learned the business, they went in and did the PlayStation. Same thing is a trick that Apple did. They worked with Motorola, built an iPod phone.
Adam: Then Apple found an operator they could work with one that needed Apple as much as Apple needed them. Steve Jobs did the negotiating and it took 18 months, but AT&T agreed to all Apple’s terms, AT&T will get the exclusive release of the iPhone, but they will be giving up other things.
Shai: And it sort of was a give and take relationship where they used that operator both to boost the operator and to be able to sort of leverage it against the others. Now the other operators didn’t want to let them in. They tried to block Apple because they knew that the moment that they give Apple what Apple wants, then Apple will own them. They knew that it’s a deal with the devil and AT&T signed that deal with the devil.
Adam: This meant that Apple now controlled the software on the phone. Operators couldn’t throw whatever they wanted on it, and in fact when people bought apps for the phone, Apple would get the money, not the operators. So once the iPhone came out, the game changed, but all the operators who weren’t AT&T weren’t going to give up. They needed feature phones as well. And meanwhile, Chen is noticing that Java apps on J2ME, that Sun is making for clients, they don’t always look great. Swing exists for desktops, but on mobile things aren’t as well thought out.
LWUIT
Adam: So Chen, against Shai’s recommendation, starts building a UI toolkit for Java Mobile. And people like it. At this time, Chen was working under Yoav at Sun Israel, a director who prided himself on getting things done.
Shai: He’s a very tenacious person. He can be difficult sometimes, but in a good way if he’s on your side. You might have had this sort of manager where he can be an asshole sometimes, but when he is on your side, he’s on your side and he’s a good person in that sense to have as a leader in your team. And he’s that kind of guy. And I like him because of that, because he’s a very go-getter sort of person.
Adam: So Yoav decides that he wants this mobile UI to be a proper real world project at Sun.
Shai: So he brought me in and I started refactoring all over the place, defining all sorts of things, but the basics that Chen essentially implemented with a few members of his team stayed the same. The only major huge change I made there was introducing the event dispatch thread, which is how modern user interfaces are built.
Adam: The next step is to find mobile app builders and see what they think of this.
Shai: So we went to Convels and Telemap, which are two local companies to build mobile apps, Telemap is a navigation app, and Convels, they essentially invented the field of voicemail. Literally the idea of voicemail was invented by them. They built that entire market. They had lots of other products of sorts and one of the products that they built was visual voicemail. The iPhone just launched and it had this feature where you could see the voicemails on the device and just click them and it played them. And they built something like that that worked with all of their voicemail solutions.
Adam: These two customers helped define the direction of the project.
Shai: They’d come to our offices and we’d go to theirs and we’d help them with the code, both literally debugging to find issues with the library and also in understanding how to design the API correctly to match their needs. Up until this point, I was working with lots of companies, lots of projects and everything, but I never gave too much credit to that step of sitting with the customers and doing those sort of interviews. Even though I worked with lots of customers. It was very technical level of things, I didn’t really understand product development. And this was my first time doing that hands-on, not working with the end result of that or something of that sort. And it was very illuminating, all of these meetings where they’d ask for various APIs and they came well-prepared with lots of ideas, and it really helped improve the final product because when we finally did launch to the general public, people were pretty amazed at how many features we had anticipated in advanced and delivered.
Adam: What the rest of Sun thought of this effort though is hard to say because Yoav decided not to tell them.
Shai: One of the nice things or bad things I guess in Sun, was the fact that every middle manager could do pretty much anything with very little management oversight. And we just started meeting with companies and because there was no charge or anything, we just wanted them to help us define what’s needed, what they liked, what they didn’t like.
Sun’s Struggles
Adam: There was a reason it made sense to keep this project a bit quiet. It had to do with how Sun worked, which Dov had explained to Shai back when he started back in 1999.
Shai: So for the people listening, I’m doing this with my hands because he showed it to me with my hand, but he’s holding his hands widely like this and saying this is Sun. Then he sort of zooms in kind of like you zoom in with a pinch and shows this tiny corner in that, this is the software portion of Sun, that was software, the entire thing around it was the hardware. And then he zooms in, this is the Java unit, again, the smallest portion, the Java unit. And then he zooms in and this is mobile and so forth. He keeps zooming in to that tiny, tiny portion until he reaches mobile, and that’s you over here, that small thing.
And it was a huge company with hardware being the actual main thing that Sun did. Sun did huge, huge servers, that’s what they did. They sold servers. And in 2001, Google sort of revolutionized that whole idea by using PCs. People didn’t actually think that you could build internet websites on PCs and actually build them to scale. eBay, Yahoo, everyone was running on Sun Hardware, all the operators you went to, Telcos, everything. It’s all like racks of Sun everywhere. So IBM dominated the banks when I went there and Sun dominated all the call companies, the telecoms, the startups, everyone was running Sun. And it wasn’t overnight, but it was fast that X86 took over, sort of wiped that market and the company that was essentially a hardware company that sold a lot of hardware had its business sort of wiped.
Adam: And so Sun had to find a way to reinvent itself and it chose Java as the path forward. They changed the stock ticker to Java on Nasdaq and tried to paint this vision that they were a company reborn. Java would run everywhere, desktop server, mobile phone, and web. They told investors they had seen dark times, but the Java revolution was well underway.
Shai: It was sort of a lost company and I mean lost in terms of it didn’t have a direction, there was sort of a management vacuum. The CEO since the founding Scott McNeely got replaced and the company was sort of running without a rudder. People got fired all the time. There were lots of downsizes. And entire markets were evaporating around. The problem was that back then the API for the desktop was Swing, and it had about 30% of the market, more or less. It had enjoyed some popularity but also a lot of derision because it wasn’t pixel perfect with the native UI and it had some issues, but was basically a very good API that had a very strong community around it.
Adam: But Sun needed something new to reinvent itself and to get people excited.
Java FX
Shai: And there was a guy called Chris Oliver. Now I didn’t know him personally, but he built something called F3, which stands for form follows function. The idea was actually pretty cool. At that time when he designed it, Flash was a big thing still and Flash dominated web UI, it sort of took over where applets sort of left off. And keep in mind that Ajax wasn’t around yet at that time. So the competitor was Flash. And he created a scene graph version of a UI library based on a declarative language similar to Java that would sit on top of Java. So you define based on the declarative language, you could design a user interface, and essentially build something that very much looked like Flash, only better because was in Java and had all sorts of advantages there in terms of the language and the structure, which looked really, really, really cool. And people at Sun loved it. And as a result, management at Sun decided to sort of bet the farm on this. They renamed it to JavaFX and gave him a huge team, like hundreds of people in total at its peak.
Adam: So JavaFX has huge teams building layers of indirection and abstraction to get the perfect developer experience. And wouldn’t it be great if you could build mobile apps using this because Moore’s law keeps making phones faster. But it turns out Moore’s Law doesn’t necessarily govern mobile phones.
Shai: Because the CPU improves, but battery doesn’t. So you still need to conserve battery and you can’t overuse CPU. But worse than that, resolutions also double in terms of amount of pixels. So you need to push more pixels into place, so you need more CPU to handle that. And you don’t really get the benefit of the additional CPU. You need to drive more pixels, you need to do more work and all higher refresh rate, same thing. So we don’t have CPU cycles to spare. Moore’s Law doesn’t apply to mobile and never has. It’s a different law that governs that. And this is something that the JavaFX team failed to understand repeatedly. The team was actually really great and I worked with an offshoot of the team that tried to get it running on mobile and I helped them do that, to get something, a proof of concept working on mobile. It was slower than molasses, and practically unusable, but we got proof of concept working.
Adam: But LWUIT, Chen’s UI toolkit, doesn’t have this problem. Its only problem is that it’s not JavaFX. In fact, it existing might hurt the narrative that JavaFX is the future. So a strategy is needed to get internal buy-in.
Shai: So we understood that if we’ll try to push it, the project will be killed because that’s the easy way to do these sort of things. Yoav did the Israeli thing, and he just got on the conference call for JavaOne. There is a conference call before the actual conference where they sort of decide what goes on the main stage. He understood the script and how things work. And he waited for the end of the call where someone says, “Okay, do we have anything else for the keynote?” And he says, “Yes, yes, I have this thing. It’s called LWUIT and it’s a library for mobile.” And oh, it sounds really interesting, prepare a presentation, and that’s it. And the VP heard about it for the first time then. And the thing is, almost everyone else on the call heard about it for the first time at that point and didn’t have time to prepare a counter-argument of why it shouldn’t be there. And this is one of the things that you can only do at a company that’s so badly managed as Sun.
The Java One Launch
Adam: So Yoav flies to California for the JavaOne conference.
Shai: And back then you had physical phones and then you have a special projector that shows you the phone. They still use that. Back then, it was the only option and you can literally, you have a camera on top with the light and you can see the phone physically projected to everyone in the audience. And he stood there with several phones and showed how the user interface updates for everything. And we did a big press release and sat with the marketing team and defined that.
Adam: LWUIT being demoed on stage at JavaOne, and the press release, and the fact that this is a new project from Sun meant that LWUIT immediately attracted attention. It showed up where snarky developers discuss things online, it showed up on Slashdot.
Shai: And back then our UI was, well, we spent a lot of time with developers so we adapted the API really nicely. But the UI was sort of very basic, iPhone reminiscent but not nearly as smooth or anything. In the comments on Slashdot, obviously people derided how bad the demos look in terms of their design and everything and theming, I’d obviously answered in the comments, “Oh, they might not have a designer.” And people made so much fun of me and, “They’re Sun, they have hundreds of designers,” and everything. And obviously we didn’t have any because it’s not how corporate works, if you don’t have a designer in your team, you just don’t have it and have no way of getting one. Sometimes the anonymous commentary in Slashdot actually knows a thing or two.
Adam: Slashdot led to traction and adoption and it became the second most popular mobile download at Sun. Samsung and Nokia adopted it and it just kept spreading.
Shai: So people used it in embedded systems, in all sorts of things over the years. But the thing is, one of the big values is that it’s very, very portable. We designed it from the start to be a very portable API. And a community member relatively early ported it to Android, I’m afraid I don’t remember, but he just submitted it. And another one did a port for RIM, for Blackberry devices. And I did a port to iOS, which was challenging because back then there was no official iOS VM.
Adam: But every step of the way, every attempt to port it to a new platform or add a new feature, they were fighting against Sun.
Shai: Legal, marketing, you name it, the default answer is no, and would we be liable? And it’s a problem in an organization, I get that there is fear of litigation and of exposure, and everything that isn’t led by a VP or something like that is immediately under scrutiny. You always need a higher up that would drag you through the stages of bureaucracy and make the people work with you. And because Yoav only made director and not VP, he wasn’t high enough to pull sway to stop that completely.
Adam: So really Sun had all the components for revolutionary software for mobile phones. They had been doing these JVMs for a while. J2ME was lower overhead than Android’s Dalvik VM, and LWUIT was a solid UI that was easy to port to new devices. But internally at Sun porting LWUIT to other platforms was a risk and got blocked. Meanwhile, operators and manufacturers to compete with Apple, they slowly start to converge around Android. And then Oracle acquired Sun and pushing things forward became even harder because Yoav left.
Code Name One
Shai: And we just couldn’t get the company to move forward. Through the years, I constantly tried to convince Chen to leave Sun and Oracle and to build a startup based on what we were doing there. And eventually he relented.
Adam: The startup would be called Codename One.
Shai: So back then we wanted to take what we did with LWUIT and build a version that worked with everything, with iOS, with Android and everything else, which LWUIT didn’t do because of politics, not because of technical issues. So we didn’t have the actual product yet, but we had a sort of proof of concept because we built the things at Sun and we said we can build them again in the open. Initially we needed the port for iOS and everything and I worked on that. And I worked on the Android port, both built all the ports and tried to get it working. We also engaged the community, we told them, “Oh, we’re leaving to build this company,” and sort of started collecting emails for public beta.
We also needed to build something that would be monetizable, something that VCs would be willing to invest in. Obviously cloud was the big idea, what occurred to me was that when we are building for iOS, we’ll need a Mac, the physical Mac machine to compile. Because legally you can’t compile an iOS application on a PC. The advantage of Java is that it’s portable. Epiphany will have Mac and the cloud and will have Windows machines in the cloud, Windows phone, et cetera, and Linux machines for Android and Dalvik and everything else. And then when people want to build a native application, they can just right click in the ID and say, I want a native iOS app. And it doesn’t send the source code, it sends the jar of the compiled batteries into the Mac that translates it to a native application, compiles it, signs it using native code and then returns a completely native application for you to install on the phone or upload to the app store.
Adam: The problem was there were very few Mac hosts at the time.
Shai: So I literally took a MacBook mini, plugged it in in my home and that’s our server, and you don’t want a direct connection from that, so it did the sort of a polling architecture. So we’d have sort of a redundancy approach and we MacGyvered the whole thing. I had multiple machines in my home representing all of the servers. This was only in the beginning. We quickly moved, found hosts and we were able to move everything away from my home. But during a hot day in the summer, we had downtime because summer’s here get really brutal, and this is a actual true story, crashes. I had three Mac mini’s in my home and then we found a decent host and were able to move there.
Adam: Hosting Windows builds was easier. But yeah, this whole thing became a giant project. They built a UI toolkit, then they needed cloud compilers that could take jars and output Native iOS code and output native Android APKs, and Windows phone apps and native apps and more. But remember Java Swing had 30% of the market share of desktop apps, which meant there was potentially a big market of developers who would want this. With Codename One, a single Java developer could take their Java skills and write native apps for all these platforms at once. So they managed over time to build the vision that Sun had always been talking about. Write once, run anywhere. But there’s a big disadvantage to not being at Sun, that is, nobody cared.
Struggling for Traction
Shai: When you issue a press release from a major company like Sun, like Oracle, like Google, you get lots of traction practically out of the box. You don’t really need to do all that much to bring people. The terrible reality of it is that in a major company like that, they just come and you don’t need to do all that much to get press, to get everything. The machine sort of works and yeah, you need to work on a press release occasionally with the PR team and do all sorts of stuff like that. And from experience, when you release something, one of the hard things as a startup, you release something and crickets. It’s really, really, really hard to get traction. Every single user is unbelievable, sisyphean process of bringing them.
Adam: So Shai and Chen learned how to make noise. They wrote books, they went and demoed at JavaOne. They went everywhere they could go to be seen. And sometimes people would ask Shai if Codename One would support JavaFX on mobile. There were all kinds of reasons why he thought this was a bad idea. And so he started making noise about that.
Shai: I wrote several articles about that and spoke my mind. Being correct and I still think I was correct about everything I said, doesn’t always get you the desired result in the end. And I was right, but I didn’t win any awards for that.
Adam: RoboVM, a somewhat similar project did support JavaFX.
Shai: That meant that they got them to be on the main stage at JavaOne and other conferences every year. Lots of people that worked for Oracle spent time promoting them over the years and giving demos of their products for them. So essentially marketing their product for them. We missed out on that. So I looked at the value of the technical product and I didn’t understand the value of the collaboration in terms of unifying the market behind a single leader. And I missed out on both. And that was also a mistake.
Adam: So Codename One did succeed as a company and it continues to have many, many customers. But while Chen and Shai hit their technical vision, the adoption fell somewhat short.
Shai: Flutter essentially occupies the place where we were. And their architecture is similar to us, although inferior in some regards. We were there first, we should have had that same level of traction. Obviously they’re Google. So as I said, this is the part that’s frustrating. When Google comes out with the product, it gains traction just because it’s Google and there’s nothing I can do about that. But they also proved that the market wants this. And we have a better of product in many regards because it’s a more popular language. Java and Kotlin are more popular than Dart. And still we didn’t get there.
What Could Have Been
Adam: So things could have turned out differently. Maybe if Sun had embraced LWUIT early on or even used their OS experience and their hardware abilities to tackle mobile in a unified way, maybe they would’ve been a player in the smartphone world today. But their incentives were a bit wrong for that and they never found their way after the hardware business disappeared.
Shai: One of the jokes I make, which is 100% correct, is we had several managers during this time and the higher up managers left, one of them left to run, essentially Nokia, and another one left to run Research in Motion. And these are managers that made really terrible strategic decisions at Sun, sort of betting the company in terms of the UI strategy in the desktop and in mobile. And the joke is that we sort of imploded and took with us other companies during that implosion with our bad management decisions.
Adam: So I guess Nokia and RIM could have taken a different path as well. Or maybe if Codename One had gotten VC backing and gotten support from Oracle and caught the hearts and minds of all the Java developers, then building one app for many platforms using the JVM, it might be the default way to do mobile development, who knows? But the thing is, Shai and Chen can never regret that they didn’t make a go of it. Chen took some convincing, but they made a solid effort.
Best Friends
Shai: That’s the one thing I really regret because he got hurt from that to some degree. Chen never would’ve done it without me. So I’m glad I gave him that experience, but we’re still friends despite running a company all these years and fighting as usual as people do. But we’re still friends despite all of that. And I think it’s also because our bond is very strong. It was relatively easy to remain a friend in these situations. I think if it was someone else, we’d probably have broken up.
Adam: So Shai and Chen are still best friends and Codename One is still around. It’s still shipping features with happy customers. It supports Kotlin now, so you can write iOS apps in Kotlin. Still, Shai thinks that Android is not as good as what Sun could have made and that better JVMs exist. And that although they did a great job, they got kind of lucky by being one of the few choices around when Apple shook up the game. And he thinks it’s a shame that the JVM is now relegated to the server side, into Android apps. Replay the early 2000s a couple of times, change a thing or two, and the mobile phone world we have would look totally different.
Outro
Adam: You can find Shai on Twitter. He’s busy doing many different things. And thank you so much Shai for sharing your story.
And if you want to share this episode around your company Slack, take a look at the episode page. There’s a full transcript, there’s some links with nice headings, you can jump around. And until next time, thank you so much for listening.
Sticher
Overcast
Apr 02, 2024
57
min