In episode 51 of To Be Continuous, Paul and Edith are back in the studio to discuss Paul’s new company, Dark. Together they explore some of the challenges that many startups face, as well as what the future of software deployment may look like.
Edith Harbaugh: So, Paul.
Paul Biggar: Yes.
Edith: How's Dark going?
Paul: Dark is going really well.
Edith: Do you have T-shirts yet?
Paul: We don't have T-shirts yet. We made T-shirts and we made swag, we made leggings in fact. Unfortunately they were all see-through.
Edith: So you're saying they were not "Dark?"
Paul: Poor. Dark is going really well.
Edith: Wait, what happened with the T-shirts? I didn't get one.
Paul: We stopped giving them away once a couple of people said they were see-through.
Edith: Were they white?
Paul: Yeah. They were white and extremely thin.
Edith: Why didn't you get black T-shirts?
Paul: We also had black T-shirts. We should get you a T-shirt, our favorite investor.
Edith: Because thank you for wearing your LaunchDarkly shirt.
Paul: I always wear my LaunchDarkly shirt.
Edith: Now I know why, because there's no Dark shirts.
Paul: Right. Yes, so Dark is at an interesting phase. We started building product around two years ago, August two years ago, so about 18 months ago. Maybe a little more than that.
We were at the, "What the hell is this product?" We had a core idea that there's all this accidental complexity in software, and we want a way to allow people to build back ends without all this accidental complexity.
We wanted people to not have to worry about infrastructure, we want people to not have to worry about deployment, for deployment to be instant, and we want people to be able to make APIs easily. Those were our core starting points.
We spent about a year iterating on various things before we came to something where we had a user that was building a business on top of Dark.
Edith: How many people were you for the year when you were just iterating.
Paul: It was growing, but largely I would say, for that year there was four of us with two of us in engineering.
Edith: So you, Allen, Ian--?
Paul: And Steffi. We were working on that for about year before we had a customer, and--
Edith: When you say "customer," were they paying you?
Paul: No. We were paying them.
Edith: You should explain to the listener.
Paul: We posted on the internet and it got quite a bit of traction on the various aggregators where people shout at each other about, "We'll pay you to be our customer."
Edith: That is really interesting.
Paul: Best thing we ever did. Holy shit. Like, I'm so shocked still at how good an idea that was.
Edith: Why do you think was a great idea?
Paul: Because you get to this point with your product, and this isn't true of every product, but this was true of our product where there's a bunch of directions that you could go.
You know your product isn't quite ready for you to market it, like you can't just give it to a thousand people and have some of them use it.
Edith: I think we had the luxury that we did not have a thousand people who wanted to use it.
Paul: Right, and same with Circle.
Edith: I say luxury in quotation marks.
Paul: I think this is a thing for people who are building a very new thing vs. what I think that you're doing on LaunchDarkly and what I did with Circle, which is there's an obvious hole and we are filling it.
With Dark it's not nearly so obvious what the thing needs to be, so we need we need customers to guide us. But it was shit.
We ourselves felt that it was roughly the right direction, but "What are we working on next? Do we need to work on the database feature? Do we need to work on the editor? Do we need to work on the infrastructure? All of it is broken in a million different ways. Which one matters?"
That was why we paid someone, to take that risk on. They would get things out of it, for example they would get not having to worry about infrastructure.
But they would also lose things out of it, like sometimes it's not going to be working and you might have to step away from it for a couple of hours while we fix things.
We paid Daniel, who was our first EIR. We paid $3,000 for three months.
Edith: $3,000 a month, or three grand--?
Paul: $3,000 a month for three months.
Edith: So you were paying him?
Paul: We were paying, yes. It was very much a runway extension for them. It's like "They're not our employees, they're not contracted. They're building their own company," and we had I think 30 applicants.
So 30 people who liked this idea, and we picked Daniel because he most fit the model or the kind of thing that we're able to support.
Edith: Yeah. That's interesting. So, our first customers were people from our accelerator.
In hindsight they weren't very good customers at all. I mean that in the kindest possible way, like they did get give us feedback about usability and ergonomics, but they didn't actually really need us.
They were just using us because I would go and bug them daily.
Paul: Yeah. Especially when you're a product that makes sense once you reach a certain size.
Edith: Yeah. It was like the two person startup who I would go bug every day like, "Have you installed it yet?"
Edith: Not our ideal customer.
Paul: Our ideal customer, there's really two markets that are our ideal customer at the start.
One is people building a client's single page web app or a mobile app, perhaps even a desktop app that needs to talk to APIs and that needs a back end to tie it all together.
That's one use case that we want, and the other use case is people building new micro services in existing micro service architectures.
So you've got essentially something that you talked to via HTTP, that's really what Dark is good at.
Edith: I like the way you've really defined your vision a lot more.
Edith: It's much tighter now.
Paul: Yeah. Because at the start it was just like, "We're trying to remove all the accidental complexity from coding and we're trying to make it 100 times easier to code, and we're trying to bring coding to a billion people." Whatever.
Edith: Or you're like, "What if you could build Instagram overnight?" It's like boom.
Paul: Right, and that's still the mission, but it's much more of a--
Edith: You've narrowed in for a couple of very specific use cases.
Paul: Exactly, yeah. So we were looking for those people, and we got a bunch of people who were like "I want to do this real time chat, or I want to do this variation on Twitter, or I want to do this real time video editing thing."
They're like "Those are outside our scope for now." They will be inside our scope, but for now it's just like literally Dark is a place that you can write code that is connected to the internet.
Edith: That's still pretty broad, though.
Paul: It is still pretty broad, yes. If everything that you need can be conducted via HTTP, we're the thing. But we need to find those people and the place that let us get to, the place that we were able to get by doing that EIR, we did it a second time with Chase.
Edith: Chase, a person? Or the bank?
Paul: Chase is a person, he makes UseAltitude.com which is a-- It's a source for flight deals.
So, rather than one of these like flight deals that you sign up for and there's a million people on them and they get used before you get them, this is someone that you pay I think it's $5 dollars a month and you get personalized deals for the routes that you want to go.
That's built entirely on Dark, and the thing that these two EIR ships let us get to was to go from the phase--
Paul: Entrepreneur-in-residence, slash engineer-in-residence. We aren't really sure which it was, but it was one of those two. Doing this, what let us get to this is, and I think I've talked on this before about the phases that companies go through.
Most companies have the "We're getting to product market fit phase, and then we are scaling or we're developing our business model, and then we're scaling." Roughly sequential order, not much overlap.
What I've realized with Dark is that our pre-product market fit phase is actually split into various different phases as well.
So our first one was like "What the hell is it that we're building?" Now the version is, "How do we make this good?"
We know what it is we're building and we know that we're able to build in it, but many of the parts of it like the editor for example is just not very good.
It's not fun to use, it has a lot of bugs that are core to how we designed it and it was good enough to prove that you could edit in production safely.
It was not good enough to be a thing that people enjoy using, so we're right at that transition point now of trying to make everything solid and good and fun, and eventually when we go public on it we want people to say "Oh my God, this feels amazing. What a great tool."
Edith: Who do you feel is your competitor for that? What are you replacing for that?
Paul: It's hard to say, because Dark is-- It's an editor, it's a programming language, and it's infrastructure. The most obvious thing that we're replacing is running a node application on AWS or Heroku, or Kubernetes or whatever.
That's a large thing, it's not like-- We're not replacing node and we're not replacing AWS, but we're replacing the using of them together, which is how they're used.
Edith: It's nice to be back at HeavyBit today, but I started to pattern match. I was thinking about Warren and his project, which he said was good but not good enough to replace anything.
Paul: The things that make Dark better than the things that come before are the overlap in the problem domains. If you compare Dark to, let's say you compare the Dark editor directly to Vim, or VS Code or whatever.
It's not very good at the things that Vim is good at. With Vim you have a single configuration to edit, many different languages, and you have these corded keystrokes that allow you to do this very powerful feats of text editing.
The Dark editor is not really a text editor, and it's not very good at being a text editor. What it's really good for being is for editing live Dark programs in production safely.
Similar with our infrastructure, we're building infrastructure that is specific to the problem of writing programs in our editor that get into production safely, and that can be maintained live in a safe way.
What the user gets out of that is the ability to just remove large swaths of what they need to do. So there is no CI pipeline as an obvious thing, there's no deployment at all. It's instant.
But it also means that it's very hard to compare directly to a thing that came before and say, "This thing is 10x better than that thing."
It's really that the experience of building back ends with Dark is significantly better than the experience of building back ends without Dark.
Edith: What's your unit of adoption?
Paul: What do you mean by that?
Edith: The two examples you talked about sound like single engineers.
Paul: Yeah, they were. Because we haven't built collaboration yet.
Edith: OK, so is it-- I ask all these questions because I'm genuinely curious, not because I'm trying to--
Paul: Of course, yeah.
Edith: Grill you or put you on the spot. Can two people start using this?
Paul: We're building some stuff internally where-- Just like apps built on top of Dark, and what we're doing at the moment is we're literally passing a baton to the next person.
At this moment in time only one person can really use Dark, and that's just that we haven't built collaboration features.
My expectation is that we'll build collaboration features in the near future.
Edith: Sorry I have all these questions.
Paul: No, questions are good.
Edith: How does source control work with this? How does versioning work with this?
Paul: We have integrated feature flags and version control. They're the same thing.
There was this realization that we had very early on that the way that people deploy software today is like a multi-stage process where each stage introduce risks that isn't actually necessary.
So, you have a change and in that change you deploy that, but you don't just deploy the change you deploy this tar ball or this Docker. Something, like this big unit of change.
You start up your application container and you shut down the other ones.
Edith: That's where blue-green deployments come over.
Paul: Right. So, that thing of getting the new version and turning down the old version used to be how software was delivered, like it used to be "It got in production now users are doing it."
Obviously you make feature flags, but let me let me continue with this narrative . OK, so you've got your thing in production now, but today we don't enable it for users immediately. What we do is we flag it.
Paul: When you enable that flag for everybody, the old code is no longer valuable, and the old code can be removed.
Edith: I could say, people sometimes ask if I have blue-green deployments or any feature flags, and my take is they're for different purposes.
Edith: Like, blue-green is "Is the code successfully delivered to a machine?"
Paul: However, once you have feature flags, what is the value of that risky deploy?
Edith: I'd say the way I break it down is a blue-green deployment is basically has this tar ball code made it out.
Edith: Then I look at feature flags as subdivision within that.
Paul: So, why is blue-green valuable?
Edith: Because machines still have wonky config variables, there's stuff--
Paul: Not in Dark.
Edith: There's still a huge risk with the whole push towards unmutable and config as code--
Paul: That's my point.
Once you have feature flags, then the blue-green deploy thing is just pure risk.
You're just trying to make sure that you don't fuck things up by deploying it.
Edith: Which still happens.
Paul: Which still happens, yes. Exactly. With Dark we remove branches, we remove blue-green deploy, we remove the get thing entirely. The only unit of change is the feature flag.
So, if you have code that's running in production, you feature flag it and you write the new thing, then you switch over to it or you test it or you enable it for yourself.
It's like your dev environment is your production environment with a feature flag maybe set for just you.
Edith: This is pretty forward-thinking.
Paul: Yeah. We started with this idea of getting the deployment process to be sub one second, and really our deployment process is about 40 milliseconds.
We had to re architect everything, there is a need to have the editor in that loop because if you're going to have the editor-- It saves and it touches a file system, and then it goes across the internet, and that's a hard chain.
So, you need to pull all the stuff in to enable that. But once you've got it all in, then what you have is "I can do a deployment in 40 milliseconds, and I can do it safely as well."
Edith: That's what Marianna Tessel talked about when she came a couple years ago about living code.
Paul: I have such a bad short term memory that I remember not a single podcast episode that we have made before this day. You're going to have to tell me more.
Edith: That's great. We have a whole library of 50+ and you can put them in your download directory.
Paul: It'll be like, "Oh my God these people sound so smart. Who are they?"
Edith: Also Paul, you know that we have transcripts?
Edith: Have you checked out the HeavyBit library lately? They have this and many other podcasts.
Paul: Occasionally people Twitter me and they're like, "In that episode where you asked about this thing--" I'm like, "I have no recollection. I've no idea what we're talking about."
Paul: What did Marianna Tessel say?
Edith: She was talking about the idea of living code, so Marianna Tessel when she was on it with us I think she was VP of engineering at Docker and now she's TO at Intuit.
Brilliant woman who I admire very much. But she was talking about her ideal, that when you checked in code it could be live instantly. With the right safeguards. Not--
Paul: That's exactly what we're building. I should go back and listen to that.
Edith: It's a good episode, you were very witty in it and you wore your LaunchDarkly T-shirt that day.
Paul: Right. So, there is version control but it's the same as feature flags and it's like a history of what the code was like before you flipped the flag.
Edith: How do you ever clean it up then, if it's just a layer of feature flags?
Paul: Why would you?
Edith: At some point it impacts readability if you just have all this stuff.
Paul: You switch over to the new thing, and then the thing that was in the flag is now just in the history.
Consider it to be old commits, almost. You rarely go back to look at your old commits, but sometimes you want to.
Edith: I buy into this. It's funny, it's very retro. I mean this is basically trunk-based development.
Edith: It's funny, I give a lot of talks about feature flagging, and I remember I gave a talk in England and somebody came up and said "We had feature flagging 40 years ago because we didn't have version control."
Paul: Interesting. They didn't just make backups with ever increasing names that they copied the directory onto somewhere else and their hard disk? They actually flagged in--?
Edith: Also this was 40 years ago and they didn't have a version control system. So every time they deployed everything went out, and so they said "We basically invented feature flags."
Like, their version of it so that when they deployed they could control it when it went out.
Edith: We certainly didn't invent feature flags, we just try to make them easier to use.
Paul: Yes. There's a realization that many of these features or amend these tools are providing the same feature set, and some of them are doing it better than others or some of them are doing it in a way that is more suitable for one environment, vs other environments.
I think Git is something which fundamentally has not been reinvented for a long time. There was RCS, then there was CVS and then there was SVN, and now there's Git.
But they're all the same tool. They all do the same job, and what we're talking about when we talk about removing accidental complexity is rather than saying "How do we make a better version of SVN?"
We want to say, "How can we reduce the job that is required of this tool to almost nothing?"
Edith: Well, just make it easy for people to focus on what they're trying to do.
Paul: Yeah, exactly.
Edith: There's so many steps in deployment. Another one of my favorite episodes is the one with Kevin Henrikson, where he talks about how he went from a two week deploy process at Microsoft down to minutes.
Paul: Yeah, no. This is coming back to me.
Edith: Yeah. Because every step he's like, "Why?" One of their steps was literally to ship it to Puerto Rico so they can get some tax break. He's like, "Can we automate that?"
Paul: Our deploy time is 40 milliseconds. Any change that you make can be live if you choose it to be live in 40 milliseconds. We had to completely rewrite how software is made, like if you think about Rails.
Rails has longer than 40 milliseconds to start up, and in fact the startup time for the Python interpreter to run Hello, World! is like 70 milliseconds.
The time to run Hello, World! in Dark across the internet in your browser is also about 70 milliseconds.
Edith: I love what you're doing with Dark, and I agree with it. I will say amongst our enterprise customers, they don't want every change to be live like that.
Edith: They want all these layers for variety of very valid reasons.
Paul: Yeah. I completely agree.
Edith: Younger me is like, "Everything should just be live," and I'm like "That's actually a really good reason that they have training of their sales force, they have training of their frontline employees, they need to notify customers."
They want to have some guardrails on if this can be deployed and ready.
Paul: Right. Deployed and ready is our thing as well.
Edith: They want it deployed and ready, but not pushed.
Paul: Yeah. Fundamentally our view on this is we want the practice of building software to be as little accidental complexity as possible, and we wanted to be-- Part of that is it being safe for whatever your use case is?
For enterprise customers, the process of releasing it sometimes has real essential complexity. Maybe it's like SOC compliance, Maybe it's GDPR compliance.
Compliance is a good example of an essential complexity that you can't get rid of by optimizing your software deployment process.
Edith: Something I've thought a lot about is--W e used to have enforcement by Box. L ike it used to be, n ow it's on the QA Box, so it's time to QA. Now it's on the beta Box, so it's time for beta testers.
Continuous delivery allows you to shrink that down so they're all the same Box, but you still sometimes want the same steps to happen right.
You still want to have the QA testers look at it, you still want to have the beta testers look at it.
Paul: We haven't got to this yet, but our expectation is that we will allow you to specify access controls for any place in your Dark infrastructure.
Edith: That's the idea of progressive delivery.
Paul: Right. So, one of the ideas of Dark is everything is written in the same tool.
As a result of that any setting that you make, for example, an access control, can be applied anywhere in the program and can be applied as fine grained or as coarse grain as possible.
Let me give an example here. One of the things that we do in Dark is we store passwords, because this is a thing.
We want people to be able to set the "Read your password" permission to "nobody," obviously.
At the same time sometimes someone needs to come in and say "The passwords aren't saving properly. What do we do?"
We have to be able to give that person a temporary permission to see some requests and that access control can be used for almost anything, as long as you use architecture product to do it this way.
So, that's the thing, we want to anywhere in your in your Dark architecture allow you to specify the needs that you have.
Edith: That's the idea of LaunchDarkly, is controlling code anywhere anytime. I feel once again we've violently agreed.
Paul: It keeps happening. What else can I tell you about Dark?
Edith: How's it going? I have my own questions. I'll say for LaunchDarkly, we always felt like we needed to make money.
I think it was partly that we were first time founders and we didn't have, we didn't feel like we could continue to raise money unless we had a track record.
I have a theory that you can raise money on three things: "Team, dream, traction." If you have a super strong one out of three, you could do it.
We felt we did not have a good enough--
Paul: If you're inexperienced you need them all.
Edith: Yes. So, what's your theory beyond not charging money yet?
Paul: The reason that we don't charge money is that we-- There is a bunch of reasons. One, Dark is not ready to be paid for. Two, we don't want to focus on pricing when we're still figuring out what it is that we're doing.
So we plan to raise our next round, our Series A, with zero in revenue. What we have to prove for that is that this is a game changing thing.
We have different metrics that we're growing, like usage of Dark but usage in the sense that people are able to build real things on this and you can look at people building real things.
And that it has features that make it possible to build those things faster, safer, better, lower maintenance, etc.
Edith: Is it a conscious decision not to charge?
Edith: I think for us, we always just associated money with value.
Paul: There's a lot of reasons to charge money, and to not. One of them is that you can't. In the early days of Circle, we charged money because no one had given us money.
And there's the risk that if you don't charge money that there's no forcing function of providing value to its users, and that kind of thing.
But what we've done is we've essentially orchestrated the entire company around extracting that value. Or, extracting that information.
I talked earlier about the EIR things that we do, that's to get real feedback at a point in the company where it's just like the product can't get feedback from the market.
Edith: I will say that we deliberately-- When we went into beta we didn't charge money because I didn't want to lock in.
Paul: Yeah. Did you lock in a certain pricing?
Edith: I didn't want to lock in a certain pricing. Our first five beta users I said "You need to pay $20 to $100 bucks to be part of our beta program." It was more that I wanted them to at least commit to that.
Paul: Yeah, exactly.
Edith: It really was not about the $20 bucks, it was like "Will you commit to $20 bucks?" If they said no, it's like, OK.
Paul: Yeah, it's really important to get that level of commitment, and money is one of my favorite ways of getting that commitment.
With Dark, if you're building your software on it you're going to be using it hours a day. We get that level of commitment a different way.
Edith: I have a personal question.
Edith: Are you having fun? This is your third or fourth startup, depending on how you count it.
Paul: Yeah, it's my fourth. Depending on how you count it, it could go as high as six. It's a lot of fun, it's a great team and me and Ellen work really well together.
We're building this thing that doesn't exist in the world that we believe needs to exist.
Edith: Isn't that fun?
Paul: It's the best part of startups.
Edith: It's so fun when you go talk to a customer and they're like, "Thank you."
Paul: Right. Occasionally our customers come in to see us and have lunch or spend the day there, and just quizzing them on "Why do you like Dark? Do you still like Dark? Would you go to something else instead?"
Some of the things that we believe are core, it's easy to believe but it's another thing to hear someone telling you, "This thing, I want this all the time for everything else that I do."
In Dark we show you values that if you're looking at a variable or something like that, we'll show you on the side a value that this variable has been recently in production.
But it turns out that it's not just neat, it's a fundamentally different way of writing software, because as engineers we spend so much time going "What is this thing?"
Tracing it through your programming to try to figure out what type it might have, and Dark it's all there at your fingertips.
So you spend your time on producing value, not trying to figure out shit that should have been provided by a tool.
Edith: Sounds fun. Are you building Dark on Dark?
Paul: Not as much as we would like. We've built one small service on Dark.
There's two other services that we're looking to move over to Dark, and I think the core engine is a thing that won't be built in Dark for quite some time.
Edith: That was a big forcing function for us.
Paul: Using LaunchDarkly in LaunchDarkly? Do you use feature flags in the thing that that controls the feature flags, or is there a sacred part of the code base that doesn't use any of your own stuff in it?
Edith: I can't say what we call it, because I get more worried about people hacking us. But we definitely have a service that is running LaunchDarkly.
Paul: Right. For us, there's this bootstrapping problem. We want to run Dark in Dark, but you don't want to be like something goes wrong six layers deep, and then you--
Edith: We have a very stable version that we run off of.
Paul: OK, like six months ago's version is like "You run this thing off to the side, so if your current thing fucks up it's not going to affect."
Paul: How often do you deploy to that?
Edith: Not as often as we like, which sounds--
Paul: I think this is hilarious that the infrastructure providers--
It is hard to use the continuous delivery that we want not because we're against it, but because there is a fundamental problem of running something on top of itself that could just break everything forever.
Edith: This may sound like a total science note, but I was hanging out with a lot of Salesforce people and they were talking about how their customers do not want more than quarterly releases.
Edith: In particular they were talking about Kubernetes, and they were like "Kubernetes has basically weekly releases, and customers don't want that."
Paul: Yeah. That's why someone runs our Kubernetes for us.
Edith: They were literally talking about how somebody had certified a version at Kubernetes and it took them 10 months to certify.
No, a customer had run this version of Kubernetes through all their internal protocols.
Edith: By the time it's ready to go live, Kubernetes is like, "That's dead."
Paul: Yeah. That's got to be fun.
Edith: I don't know who is wrong in this situation.
Paul: Yeah, it seems that everyone is right in some way, but it's all wrong in a bunch of ways.
Edith: Kubernetes is still moving pretty quickly. They're like, "We're not going to freeze for 10 months."
Paul: Also that there are just security releases.
Edith: Yeah. So Dark, you're still having fun?
Paul: At Dark we're still having fun. I think we've identified the core challenges that remain to get to market, and none of them are risky anymore. They're just-- We got to get it done.
We're hiring for this, by the way. But it's all to enable this vision, we need this. It doesn't matter what version that comes in .
Or the other one that we have is, "We build a version of this and it's shit in this way. We need to build a new version that doesn't have that thing."
None of it is, "We don't understand how to build this product anymore."
Edith: What are you hiring?
Paul: We're hiring infra and front end, is our is our main things, but like software generalists and engineers.
Paul: Engineers in San Francisco, unfortunately.
Edith: This is interesting. Did you decide not to do remote?
Paul: We decided that during the phase of our business that we're inventing something new, the high bandwidth communication of being in the same room is essential.
Edith: Did you try to do remote?
Edith: Yeah, I agree.
Paul: Circle was largely distributed, so I have a lot of the experience of doing it. But we will go distributed at a point in the future when we hit another shift in what the company is.
Edith: It's funny, because remote is all the vogue right now.
Paul: Yeah. Are you remote at all? Distributed?
Edith: We have a couple remote employees, but we are very deliberately all in one room for the reasons you just cited.
Paul: Of course you're in Oakland now, so it's easier to hire over there.
Edith: Yeah. I mean back when we were at Heavybit, I would go out and visit prospects. I would come back, I would get coffee with the team, and when I say "Team," we were five people.
Paul: "Here are all the things that I've learned from the conversation."
Edith: Yeah, and it was just organic. It was just how we could move very quickly back then.
Paul: We got a lot of the same thing. It's just someone has a new design for something or they write a new doc about how certain things should work, and you just need very fast feedback.
So being in the same time zone, the same working hours, the same room, they're all an essential part of that. I look forward to the day when we no longer have those-- Where it's not such a strict requirement for those.
Edith: But it's really key, I think, in the early days. Otherwise you have to build another rhythm.
Paul: Yeah. Completely.
Edith: It was nice to hear about Dark.
Paul: Yeah. I'm happy to tell you things about it. We recently wrote a blog post on our Medium, DarkLang on Medium. Also, @darklang on Twitter.
Paul: If you message us on Twitter, I'm sure we can find a T-shirt.