September 19, 2018
Ep. #48, Resolving Accidental Complexity with Dark
In episode 48 of To Be Continuous, Edith asks Paul what it took to found his fourth startup, Dark.
Jeff explains that unlike a lot of other open source tooling Spike was designed to be both flexible and also not configuration heavy out of the box. Jeff also talks about the different factors that prompted the transition from Roots to Spike such as the complexity of the core code base that made it difficult for contributors to build plugins.
About the Guests
Jeff Escalante is Lead Environment Developer at Carrot Creative. Jeff works full time on open source node.js projects for Carrot. Jeff also teaches part time at Thinkful and is active on AirPair helping people out with front-end and node.
Brian Douglas: Welcome to another installment of JAMstack Radio. In the room we've got Matt Biilmann once again. Matt?
Matt Biilmann: Hey, great to be back.
Brian: Cool, and then on the line we've got Eli Williamson once again, as well.
Eli Williamson: Hey listeners, another day to talk JAMstack.
Brian: Cool. And as our special guest we have Jeff Escalante.
Jeff Escalante: Hi.
Brian: Jeff, do you want to introduce yourself?
Jeff: Sure. I am Jeff. I'm calling in from New York. Super excited to be here, and talk about a lot of static stuff.
Brian: Cool, so I originally reached out to you because I think you gave a talk at JAMstack Dallas, was that correct, very recently?
Jeff: Sort of. It was as a remote talk I guess, the first one of that type that I've done. I kind of just Google Hangouted in and took questions and did a little bit of coding demos for it, which was fun.
Brian: Cool, and I think the subject of the talk was around Spike, the static site generator, was that correct, as well?
Brian: I guess first question, what is Spike?
Jeff: Well, it's a static site generator. I guess usually the way I introduce it is within the context of like the work that I do at my company, just because I'm sponsored to work on this project by my company. And so it sort of was built out of the needs that we have. The major difference is that I work at an agency, which means that we're not working on one product really ever, and we don't do any like long term maintenance type things.
We essentially will take on small medium size projects for a variety of different companies and then we'll build them out and deliver them and that will be the end of it. So if it's a longer running project we'll hand it off to a different team of developers for maintenance. And if it's like something like a marketing campaign or something that wraps up at a certain point then we'll wrap it up and send through analysis.
So, a lot of open source tooling these days is built out by larger product companies, and so we have kind of a different set of problems that we encounter, just because the scope of our work is different and what we're trying to solve is a totally different type of problem. So the way that we build our tools is a little bit different, as well. So for Spike specifically, it sort of has to be built to be both flexible and also not configuration heavy, because depending on who is working on the project or who we are handing it off to, there could be a request for the project to be built in any type of different architecture or different language.
But at the same time, we're constantly starting new projects from scratch. And so having the type of tooling that requires a lot of configuration is really bad for us, just because it continuously sucks up a lot of time. So
Our major goals with Spike were both to be flexible enough to fit any type of configuration needs that might come up as part of a client project while also not being too configuration heavy out of the box.
And I know it kind of sounds like those two goals are at odds, but we've done our best to kind of like make them work together.
Brian: So when you say we and you work at an agency, what agency are you currently with?
Jeff: Sorry, I should've mentioned that. I work at a company called Carrot Creative and we are owned by Vice Media, but we sort of operate independently for the most part.
Brian: Cool, and I'm pretty sure a lot of the listeners are probably racking the brains, because they've heard of Vice Media and they've heard of probably Carrot. I mean you guys have some pretty nice size clients and pretty notable clients, as well, recently. Are you able to say who you've worked with in the past?
Jeff: Yeah, we've worked with a ton of people throughout the years for sure.
Brian: Yeah, I mean just from the website I saw that Adidas was one of them and Cartoon Network was another one.
Jeff: Yeah, we've done a lot of work for Rick and Morty specifically. Adult Swim, Netflix we've done work for, Suave, Red Bull, Clinique, American Express, NBA, MLB, Budweiser, Airbnb, Disney, Facebook, Lipton, Magnum. There's quite a lengthy list. Rolex also in there. So we've done work for pretty much everybody.
Brian: Yeah, that's an impressive list.
Eli: Yeah, absolutely. I'm curious, Vice has been kind of in front of mind lately because of Viceland launching almost a year now I guess. I'm curious if you've been involved with any of that kind of online frontier. I know they're certainly breaking ground on syndicate. I'm curious to hear if you've been involved or Carrot for that matter has been involved in any of the online marketing or static sites associated.
Jeff: That's a good question. We haven't been as much involved with Vice's online presence as I would say some of us would like to be, for sure. We were acquired by Vice a number of years ago, but the way it went was essentially that they acquired us and then we continued operating as usual with them as like a priority client. So people will come to Vice and Vice is known for the quality of their content.
And so companies will come to Vice and ask for them to produce content, and they will say, "Sure, as long as you pay me for it," and then they'll need to deliver that content in some type of container, which usually ends up being some type of website or something like that. And so that was kind of the role that we ended up coming into at the beginning was just like Vice is our content creation experts.
And then they would come to us in order to build the shell and the container around the content that is delivered for their like clients rather than like their primary publication and stuff. Recently we've started working with them a little bit more as far as their own properties, but we're kind of just getting our toes wet with that stuff. So, we might have some more updates to come there, but the integration has been moving along slowly but surely, and I can't say that Carrot directly has had a major impact on any of their web properties thus far.
Eli: Very cool, but for some of the other clients you've used Spike in production, correct?
Jeff: Yeah, we've been using Spike in production for well over a year at this point, and it's been great. It's really helped us to move along really quickly just because it really is built to suit exactly our needs. And the fact that I work here and that I built it certainly helps a lot, as well, because if there is a need that isn't suited we can just earmark a little bit of time to push it in that direction.
Eli: Absolutely. And my understanding is you worked on Roots, which is sort of a predecessor of this. Can you talk a little bit about the difference between the two? I personally have been a huge fan of Roots since the get go. In fact, if Roots hadn't existed I don't think I would've ever found my spot in Netlify.
In fact we spoke many moons ago when Roots shifted to Netlify as the preferred hosting platform and I haven't looked back. In fact, now I work there. I guess I owe you a big thanks, but I am curious to see how this evolution of static site processors has gone from Roots to static and what are some of the differences there?
Jeff: Yeah, wow, that's great. I didn't know that, but that's amazing to hear.
Eli: Thank you.
Jeff: Yeah, well you're welcome. Thank you for using it. I appreciate it. It's always good to hear about people using stuff that I made. It's a great feeling, as I'm sure you all know. So as far as the transition from Roots to Spike, I guess there were a few different factors that pushed us over the line here, and I'll go through them one by one. So the first one was that we had a plugin system for Roots.
And it was okay, and you could do almost anything that you wanted with it. However, in order to use it you really needed to have a good understanding of the core code in Roots. And the core code is very complex. It's a full compiler that was set up to run like as much async as possible.
So really Roots is the way that I learned Node. And it pushed me pretty far down the line with like asynchronous management.
It started out being like using regular callbacks, and then it got to the point where that was too much and I used the async library. And then it got to the point where that was too much and I used promises, which I still love to this very day. But like what I'm trying to get at is that the core of Roots was very complicated and not very many people understood it. And that made it really difficult for people to build plugins.
And so people would often ask for stuff or features and I would really want to say like, "Why don't you contribute this feature if you want it?" which is frequently my response to people asking for features because open source, right? So the unfortunate part about that was just that nobody really was able to because I was like, "Okay, why don't you contribute this feature, but also you need to like understand how this whole project works if you really want to be able to build a plugin that does it."
And it was a good change. It's something we embraced, but the module bundlers sort of have their own like watch systems built in. So it was difficult to kind of like figure out how we could integrate a module bundler with its own system of mapping dependencies and watching with our own system that also watched the static files and tried to figure out how they were built.
And consistently that was kind of a problem with our integration is that since you sort of had to run the two side by side it was always difficult to work with module bundlers with Roots. Another thing was the way that we handled languages. So we had this project called Accord. And it still exists and still is in use pretty widely actually, but essentially what it was was that with Roots we had the same type of requirements as we do with Spike, which is that we need to be able to use anything.
We can't tie it to one specific language or one specific framework. We really need to be able to swap things in and out. But in order to be able to do that there needs to be a consistent interface to each language. So if you want to be able to use like HAML, or like Jade, or like any of the other templating languages there needs to be some sort of consistency that you can plug them into, right?
Because they all did the same thing, but since they were all independent languages they were implemented differently. And so Accord was like if you want to add a new language to Roots you would have to write an adapter for a cord and then it would work, right? And this was fine, but
It added some overhead in that people had to understand how Accord works, and it kind of was an involved process. And like we didn't have a huge community of people writing tons of Accord adapters. So this was like sort of the pain point for us.
And so at the same time we had been sitting on a stack of Jade and CoffeeScript and Stylus, which was our default out of the box with Roots. And these are great, great languages that did a lot for the community and did a lot to sort of inform changes that have gone into web standards, which was awesome. But also during the time that Roots was alive and around the community started shifting away from some of these languages and towards stronger options. So people started shifting away from CoffeeScript and towards using ES6 through Babel.
Brian: It sounds like the simplicity thing that you were talking about with Spike, it sounds like you guys get that with Spike, because I actually looked at the stack of Spike and you guys are using things like webpack and Babel and stuff like that. So it makes sense to have other people come onto the project.
Jeff: Exactly. So the difference with a project like Babel or PostCSS is that they're not fixed spec projects. If you take a CSS file for example and put it through PostCSS, it comes out as exactly the same file, right? You get no effects at all. However, once you start wanting to make certain transformations to the file, you can write a plugin that essentially will parse your CSS, it'll give you the abstract syntax tree, and you can write a plugin that works with that tree, makes the changes you want, and then it will generate back into regular CSS.
We started to really see a lot of incredible potential in them because instead of having a collection of large fixed spec languages that were all independent and all had different interfaces, what we could have is essentially just one language, one library. And through this library and through the plugins you can piece together whatever language you want.
But what we shipped Spike with was a shift from where Accord would say, "You can write an adapter for any individual language," with Spike you just get these three base plugin platforms. And we say, "You can do whatever you want. Here's our recommended defaults, but if you want to change them you can just take from any of the giant library of community plugins or write your own plugin to make this behave how you want."
So the problem of the common adapter interface was solved, and also it gives you a lot of enhanced capabilities because you can essentially write any feature you want and it's accessible to anybody.
Eli: I think the thing that really comes to mind when experiencing Spike is the plugin platform. It really is flexible to make this processor do what you want it to do. But of course there's certainly things that kind of emerge as common plugins. You know, I look at the NPM list of plugins, and I see CMSs. I believe there's an auth thing in there. I know this kind of parallels to some of the work that Matias did on smashing in kind of creating some of our own APIs or plugins if you will around some needs. Matt, do you want to elaborate a little bit on that?
Matt: Well, I mean I think there's a parallel to these things that are not strictly the same, right? So we haven't done any work on the build tools side of things as such, but more on the ecosystem around the build tool, right? And I think that it's a little bit of a parallel between these moves from having these very specific languages that are all or nothing, and to a system that are more based on smaller parts that you can pick and choose.
And to the general trend of the shift that happens when you start decoupling the backend and the front end and moving from a model where where you used to have these very big monolithic apps like WordPress or Ruby on Rails, or anything like that, where you always depended on a very specific ecosystem for that monolithic platform. So if you were making a system with WordPress, you were sort of inherently based on what plugins are there for WordPress and what's in this ecosystem, right?
And everything would in the end be one big monolithic block that had various capabilities. And what we saw in that, like what we did around the smashing project and what we've been working on in general in that space has been building these small microservices that are standalone and then could be combined in very different ways, but that again have those kind of concepts where you find some small concepts that can drive a lot of freedom,
So the concept for example of saying, "Okay, if we use JSON web tokens as the mechanism for identifying a user and for carrying around the information about that user, then we're starting to have all these systems that can potentially understand the JSON web tokens and can suddenly be used together even if they never knew about each other or were not built together."
And then you have this, start having this pluggable layer where you can use a system that uses off zero and just use JSON web tokens and talk to a system like API GoCommerce and understand each other, or you can use our open source GoTrue API and just use the JSON web tokens there and combine it with other things. And you can use these tools together with Spike, and you can use them together with Hugo or together with any kind of build tool.
It's sort of the same journey to go from one very big, very cohesive and tightly coupled monolith into a system where we just in general move to more decoupled units and think more about how can each of these specific paths be swapped out and used together.
Jeff: Yeah, so that's definitely a great point and was certainly a driving factor behind some of the stuff we did. But even more interestingly on the flip side what we were thinking about as well is that as you move off of the monolithic Rails like type of structure into a more smaller, modular type of system, as much as there are a lot of benefits, which there certainly are, there also are drawbacks, right?
And the primary drawback is just that, well, I started off actually as a Rails developer, so this one definitely is true to my heart and I don't know if anyone else remembers this, but at the time when Rails was really the thing, the hotness sort of speak, there was a glorious era where since the conventions were so strong and since all Rails apps were so bound to convention, you would get a situation where somebody could for example work at one company on their Rails app and then move to a different company and jump right in and be totally comfortable immediately getting to work because of the Rails conventions.
So when you move to a more modular system, you do put yourself in a situation where you might actually lose some of those benefits of standards and familiarity for developers.
So this is something that we thought about a lot when we were working on Spike. And what we ultimately ended up landing on was that the best kind of like compromise between those two would be to offer the unlimited modularity and customization, but also out of the box present essentially a preset. Which is what we call a standard, but probably most people are familiar with these through Babel presets.
And I assume that many listeners have used Babel Preset Env for example, which is the main one that they push. If you look into the source of that there's over 100 plugins that actually make up that preset. But, you don't have to think about that. You don't have to configure each one. They just get wrapped up into what essentially is a fixed spec language, right, just like the monolithic languages of old, but it's not all or nothing, right?
You can take that preset and use it. Alternately, you can build up your own preset out of plugins or add an extra plugin to add an extra feature, etc., as you need it. So that balance has been something that's been amazing for us when we're working on customizing different types of builds of Spike.
Matt: Again, it's a very close parallel to the whole reason we started building out Netlify in a way, right, which was also seeing that this decoupled way of building things was really, in our opinion, the future of the web and the right architectural approach to how to build things. But at the same time, of course, there was a bit the same, right? When you took this and said, "Okay, so how do we build things?" Suddenly you had to pick all these different services, right? Like you had to set up the build automation. You had to set up a CDN. You had to figure out the hosting.
You had to solve pre-rendering if that was what you needed. And the list goes on, right? So a big part of our mission has sort of been to to help start creating these presets for the whole stack in a way where you still have that freedom to swap out any layer or to combine things that you innovate and invent and so on, but where we can at least also give people this coherent experience of like here's a stack that you can just get started with and that you can work with across projects and so on.
Jeff: Yeah, that's really great, and definitely is one of the main reasons why we are huge fans of Netlify over here at Carrot and certainly bank on it pretty extensively for a lot of things.
Matt: We are fans of you guys, as well, so it's mutual.
One of them is the watcher and the cache system and everything are just webpacks. We didn't need to run a separate one and a whole separate system. We didn't need to write a compiler. The compiler is webpacks. So the actual logic in Spike is very, very, very small. All of Spike is just a wrapper for the configuration file and a webpack plugin. The whole of the project is no more than like 500 lines honestly, and you can go look at the source and see this. It's very accessible.
And also we have the entire plugin system for webpack available for use with Spike out of the box. So the plugin API is great. A lot of people are familiar with it. It's a very strong community standard, and you can take any plugin and just slot it right in it and it will work really nicely.
So for us this was a really great bonus, because it reduced the amount of code in the code base by an enormous amount, and reduced the amount of maintenance that we had to do and like responsibility we had for the code and off loaded that to some of the absolutely brilliant maintainers and creators of webpack, and also increased compatibility and the number of packages and kind of plugins that you're able to use it with.
So the whole of Spike is just two concepts. It's basically take webpack and extend it to compile static sites, and default loaders into PostCSS, Babel, and Reshape so that you sit with intelligent defaults of these plugin platforms to start building on top of.
And off of that it's been a really big upgrade from Roots for us and has allowed us to like push a lot further than we were able to with Roots in the past.
Matt: How many like roughly projects do you think you've built by now with Spike?
Jeff: Like us at Carrot or like everybody?
Matt: Just at Carrot.
Jeff: 30 to 50 maybe. I'm not sure. It's hard to say. Pretty much all of our projects go out on top of it, because we'll always use the JAMstack is what it comes down to. And we've been proponents of this from before you guys even came up with the term. We had been splitting our apps in that manner to just running an API and running a static front end. And so pretty much all of our work that goes out is going to.
Unless it was a client that specifically asked for something like Rails, which does happen to be fair, all of our work that goes out and when it's up to us it'll go out on a JAMstack and that means we're running the front end on Spike.
Brian: Cool. I think we've kind of covered I guess the gamut on Spike. I'm actually super excited to actually try this out. I just created a new project before we started this call just to make sure I had my feet wet with it, but I'm very impressed. And after hearing you talk about the simplicity of it, again, that's like the biggest key of this is like being able to pass this along to somebody.
If you're working as a contract for somebody within Carrot and then passing it along to whoever is going to own this, whatever company per se, having those developers understand what that is, I think even though this is a module based static site generator, having that consistency is almost like having that convention. So if you guys are doing like 30 to 50 sites so far with the short lived amount of time Spike has been around, I think that's pretty huge.
Jeff: Yeah, definitely. And we have experimented with other static site generators and certainly don't have any beef with any of them. It's just that every one of them has kind of a particular strength and like something that it was made for. And so for the types of things that we run up against, Spike was made exactly for that. So if there is anyone listening who works for an agency or perhaps is a freelancer, it might be something that also would help to solve your problems in the same way.
Brian: So if someone wanted to trial Spike, is there a place they can go, like a website that you could share?
Jeff: Yeah, so we have like a marketing site that's like not 100% done, but it's spike.cf, and that will get you to the right place. Generally we don't have a lot of marketing materials, just because we've done a lot of open source work in the past at this company and have kind of found that as soon as we start publicizing stuff it doesn't necessarily give us a lot of benefit straight off the bat but it takes a lot more time.
So like we'll get a bunch of issues, we'll get a bunch of people asking questions, we'll get a lot more people who are just getting started and are not willing to look through more complex docs or look through the source to figure things out, and we just have had a bunch of work recently and really haven't had a lot of time to have a gap where we can create really polished marketing materials.
So we have a Slack room. It's actually a really great community. And I think the most accessible way to get into it is to come join our community, and I can give you guys a link to that. It's just like a public Slack organization. But there there's a bunch of people who use Spike and are experienced with it and were very, very friendly, very encouraging of contribution. We really try to embrace the open source spirit.
And so I think probably the best way to get started is to come say hello, join our community. You can install Spike with just npm install spike with a global flag. That'll give you the command line tool. And then just run spike new and the name of your project and that will get you started with the template. But past there it kind of is like, you know what, what are you trying to do?
And we also have a doc site, which is fairly thorough, and then we have a Slack community that's very quick and responsive to help out. So between those three, I think it should be pretty straightforward to get started with Spike.
Brian: Cool, yeah, I definitely encourage listeners to try it out. I mean if they've gotten this far through the entire episode and you haven't looked at the site yet, I highly recommend you check it out.
Brian: No, no, it was very enlightening. I know Matt's eyes were like raising, because he's basically been preaching the same thing to us in all his talks as well, too, as well. So with that being said, I want to transition us to picks as we're actually closely hitting the time limit. So these are like JAM picks, things that keep you going. It could be music picks or technology picks, and then with that being said I'll start.
Since you guys are all my guests I'll go first. One thing I wanted to mention I actually was working here downtown in SF. Our office is actually at Dark Patch, but today I had some meetings. So, I wanted to shout out to Onigiri, which is a restaurant here, which I think it translates to rice balls. So it's a Japanese shop that they make rice balls, which is like I've never heard of this before so people are probably laughing at me while I'm trying to explain it, but it's just like basically triangle shaped sushi with stuff in the middle.
They're amazing, and they're actually relatively cheap for downtown SF, so like a couple bucks for an entire triangle. So definitely check that out if it's in your local Japanese food restaurant. Also want to throw out a project I was working on. It's called Open Sauced. I haven't announced it officially yet, but by the time the podcast comes out we'll have a blog post about it.
But, it's an open source project that's open source to manage your open source projects, and it's built on GraphQL, Apollo, and I'm going to add style components just for why not. And it's also built on React. So, if you are listening and you need help managing your open source projects and you want to help open source and help with this, check it out. It's called Open Sauced. And Matt, did you have any picks for us?
Matt: Oh man, I don't know. Like I'm all out of picks right now.
Brian: Okay, you can take a pick pass if you want.
Matt: Don't pick the new MacBook unless you have lots of USB-C dongles, because it's frustrating. I'm out of power right now and I can't find anyone that has something I can charge it with.
Brian: We were joking before, Apple is raising the standard by making sure no one has the standard until they're forced to make the standard a thing.
Matt: But it has a touchpad.
Brian: Yeah, a touchbar. That's basically that's my point. Eli, do you have any picks for us?
Eli: Well, I guess I would second the Open Sauced project. I got the privilege to work on that a little bit and it was quite fun, so go check that out. But, in kind of my own design geekery, I've been really intrigued by Microsoft's Fluent Design System. It just dropped not too long ago, but it's as a designer with the background UX, I get quite fascinated by design systems. They have some interesting principles.
Centrally I'm trying to help define the rules for kind of in between physical and digital worlds. Things aren't necessarily new, but are defined in a way that's quite unique to merging hybrid products. I've really been geeking out over this, and I'm really excited to see how it continues to develop.
Brian: Cool, yeah, the previous episode we had some guys from Salesforce and Braintree talking about design systems, which is it's a big thing. Also, Active Ingredients had a whole talk on design systems, as well. The videos will come out shortly. So, I will definitely check that out. I'm very intrigued about this whole architecture and the movement with the design.
Eli: Yeah, same.
Brian: Jeff, did you have some picks for us?
Jeff: Yeah, I have two quick ones, and they're both tech stuff. So sorry.
Brian: Whatever you're jamming on.
Jeff: The first one is a sort of another like semi Spike related thing, but it is not exactly. So we have a system that we've been tinkering with recently. So right now a lot of people in order to render their React apps faster will server side render it, meaning that they'll set up a server, they'll ship their React codes to the server and to the client, and then the server will render an HTML version and like send it straight down.
And we made a plugin for Reshape, which is like the Babel of HTML essentially we were talking about earlier. And this plugin will render your component to static, and then you can require the same component in on the client side and initialize it there. And it has a little utility for kind of like rehydrating the initial state that you put into your HTML.
It gives you the same benefits that you can use React without having a performance toll whatsoever or having that like blank HTML with the empty root, but at the same time you don't have to set up a server, you don't have to run a server. You can still just deploy it as flat HTML files and it works really well.
So this is really cool. We're super excited about it, and I've been making some videos. We just kind of finished it up like last week. And so that can be found if you just search on GitHub for reshape-preact-ssr, name in progress. And there's plenty more details on this, but we're super excited about it just because like it's static and that's great.
Matt: It's a very good approach I think like also because Node inherently is so based on the event loop, which means that CPU intensive actions are really expensive and can block the whole server, right? And running React on the server for every single request is like just the definition of doing something really CPU intensive, right? So you'll see that a lot of the dynamic server side rendering. And then it also means that in reality they often become quite fragile and sometimes under load they'll be really, really slow to deliver that HTML because you'll have a blocked event loop that's just actually queuing up requests.
Jeff: Exactly, and there's just more complexity, overhead costs. Like for us like we're excited about this because this seems almost like as significant as a shift from running a dynamic app to running like a JAMstack app. And that like you cut the cost a ton, you cut the speed to be a lot faster, it becomes a lot simpler to set up. So we're super stoked about this and I hope that people will come check it out.
And this project called JSS is as close as I've seen to essentially what the equivalent is to a React for CSS, which is really cool, and I'm excited about the potential of it.
The only downside I see is just that switching to a system like that means that you'll be running into a little bit of unfamiliarity since it's not just using regular CSS. But at the same time, React doesn't just use regular HTML, and over time people have kind of adopted that and realized that it really is a great system. So definitely check that out. I think it's really neat, and we might see some more cool integrations of that in the future.
Brian: Cool. Well thanks guys for all those picks. I got a lot of homework now. And Jeff, thanks for coming on and being our guest talking about Spike, and talking about Carrot, and what you guys did and how you guys got here.
Jeff: Thank you for having me.
Brian: Much appreciated. And Eli and Matt, thanks again for jumping on and this conversation. And listeners, keep spreading the JAM.