Deployment with Luke Melia, Aaron Chambers, and Mattia Gheda

Luke Melia, Aaron Chambers, and Mattia Gheda john Taras and Charles to discuss all things deployment!

Luke Melia: Luke has been working with Ember since it was under early development as Sproutcore 2.0. Ember.js powers a SaaS company he co-founded, Yapp, and they funded their business for a couple of years doing Ember consulting under the Yapp Labs moniker. They’re full-time on product now, and his engineering team at Yapp (currently 3 people) maintains around 6 Ember apps. Luke helps to maintain a bunch of popular addons, including ember-cli-deploy, ember-modal-dialog, ember-wormhole, ember-tether, and more. He started the Ember NYC meetup in 2012 and continues to co-organize it today.

Aaron Chambers: Aaron Chambers: Aaron is the co-author of EmberCLI Deploy and is currently an Engineer at Phorest Salon Software, helping them move their desktop product to the web platform. He's been using Ember for 5 years and maintains a number of plugins in the EmberCLI Deploy ecosystem. Aaron loves trying to work out how we can ship JS apps faster, more reliably and with more confidence.

Mattia Gheda: Mattia is a Software Engineer, Ember hacker, Ruby lover and Elixir aficionado. Currently he works as Director of Development for Precision Nutrition where Ember, Ruby and Elixir power several applications. He loves meetups, organizes Ember.js Toronto and co-organizes Elixir Toronto.

Resource:

• Immutable Web Apps

---

Please join us in these conversations! If you or someone you know would be a perfect guest, please get in touch with us at contact@frontside.io. Our goal is to get people thinking on the platform level which includes tooling, internalization, state management, routing, upgrade, and the data layer.

This show was produced by Mandy Moore, aka @therubyrep of DevReps, LLC.

Transcript:

CHARLES: Hello and welcome to The Frontside Podcast, a place where we talk about user interfaces and everything that you need to know to build them right. My name is Charles Lowell, a developer here at the Frontside. With me also co-hosting today is Taras Mankovsky. Hey, Taras.

TARAS: Hello, everyone.

CHARLES: Today, we have three special guests that we're going to be talking to. We have Aaron Chambers, Luke Melia, and Mattia Gheda who originally met collaborating on fantastic open source library that we, at the Frontside, have used many, many times that saved us countless hours, saved our clients hundreds of thousands of dollars, if not more. ember-cli-deploy. We're gonna be talking not about that library in particular but around the operations that happen around UI. So, welcome you all.

LUKE: Thanks, it's great to be here.

CHARLES: Like I said, I actually am really excited to have you all on because when we talk about the platform that you develop your UI on, something that often gets short shrift in communities outside of the Ember community is how do I actually deliver that application into users' hands. Because obviously, we don't want it to be working just on our laptop. We want it to be delivered to our users and there are myriad ways that that can happen and it's only gotten more complex since the last time we talked which must have been like three or four years ago. I kind of just have to ask, I think that what you all were talking about then was cutting edge is still cutting edge now but there must have been some pretty incredible developments like in the last three or four years. What have been kind of the new insights that you all have?

LUKE: I think that what we realized as we got started with ember-cli-deploy and a project kind of came together as a combination of a few different open source efforts, something that Aaron was working on, something that our collaborator Mike was working on. We decided to come together under one umbrella, joined forces. And what we realized pretty soon is that deployment needs vary a ton between companies. And so, we are coming from this background in Ember community where we had this attitude where nobody is a special snowflake. We all kind of have the same needs for 90% of what we do. And that's true. I really believe in a lot of that Ember [ethos]. But when it comes to deployment, you know what? A lot of companies are special snowflakes or it's at least is much more fragmented than kind of our needs around on the JavaScript side.

And so, what we decided to do was to try to evolve ember-cli-deploy into a platform essentially, an ecosystem that could let people mix and match plug-ins to do in their organization without locking them into an opinion that might simply be a non-starter in their org.

CHARLES: It's hard enough to have opinions just around the way that your JavaScript code is structured but when it comes to rolling out your app, it really does encompass the entire scope of your application. So, it has to take account of your server. It has to take account of your user base. It has to take account of all the different processes that might be running all over, distributed around the Internet. Maybe somewhere on AWS, maybe somewhere on Legacy servers but it has to consider that in its entirety. So, it's having opinions that span that scope is particularly difficult.

LUKE: Yes. And so, you mentioned a bunch of technical details which are absolutely forcing factors for a lot of words in how they do their deployments but what we found in talking to people that there are also people in political aspects to deployment in many cases. Engineers kind of own the JavaScript code that's running within their app, more or less. But when it comes to pushing the app into the world and a lot of companies, that means they're interacting with sysadmins, ops folks, people who have very strong opinions about what is an allowable and supportable way to get those deployments done and to have that stuff exist in production. And so, we needed to come up with an architecture that was going to support all these kind of varied use cases. And so, we came up with this system of essentially a deployment pipeline and plugins that can work at various stages of that pipeline. And that ecosystem has now grown quite a bit. It's actually, I don't know Aaron if you and Mattia would agree but I think it's probably the best decision that we made in this project because that ecosystem has grown and evolved without us needing to do a ton of work in maintenance. And it's been really great. I think Mattia, you pulled some of the current numbers there.

MATTIA: Yeah. I pulled some numbers just yesterday and we have currently 150 different plugins attached to themselves, to different parts of the pipeline. So some of them are about how to build the assets, some of them are about how to compress them, some of them are about shipping. And they allow people to ship it with different ways like we are seeing [inaudible] with just simply Amazon APIs or Azure APIs and some of them even are about just how to visualize data about your deployments or how to give feedback to the user about what was deployed and represent the information.

And then this is kind of a bit more detail, probably specific to us but we also added this idea of plugin packs. So, in order to help people define their deployment story, we created this ideal of plugin packs. Plugin packs are simply group of plugins. So, plugins grouped together. As a user, if you want to implement what we call a deploy strategy, you can simply install a plugin pack and that will give you all the plugins that allow you to deploy in a specific way. And that's kind of like an optimization that we added just to make it easier for people to share deployment strategies, share ways of deploying applications.

CHARLES: Right. It's almost like an application within the framework.

MATTIA: Yeah, exactly. But to stay on the community side, I think that the interesting part about what Luke was saying which was a great success for us is that all we maintain as the core team for this project is the core infrastructure, so the pipeline and a thousand of plugins. Everything else is community-based. And often, even in my day-to-day work, I end up using plugins that I didn't write and that I don't even maintain. But because the underpinning of it were designed especially by Luke and I are flexible enough. It just like has been very, very stable and very, very reliable for many years. So, I will say definitely the idea of like in the spirit of what Ember is, I guess, creating a shared ecosystem where people can add what they want and extend what was provided has been the one single biggest win of this project.

CHARLES: One of the things that I'm curious about is we've talked about how you're allowing for and kind of embracing the fragmentation that happens in people's kind of the topology of their infrastructure. What do you see is the common threads that really bind every single good deployment strategy together?

MATTIA: My biggest thing here, and we actually have some shared notes about this, but my biggest thing about this is the idea that building and deploying an application for me is divided into three parts. There's building part where you have to decide how to compile your JavaScript application and how to produce some sort of artifact. There is the shipping part where it's about deciding where you're going to put the artifact. And then there is the serving part which is how you show it and deliver it to your users. I think that these three are the underpinnings of any deploy strategy. What we did with this project is just acknowledging that and give each one of these a place. And so, the entirety of what we do in what Luke defined as a pipeline is simply give you a way to customize how you build, customize how you ship, and then customize how you serve.

So yeah, I think that that's kind of the root. And the question that everybody that wants to deploy a modern JavaScript application have to ask themselves is how do I want to build it, how do I want to ship it, and how will I serve it to my users. And these things are completely independent one from the order in the sense that you can have something build it, something ship it and something serve it, and that's what we end up doing in most of our deployments, I find.

CHARLES: It's good to think about those things as soon as you possibly can and make sure that you have all three of those bases covered really before you start adding a whole ton of features.

TARAS: Sprint 0, right?

LUKE: In Agile, we call Sprint 0 the phase the thing you do right in the beginning. You've got a skeleton of an app and then you get the deployment infrastructure going, you have the test infrastructure going, so that there is no task within your actual feature development where you have to do those things. And I think that can be a valuable concept to embrace.

I would just add to Mattia's three points that for deployments, to me, some very simple qualities of a good deployments are repeatability. You need to be able to reliably and consistently run your deployment process. Sounds simple but there's plenty of operations that have run up way too long on manual deployments. So, we don't want to see those rollback capabilities if you have a deployment that you realize was a mistake right after it gets into production. I'm sure none of us have ever experienced that.

CHARLES: That never happens to me.

LUKE: You want to have a method to roll that code back. That's something that can be remarkably complex to do. And so, having some guardrails and some support mechanisms to do that like ember-cli-deploy provides can be really useful. But whatever your approach is, I think that's a necessary quality. And then I think we start to step into kind of more advanced capabilities that a good deployment architecture can provide when we start to think about things like personalization, A/B testing, feature flags, these kinds of things. And that requires more sophistication, but you cannot build that on a deployment foundation that's not solid.

AARON: I think for me, one of the things I've been really thinking about a lot lately, it's a bit of a mindset shift, I think, to get to where the things Mattia was talking about separating those different parts of deployment. And so, I really start to realize the traditional mindset around deployments like I build some stuff and I ship it to the server and then the users get it. But if we can actually stop and actually split our understanding of deployment into two separate phases. One is the building and the physical shipping of the files; and the other one's actually making them available to people. You open up this whole world of our features that you wouldn't normally have. So to be able to actually physically put stuff in production but not yet have it active, as in users don't see it yet but you can preview those versions in production against production databases. And then at some point after the fact, decide, "Okay, I'm now going to route all my users to this new thing," And to be able to do that really easily is massively, massively powerful. And so, to me, the thing I've been thinking about lately is it is a small mindset shift away from packaging everything up and pushing and overwriting what's currently there to being something, again like Luke said, immutable deployments where everything we build and ship sits next to all the other versions and we just decide which one we want to use to look at it any time which leads into then, I guess, A/B testing, feature flags and things. So I guess deployment really is not so much about the physical shipping, that's one part of it. To me, deployment now is shipping of stuff, as in physical deployment and then the releasing it or enabling it or activating it to users.

CHARLES: Or routing it. Sounds like what you're describing is an extraordinarily lightweight process.

AARON: It is, yeah.

CHARLES: To actually route traffic to those files.

AARON: It is. It's incredibly lightweight. That's the amazing thing about it. When you think about it, you're building a few JavaScript files and CSS files and images and putting them on a CDN, and then you just need a tiny web server that basically decides which version of the app you want to serve to people. There's not much to it at all, really.

CHARLES: I mean that's absolutely fascinating, though the capability that you have when you have the ability to have these versions, the same versions or different versions of your application sitting along next to each other and being able to route traffic. But it also seems to me like it introduces a little bit of complexity around version matching because only certain versions are going to be compatible with certain versions of your API. You have different versions of your API talking to the -- so the simplicity of having kind of mutable deployments, so to speak, is that everything is in sync and you don't have to worry about those version mismatches. Is that a problem or this could just be me worrying about nothing? But that's kind of the thing that just immediately jumps out to me is like are there any strategies to manage that complexity?

LUKE: To me, what you're describing, I kind of think of as a feature not a bug. And what I mean by that is that it is very simple to have a mental model of, "Oh, I have a version of my JavaScript code that works with this version of my API." And as long as I kind of deploy those changes together, I'm good to go. The reality is that that's impossible. The JavaScript apps that we write today, people are using anywhere from two seconds at a time to two days at a time. It's not uncommon these days to have some of these dashboard apps. People literally live-in for their job eight hours a day, nine hours a day, keep the browser tab open and come in the next morning and continue. And so, obviously there are some mechanisms we could use to force them to reload that kind of thing. But at some point in most apps, you're going to have a slightly older version of your JavaScript app talking to a slightly newer version of your API for either the span of a minute or perhaps longer depending on their strategies.

So to me, the process of thinking about that and at least being aware of that as an engineer thinking about how your code is going to get from your laptop into the world, I think it's an important step that we not paper over that complexity and that we kind of embrace it and say, "Hey, this is part of life." And so, we need to think about just like we need to think about how your database migrations get into production. That's not something that you can paper over and just have a process that it's going to take care of for you. It requires thought. And I think that this, in the same token, how different versions of your JavaScript app are going to interact with your API requires thought. An exact parallel also had different versions of a native mobile app that go into the app store. How did those interact with different versions of your API? So, I think you're right. There's complexity there. There's ways that we can try to mitigate...

CHARLES: Keep repeating ourselves if we think that that [inaudible] actually doesn't exist even in the simple case?

MATTIA: Yeah. I think that that's to reiterate what Luke is saying. That's exactly the point. You can pretend it's not there but it is and you have no way to avoid it. Once you ship something to a browser, you have no control over it anymore. And so, you have to assume that somebody is going to be using it.

LUKE: Aaron, I think you too, I don't know if you can share it. But you recently told us some stories about kind of what you encountered in your work about this and of how long people were using versions and stuff.

AARON: Yeah. Something that we hadn't sort of put a lot of thought into. But the last place I worked at, we had quite a long lived app and we're using feature flags and we're using launch [inaudible] something and it gives you a list of flags and when they were last requested. And there were also flags that we removed from the code and it was just a matter of waiting until all the users had the most current version of the app and weren't requesting the flag anymore. But this one flag just kept getting requested for months and we just could not work out why. It really sort of opened my eyes up to this exact problem that these long lived apps set in the browser and if you have someone that just doesn't reload the browser or restart the machine or anything, your app can live a lot longer than maybe you actually realize it is. So we're shipping bug fixes, we're shipping new features, and we're all patting ourselves in the back. We fixed this bug but have we really? If your users haven't reloaded the app and gotten the latest version, then you haven't actually fixed the bug for some number of people. And it's really hard to tell them as you think about this and put things in place, really hard to tell what versions are out in the world, how many people are using this buggy version still.

CHARLES: Yeah, that's an excellent point. I haven't even thought about that. I mean, what is the countermeasure?

AARON: We hadn't made it until we came across [crosstalk].

CHARLES: It's nothing quite like getting smacked in the face of the problem to make you aware of it.

AARON: That's right.

CHARLES: So, what's the strategy to deal with that?

AARON: I guess for me, my learnings from that would be from very early on thinking about how we're going to encourage people to reload, to start with, and maybe even have the ability to force a reload and what that means but then that has gotchas as well. You don't want to just reload something when a user is in the middle of writing a big essay or something like that. But definitely thinking about it from the start is one of the things you've got to think about from the start. But I guess something that I'd like to implement and I've kind of thought through but not really explored yet, but the ability to see what versions are out there in the world and there are things I've been thinking about in terms of this little server that serves different versions. Maybe we can start having that kind of tracking what versions are out there and who's using what and being able to see because it would be great to be out to see a live chart or a dashboard or something that sort of shows what versions are out there, which ones we need to be aware of that are still there and even what users are using and what versions they can maybe even move them on, if we need to. But there's definitely a bunch of things that aren't immediately obvious. And I don't know how many people actually think about this early on, but it's critical to actually think about it early on.

MATTIA: Yeah. I was going to share what we do which is very similar to what Aaron said just maybe for the listeners to have some context. The first thing you can do is basically what Gmail does, which is every time a web app sends a request, an [inaudible], it will send the version of the app with the request and the backend can check. And the backend can check if you are sending a request from the same version that is the most recent deployed version. And if it's not, this sends back a header and the same way that Gmail does, it will display a pop up that is like, "Hey, we have a new version if you want reload." And on top of that [inaudible] is that we have a dead man's switch. So if we accidentally deploy a broken version, a part of this process, the frontend application tracks in the headers. And if a special header is sent, it force reloads, which is not nice for the user but it's better and sometimes is critical to do so.

CHARLES: Right. I remember that was that case where Gmail released something. They were doing something with broken service workers and the app got completely and totally borked. I remember that my Twitter blew up, I don't know, about a year ago I think, and one of the problems I don't think they had was they did not have that capability.

MATTIA: I mean, you learned this the hard way sadly. But I think these two things are definitely crucial. And the third one, [inaudible]. I thought, Luke, you had the ability that Aaron was talking about like tracking versions in the world. And I think that's more useful for stats so that you know how often your users update. And then you can make the design decisions based on that and based on how much you want to support in the past.

LUKE: Yeah. We haven't implemented that but it reminds me whenever there's a new iOS version, we do a bunch of mobile work in the app and we're always looking at that adoption curve that's published. A few different analytic services publish it and say, "OK. How fast is iOS 12 adoption? How fast are people leaving behind the old versions?" And that helps to inform how much time you're spending doing bug fixes on old version versus just telling people, "Hey, this is fixed in the new OS. Go get it." But if you are able to see that for your own JavaScript apps, I think that would be pretty hot.

CHARLES: Yeah. Crazy thought here but it almost makes me wonder if there's something to learn from the Erlang community because this is kind of a similar problem. They solved 20 years ago where you have these very, very long running processes. Some of them there's some telephone servers in Sweden. They've been running for over a decade without the process ever coming down. And yet they're even upgrading the version of Erlang that the VM is running. And they have the capability to even upgrade a function like a recursive function as it's running. And there's just a lot of -- I don't know what the specific lessons are but I wonder if that's an area for study because if there's any community that has locked in on hot upgrade, I feel like it's that one.

LUKE: That's a terrific analogy. I bet we could learn a ton. Just hearing that kind of makes me think about how kind of coursed our mental model is about updates to our JavaScript apps. We talked to Aaron, we're talking about kind of this idea of a mutable apps and you have different versions side by side. But the idea of being able to kind of hot upgrade a version with running code in a browser, now that's an ambitious idea. That's something I'd say, "Wow! That kind of thing would be a game changer."

CHARLES: Yeah.

AARON: Makes me feel like we got a whole bunch of work to do.

CHARLES: We welcome them. I'm always happy to give people plenty more work to do. No, but how they manage even being able to do migrations on the memory that's running. I don't know if it's something that's going to be achievable but it sounds kind of like that's the direction that we're heading.

LUKE: It does, as these apps get more complex and they continue to live longer. The idea, the work arounds that Mattia mentioned about kind of showing a message and having a dead man's switch, these are all certainly useful. And today, I would say even like the best practice but they were not what you would want to do if you could magically design any system. If you're taking a magical approach, the app will just be upgraded seamlessly as a user was using it. And they would be none the wiser, the bugs would be fixed. End of story. There's no interruption to their workflow. At least for me personally, I don't really think about that as a possibility but I love the Erlang story and analogy and to say maybe that is a possibility, what would it take? I would obviously take a collaboration across your JavaScript framework, perhaps even JavaScript language features and browser runtime features as well as your backend and deployment mechanism. But I think it's a great avenue for some creative thinking.

CHARLES: I'm curious because when we're talking about this, I'm imagining the perfect evergreen app but there's also feels like there's maybe even a tension that arises because one of the core principles of good UI is you don't yank the rug out from underneath the user. They need to, at some point and we've all been there when the application does something of its own agency, that feels bad. It feels like, "Nope, this is my workspace. I need to be in control of it." The only way that something should move from one place to the other without me being involved is if it's part of some repeatable process that I kicked off. But obviously, things like upgrading the color of a button or fixing a layout bug, those are things that I'm just going to want to have happen automatically. I'm not going to worry about it. But there is this kind of a gradation of features and at what point do you say, "You know what? Upgrading needs to be something that the user explicitly requires," versus, "This is something that we're just going to push. We're going to make that decision for them."

LUKE: Yeah, it's a great question. One of the things that I'm curious what you all think is when you think about the mental model that our users have of working in a browser app, do you think that there is a mental model of, "Oh, when I refresh, I might get a new version." Do people even think about that? Or are they just like your example about a button color changing as kind of a minor thing. I don't even know if I could endure stack. We've all been I think in situations where you do a minor redesign and all of a sudden, all hell breaks loose and users are in revolt. Take the slack icon. So, I think it's a fascinating question.

CHARLES: I don't know. What's the answer? Do you always ask for an upgrade just observing? I don't have any data other than observing people around me who use web applications who don't understand how they actually work under the covers. I don't think it's the expectation that this code, this application is living and changing underneath their feet. I think the general perception is that the analogy to the desktop application where you've got the bundled binary and that's the one you're running is that's the perception.

AARON: I'd say the difference there is that, and with all these new ways of deploying, we're shipping small things faster in multiple, multiple times a day or even an hour. So it's not the sort of thing you really want time to use. There has been an update, need to upgrade as well. And that's the difference between the desktop mentality. And if that's the mentality they have, it sits quite a bit of a shift, I guess.

MATTIA: It makes me think that one of the tools that the users -- if you take a look at the general public, there's probably one tool that everybody can relate to which is Facebook. So, I think if there is a way to say what do people generally expect. There is a business user which I think we are often most familiar with but the general public, probably what they're most familiar with is what happens in Facebook. And I don't use Facebook almost. I haven't used it for a couple of years but I wonder how much of what people experience in Facebook actually impacts the expectations around how applications should behave.

LUKE: I think that's a really good question. I do think your underlying point of you have to know your own users I think is an important one also. Obviously, some folks are going to be more technical than others or some audiences will be more technical than others. But I would even question, Charles, your suggestion that people think of it kind of as like a binary that it stays the same until you refresh. I think people have an idea that web apps improve over time or sometimes they get bugs but hopefully that they improve and change over time, and that there is a tradeoff there that means sometimes there's something new to learn but at the same time, you get new features. But I don't know that people necessarily associate that with and it happens when I hit reload or it only happens when I open a new browser, like I don't know that it's that clear for people in their head.

CHARLES: Right. I can see that. But the question is if the evolution is too stark, I think people tend to get annoyed. If they're in the middle of a workflow or in the middle of a use case and something changes, then it gives it a feeling of instability and non-determinism which I think can be unsettling.

LUKE: Definitely. We all value, as engineers, we value getting into that flow state so much of like, "Oh man, I'm being productive. I don't have any distractions." And you kind of owe that to your users also to be able to let them get into that state with your app and not be throwing up, "Hey, there's new stuff. Reload." "I'm in the middle of something. Sorry."

CHARLES: Yeah. I definitely do the same thing. Sometimes, I let iOS be bugging me to upgrade for a month until I finally start to feel guilty about security and actually do the upgrade.

LUKE: Right.

CHARLES: Although once they started doing it at night, it actually made it a lot better.

LUKE: That's an interesting idea, too. I think there's a natural tension between the lower integration risk that we have as the engineering teams of shipping very frequently. Aaron mentioned shipping a dozen times a day. We certainly have been there and done that as well. I would say on average, we ship a few times a day. But the reason that we do that is because we know the faster we get code into production, the faster we can trust it. So it passes all your tests, it passes your [inaudible], but you don't really know if you're being honest. You don't really know until there's thousands of people using it in production. And yet, this conversation makes me think about there is a tension between how frequently you do that versus your users' kind of comfort level and expectations.

CHARLES: And maybe there is a thing where you can kind of analyze on a per user basis how often they're active in the application and try to push updates on times that are customized to them.

LUKE: Like when a user has been idle for 30 minutes or something like that.

CHARLES: Yep. Or even like track trends over months and see when they're most likely to be idle and schedule it for them.

LUKE: Good point.

CHARLES: Something like that.

TARAS: I have an idea. We should introduce screen savers into web apps. And so when the user stops using the app, just turn on screen saver and do the upgrade.

LUKE: I can see the VC patch. It's after dark but for the web.

CHARLES: Enter install flying toaster.

AARON: It does that to open up the idea as well of that automated checking that things are okay well after the fact because it's all right to sit there maybe activate something and sit there even for an hour and make sure there's no bug request coming in. But if no one's actually received your app, then of course it's not going to come in. And it's very easy to kind of move onto the next thing and forget about that. I guess it's not something that I've ever put a lot of time in and I haven't worked anywhere that's had really great automated checking to see is everything still okay. And I guess it's an interesting thing to start thinking about as well an important thing.

CHARLES: Like actually bundling in diagnostics in with your application to get really fine grained information about kind of status and availability inside the actual app.

AARON: I'm not exactly sure, really. I guess I maybe wasn't thinking about inside the app but I don't know what it looks like exactly. But there is that element of shipping fast and getting stuff out there. But are we really making sure it all works later on when everyone's actually using it.

LUKE: Yeah. This by the way, I just wanted to say when we were talking earlier what are the essential qualities around deploying an app. And this reminds me that one thing that we didn't mention but is very simple is your app should have a version. And it should be unique and traceable back to what was the GitHub, git commit that was the origin of that code. It's just a very simple idea but if you're going to be analyzing errors in production when you have multiple different versions of your JavaScript app running, you're going to need to know what version caused this error and then how do I trace it back and make sure that the code that I'm trying to debug is actually the code that was running when this error happened.

CHARLES: Do you only use just [inaudible] or do you assign like a build number or using SemVer? What's a good strategy?

LUKE: In our case, we use git tags. And so, our CI deployment process for our Ember apps basically looks like this is we work on a PR, we'll merge it to master. If it builds, the master gets deployed into our QA environment automatically using ember-cli-deploy from our CI server. And then once we're happy with how things are on QA, we do git tag or actually I use an ember addon called ember release that does that tagging for me and I'll tag it either in patch minor or major, roughly [inaudible] although it doesn't matter that much in the case of apps. When there is a new tag that builds green on CI, that gets deployed automatically into production by ember-cli-deploy. And so, that's kind of a basic flow. That tagging, just to be clear, the SemVer tag is just going to be number.number.number. You can get more sophisticated than that and I think both Aaron and Mattia have a system where even in the PR stage, there's automatic deployments happening. So maybe one of you want to mention that.

AARON: We're slightly different. Every time we push the pull request, that gets deployed to production. We're able to preview up pull requests in production before we even merge into master which we find super useful to send out links to stakeholders and maybe people that have raised bugs just to get them to verify things are fixed. And then at the point that it's all Google merged, the pull request to master which will automatically do another deploy which is the thing we'll ultimately activate, we activate it manually after the fact. We just do a little bit of sanity checking but we could automatically activate that on merge to master as well. But yeah, the being able to preview a pull request in production is super powerful for us.

CHARLES: That is definitely a nice capability. It's hard. It's one of those certain workflows or patterns or tools that you remember life before them and then after them and it's very hard to go back to life before. I would definitely say kind of the whole concept of preview applications is one of those.

AARON: Absolutely. It's a daunting concept if you're not there, previewing something that's essentially a work in progress and production. And there are some things you want to be careful with, obviously. But for the most part, it's a super valuable thing. As you say, it's a world where once you're there, it's very hard to step back and not be there again.

CHARLES: So I had a question about, we talked about I think it was 162 plugins around the ember-cli-deploy community. What for you all has been the most surprising and delightful plugin to arrive that you never imagined?

MATTIA: That is a good one. I'm pulling up the list. What I can tell you, for me, it's not about a specific plugin. The surprising part was the sheer amount of different strategies that people use for the shipping part. At least, I found that the build part is similar for most people, like most people want to do the things that you're supposed to do. So, you want to build your application and then you want a minify it in some way. And there's a bunch of options there from gzip to more recent technologies. But the way people deliver it to servers and the difference in the solutions, that I think for me has been the biggest thing, where people that ship [inaudible] are people that ship directly to Fastly, people that use FTP files, people that use old FTP, people that use our sync, people that do it over SSH. We have people that ship stuff directly to a database because some databases actually have great support for large files. So we even use it as a storage. We have people that do it in MySequel, people that do it in [inaudible].

CHARLES: It's actually storing the build artifacts inside of a database?

MATTIA: Yes, I've seen them in that. It's kind of interesting like the solutions that people ended up using. And so for me, I think that that's been the most fascinating part. Because as we were saying at the beginning, I'm just seeing now, we even have one for ZooKeeper. I don't have an idea what this does but it's probably related to some sort of registration around the seven-day index. That, to me, I think has been the biggest surprise. Everybody ends up working in a different environment. And so, that flexibility that users need has been by far the most surprising one.

AARON: I think that's also been one of the challenging things, one of the enlightening things for me. I think in the Ember ecosystem, addons and even just Ember itself, it's all about convention of configuration and doing a lot of the stuff that you do for you. I think people expect them to see a lot of point to do the same thing. But the key thing here is it just really automates all the things you would do manually and you need to understand exactly what you want your deployment strategy to be before use them to see a lot of [inaudible] could do for you. You need to decide do I want to install my assets on a CDN and do I want to install my index in Redis or in console or in S3 bucket. You need to know all these things and have decided on all these things and then ember-cli-deploy will make that really easy for you. And this is one of the educational things, I think we still haven't even nailed because there are always people that want to know why this doesn't work. But deployments are complex thing and as what you were saying Mattia, there are so many different variables and variations on doing this that there's no sensible configuration ember-cli-deploy could really provide out of the box, I guess. And so, that's why we ended up with a pipeline that gives you the tools to be flexible enough to support your strategy.

LUKE: I think the closest that we come to the convention is that if any app is using ember-cli-deploy, you can run ember deployment targets or ember deploy prod and ember deploy QA and you can expect that that's going to work. What you don't know is how has it happened to be configured in this project.

Charles, your question about kind of the most surprising thing that's come out of the ecosystem. For me, I would say -- Mattia mentioned plugin packs earlier which are groups of plugins that kind work together well. And so, we've seen some plugin packs like you might expect, like an AWS pack for deploying to AWS. But the more interesting ones to me that we've seen a lot of, companies open source their plugin packs. So what you naturally fall into as a company that's adopting ember-cli-deploy that has multiple ember apps is that you are going to develop your own plugin pack for internal use because generally speaking, companies follow the same deployment pattern for each of their apps. There's usually not any reason to vary that. So then the new thing that happen on top of that is people said, "Why don't I make this open source so other people can kind of see how we do it?" And that's been a really delightful part of the process to kind of get a peek into how other organizations are orchestrating their deployments. And if people are curious about kind of looking at that themselves, you can go on NPM and look for keyword ember-cli-deploy-plugin-pack and pull up all of those. And you can kind of poke around and see what different companies have open source there.

CHARLES: I actually love all three of those answers because it really is for me when you have a constellation of people around a particular problem, it's the surprising solutions that emerge that are some of the most exciting that would have lain hidden otherwise. It would have been kind of buried beneath the source of company A or company B or company C but actually having it all out in the open so that you can inspect it and say, "Wow, where has this solution been all my life?" Something that you have never imagined yourself.

LUKE: It's so funny that you mentioned that because that actually is the origin story way back, I'm talking like 2013 probably when we were very early Ember adopters and we were trying to figure out how do we deploy this thing. We're deploying it with our Rails app, like literally deploying the Ruby code and the JavaScript code together which took forever which is a disaster. And I heard through the grapevine, just exactly what you're saying Charles, where the good ideas are kind of hidden inside of a company. I heard through the grapevine that Square had this approach that they were using where they would deploy their JavaScript assets and then deploy their index HTML file, the contents of that file, into a database, it's Redis in their case, and serve then it out of there. And it empowered all of these interesting situations of like having multiple versions, being able to preview the release, et cetera. And so we then set out to copy that idea because there was nothing open source. So we had to create it ourselves which we did in Ruby which we made it inaccessible to many JavaScript shops in the first place. Then the evolution of that kind of over time and of Mattia and Aaron and Mike and the rest of the community kind of talking together has now moved this into the open source sphere where these ideas are more accessible and we've created an ecosystem encouraging these ideas to stay out in the open. It's so true that there's just gems of ideas that have been created by really brilliant engineers inside of companies that could be benefiting so many people, they just haven't seen the light of day yet.

CHARLES: Yeah, that leads me to my next question. I would say most of the ideas that we've been talking about today really, except for the build part, how Ember specific are they? Obviously, the ember-cli is a great resource and has a lot of great opinions for actually building the JavaScript assets themselves. But the second two phases of the pipeline really can vary freely, if I understand correctly. And so, have you all ever thought about trying to maybe kind of abstract these processes and these plugins so that these same ideas don't remain not just inside of a company but inside a community that spans a set of companies so that it's available for a wider audience? How integrated is it with Ember and what kind of effort would that be?

LUKE: That's a great question. Aaron or Mattia, one of you guys wanna talk a little bit about the history here?

MATTIA: Yeah, sure. We've been thinking about this as well. In the past, a guy on the ember-cli-deploy team, Pepin, has actually started this effort and it kind of prototyped this very idea of separating the ember-cli-deploy part from Ember CLI itself and make it a bit more generic. And he started a project called Deploy JS which I can give you a link for the show notes later. I don't think that the project is currently maintained but definitely the start of the effort is there. And the funny thing is that it was surprisingly easy. I think that we didn't get there mostly because we just all use Ember at work. As you know, open source is mostly motivated by the needs that an individual or a group of people have. But if any of the listeners were very interested in this, I think they should definitely get in touch and we will be happy to talk to them and see what can be done here.

AARON: And also if you look, there's actually a plugin called ember-cli-deploy-create-react-app and there's also ember-cli-build-view. So it can and is being used to deploy non-Ember apps which I think is super interesting because the only real Ember part of it is just using the CLI to discover the addons and plugins. And from then on, it's really out of the hands of Ember. But it sort of leads into a little bit, Luke mentioned this concept of immutable web apps. And I've been thinking a lot about this lately because a deployment strategy that ember-cli-deploy use as an example a lot and it's kind of become [inaudible] ember-cli-deploy in ways, the lightning approach which is this whole idea of splitting or putting your assets in CDN and your index HTML separately maybe in Redis and serving that. I've been trying to work out how I can talk about this to the wider JavaScript community in a non-Ember way and knowing full well that the concept of lightning deployment means nothing to anyone outside of Ember. Just by chance, I was just talking to some people and this terminology of immutable deployments kind of rose. I started searching around and I come across a website called ImmutableWebApps.org which was just scarily the same as what we've been talking about for the last three or four years with ember-cli-deploy. And a way to boil down at a framework-agnostic level, what are the key points that you need to consider when building a JavaScript web app to make it immutable. And it was just really amazing seeing it. This website was put up 3 weeks before I did my Google search coincidentally. And it's basically word for word what we have been talking about the last three or four years, So, someone else in the other side of the world's been coming up with the same ideas in their company like you were talking about earlier and we've reached out to him. I guess that, to me, is sort of the way forward that I want to sort of pursue to try and get these ideas out in a framework-agnostic way to the rest of community and say, "Hey, we thought about deployment in this way. Have you thought about building your app in this way to give you these sorts of capabilities?"

CHARLES: I think the wider community could definitely benefit from that because most of the blog posts and talks I've seen that concern themselves with deployment of single page applications, it's still much more of the tutorial phase. Like, "This is how I achieve getting this deployment strategy." Not, "This is how I repeatedly encode it as a program," and leverage it that way. And so, definitely getting that message out to a wider audience, I think it's a -- what's the word? It's an underserved market.

LUKE: Yeah. I really like this idea also. I think about this ImmutableWebApps.org, if you look at it. It's sort of a manifesto with conceptual description of what are the qualities that your app has to have to qualify as an immutable web app which I think is kind of a funny idea but one that people can start to get their heads around and compare that description to their own apps and say how do I hit this or fall short. And it reminds me a lot of kind of the idea of Twelve-Factor App which is an idea that I think came out of Heroku originally. And it's an idea of a backend app that is portable to be able to easily move across different hosts and easily be scalable to different instances of [inaudible] if your app obeys all of these things then it's going to do well under those circumstances, that will satisfy those needs. So, I think it's a great way of thinking and probably maybe even a better entree into the conversation with the wider community than a library, this is certainly a library called ember-cli-deploy.

CHARLES: This is a fantastic discussion. It's definitely reminded me of some of the best practices that I haven't thought about in a long time and definitely opened my eyes to some new ones and some new developments. So often, we can be focused on how our apps work internally, like how the JavaScript code works that we can just kind of -- what's the saying -- lose sight of the forest through the trees or I can't remember. It's like too busy looking down the end of your nose to see past your face. I've probably mangled both of those adages, but maybe 60% of two mangled adages is at least equal to one. This is something that we need to be thinking about more, that everybody needs to be thinking about more. This is actually a very exciting, very useful problem space and I'm just really grateful that you guys came on to talk to us about it. So, thank you, Mattia. Thank you, Luke. Thank you, Aaron.

MATTIA: Thank you so much. It was a great time.

AARON: It was a pleasure. Thanks for having us.

LUKE: Thanks so much.

CHARLES: Thank you for listening. If you or someone you know has something to say about building user interfaces that simply must be heard, please get in touch with us. We can be found on Twitter at @TheFrontside or over just plain old email at contact@frontside.io. Thanks and see you next time.