096: Outside-In Development

Show Notes:

• 01:02 - Why outside-in development?
• 05:50 - Best Practices and Implementation
• 09:35 - API Iteration and Design
• 18:31 - Is outside-in development a timeless approach to software development?
• 24:10 - Outside-in Creation
• 28:37 - Summarizing Outside-in Development and What it means to the Frontside

Resources:

• Sketch
• InVision
• Balsamiq
• pretender
• Ember CLI Mirage
• bigtest

Transcript:

CHARLES: Hello everybody and welcome to The Frontside Podcast episode 96. My name is Charles, I'm a developer here at the Frontside. And your podcast host in-training with me today is Jeffrey.

JEFFREY: Hello.

CHARLES: Hello, Jeffrey. And Arash.

ARASH: Good morning.

CHARLES: Yeah, how are you doing?

ARASH: Doing great. Thank you.

CHARLES: All right. So, today this is going to be a little bit of an internal powwow where we're just going to talk about some of the patterns that we see as we develop software and just kind of a little bit of a chat on the way we do it. So, we're going to be talking today, I guess we could do like a little bit of a spoiler and just kind of co lead with what we're talking about.

ARASH: Which is...

CHARLES: Which is...

ARASH: Drum roll...

CHARLES: Outside-In development. And it's kind of the way that we've just naturally gravitated towards, towards developing software here.

ARASH: So first question is why we gravitated toward that.

CHARLES: I could just tell you kind of our personal history on this. Is that, I guess it was, would you say like around 2007, something like that kind of the pattern of like REST services got like really well established.

ARASH: Sure, that's accurate enough.

CHARLES: Yeah. It's like around mid-2000's. And so people just went kind of, you know, cuckoo for Cocoa Puffs except APIs instead of Cocoa Puffs. So it was like all the talk was about your API, what's going to be your API, and how are you going to present this to the world? And oh my goodness, you're opening up your system for extension by developers and it's really fantastic. So, it kind of became the norm, I feel, from that point forward to be thinking about what is the API that I'm going to be presenting rather than what's the application.

ARASH: That's your starting point.

CHARLES: Yeah, that's your starting point. So a lot of development and getting like a lot of frameworks came along to make API development really simple, regardless of run time. It was really just really simple to just set up API. And so, a lot of people did that and I think there was value in that and that you're thinking kind of about your external domain model, like how you're going to present yourself to the world. And so you really are focused on the constraints and the abstractions that you want to present and how you want to hide all the messy complexity behind your system. But the problem that you get into is in those days, especially on the web, the clients were very closely aligned with the APIs. If you're coming like from a Rails backend, you almost had one for one between your pages and your API.

If I've got a user's end point, I'm going to have a user's management page. And I think as clients became more stateful and more of the kind of interactions were ported over to the client, that synchronization and one to one mapping between what your API look like and what your interface look like, that started to crumble and disintegrate. And now I think that it's actually quite, I mean there are still clearly analogs because you're talking about the same entities. But your client is really now a full-featured application. And it's got complex rendering logic, it's got complex state management, it's got all these things that happened completely and totally separate from your API endpoints.

JEFFREY: And your UI ends up being aware of relationships between models. So yeah, it's just so much more sophisticated than what we used to need to do when everything was server-side rendered. When you're dealing with a heavy client side app, your API needs to be quite a bit more flexible really.

CHARLES: Right. And so what we were finding was that we'd often have to change the API significantly in order to support complex interaction on the client. But the problem is, is if you've started with your API, you've invested a lot of development, you've invested a lot of design and you've really laid down not just that design, but also the infrastructure and the operations to make it real, it can be very hard to change. And so, it can exclude a lot of the desirable interactions that you would like to have merely by virtue of the fact that it's set in stone. So I think that's kind of why we found ourselves...

JEFFREY: Why we discovered that inside out wasn't quite working for what we're doing anymore.

CHARLES: Yeah, exactly. It was like having a rock solid API was actually a drawback rather than an asset, whereas 10 years earlier it was an asset. And so it really, kind of having that insight made us step back and wind it back and say, "OK, where is the natural starting point? Where do we want to start?"

ARASH: So philosophically, what does Outside-In development mean to the Frontside?

JEFFREY: It means that instead of starting with the API, looking at actually the workflow is instead we start with what's the UI. What is the client side app going to look like and what are the needs of that client side app? And that drives the development of the API rather than solely, I guess you could say, the business models that would have previously been the initial driver for what that would look like.

CHARLES: Right, exactly. Letting the business models be kind of a function of the desired interaction or the desired experience and then letting it proceed from there. So really is yeah, it's starting from kind of what does the person using your system going to touch rather than what is the computer using your system going to touch.

ARASH: And so if one of our listeners is interested in Outside-in development, what kind of tenets would you recommend they follow and what are some of the best practices you can put into place to align well with this kind of thinking that we're talking about here?

CHARLES: I think that's a good question because there's obviously the philosophy of it, but then there's the practice and the implementation and what does that mean. And there's a lot. There's a lot of just kind of the way you approach the problem and then the tools that you're going to use. I don't know, Jeffrey, which ones should we tackle first?

JEFFREY: Let's tackle the tools. Why not? We'll go backwards. We probably should cover other things first, but why not?

CHARLES: We'll tackle the tools? All right, and this is actually a question I want to throw your way too is at what point do you...because you want to start really with having a good design, like a good wireframe? Do you prefer to start with say like something in InVision or Sketch or do you like to proceed straight to a working implementation?

JEFFREY: For me, it usually depends on how complicated of an app it's going to be. If it is something that's, I'm just dealing with like some crud operations, a lot of times I'll just go straight to the HTML and like starting to build that in actual JavaScript and we'll probably talk about the fake backend for that. But yeah, it versus something that's a little more complicated that has maybe a more novel approach or particularly interesting workflow or something that's going to be pretty difficult to code. That's when I'll start on in Sketcher, InVision, or a tool like that.

CHARLES: I think in the last project, we used Balsamiq. I mean, we don't want to get caught up on like tools, but I think that's a great tool because the other thing we did is it wasn't just one designer doing it. We actually had all of our developers going and writing Balsamiq sketches for the features that they were going to be implementing because it really means that they're going to actually put themselves in the shoes of someone using it and they're going to be thinking about what it's like to be a user. Because they've got to start drawing the boxes and lines and the buttons that people are going to click in and the text fields in which they're going to communicate with the system.

JEFFREY: And what was awesome about using a sketchy type of style like Balsamiq is that it removes the developers from thinking too much about having to make sure this headline is exactly the right font size, that I'm using all the exact right colors. Like, don't worry about that yet. Let's just get the wireframes in this sketchy style first and then we can work from there.

CHARLES: Because it really is. It's very valuable to have this process for yourself as one single developer to be able to view it from the outside in. And I think that looking at it one way, that's truly what it means to be full stack is that you can put yourself in the shoes of every piece of the system at each point. So I can put myself in the user's shoes, I can put myself in the client's shoes. When I say 'client', like the actual browser, the browser's shoes. I can put myself in the server shoes and I can have that perspective of each part.

JEFFREY: So that got me thinking about what browsers would wear what kind of shoes, but we'll leave that conversation for another time. Who'd wear the Chrome shoes?

ARASH: Chrome is definitely Nike, for sure. Speed.

JEFFREY: So back to the tool.

CHARLES: Just like Safari, like Hugo Boss.

ARASH: It's like, yeah, that's pretty close. Netscape is like K-Swiss or something like that.

CHARLES: It's got to be some defunct brand.

ARASH: BK Knights.

JEFFREY: Mosaic is the LA Lights.

CHARLES: Those were great.

JEFFREY: So once I have this design, I'm starting to move to code and I'm starting to build some UI, how can I iterate on my API from there? Because I'm starting to actually build a real app, I want to connect to something, I need a data store. What can I do there?

CHARLES: We've talked about this a lot, I think on this podcast, really over the years. And this is a technique that we got from the Ember community. There's a stack of fantastic libraries. There's Pretender which allows you to stub out XMLHttpRequests using a DSL, kind of like Node Express. And then based on top of that, there's basically an entire fake API layer which allows you to build, I would say it's not even really a stub at this point. It's like actually a prototype server implemented entirely in your browser. It's called Mirage.

JEFFREY: So, you think about it, almost like starting in Sketch and then going to HTML. You're starting in some pretty simple JavaScript and then you're working your way up to what that API design looks like.

CHARLES: Right. And so that lets you experience all of the nuance of API designs. So it comes with like you can experiment with different serialization formats, what are my property's going to look like? You can experiment with how do I load related data? So, if I've got a list of users and I want to get all of the comments that they've ever posted on my site, you might want to load those at the same time. So whether your API's going to support that and how it's going to support that, you can't really know until you actually have a consumer of that API. So what Mirage lets you do is it lets you consume an API you haven't written yet and you get to feel what it's like on the client and you get to make changes to that interface at a very, very low cost.

So there's no deployed infrastructure, there's no automatically generated documentation. There's no real consumers yet. There's no multiple clusters running in the Cloud and it's very, very, very lightweight. It's just running right there in the browser. And so what that lets you do is if you have an insight about the shape of the API, it lets you validate that insight. Or if you have an assumption that you have about your API that actually is going to be detrimental to your experience, you get to have that assumption invalidated very quickly. And so at the backend of this process, you end up with a backend that is optimally shaped to serve the experience that you're shooting for.

ARASH: So once you've got that, how do you reconcile it against is this a good API that I want to expose externally because maybe it's not.

CHARLES: Like what do you mean?

ARASH: Imagine that we've built the UI and we've iterated on the API in a fake stubbed way with Mirage or some similar tool and we've got to a state that we're pretty happy with, with the API design that works for this UI. Now, imagine we want that API to also be available to other things.

CHARLES: Right.

ARASH: How do we reconcile that?

CHARLES: That's actually, I think that's kind of an open question right now, right? I mean, we don't really have a good answer to it because it is a stubbed API. And ultimately you're going to have a real API. So at some point, you do have to say like, this is the shape that I want. I'm going to go ahead and create these things and I need to make sure that those two APIs actually line up. Is that what you're asking is like, how do I make sure that they line up?

ARASH: Sure. That's one piece of it.

CHARLES: Oh, that's one piece of it. OK, so this is actually...we'll explore this process. This is actually like a problem, I think, with the stubbed approach is you have your set of stubs and then you have kind of the real McCoy which eventually will come along. So, it's great to have a server running inside the browser, but at some point you're going to need an actual server that's running in the Cloud.

ARASH: Will you? Like what does the future hold?

CHARLES: I mean, yes because the data needs to move, right? The data needs to move off your laptop. Honestly, Arash asked a good point if you're making an offline application. So if you're making an offline application, this actually allows you to bring a persistence architecture to the browser that is like every other... you don't have to make a special one off case. You can treat your backend like a backend. It's just running on the frontend. So that's probably a corner case, but it's worth pointing out that there used to be like back in the day, if you're using something like Microsoft Word or Excel or some other offline app, you can go a long way without having any internet connectivity, but you get to build it in exactly the same way. And then when you are ready to have internet connectivity, you've got the architecture in place.

So, the future might not hold that and you might actually be able to use this in production. I would say it's a minority of cases. Most time you actually are going to want to have a server component. Let's say that you do, now you've got your real API and then you've got your prototype API that you run your tests against or that you're running against, and they need to line up because your frontend is going to be using both of them kind of throughout the development process.

So there are a couple of strategies to deal with this. I'd say one is you run automated tests against both versions and that's a whole another subject of having testable APIs. Like most APIs that we have that we develop aren't set up to do end to end testing including a lot of the ones that we've written, although I think going forward now that we have a better handle on it, we can do that. We can definitely unpack that subject at some later point like making testable APIs.

But I would say that the other strategy that you can use is use some sort of third party verification mechanism where you essentially record a bunch of interactions between your application and the fake API. And then you take those recorded applications and have them in some sort of repository and you can play them back on your real API and make sure that the responses generated by your real API for those recorded interactions to match up. So it's kind of like, I don't know, it's just a machine to make sure that the APIs stay in sync.

ARASH: That's a domain that we need to get better at.

CHARLES: Yeah.

ARASH: We're only scratching the surface there.

CHARLES: We're definitely only scratching the surface there. But the power of being able to develop without a real API first is enough so I think it makes the exploration warranted. There are some established tools in this space. The biggest one that I know of is called Pact. It's both a library and a repository. So it's kind of like got a...I don't know if there's a central server for it, but it's kind you record interactions, you upload them to a Pact server and then you can verify. So you can actually have a lot of consumers. And so, it's designed not only for making sure that your stubbed API works, but it's also for making sure that you just don't make breaking API changes. You can actually collect a lot of data about how people are using your API and then you can use that as a repository for when you make a change in your API to verify that'll work for all these other people.

So if I'm just some random person or some random developer using your API, I can...kind of like how you record [inaudible] statistics and stuff to make the Apple experience better or whatever. It's very common for developers to ask, to record for certain anonymous data. You can submit anonymous data in the form of like Pacts. And so that can help an API developer verify whether the change they're going to make is going to break clients out there.

So that's a little bit of a sidetrack there, but it's something that you start to think about a lot more when you start to develop in this fashion.

ARASH: Did you feel like Outside-In development is a timeless approach to software development?

CHARLES: I do. I feel like Outside-In, it actually even pervades more of software development than actually like the building the software. Like I feel that the longer you go in software development, you realize that the highest value activity is to front load understanding. So whether you're submitting a pull request, whether you're working on a work ticket, whether you're documenting code or even writing a method, the highest value activity that you can be engaged in, in each one of those points is understanding what the hell you're doing.

ARASH: And why, too. Right?

CHARLES: Yeah, exactly. Why? Understanding the motivation. What's your prime directive? What is the context on which you're entering into this piece of work?

JEFFREY: I'm going to go with a bit of a contrarian approach.

CHARLES: OK.

JEFFREY: I'm going to go with a "maybe"...

CHARLES: Maybe?

JEFFREY: Maybe this is timeless.

CHARLES: Maybe it's timeless?

JEFFREY: Because I think historically, so much of software development was working with constraints. And the particular types of software that we're working on, the most important constraints are on the frontend. What does the end-user experience look like? But historically, that hasn't been the case. Historically, the primary constraints have been what can this technology stack that I'm working on actually do?

CHARLES: Like what is the computer experience that I can support.

JEFFREY: Yeah. And so that was actually the bulk of the work was like how can I actually get something that I want out of this giant machine in a room with me? So, maybe it is timeless, maybe not. But I think it is the way going forward.

CHARLES: OK. So I agree that there's an interplay. There is kind of a yin and yang cycle. And I don't mean the cycle of we go to where we have to think about this and then we go to where we have to think about that. But I take your point, Jeffrey, but I think of something like Super Mario Brothers, which is I would say both a miracle of experience but also a miracle of technology in the sense that I think it was hand coded in Assembler on an 8-bit controller with who knows how much memory, the whole thing. So taking that into account, they had to think very strongly about what the computer could do at that point. They were operating under some just incredible constraints. But at the same time, they were thinking about what...they very clearly were focused on what is the coolest game that we can make at this point.

So you're absolutely right. You have to hold both inside your head. You don't want to go crazy. I mean, we could sit down on the Balsamiq thing and be like, the first thing we're going to do is we want a VR room with...okay, no. Let's dial it back.

JEFFREY: How do you wireframe VR? I've never seen anybody try do that.

CHARLES: The Frontside Podcast episode 97: Wireframing VR.

JEFFREY: Interesting.

CHARLES: How we wireframe VR. So yeah. I agree. I take your point. You do have to hold both in your head at the same time. But I would say start with one a little bit and then see where that can take you, that the task of gaining understanding about what you're trying to build [inaudible].

JEFFREY: Yeah.

CHARLES: If you try to understand what you're trying to accomplish, then you can try to push the tech stack towards that direction and maybe push a little progress forward. And then if you've accomplished something that you couldn't accomplish before, then you can start dreaming about even better experiences. And so kind of moving...like the wheels on He-Man's magic bus or whatever.

JEFFREY: This is out of my domain.

ARASH: I'm so [inaudible] backwards right now.

JEFFREY: Our references are not timeless.

CHARLES: Haven't you seen the...what's the one where he's like singing the song like the 4 Non Blondes song? And I said, hey, yeah, yeah. You haven't seen the He-Man, like the cover of 4 Non Blondes?

ARASH: Oh, like they dubbed his...

CHARLES: Yeah, no one's seen it? Oh, man.

ARASH: It's starting to sound familiar.

CHARLES: OK. I mean, this is like we're talking...

ARASH: This is like 2006 internet?

CHARLES: No, this is like 2010 internet. It's not the ancient past here.

ARASH: I'm going through my internet filing cabinet in my brain right now.

CHARLES: Okay. It had particular significance because the friends from attorney were also part of my childhood and I realized I'm unique in that aspect. But they basically, in those days they had the Hanna-Barbera cartoons were very cookie cutter, so they basically took the Mystery Machine from Scooby-Doo and they kind of re-colored it and they put like weird tank treads that like they put really funky kind of weird futuristic He-Man wheels on it. And it went from the Mystery Machine to He-Man's kind of a ride.

ARASH: They repurposed all the illustrations.

CHARLES: Exactly. Hey, you got to save time.

ARASH: Save time somewhere.

CHARLES: Doing your framework.

ARASH: Kids need cartoons.

CHARLES: That's right. So anyway, that's how the wheels on He-Man's Mystery Machine worked.

ARASH: This has been interesting for me because when we first started this conversation, we're looking at it as Outside-In development. But we've talked so much here about design and creation that it almost starts to feel like Outside-In development is almost too narrow. Like it's really what we're talking about here is Outside-In creation of anything useful and valuable to people.

CHARLES: Yeah. It's easy to overlook like whenever you want to create something, you get so focused on the actual building of it or the making of it happen that you kind of lose sight of where it is that you're going. And I think that sometimes it's important to be able to create without knowing where you're going to kind of push the paint around the canvas, so to speak. I think that's a valuable activity, like sometimes it's important to just code without really having any clear direction. You're just kind of following your instincts. Then it's the same way you look at Picasso's study in charcoal or whatever. There'll be like a picture of a horse leg and Picasso's like, "Oh, there's a horse leg. Man, how do I capture a horse leg?" And not really having much intent. And so that's important. But I think when you're building systems or you're creating something that you want to have a real lasting impact, you do need to consider it holistically and you really need to understand why it is that you're doing what you're doing.

JEFFREY: Part of what's awesome about working in software is that unlike Picasso who had to start a new piece, we can keep working on the same piece and be able to paint over it multiple times. And you can start with that outline of like, "Hey, this is the vision that we see this particular thing going." But then don't make space for like, "Let's just push the paint around and see what happens in this little section," and that's fine.

CHARLES: Yeah. And I think that because software is so malleable that it can end up capturing a lot of what is typically considered design. I think this has kind of also been a theme that we've talked about kind of over the last year or so is that over the last 20 years we've kind of seen software capture a lot of what were once separate practices. So when I was first starting out, it was quality. The line between QA and development was very stark certainly at the beginning of my career. And there was a fusion that was happening right as I joined of QA and development. And we saw basically the testing revolution, realizing the testing needs to be brought into the center of development, which needs to be put forth first if you want to have quality, be something that you want.

And then, a revolution that I saw kind of in the middle of my career is seeing operations. It used to be certainly for many years into the beginning of my career, like the people who maintain the software were very different from the people who wrote it and there was a big divide there. And with the Dev Ops movement, it's kind of the realization that no, if we want to have healthy operations, we need to understand what healthy operations are at the very beginning and we need to like put them at the beginning of the process. And I think that what we're seeing now, maybe a revolution that hasn't quite happened yet, but we're on the cusp of is seeing design burrow its way to the heart of development where it's like we want something to be beautiful. We want it to be frictionless. We want it to be delightful to its end users. It needs to be there from the get go and developers need to be thinking about it. It's not someone else's responsibility. It's you as the primary creator.

JEFFREY: It doesn't matter how elegant of an API you've created if the end user experience is just not there.

CHARLES: Yeah. So it's all about tearing down. There's been a series of walls that have come down that I've witnessed. And so I think this is one that's in the process of crumbling.

ARASH: So, this has been an awesome talk. How would you summarize Outside-In development and what it means to Frontside today and in the future?

CHARLES: I think that it really is about making sure that the understanding is solid. That's the core piece, frontloading understanding, realized through a stack of tools. So the tools we use are some sketch and design tool. We'll sketch out the idea we really like, ask questions and try and understand it and firm up in our idea, like have a vision of what the experience is going to be like.

Then the next step is to write some acceptance tests. So to define done, define what your criteria are going to be so that if I run these automated tests, the code will in fact realize the experience that I'm dreaming about. To do that, we've actually been assembling a suite of tools over the past year to do that and we've actually started releasing them to the world. So, it's still in a...I don't want to say alpha because we're using them in production systems, but it's much more of a work bench at this point than a framework, if that makes any sense.

So we have about four related tools under the big test tent. We have interaction library for stubbing out, gestures, mouse clicks, keyboard events, scrolling. We have assertion library that works with any other source. It's really an assertion extension or extension wrappers to make your assertions impervious to asynchrony. We've extracted Mirage from Ember CLI Mirage so that you can use it in any project, React, Vue, Ember, what have you. You can check it out at big test. It is for early adopters right now, but it is in production systems and if you want to acceptance test anything, command line, back client, what have you, if you're willing to write some JavaScript, you can write those acceptance tests and have them be robust. And whatever experience you're trying to create, you can validate it with that. And so, that's why we created it.

JEFFREY: And that's really a good marker of where we are in our experience with Outside-In development right now is we've been doing this in practice for awhile at this point and we're crafting tools to help us be even better at it and sharing those.

CHARLES: Right. And so we're pretty excited about these tools because I think they bring solutions to a lot of the problems you're going to encounter. So there's a lot more work to do and we're excited to do it. Head on, check those out. And we'll go ahead and wrap up. Just a few quick announcements.

If you're going to be at Assert(js) tomorrow, the Frontside is going to be there. I'm going to be there. Mr. Wil Wilsman is going to be there. So if you're there, please do give us a shout and we'll hang out. And then on the next podcast, we are really, really excited. I'm both nervous and I just can't keep a lid on it. We're going to have Erich Gamma on the podcast to talk about the language server protocol that he's been working on.

If you don't know Erich Gamma, he's one of the Gang of Four that wrote The Design Patterns Book. He was the primary architect behind Eclipse and most lately VS Code. So, if you've used any of those projects or benefited from them, which I have benefited immensely from all three, it's going to be really, really exciting. So, that's something to look forward to.

And with that, I will say goodbye to you, Jeffrey.

JEFFREY: Ciao!

CHARLES: And to you, Arash.

ARASH: Adios!

CHARLES: And everybody listening along at home or in your cars or cleaning your kitchen as I do when you listen to podcasts, we'll see you next time. If you want to get in touch with us, you can always give us a shout on Twitter, we're @thefrontside or you can drop us a line, contact@frontside.io.