Jeff Terrell: Okay, a couple of announcements, I am not going to publish tech talks, by default, just to try to minimize whatever stress, you might feel about presenting to other people.
Jeff Terrell: However, could be a useful, you know portfolio thing to have a published talk somewhere on YouTube that you could link employers to or something like that so.
Jeff Terrell: If you're interested in that i'd be happy to publish your talking in the Channel that I publish all the lectures to.
Jeff Terrell: And or share the video with you, if you wanted to post it somewhere else so just let me know if you'd like to do that and note that you'll need kind of buy in from all of your team just because everybody will be on that recording.
Jeff Terrell: also keep in mind that some people will be remote for the tech talks.
Jeff Terrell: So you'll have to plan for that, if you do in person participation only.
Jeff Terrell: I suppose that's fine in terms of grading but you might not have all that many people here so that just you know, be aware.
Jeff Terrell: And in order to convey well enough to be zoom participants, you will need to use a MIC of some sort there is this.
Jeff Terrell: handheld MIC that we used for the midterm presentations there's also a level your MIC that you could you know put in your pocket and clip the little MIC to your shirt or something so that you can be hands free.
Jeff Terrell: Both of these require batteries and a little bit of setup which I can handle, for you know the podium here so.
Jeff Terrell: I don't know, yes, something else to be aware of and i'll start the zoom meeting you can either project there's a computer here that you can actually log into with your onion and I might actually go ahead and log into it.
Jeff Terrell: If I might do that anyway, and then, if nobody is using it then maybe i'll stop doing it, but but that'll be available, so if your slides are available from your project website.
Jeff Terrell: You can just use the computer here, but I know some of you will have some sort of activity that requires certain software to be installed.
Jeff Terrell: Or at least that's what's typically happened in the past, or maybe even some DEMO or something that you have already installed software for on your laptop.
Jeff Terrell: And so you will of course be able to plug in your laptop as well i'm not as concerned about switching times for tech talks because there's only going to be three per day, as opposed to 13 like from mid term presentations.
Jeff Terrell: i'm trying to think anything else to say oh yeah I mean I think one thing i'll say and i'll probably email this out as well.
Jeff Terrell: I was thinking you know I don't want to be at all insensitive to anybody, and I understand there are very good reasons to watch class remotely.
Jeff Terrell: But I really think that like you get you tend to get more out of stuff that you go in person to so I know it can be challenging or maybe even impossible for you in certain situations.
Jeff Terrell: Sometimes it's challenging just to motivate yourself to get out of your room and actually come to campus and I get that I, I was.
Jeff Terrell: kind of notorious in when I was in college for skipping class all the time and sleeping through like 12:30pm classes and things like that so.
Jeff Terrell: So I get the idea of not coming to class and there's no judgment there but.
Jeff Terrell: But you know I think you'll get more out of the tech talks and just more out of experience and feel more connected to the class in general, if you actually come, so I would encourage you to keep that in mind, and maybe try to make the effort.
Jeff Terrell: Any questions about announcements or anything so far.
Jeff Terrell: Okay.
Jeff Terrell: Quick mentor tip before we dive into the topic for today.
Jeff Terrell: You could ask her mentor how they actually communicate with clients So what do you do in your job, do you have a bi weekly client meeting, do you DEMO the features that you finished that week, do you do DEMO videos are doing do in person DEMO.
Jeff Terrell: How do you run the meetings, is there a particular agenda that you follow do you do a retrospective with your clients.
Jeff Terrell: Do you have estimates for your upcoming tasks already done, by the time you meet with your client.
Jeff Terrell: Do you invite them to your board or whatever you're using for project management, you know these are things that might be useful to ask her mentor so feel free to do that or not, as you like, just an idea.
Jeff Terrell: OK, so the topic for today is driven by a question that I got from a few of you.
Jeff Terrell: In the takeaway questions from last time, which I think is a really good and insightful and appropriate question to ask at the conclusion of last monday's lecture which is okay yeah I see some things about testing, we saw testing.
Katherine Li: Have.
Jeff Terrell: A pretty simple set of functions standard deviation mean and some.
Jeff Terrell: But how do you translate that to an actual APP.
Jeff Terrell: Right, the code base is going to be a lot bigger there's a lot more going on there's actually a user interface, how do you test that and that's something that I would not consider me to have equipped you for yet, but I hope to do so in this lecture.
Jeff Terrell: we'll get into some of the.
Jeff Terrell: details here in a bit, but I want it before I dive into the problem that will motivate our discussion of this topic, I wanted to mention that.
Jeff Terrell: This lecture represents.
Jeff Terrell: A lot of years of programming and APP development and arguing with people who maybe have different opinions than me.
Jeff Terrell: and discussing these these topics and a lot of it just wrestling for me to get to the point where i've wrapped my head around this stuff well enough that I feel like I can present it in a coherent way.
Jeff Terrell: I think there's a lot of value in this lecture.
Jeff Terrell: comes from a lot of struggling on my part to just get to that point of understanding things well enough so you'll hear some of my opinions about application architecture and design of code and things like that.
Jeff Terrell: To me, a lot of times opinions can be where the real value is especially somebody who's pretty experienced hearing their opinions about what the right way to do things and the wrong way to do things are.
Jeff Terrell: Even though right and wrong or words that I have shied away from in the Semester and I think that those are words that any good engineer will shy away from.
Jeff Terrell: Most engineers who have some experience will have some opinions about what are you know more or less effective ways of getting your work done or building maintainable.
Jeff Terrell: applications and that kind of thing, so this is where we get into some of that from me, this is where I share some of my opinions, based on my experience, these are things that not every software engineer is going to agree with, in fact, some would vehemently disagree with me.
Jeff Terrell: But I stand by what i'm saying and i'm convinced enough to to teach some of these things to y'all today, even though perhaps some would take issue.
Jeff Terrell: So right, the question that drives this is how do I test real code like application code so let's begin here.
Jeff Terrell: i'm going to use another it's it's not quite as trivial as last time, but it is a relatively trivial example that I think nevertheless highlight some of the issues that you encounter in testing real application code.
Jeff Terrell: So it's still going to be relatively small amount of code, but I think we're going to get into some good issues.
Jeff Terrell: And I think it's really not about the amount of code in the examples, although there probably is some minimal threshold, you have to hit in order to get into some of these issues.
Jeff Terrell: So I think even the approach that we use last time can scale well to larger code bases.
Jeff Terrell: With a certain asterisk that will kind of be the bulk of what I talked about today.
Jeff Terrell: So so yeah remember to test the variation where it lives don't always rely on your end to end tests for every amount of variation.
Jeff Terrell: And you can scale the approaches that I showed you last time even to larger code basis, however.
Jeff Terrell: it's not obvious how to test many kinds of real world application code, so I have four examples here in the express js framework for node js back ends.
Jeff Terrell: You will often say something like response set set set status 200 and actually it might be response status, I don't remember I just took this by memory, so how would you test response set status, maybe there's some way that response exposes what it statuses.
Jeff Terrell: you'd have to look at the API to figure out if that's something that you can get.
Jeff Terrell: But might not be obvious right away, how to test that.
Jeff Terrell: or let's say you have a user model and the user model has a session model attached to it, so you can say user session, and then on your session, you can save that session to the database, so you call the SAVE to database method, how would you test that.
Jeff Terrell: Maybe you need to actually fire up a database.
Jeff Terrell: not sure that could be complicated.
Jeff Terrell: or common thing that you'll do in the front end is you'll fetch things from the back end so maybe the front end needs to fetch the list of posts.
Jeff Terrell: So it calls fetch posts from backend and in javascript this is returning a promise so, then you would chain adopt then method on to that promise, so that when that promise resolved, you would do something with the posts.
Jeff Terrell: But how do you test that.
Jeff Terrell: not really clear maybe you know as you get more sophisticated with tests, you might have some good tools that you've built into your toolbox and know how to do this, but, at the outset yeah, how do you do, that not really clear.
Jeff Terrell: or something that you'll need to do if you use react is you'll need to render your application somehow.
Jeff Terrell: And need to render it into the page, so the way that you would do this as you'd say react, Dom render you would give it a rack react component.
Jeff Terrell: And then you would tell it where to mounted into the document of the page so it's a document get element by ID give us some ID that would give you the the tag.
Jeff Terrell: Or the element, but you'll need to do that and how would you test that that's doing what you expect.
Jeff Terrell: If you had a Cypress test, you would you would be testing that.
Jeff Terrell: Because Cypress is doing the end to end test of a browser APP but apart from that, how would you test that react react on render is doing what you needed to do.
Jeff Terrell: So we could use test doubles for these we could intercept some of these calls that are a little bit challenging to to handle and some would say that's the right thing to do, but is that the best way is that the only way.
Jeff Terrell: So let's get into a an example and it'll be important that you understand this example to follow the rest of this lecture.
Jeff Terrell: So so let's say that we had this, this is actually coming from a real problem that I encountered I first gave a version of this talk is a lightning talk at the place that I used to work because I encountered code that.
Jeff Terrell: That looked a lot like this i've simplified it and cleaned it up a little bit, but.
Jeff Terrell: It it was actually doing this, and this is yeah, this is a real world problem that I ran into a few years ago.
Jeff Terrell: So we had some code that generated a javascript module index for all the different files under a certain directory.
Jeff Terrell: And we'd save that module and index to index, such as, and this is a common thing that a lot of javascript programs will do is they will.
Jeff Terrell: Have a single export.
Jeff Terrell: That has all of the functionality within it, and so you you've broken it up into different modules, but you have an index module that you can import to get everything.
Jeff Terrell: And so the result of what's stored into index, such as after running this might look something like this module exports equals and then you have a big shoot those should not be equal signs on lines 234 there should be colin's anyway.
Jeff Terrell: You have a big object javascript object here.
Jeff Terrell: And each property is the result of requiring the corresponding module.
Jeff Terrell: That makes sense yeah pretty straightforward.
Jeff Terrell: Okay.
Jeff Terrell: Questions about the example because it's important to get this before we move on.
Jeff Terrell: I need to explain it again feel free to say so.
Jeff Terrell: Okay.
Jeff Terrell: Alright, so to do that there's a couple of pages of source code that i'll show you.
Jeff Terrell: This is fairly simple as just getting the source paths getting getting all of the so kind of on the right hand side of all those equal signs, which should be colin's we say require this require that whatever.
Jeff Terrell: This first function is just going to get a list of all of those paths that we need to require.
Jeff Terrell: So this is javascript syntax obviously line numbers are not part of the code here but.
Jeff Terrell: it's a function with no arguments and we actually Shell out to the find command.
Jeff Terrell: to figure out what all those paths are.
Jeff Terrell: And that's that line three child process that exact sync find command.
Jeff Terrell: We split the lines you split the output into lines, I should say.
Jeff Terrell: Now we filter on line five so that we only get non empty lines if there was a novel or undefined or something like that or an empty line, then we would filter that out remove it.
Jeff Terrell: And then line, six and seven are filtering out anything that matches spec I forgot to update that actually okay that should.
Jeff Terrell: Basically, the intent there is to filter out any javascript files that are actually test files so we're not interested in, including those test files in the module index that we create.
Jeff Terrell: So we filter those out and then, finally, we sort them all and return the result.
Jeff Terrell: Questions about this code.
Jeff Terrell: Okay.
Jeff Terrell: This is the rest of the code and my question to us We walked through this is what's wrong with this code alright see if you can find the problem here.
Jeff Terrell: So this is the way to do.
Jeff Terrell: file output in node js you have to import the fs module and then you would say create right stream like we have online one you give it a filename like index js and that gives you a stream.
Jeff Terrell: And then you have to say stream out once open and give it a call back once that Stream is opened, it will execute that callback.
Jeff Terrell: Okay, and the callback is in lines three through 11.
Jeff Terrell: So first on line three we write module that exports equal open brace new line first line remember we're producing output that looks like this.
Jeff Terrell: So basically going to write the first line we're going to write all the middle lines and then we're going to write the final line.
Jeff Terrell: yeah so line three is the first line line 10 is the last line line 11 is going to close the stream.
Jeff Terrell: And then lines four through nine are actually doing the output for each one of those source paths that we found.
Jeff Terrell: So get source paths were calling that function, we saw on the last slide and for each one of those.
Jeff Terrell: we're going to have a file name and we're going to figure out what the exported symbol is so this would be like online to here, this would be a sink command we're trying to strip off all of the path elements.
Jeff Terrell: And then we strip off the extension and we make a require statement, and then we write that that line for that statement so.
Jeff Terrell: You can pry think probably follow what this code is doing without too much trouble, even if you're not familiar with javascript or even if you're not familiar with the fs module in node js but does anybody see a problem with this good.
Jeff Terrell: Okay, if you don't i'm not necessarily just wait for somebody to answer but.
Jeff Terrell: Anybody see an issue here.
Jeff Terrell: So the problem that I see.
Jeff Terrell: Is it's not actually a logic error really it's a question of how do I test this.
Jeff Terrell: How do I test this code, this is not easy code to test or it's not clear like how I would start on the testing approach.
Jeff Terrell: So we're starting to get into.
Jeff Terrell: The motivating question of the of the classic how do I test real world application code.
Jeff Terrell: This is trying to get you know less obvious how you would test it like it would be in real code okay so So how do we test that we're generating the right index file.
Jeff Terrell: One approach would be just run the script let it do whatever it does and then see if the resulting index js file is correct right, I mean that would work.
Jeff Terrell: But there are some problems with that approach.
Jeff Terrell: One is what if there's already an index js file that you actually need to keep around and you don't want to cover that every time you run the tests.
Jeff Terrell: That could be an issue.
Jeff Terrell: And what if we if we just run it then it's going to find javascript files in our current project.
Jeff Terrell: But one of the effects that that's going to have is anytime we change the list of files in our project, you know we add a new module and new javascript file, or we delete a module that we don't need anymore that's going to break the test.
Jeff Terrell: Right, we got a brittle test, because now that index js file that results from the test run it's going to be different, because now it has that extra module in it we've broken our test.
Jeff Terrell: And another question would be like should we mock get source paths.
Jeff Terrell: Or maybe child process on exact sync on the find command If so, then we can actually control what's in that index file so that would solve the problem of.
Jeff Terrell: A brittle test, based on what the actual javascript source pads are because then we would control in the test what the source paths are.
Jeff Terrell: Okay, so so this approach would work but it might be a little bit involved, we might clobber an index js file, that we need to keep around or we don't want to get clobbered we would have to remember to.
Jeff Terrell: Remove whatever the output file was after we're done running the test.
Jeff Terrell: Otherwise we're just.
Jeff Terrell: Having a maybe a non version controlled file that gets it's just kind of in the way whenever we look at good status.
Jeff Terrell: Another approach would be to use a mock to intercept the rights to index js and accumulate the output.
Jeff Terrell: Okay, so let's go back to this code here so basically we would say online one we would mock out fs create right stream so that it's not actually opening a file for writing but, instead, it is just we're going to intercept it and return our own thing.
Jeff Terrell: So we could do that, we saw mocks last time doing that, by itself is not hard but notice what all you have to do to actually make that approach work.
Jeff Terrell: Whatever you return from your mock needs to be something that has a once method.
Jeff Terrell: It also needs to have a right method and an end method.
Jeff Terrell: And you need to make sure that whatever the ones method actually is it's not just to know.
Jeff Terrell: It needs to maybe track what arguments that gets called with and that second argument needs to be a callback that does get called.
Jeff Terrell: And the right method needs to actually keep track in a state full way.
Jeff Terrell: What strings it gets passed.
Jeff Terrell: And needs to accumulate a string every time we call right needs to add to the accumulated value and then somehow expose the accumulated value to the test, so that the test could actually say did we generate the right output or not.
Jeff Terrell: So that would be a lot of work.
Jeff Terrell: All right, that'd be a lot of work to do, mocking this would give a safety, because then we're not actually collaborating anything on the file system if we mocked up with the input and output side, then we'd Have a nice isolated unit test, even for something like this, but that gets involved.
Jeff Terrell: And the key idea that's going to drive this lecture is that restructuring the code can make testing a lot easier.
Jeff Terrell: And so today i'm going to tell you how you can restructure code to make testing easier.
Jeff Terrell: there's basically two things that we want to do in this particular case, to restructure our code and after we see these things and get a feel for them i'll talk about in general, what we need to do.
Jeff Terrell: The two key changes one thing if we could separate the string building from the actual writing to the file.
Jeff Terrell: So all those calls to stream right.
Jeff Terrell: What if we just accumulated a string built up a string and then at the end we just dumped that entire string to the file.
Jeff Terrell: That would make things easier to test and we'll see why.
Jeff Terrell: The second thing is to use dependency injection for the source paths.
Jeff Terrell: You know, remember dependency injection we talked about that on Monday.
Jeff Terrell: So, right now, this code is depending on the get source paths function.
Jeff Terrell: But what if we made, I mean this is actually isn't a function, yet we would need to wrap all of this in a function, maybe we'd call it main or something like that.
Jeff Terrell: and whatever function, we needed to do we would need to call get source paths in the calling function and pass that in as an argument to the code that we have here the function that we would have here.
Jeff Terrell: And that way it'd be a lot easier to test, because then our test could pass in whatever paths that wanted to there wouldn't need to be any marketing involved.
Jeff Terrell: So those are the two changes that we want to do that will make testing a lot easier.
Jeff Terrell: So let's isolate the right operation stuff one we're going to separate the string building from the actual right to file operations let's look at the right to file operations.
Jeff Terrell: In fact let's just make it a single right to file operation, this is really focused just give me a string and a file and I will open the file dump the string to the file and close the file.
Jeff Terrell: So now we've done all the work of the fs module we're calling create right stream and calling once right and and on that stream and that gets the job done.
Jeff Terrell: notice that it's really easy to understand what this function does right.
Jeff Terrell: In fact, you can read the name and especially if you read the name and the parameter names.
Jeff Terrell: it's really clear like I know what this does I know when to use it, I know how to use it.
Jeff Terrell: So it's easy to get your head around which is really important.
Jeff Terrell: Okay, so going back to the other piece of this, we wanted to separate the string building from the right operation we've got the right operation, Nice and contained and isolated, how do we isolate the string building.
Jeff Terrell: there's a couple of functions that we can use here.
Jeff Terrell: This is kind of extracting the logic.
Jeff Terrell: And separating it from the right operation.
Jeff Terrell: Here we have a function that takes a file name, maybe I should say path, but anyway file name.
Jeff Terrell: And it's going to do those computations to get the strip off the directory names strip off the extension and then we're going to have this require statement and then we're going to have the line in our output here that's what we're actually returning from this function.
Jeff Terrell: So that's a piece of it, and then the other piece is if we have an array of file names and we'd have that first line of the output and the last name of the output.
Jeff Terrell: And in the middle, we would have.
Jeff Terrell: a map of all of the lines we would say um.
Jeff Terrell: Basically, for each file name, we went to call file name to required line on that file name and I guess it's an array of strings and we want to join all those strings together with no separate years.
Jeff Terrell: i'm going to assert that this code is really easy to test, do you see why.
Jeff Terrell: Do you agree.
Jeff Terrell: Is it look easy to test.
Jeff Terrell: I think it's easy to test because.
Jeff Terrell: These functions, they take in certain arguments and they return something and those kinds of tests are really easy to write you just call the function give it whatever arguments, you need to give it.
Jeff Terrell: and assert that the output of the function of the return value of the function is what it needs to be so this this tests are really easy.
Jeff Terrell: we'll dig into more reasons why and get more precise about why it's easy and a bit.
Jeff Terrell: Okay, so recall questions about this first we we separated the string building from the actual right to file operation questions about that.
Jeff Terrell: Okay.
Jeff Terrell: Now we want to use dependency injection for the source paths.
Jeff Terrell: So, in other words, we want to pass source paths as an argument.
Jeff Terrell: So here we're going to split up the get source pass.
Jeff Terrell: code, a little bit make this a little bit more clear get paths under directory with extension give it a directory give an extension.
Jeff Terrell: And then we do this, child process that exact same thing that we saw before.
Jeff Terrell: And do are filtering only empty lot or non empty lines you sort the results okay.
Jeff Terrell: Then we have a helper function is the path that we're giving a source path or not, and this is basically just saying, can I match that test.js against that path, and if I can then this is not a source path and vice versa.
Jeff Terrell: And then get source paths it's kind of the main thing that will actually call here it's going to call get paths under directory with extension.
Jeff Terrell: it's going to look in a source directory for files, with the extension js and we're going to filter that list with a source path so that we are only returning source pass and not test pass.
Jeff Terrell: And then.
Jeff Terrell: We have some sort of a main function.
Jeff Terrell: function and no arguments we get our source pass from gets source past that's what we just saw.
Jeff Terrell: We passed that this is our dependency injection stuff we pass that to build index string.
Jeff Terrell: And that gives us our index string and then we can dump that index string to the file index js.
Jeff Terrell: isn't this really easy to read y'all.
Jeff Terrell: That main functions really clear what it's doing.
Jeff Terrell: So this is a really nice sort of you know, sometimes you get like into the weeds for code and you can't see the forest for the trees you can't really figure out what is this supposed to do, but this is a really nice bird's eye view of what the overall program is kind of do.
Jeff Terrell: Okay anyway questions about dependency injection That was the second change that we made to our program to make it easier to test we'll talk about why it's easier to test in a minute.
Jeff Terrell: Okay.
Jeff Terrell: Okay, so if we if we test mean we have all the same problems right it's it's still going to be writing the index js we're still relying on the state of the file system, so all the same problems that we had originally still apply.
Jeff Terrell: But main is so simple right it's like three lines of code that I don't know that there's really a lot of value in testing this.
Jeff Terrell: Get pads under directory with extension that does depend on the file system state.
Jeff Terrell: But this is a pretty generally useful function.
Jeff Terrell: If you got this right or if you found somebody else up had a function like this.
Jeff Terrell: You could use it and you know, maybe, maybe it's pretty well vetted maybe you've used it in other projects before you know it works, maybe you don't need to test it because you're confident that it works if you did want to unit test it let's go back to the code.
Jeff Terrell: You would just need to mock child process that exact sync to make sure that that returns what you want, and then you wouldn't actually be needing a file system state at all.
Jeff Terrell: His source path is really easy to test that's just a regular expression match against the path that you give it.
Jeff Terrell: You give it a path and it returns you the true or false and so you can make assertions on that.
Jeff Terrell: If you want it to test to get source paths all you'd really need to do is market paths under directory with extension if you if you mark that then it'd be pretty easy to test kits source paths really all get source passes doing is calling.
Jeff Terrell: You know filtering based on whether their source path or.
Jeff Terrell: filename to require line and build index string are really easy to test.
Jeff Terrell: With no mocks required.
Jeff Terrell: there's nothing that these things are doing that you would need to mock give it a filename you get back the require line.
Jeff Terrell: You give it a list of file names you get back the entire index string that you're going to dump to the file, so you can you can write test for this pretty easily.
Jeff Terrell: Now dumps during the file is thing that would be pretty hard to test.
Jeff Terrell: For the same reasons that we saw originally right we'd have to either let something get clobbered and maybe we clap or something that we know we don't need and that would be all right.
Jeff Terrell: We have to remember to delete it afterwards, or maybe we try to mock all that fs module stuff create right stream and.
Jeff Terrell: open and right and and and all that kind of stuff but again kind of like with get paths under directory with extension downstream to file is a really broadly useful function.
Jeff Terrell: So if we got that right once we could use that probably in a variety of projects anytime we needed to do that kind of thing.
Jeff Terrell: And we might decide kind of like I said with mean we might just say it's not really worth it, to have an automated test for this, the code is simple enough that I can see that it's correct and i've tested manually and it works and we're good here.
Jeff Terrell: So.
Jeff Terrell: You don't have to test everything.
Jeff Terrell: And it's fair to say that all we've done is shifted the testing difficulties around but.
Jeff Terrell: that's actually really valuable to do that can be the key to making your whole program more effective is just to shift these things around.
Jeff Terrell: And, in particular, if you can separate the logic of your code.
Jeff Terrell: From the difficulties now define difficulties more precisely and.
Jeff Terrell: That means that you can test the logic more easily.
Jeff Terrell: And that's huge.
Jeff Terrell: The logic here, in this case would be really mainly these two things.
Jeff Terrell: So I can test those super easily get a lot of bang for my buck here that's the the tricky logic to get those changes just right.
Jeff Terrell: And after that, I mean you have stuff maybe it's harder to test but.
Jeff Terrell: we'll still be able to test, most of our code, with some pretty simple tests.
Jeff Terrell: So a really key insight at this point is isolating the testing difficulties can enable much easier tests.
Jeff Terrell: Question questions so far.
Jeff Terrell: Okay.
Jeff Terrell: Okay, let me get a little bit more precise about testing difficulties what What do I mean by testing difficulty.
Jeff Terrell: In our starting code, it was the right operations.
Jeff Terrell: That was the the main difficulty that we encountered and to a lesser extent, the file system querying that we did to figure out what javascript files were present.
Jeff Terrell: But that was the difficulties that we had with this particular example what just in general, what constitutes a testing difficulty.
Jeff Terrell: And this is where we start to get into my opinion about code and ways to structure it.
Jeff Terrell: My opinion is that most difficulties with testing are caused either by poorly managed effects.
Jeff Terrell: or by implicit dependencies and I have found this to be true.
Jeff Terrell: When I thought about it, I guess, I first gave this about four years ago, this version of this talk.
Jeff Terrell: I thought about it, then, and that seemed to be a good encapsulation of my experience and ever since then i've i've always thought about okay testing feels kind of hard right now is it still one of these two things and yeah I think it is so.
Jeff Terrell: I think this is true, most difficulties with testing are caused either by poorly managed effects.
Jeff Terrell: or by implicit dependencies let's break that down what does it affect and effect is anything destructive.
Jeff Terrell: And there's a lot of different things that you can destroy in your code destroy isn't pejorative like it's it's okay destroying things as part of what you need to do, but you might be destroying the contents of a memory cell.
Jeff Terrell: Might notice that more if you're in a language that's more acquainted with the memory model like see.
Jeff Terrell: But yeah you can destroy the contents of a memory cell, you can destroy the screen or a certain parts of the screen, for example, when you print something to the console you're actually affecting you know the pixels that are being thrown on the screen.
Jeff Terrell: You might destroy the contents of a file like we saw with the index, such as we might be collaborating whatever was there before.
Jeff Terrell: In a react case as one example, you will often be destroying pieces of the document of a webpage.
Jeff Terrell: Anything that gets reactivity rendered and comes up with something different, maybe you have to delete Dom elements in order to make it updated.
Jeff Terrell: You might be.
Jeff Terrell: destroying a row in the database anytime you save something or update something you might be destroying the value of a variable so reassigning a different value to that same variable.
Jeff Terrell: And importantly effects propagate all right so anything if you have an effect in one of your functions and some other function calls that function.
Jeff Terrell: The calling function is also effectual now it's been sort of contaminated by the presence of calling that that effect full function and that goes, all the way up your call stack.
Jeff Terrell: So you can end up in a situation where every single function in your code base is an effect, and this would be an example of what I would say is a poorly managed effect you're not managing it well.
Jeff Terrell: So in the old code, we saw streamed open and stream right or effects and the new Code, the dumps string to file is our only effect.
Jeff Terrell: Okay second thing your questions about what an effect is.
Jeff Terrell: Okay.
Jeff Terrell: second thing that I think we need to be careful about.
Santosh Nukala: To enable testing.
Jeff Terrell: And by the way.
Jeff Terrell: Why do we enable testing let's remember that just for a second testing helps us move fast it helps us get into a creative flow it helps us discover when things break.
Jeff Terrell: It helps create maintainable code and keep code simple you know, there are very good reasons to test and i'm assuming that you remember that, but it may be worth saying again why we care so much about.
Jeff Terrell: Restructuring our code to make testing easier.
Jeff Terrell: Okay, so the second thing we want to avoid is an implicit dependency, which is anything that queries some external state that isn't explicitly passed in as an argument to the function or method.
Jeff Terrell: So the file system would be an example of this.
Jeff Terrell: We just call that child process dot exact command function and reached out and directly query the file system.
Jeff Terrell: And a database can be an implicit dependency we're just issuing a database query.
Jeff Terrell: The current document in a browser tab just whatever is there if we say document get element by ID you know, maybe that returns, an element, maybe it doesn't maybe it returns something different than it did last time.
Jeff Terrell: Because the document has changed.
Jeff Terrell: At a finer grained level variables that are available in scope, but not past as arguments.
Jeff Terrell: would be an example of an implicit dependency.
Jeff Terrell: However, I would modify that a little bit that if it's not available to other code outside of the function or method, then it's not external state, I would say that's more of like internal state which.
Jeff Terrell: I don't know actually maybe that still counts, I don't know it's it's a little little bit of a Gray area but.
Jeff Terrell: And again, just like with effects implicit dependencies will propagate they will taint everything that caused them with that implicit dependency.
Jeff Terrell: And so you can end up with large swaths of your code base tainted pie and implicit dependency.
Jeff Terrell: Because anything that cause, something that has an implicit dependency itself also has an implicit dependency.
Jeff Terrell: So in the old code child process exact sync wasn't implicit dependency on the file system and the new code we change that to be get passed on a director with extension.
Jeff Terrell: question about what I mean here with implicit dependencies.
Jeff Terrell: or anything else so far.
Jeff Terrell: Okay.
Jeff Terrell: it's important to note, you cannot eliminate these things from your code base and might be tempting to think, or you might be inferring something that I want to be careful to, not to say i'm not saying.
Jeff Terrell: i'm not trying to eliminate these things you have to have some of these things in order to have your program do anything so, for example, like.
Jeff Terrell: You got to change the screen, if you want to have a user interface at all right that's an effect so you've got to have an effect you got to have various effects, but the question is where, in your code base to these effects, live.
Jeff Terrell: And if you can identify those effects and those implicit dependencies and situate them carefully in your code base that can make all the difference between a well factored easy to maintain easy to test code base, and one that gets really challenging and messy.
Jeff Terrell: So let's take some time and look at identifying effects and implicit dependencies.
Jeff Terrell: How do we identify effects here's some tips, is it known to be effective, so, for example, any print statement and whatever language you use is is affected right it's absolutely In effect, we know that it's an effect because it's writing something somewhere.
Jeff Terrell: And so any function that calls a print statement is itself a faithful.
Jeff Terrell: and any function that calls one of those functions is a faithful and so on, all the way up the chain.
Jeff Terrell: Okay, another.
Jeff Terrell: effect would be if something is clearly affecting an existing value so, for example, if you have an array and you say array sub index one equals such and such that would absolutely be in effect you're changing the value associated with that array index one.
Jeff Terrell: And, in general I don't know if you are aware of this distinction in programming but anybody kind of familiar with the distinction between statements and expressions.
Jeff Terrell: Anybody seen this before now.
Jeff Terrell: Okay.
Jeff Terrell: i'm just i'll show you some statements here but statements tend to be effectual expressions tend not to be.
Jeff Terrell: we'll just go we'll look into this, a lot more, and my programming languages class next semester, by the way, if you're interested in that.
Jeff Terrell: But how can we identify statements and one of the easiest ways is to just know which lines clearly don't contribute to a return value.
Jeff Terrell: If something is contributing to a return value in some way, then we, we know that it's useful for return value and it's it's probably not effective.
Jeff Terrell: Now, this is a bit sloppy it's not true to say that lines that don't contribute to return value our statements and everything else is an expression, but this is just kind of a rule of thumb that can be helpful for us to identify effects.
Jeff Terrell: So, for example, line three here that's just doing something, and we don't care what stream right returns were never saving.
Jeff Terrell: Whatever value is returned from streamed out right and line three so that's a big clue that either, this is a completely useless line or that we are executing it for its effect, rather than executing it for its return value.
Jeff Terrell: that's exactly what we're doing right we want stream to our right to do something that's why we're calling it.
Jeff Terrell: we care about its effect, not its return value.
Jeff Terrell: Can you see from the code that clearly we're throwing away the return value, and so, therefore, this is an effect.
Jeff Terrell: You can tell that just by looking at the single line right.
Jeff Terrell: there's another stream right down here we can see that we're not returning anything from this function here.
Jeff Terrell: We don't we don't care about the return value of street art right in this case either just throwing it away so that is executed for its effect.
Jeff Terrell: same thing was streamed out right and streamed and on lines 1011 we're just throwing those return values away so those are also effects it's clear to see.
Jeff Terrell: Questions about this identifying effects.
Jeff Terrell: Okay.
Jeff Terrell: Now let me just take a step back here and say I don't expect that you know after watching this lecture that all of a sudden you'll be way better at doing this kind of stuff and identifying effects and managing them and all that.
Jeff Terrell: that's not probably what will happen this requires practice you're not going to get good at it overnight.
Jeff Terrell: It takes time to cultivate mindfulness about effects and implicit dependencies and to get experience with what things are easy or hard to test.
Jeff Terrell: But if you, you know take this seriously and try to control your effects, then this will seriously level you up as a software engineer really I think take you to the next level.
Jeff Terrell: also remember read green refactor remember that you can write sloppy code fill it with all the effects, you want to start with just get it working.
Jeff Terrell: And then you can go back and fix it later, you know, take the effects out restructure as needed that kind of stuff don't underestimate your ability to revise.
Jeff Terrell: And we will talk in a bit about how to manage these testing difficulties, both the effects and the implicit dependencies before we do that let's do a little quiz which of these are effect.
Jeff Terrell: So i'm going to leave this up in just a SEC, but i'm going to go and.
Jeff Terrell: activate this poll so polivy.com slash Jeff terrell and give you I don't know, maybe two three minutes to tell me which ones of these you think are effects.
Jeff Terrell: yeah should we assume some function has.
Jeff Terrell: No effects itself we don't know we don't know we can't actually tell for number eight we don't have enough information we'd have to look and see and the implementation of some function, do we have effects there.
Jeff Terrell: So good good good question there.
Jeff Terrell: Okay, so let's just walk through these a little bit just give you some practice here and i'm going to try to move fast i'm kind of running short on time.
Jeff Terrell: This is a statement actually because it is an assignment but it's not an effect we're not actually destroying anything by doing this.
Jeff Terrell: Number two probably as an effect, we would expect that any method that starts with sat is going to destroy the old state.
Jeff Terrell: Of that object and replace it with some new state.
Jeff Terrell: Returning is not an effect but printing is printing is going to.
Jeff Terrell: Do something somewhere.
Jeff Terrell: it's going to push some output somewhere.
Jeff Terrell: number five is, in effect, we are modifying my array by adding a new element to it.
Jeff Terrell: We don't know.
Jeff Terrell: don't have enough information to say anything about six or eight.
Jeff Terrell: Seven absolutely is an effect, we are changing my map.
Jeff Terrell: we're mutating it, not as an effect.
Jeff Terrell: Maybe we are overriding what was already there, maybe we're just adding a new key value pair to my map, but that is an effect.
Jeff Terrell: Questions about anything here.
Justin Bautista: I just want to clarify really quick, if I may ask a question.
Justin Bautista: yeah um so i'm a little bit ago on the slide about how to identify statements.
Justin Bautista: So statements.
Justin Bautista: Do contribute to the return value of the code right.
Jeff Terrell: And no a statement does not contribute.
Justin Bautista: what's not okay okay.
Jeff Terrell: yep.
Jeff Terrell: Sorry, if that wasn't clear.
Justin Bautista: Thank you, thank you.
yep.
Jeff Terrell: Other questions.
Jeff Terrell: Okay.
Jeff Terrell: Alright, so let me talk about pure functions a pure function is basically what we've been driving at for most of us lecture a pure function is something that does not have effects and does not have implicit dependencies.
Jeff Terrell: The implications of pure functions are huge.
Jeff Terrell: Because it has no effects and no implicit dependencies it is fully deterministic.
Jeff Terrell: It is always going to yield the exact same value given the same arguments, no matter what.
Jeff Terrell: And that is huge, that means it's so much easier to understand, to use compose with other things to reason about to test.
Jeff Terrell: it's just simpler.
Jeff Terrell: If it's fully deterministic that means it's independent of time.
Jeff Terrell: In fact it's independent of everything except whatever arguments you you pass to it when you call it.
Jeff Terrell: So pure functions are really the name of the game when it comes to code maintain ability, in my opinion.
Jeff Terrell: And the most important idea of today, if you only walk away knowing one thing from this lecture make it be this make as much of your code pure functions as possible.
Jeff Terrell: That will help you test easier.
Jeff Terrell: And that's really that's really what it's all about.
Jeff Terrell: In other words, isolate the effects and implicit dependencies to as few functions as possible and let those be on the outside of your APP so that they're not being called by other functions, unless you absolutely need to.
Jeff Terrell: And this is the idea behind what's called the functional core imperative Shell, this is an architecture principle.
Jeff Terrell: it's not exactly an architecture per se but it's a way of structuring your code.
Jeff Terrell: That makes I think it easier to maintain easier to test etc so functional core imperative Shelby idea here is think of an m&m.
Jeff Terrell: And m&m its core, you know the bulk of it is made up of chocolate right, but then it has that hard candy Shell on the outside.
Jeff Terrell: And the idea here is that your APP should be the same way, the pure functions should be the chocolate that should be the bulk of your of your code most of your coach be pure functions and then a thin layer on the outside.
Jeff Terrell: Are the effects and the implicit dependencies that you need.
Jeff Terrell: But that's the thin candy Shell, the bulk of everything is the pure functions.
Jeff Terrell: And you try to keep those those imperative things to that to a minimum.
Jeff Terrell: Now let me also say this is again getting into my opinions here and, obviously, people will disagree with me I don't like object oriented programming.
Jeff Terrell: I don't like it, I don't think it is a good way of structuring programs.
Jeff Terrell: And if that sounds controversial it's because it is but there's a growing number I think if people generally speaking, by the way, if you.
Jeff Terrell: Look at functional programming that's very much in line with what i'm talking about So if you have ever seen that term that's kind of what where my opinions come from it, I tend to align with people who identify as functional programmers.
Jeff Terrell: I think that functional programming is kind of the next wave of programming and it's going to take over object oriented programming more and more as time goes on.
Jeff Terrell: And i've been trying to basically make the case not not to sort of further my cause and the religious wars, but because I really do think it's better and I wanted you all to be exposed to it, and I wanted to be able to talk about why.
Jeff Terrell: And so I had to set the stage here a little bit, but if you think about it right in object oriented programming, which all of you know at this point objects have typically they have internal mutable state.
Jeff Terrell: In some cases they don't there are things like value objects that don't have mutable state, but they might have some state that's immutable.
Jeff Terrell: That would be an exception of this but, generally speaking, most of the time when you're doing object oriented programming you're relying heavily on mutable state each object has mutable state.
Jeff Terrell: If you have mutable state what that means is that all state affecting methods are effects.
Jeff Terrell: You have destroyed the old state and you replaced it with some new state and that's an effect and therefore everything that calls, you know, a setter, for example, is going to be an effect it's going to be in effect for function.
Jeff Terrell: And all state reading methods are implicit dependencies.
Jeff Terrell: Even if all you're doing is reading the value of that state you're not dependent on any previous caller have a writing function or writing method, I mean.
Jeff Terrell: Somebody calls, you know different writing methods of this object and whoever did before it got to you has affected what you get back from that reader method.
Jeff Terrell: So, in other words the difficulties that we have with testing are all over the place, with object oriented programming it's just the core of how you actually do any programming at all and an object oriented paradigm.
Jeff Terrell: And for this reason, methods and objects are harder to reason about than pure functions, they are not deterministic.
Jeff Terrell: They are dependent on implicit things that have happened in the background, and they have effects.
Jeff Terrell: So it's harder to reason about Tara to test pure functions are better in these regards, and that is a very important thing the fundamental building block of your entire application should be easy to reason about it should be as easy as possible to reason about.
Jeff Terrell: So that's why I don't like object oriented programming.
Jeff Terrell: So how do we actually manage this i'm going to end with enough time for y'all to take quiz five.
Jeff Terrell: The basic idea, and this again think about the functional core imperative Shell, the imperative stuff the effects and things like that are on the outside of the m&m.
Jeff Terrell: so effectively what we want to do is, we still have our effects we still have our implicit entities, but we're going to try to move them to the outside of the APP.
Jeff Terrell: So that they're not tainting everything that's in the APP that's calling them all the way up the chain.
Jeff Terrell: So any implicit dependencies we do dependency injection with we passed that data in as an explicit argument to functions.
Jeff Terrell: OK, and now it's no longer the functions problem to do this, implicit dependency, but it's the collars problem to use that implicit dependency.
Jeff Terrell: And so we've basically taken one step closer to the outside.
Jeff Terrell: closer to the imperative Shell right so now the color has to do the implicit dependency stuff but then we can do the same thing again on the calling function and say let's move.
Jeff Terrell: The implicit dependency outside of this function and onto its color and we push it all the way to the edge of the Program.
Jeff Terrell: And then instantly imperative Shell same thing with the facts and, generally speaking, the effects.
Jeff Terrell: not always but but can be at the end of your function, so your function will just return the data that the effects needs like we saw with that string building example we don't have to write every single time we just have to return the built string.
Jeff Terrell: return that at the end that's the data that the effect needs, and then we make the collar actually execute the effect.
Jeff Terrell: And then we can iterate on that we've pushed that one step closer to the edge, but we can iterate on that make the color more pure and it just returns, the data and we push it all the way, until we can't push it anymore, and then it's at the edge.
Jeff Terrell: So that's the basic strategy here.
Jeff Terrell: pushing this stuff to the imperative Shell and eventually at the end you have these tightly focused effects like what we saw with our rewritten example.
Jeff Terrell: And all the external stay reading functions there's tightly focused, you might not even need to test those things and they're all at the outside of your APP not tainting the logic.
Jeff Terrell: The core of your code, which is how easy to test, these are all pure functions are easy to test for us to reason about they're easy to propose.
Jeff Terrell: i'm just a couple more slides and then i'll finish up here.
Jeff Terrell: Now, your framework might require a lot of effects from you.
Jeff Terrell: So the example that we saw earlier response status 200.
Jeff Terrell: In express js again not quite sure of exactly what this is, but this is an effect right we're we're mutating the state the internal state of response.
Jeff Terrell: This is an effect, and this is the only way to do that and express.
Jeff Terrell: and other frameworks are going to have similar things if they don't if they aren't built with sort of functional programming principles in mind.
Jeff Terrell: Then they're going to have these mutating effects that you have to do and that's going to taint all of your functions that need to do something like that right.
Jeff Terrell: In in a lot of cases, you can find alternatives that are more friendly to functional programming.
Jeff Terrell: But, in some cases, you might not be able to actually have a big question mark in my own understanding about what mobile Apps look like in this regard, I think, mobile Apps.
Jeff Terrell: Especially like android Apps and Java and caitlyn are going to be highly dependent on effects and so it's going to probably be pretty hard to do a native mobile APP without having.
Jeff Terrell: code that's just generally fairly difficult to test.
Jeff Terrell: i'm not quite sure what swift looks like in this regard, if anybody knows i'd be interested to hear but anyway yeah you might your hands might be a little bit tight here, depending on what constraints, you have, for your APP so that's worth saying.
Jeff Terrell: But yeah so the main idea i'm just to sum it all up is identifying effects and implicit dependencies.
Jeff Terrell: and moving them to the outside of your APP in that imperative Shell will make testing and maintaining your code far easier, so that answers the question you know what about real world code.
Jeff Terrell: devin says, you can use F sharp for functional mobile Apps because.net can compile the native mobile code now that's cool okay.
Jeff Terrell: that's neat yeah I don't know what like, for example, the API calls like, how do you paint stuff to a screen or something I don't know if.
Jeff Terrell: F sharp would give you bindings that would be more functional or, if you would still have to do like imperative stuff at that point but but that's good to know at least that you can do F sharp that's me.
Jeff Terrell: Okay might have time for maybe one question if there is one, and then we'll switch to the quiz.
Jeff Terrell: Okay, maybe, maybe there aren't any questions anyway.
Jeff Terrell: So stick with five.
Jeff Terrell: i'm going to stop the recording here.