Luke SinghamTechnology, Data Science and other ponderings2023-12-01T00:00:00Zhttps://lukesingham.com/Luke Singhamlukesingham@gmail.comStartup Reflections2023-12-01T00:00:00Zhttps://lukesingham.com/startup-reflections/<p>These are some of the tough lessons lessons learned from my start-up journey. Writing this has been incredibly cathartic.</p>
<p>My top two key-takeaways are:</p>
<ol>
<li>
<p>Validate the problem and willingness to pay upfront i.e. find someone that says 'how can I use this now? Please take my money'</p>
</li>
<li>
<p>Stress test your co-founder relationship.</p>
</li>
</ol>
<h1 id="product">Product <a class="direct-link" href="https://lukesingham.com/startup-reflections/#product">#</a></h1>
<p>For a little context, our two sentence description of ilumadata was:</p>
<blockquote>
<p><em>ilumadata is Retool for data pipelines. We're building a cloud editor for enterprise data teams that makes it easy to build, test, and launch data transformations in seconds.</em></p>
</blockquote>
<p>Lessons:</p>
<ul>
<li>Read <a href="https://www.goodreads.com/book/show/52283963-the-mom-test?ac=1&from_search=true&qid=hsNb4HuTjb&rank=1">the Mom test</a> first, it's a quick read and will save you a lot of time. To validate your product, you need to:
<ul>
<li>Find and validate the problems your target market has. Ask, <em>what is your biggest problem? Have you tried fixing it? How often does this happen? What is the impact?</em></li>
<li>Get some sort of commitment. <em>Will they sign a Letter of Interest (LOI)? Introduce you to their boss? Pay money now?</em> Yes you can do this without a product.</li>
</ul>
</li>
<li>For ilumadata the adoption cost was greater than the value add. See the <a href="http://www.startupengine.org/2013/08/the-collision-installation.html">Collison install</a> for how Stripe reduced their initial adoption cost.</li>
<li>The difference between iterating on your product based on discovery calls and no users vs. <em>paying</em> users is huge.</li>
<li>Ultimately ilumadata was a <em>nice to have</em> - a proliferation of data tools, and a change in the macroeconomy meant no space for nice to haves.</li>
<li>Having domain expertise can de-risk the venture. A lot of seed funding is based on a solid team with a domain insight. On reflection, was our insight shallow? Maybe, there are a million reasons for why startups fail. The big product de-risk is asking, do people want this? No, do they <em>really</em> want it? I <em>thought</em> I de-risked this early on with some discovery calls, but in retrospect they failed the <a href="https://www.goodreads.com/book/show/52283963-the-mom-test?ac=1&from_search=true&qid=hsNb4HuTjb&rank=1">Mom test</a>. To de-risk efficiently, try selling a product that doesn't exist.</li>
<li>Iterate faster - constantly update what you are trying to learn from the next call. What is the hypothesis you are trying to prove/disprove?</li>
<li>There isn't a definitive way to know when to pivot. There are plenty of examples where founders stuck to their initial idea and it worked out e.g. Retool and Airbnb. In our case we did not have enough traction to justify sticking to our initial idea.</li>
</ul>
<blockquote>
<p>"if a company makes a product that people dislike yet still buy, then presumably they'd buy a better alternative if you made one" - <a href="http://paulgraham.com/superlinear.html">pg</a></p>
</blockquote>
<ul>
<li>đ When we started, no one liked dbt cloud, the mistake was thinking they were an indicator of success. Yet a year later, based on our user interviews dbt cloud did not appear to have high uptake of their paid cloud product. 7 months into our venture they had laid off 50% of their workforce and doubled their pricing. dbt core solves some real pain points, but as far as I can tell, dbt cloud is a nice to have. dbt has recently overhauled their cloud product, so perhaps they have found a way to make it a must have - time will tell đ¤ˇââď¸</li>
<li>Does your product need SOC II? We thought we did. We had found an evangelist user at a large US heathcare company who was keen to use ilumadata. Maybe it's going to be our first enterprise sale! In prepping us for the sales process they said we needed SOC II. So we rushed out and got it in two weeks. The lead went cold, completely ghosting us đť. But we'd just forked out 20k to get our shiny SOC II badge. In hindsight, before getting SOC II, we should have been in front of the <em>buyer</em> and got a commitment from them. SOC II companies sell early stage startups on fear 'Don't let SOC II be the reason you lose your first big deal'. If you are really solving a problem, you should get a commitment from the <em>buyer</em> before you get SOC II.</li>
<li>We were building a data platform, a data operating system, a <em>single pane of glass</em> for data infrastructure. Rather than tackle one single easy to communicate problem, we were solving many smaller problems. We'd tell ourselves certain features were table stakes. For early stage, I'd avoid being a platform. It's hard to communicate, hard to sell and hard to build.</li>
<li>Start quantifying the top problems from each discovery call. <a href="https://maze.co/guides/card-sorting/">Card sorting</a> is a great exercise to do this. Put 5-7 problems you think your persona has on a kanban board, share your screen and ask them to sort them from most painful to least. Several months in, we realised that we weren't addressing the most significant or painful problem. Card sorting also gets people talking about their problems.</li>
<li>In an attempt to breach the chasm between the data team and the rest of the business, we fell into it. The product was too technical still for business folk and too light touch for those tech inclined data people.</li>
</ul>
<h1 id="co-founders">Co-Founders <a class="direct-link" href="https://lukesingham.com/startup-reflections/#co-founders">#</a></h1>
<blockquote>
<p>Here's a guide on: <a href="https://lukesingham.com/how-to-find-a-co-founder/">How to find a co-founder</a></p>
</blockquote>
<p>Co-founding a company is the business version of marriage. The first signs we didn't work well together occurred during the YC batch. We had left the happy place of coding as a side project, and now were making all sorts of time pressured decisions to make this a successful company. Our communication styles were very different. Founders are literally the foundation of the company, not working well together always meant a separation was inevitable<sup class="footnote-ref"><a href="https://lukesingham.com/startup-reflections/#fn1" id="fnref1">[1]</a></sup>.</p>
<p>Lessons:</p>
<ul>
<li>Perhaps the most important factor in an early stage startup, <em>really</em> invest time here building and testing the relationship.</li>
<li>Make sure you go through and write down your answers to <a href="https://proof-assets.s3.amazonaws.com/firstround/50%20Questions%20for%20Co-Founders.pdf">First Mark's 50 Founder Questions</a>. It prompts both you and your potential co-founder to reflect on some hard questions, but also helps uncover where there are discrepancies between the two of you. Do this before you start working together. We didn't do this.</li>
<li>It is a marriage, your incorporation documents are your marriage certificate and it takes significant legal fees, stress and heartache to get divorced. Would you get married to someone you haven't really spent time getting to know?
<ul>
<li>Work together - starting to code on the idea is unlikely a good stress test. It's just one area of what you need to work on as founders and as tech co-founders you're in your happy place! It tests whether you work as an engineering duo. But what about when you're stressed, tired, and have to make a decision on product direction? I know founders who have purposely set up 12hr days working side by side with a packed agenda to stress test. Others have gone on a two week working 'holiday' together - and then chosen not to continue, a lot better than years later.</li>
<li>Pick things you disagree on and commit to debating passionately. If you can't do this productively now, how will you in the future?</li>
</ul>
</li>
<li>Listen to any negative hunches you might have - this will likely get worse later on. Would you hang out solo with them, just for fun? If not, why not?</li>
<li>Pre-define areas of ownership. I made the mistake of everyone does everything, which is often true in startups, but everyone doesn't have to <em>agree</em> on everything. Seeking consensus on operational items should perhaps be open to a round or two of feedback, but ultimately the decision to execute and move fast lies with the owner. Your co-founder should be comfortable with this. Areas of responsibility and how you resolve deadlocks is covered in the <a href="https://proof-assets.s3.amazonaws.com/firstround/50%20Questions%20for%20Co-Founders.pdf">50 Founder Questions.</a></li>
</ul>
<h1 id="sales">Sales <a class="direct-link" href="https://lukesingham.com/startup-reflections/#sales">#</a></h1>
<p>We initially had an enterprise sales motion. Following several months of zero traction, we shifted our focus down market, targeting small to medium enterprises. Take the following lessons with a grain of salt, because ultimately we didn't make lots of sales! đ</p>
<p>Lessons:</p>
<ul>
<li>Enterprise sales is hard. Super long sales cycle. Lots of touch points. I've had a few emails an entire year later saying they are now 'interested' in buying.</li>
<li>Pre-Product Market Fit both founders should be doing at least some selling. Stick your noses into the market. If you're not selling, you're not learning. And if you haven't found a 'hair on fire' problem, then building may be a waste of time. But sure does feel productive!</li>
<li>Customer onboarding - ensure technical founder/s are present for this. Feel the customer onboarding pain, so it gets prioritised appropriately against other engineering pains.</li>
<li>Read the Mom test first, oh I said this already! Have you read it yet?</li>
<li>You can sell first, then build. Yes, sell a product that doesn't exist. Do outbound "we're building X that solves Y" - this is your hypothesis that you should be able to accept or reject after 5 decent calls. Focus on the person's problems and if what you're building solves a genuine pain point, be transparent about the development progress and get a commitment e.g. a meeting with their boss or an LOI or a design partnership!</li>
<li>There were a couple of hunches I felt when selling to a technical audience. Firstly, they were more interested in <em>how</em> you engineered something - were they wondering how to build it themselves? Secondly, technical teams trust open source more. Which is related to the ease of adoption.</li>
<li>Depending on your product, the user and buyer may be different people. This makes it harder for you to sell. As you need to win both to be successful.</li>
</ul>
<p><strong>Buyer</strong></p>
<ul>
<li>We took too long to realise the initial problem we were solving was not the top priority. There was much greater concern with data quality. Here's the results from the card sorting exercise of 88 calls. The problem we started solving placed 6th đ¤Śââď¸.<br />
<img src="https://lukesingham.com/content/images/2023/top-problems.png" alt="top data problems" /></li>
<li>Start with buyer validation, not user validation. Getting a commitment from a buyer is better than the commitment a user can offer you.</li>
</ul>
<p><strong>User</strong></p>
<ul>
<li>We spoke to many users and got introductions to their bosses. But initiating contact with users to reach the buyer, involves engaging multiple people within the same company to identify someone willing to intro you. Start with the buyer, at least you can qualify them faster, <em>Please Sir, do you have budget?</em></li>
</ul>
<h1 id="investors">Investors <a class="direct-link" href="https://lukesingham.com/startup-reflections/#investors">#</a></h1>
<p>I'm extremely grateful to everyone who took a bet on ilumadata. We raised a small round ($720k) from a mix of angels and a VC.</p>
<ul>
<li>Don't be afraid of owning your failure i.e. don't let ego get in the way. I let this get in the way of regular investor updates, I'd think <em>I'll wait for some good news</em> (that never came!). I should have been more open with investors, they are there to help you. I was afraid of being judged, but in retrospect I was judging myself more harshly than they would have.
<ul>
<li>Many investors will go into bat for you. If you're not open with them, they can't help you.</li>
</ul>
</li>
<li>We raised a small round but it cost 2 months of time. Did we need to raise? Whilst in YC "The mantra was if you <em>can</em> raise at Demo Day, you <em>should</em> raise" (see <a href="https://www.lennysnewsletter.com/p/startup-to-exit-lessons-from-a-first">another YC founder's experience of this</a>)
<ul>
<li>We could have proven this idea didn't work with only the YC investment. In fact, we could have validated it with no money!</li>
</ul>
</li>
<li>Don't be driven by investors, they are not the buyer.
<ul>
<li>If they ask for a product roadmap and you are pre-PMF and pre-seed, this is a waste of time. Spend time on getting traction. I remember Siebel during our batch saying "chuck out your Product Roadmap!"</li>
<li>I've filtered out potential co-founders based on focusing on 'investor traction' rather than customer traction.</li>
</ul>
</li>
</ul>
<h1 id="market">Market <a class="direct-link" href="https://lukesingham.com/startup-reflections/#market">#</a></h1>
<ul>
<li>You don't want to be working on a 'nice to have' at the best of times. But if you are and you have a enterprise sales motion, recession fears and major tech lay offs mean most enterprise doors are closed.</li>
<li>One early thought was - 'if we improve productivity even by a tiny fraction like 5% then using ilumadata is an obvious win'. Firstly, your adoption cost needs to be low to prove this, ours was not. Secondly, even this is not enough, you need to solve a problem and 'improving efficiency by 5%' is not a problem, let alone a painful one.</li>
<li>Market Saturation - so many tools in this space, and it only got worse as we were out there. We thought our product was unique or at least sufficiently different, then we started finding many companies doing the same thing.</li>
<li>Outside of the real core of a data platform, cloud data-warehouses and BI, who is getting decent revenue? I had a couple of investors that had invested elsewhere in the modern data stack that many of the big names in that space weren't making much in the way of revenue.</li>
</ul>
<h4 id="see-also">See also <a class="direct-link" href="https://lukesingham.com/startup-reflections/#see-also">#</a></h4>
<ul>
<li><a href="https://www.cascade.io/epilogue/the-data-and-analytics-market">Cascade - Similar startup that failed</a></li>
<li><a href="https://benn.substack.com/p/category-collapse">Category Collapse</a></li>
<li><a href="https://pmarchive.com/guide_to_startups_part4.html">Guide to startups</a></li>
</ul>
<h1 id="legal">Legal <a class="direct-link" href="https://lukesingham.com/startup-reflections/#legal">#</a></h1>
<ul>
<li>Cease and desist - we got one. It scared us and distracted us for at last a month. Don't be scared of this. Unless - you really did break laws!
<ul>
<li>Our YC Partner congratulated us, saying it's a sign you're doing something right. It's a badge of honour.</li>
<li>Big incumbents sometimes monopolize the leading law firms by simultaneously assigning legal matters to all of the top-tier firms. That way, the law firm you're using will have a potential conflict of interest and need the company that sent you the C&D to sign a waiver. Which of course they won't do! It's like a game of monopoly, and they've parked hotels on the top end of town.</li>
<li>Get referrals to boutique specific law firms - we landed an incredible litigation firm that was very reasonable on the company purse.</li>
</ul>
</li>
</ul>
<h1 id="self">Self <a class="direct-link" href="https://lukesingham.com/startup-reflections/#self">#</a></h1>
<ul>
<li>Mental health - I'm not someone who has had any diagnosable mental health issues. Doing a startup is very hard. Everyone tells you this. But the lived experience is very different. Facing hundreds of rejections, many iterations, a hard pivot, co-founder separation and eventually a wind-down is tough. I remember seeing the 'trough of sorrow' graph before going through YC, it's very real!</li>
</ul>
<p><img src="https://lukesingham.com/content/images/2023/trough-of-sorrow.png" alt="trough-of-sorrow" /></p>
<ul>
<li>Sales Led Growth is hard! Not sure if it would be different with better traction - a bit of dopamine goes a long way.</li>
<li>I'm definitely not a perfectionist - if it works, let's ship it! I'm very comfortable with things being rough around the edges.</li>
<li>I learnt more about what <em>I need</em> from a co-founder. Much like your first non-platonic relationships, you learn the hard way what works and doesn't work for you.</li>
<li>I found talking to other founders really helpful, normalising the struggles unique to startups, hearing about what they have tried, what's worked and what hasn't - it's something I do regularly now, but could have done more through the hard times. This is a huge benefit of going through an accelerator, to forge your network of entrepreneurs.</li>
<li>CEO - this role is tough, and it's a role I don't particularly relish as a full-time position. A lot of startups in our batch had the fight over the role, many have co-CEO's until your group partner finally yells at you enough to change! For us, CEO was really who drew the short straw! Early on, I split my time between sales and dev work. After all, we had an MVP to build. 6 months in and the code base was still 40% written by me. This felt productive, but in retrospective validating the problem and willingness to pay would have been time better spent. The last few months of the venture I was pure sales/product, soft pivoting through problem areas in the modern data tooling space. When the business/tech ratio became 100% business for me, that's when I realised I didn't want to be CEO.</li>
<li>There is an inherent duplicity that must be maintained, startups are innately difficult and full of uncertainty. But you must project confidence and optimism when raising funds, when selling to customers and to (potential) employees. All whilst you might be plagued by doubts or recovering from the last bruising 'No' you were banking was a 'Yes'! I found this balance hard to maintain.</li>
<li>I sought consensus. Given the co-founder relationship was already strained, I didn't want to add fuel to the fire. What I should have done, was make clear areas of responsibilities, provide a short time frame for feedback and then execute. The trade-off is speed vs. consensus, and for early stage startups you don't have much time.</li>
</ul>
<h1 id="essential-reading">Essential reading <a class="direct-link" href="https://lukesingham.com/startup-reflections/#essential-reading">#</a></h1>
<ul>
<li><a href="https://docs.google.com/document/d/1pUdqNJf0hchC96ZW142vE7xKPNsWZpw_GF_PIf9pJ7k/edit">startup primer</a></li>
<li><a href="https://evis.substack.com/p/we-grew-from-0-to-150k-in-6-months?r=7zecw">https://evis.substack.com/p/we-grew-from-0-to-150k-in-6-months?r=7zecw</a></li>
</ul>
<h1 id="footnotes">Footnotes <a class="direct-link" href="https://lukesingham.com/startup-reflections/#footnotes">#</a></h1>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>One exception to this is if you find PMF - it solves everything. I've heard of founders absolutely hating each other and that trickling down into very poor company culture. But when you're on a rocket ship to a big payout, people will stick it out đĽ! <a href="https://lukesingham.com/startup-reflections/#fnref1" class="footnote-backref">âŠď¸</a></p>
</li>
</ol>
</section>
How to find a co-founder2023-12-01T00:00:00Zhttps://lukesingham.com/how-to-find-a-co-founder/<p>I recently went on the hunt for a new co-founder. This ended up being my process.</p>
<h1 id="starting-principles">Starting principles <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#starting-principles">#</a></h1>
<ul>
<li>
<p>Meet as many as you can in parallel. Like a sales pipeline, you wouldn't only talk to a single customer at a time, nor should you for founders.</p>
</li>
<li>
<p>Give yourself a deadline, goal setting matters, it also creates urgency so you can close.</p>
</li>
<li>
<p>It takes longer than you think, several months for me and I know many others that took a lot longer.</p>
</li>
<li>
<p>Decide on some initial parameters:</p>
<ul>
<li><strong>Geography</strong>: Initially for me, this was East coast US to Europe for some timezone overlap. This narrowed to UK and western Europe. I realised having the Atlantic ocean between us was going to be too big of a hurdle to de-risk the relationship.</li>
<li><strong>Profile</strong>: I started with 'either a CEO or CTO type'. Typically you'll have a CEO/CTO founding combo. Being technical myself and having <a href="https://lukesingham.com/startup-reflections/#self">reflected on my previous venture</a>, I narrowed this to CEO.</li>
<li><strong>Idea or not</strong>: I was looking for someone with an idea, validated ideally.</li>
<li><strong>Quit their job</strong>: candidates that haven't quit don't have the same urgency. They are comfortable. And sometimes flirting with the idea of startups rather than being ready to commit now.</li>
<li><strong>Not an existing duo</strong>: you want to be on as equal terms with your co-founder(s) as possible. After meeting a few duos, I realised it was unlikely I would feel on the same level as the original duo.</li>
</ul>
</li>
</ul>
<h1 id="the-process">The process <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#the-process">#</a></h1>
<p>This is somewhere between dating and hiring someone. You want to keep it natural and a lot of it is about getting to know the other person. But you have a deadline and a business to start!</p>
<h2 id="top-of-funnel---get-swiping">Top of funnel - get swiping <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#top-of-funnel---get-swiping">#</a></h2>
<p>In dating the 'old way' was being set up by friends. The 'new way' is dating apps. Both work for co-founder leads, but just like Tinder, you'll get many more leads through <a href="https://www.ycombinator.com/cofounder-matching/">Y Combinator's co-founder matching app</a>.</p>
<h2 id="1.-the-first-date">1. The first date <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#1.-the-first-date">#</a></h2>
<p>This is the initial meet and greet. I kept these to a 30min video call or a quick in person coffee. The goals:</p>
<p>â Get to know them. Do you vibe? Yep, it's a lot about vibes. If you don't vibe on a first date with someone, you don't go on a second date.<br />
â Do you respect them? This is really about whether you see them on eye level.</p>
<h2 id="2.-the-written-assignment">2. The written assignment <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#2.-the-written-assignment">#</a></h2>
<p>The aim here is to have the most time efficient process to get to know your potential co-founder. The <a href="https://proof-assets.s3.amazonaws.com/firstround/50%20Questions%20for%20Co-Founders.pdf">50 Founder questions</a> get you to introspect on <em>why</em> you're founding a company, reveals your areas of expertise and raises thorny questions early like <em>how would you resolve a stalemate?</em> The best founders were already aware of the 50 Founder Questions. If someone hadn't heard of them, I'd give them time to get back to me.</p>
<p>â Exchange <a href="https://proof-assets.s3.amazonaws.com/firstround/50%20Questions%20for%20Co-Founders.pdf">50 Founder questions</a> - ensure values, expertise and thinking alignment.</p>
<h2 id="3.-more-dates---discuss-the-founder-questions-and-any-other-questions">3. More dates - discuss the Founder Questions and any other questions <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#3.-more-dates---discuss-the-founder-questions-and-any-other-questions">#</a></h2>
<p>If the potential co-founder's responses revealed a gap too large, I'd just say so in an email. If you only have minor flags, question marks and areas you want to dig deeper - have as many calls to work through them as required.</p>
<p>â Each meeting is increasing your confidence in each other, to the point you are ready for a work trial.<br />
â With a view to commencing a work trial, if the co-founder is committed to an idea, get across the idea.<sup class="footnote-ref"><a href="https://lukesingham.com/how-to-find-a-co-founder/#fn1" id="fnref1">[1]</a></sup></p>
<h2 id="4.-part-time-work-trial">4. Part time work trial <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#4.-part-time-work-trial">#</a></h2>
<p>By this stage you've worked through values, any possible question marks you've had about each other and are ready to test out what it is like to work with each other. I did this with several people at the same time. Using the dating analogy, this is that tricky and tiring phase where you haven't committed to one person yet. You're switching contexts and trying to not mix up details about each person! The work trial can look quite different depending on each person. For example, with one they hadn't found an idea, so the working together was about making progress through the ideation phase. For another, it was about doing joint discovery calls and validating ideas.</p>
<p>â Complete a timeboxed work trial together.</p>
<h2 id="5.-commit">5. Commit <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#5.-commit">#</a></h2>
<p>This doesn't mean you have incorporated. But from step 4 you need to pick one person to move forward. And hopefully you have a strong sense of who the right person is. Having committed to each other, set yourself ambitious goals that are aligned to each of your primary areas of responsibility. For me that looked like developing a prototype and for my co-founder it was securing design partnerships. If this all goes well, you're already a company, you just need to sign the dotted line.</p>
<p>Good luck!</p>
<h1 id="other-inspiration">Other inspiration <a class="direct-link" href="https://lukesingham.com/how-to-find-a-co-founder/#other-inspiration">#</a></h1>
<ul>
<li><a href="https://whalesync.notion.site/TEMPLATE-Cofounder-Trial-Period-cbc8b29c068c4150b9ccd71f40291f6e">Whalesync's trial period template</a></li>
<li><a href="https://docs.google.com/document/d/1rXUOP-FcnIE8eNTkKlELkakZ-MLaIvEyIxUlBOLNZPw/edit">YCs doc on co-founder matching</a> my highlights:
<ol>
<li>Someone you enjoy spending time with and get along with. <strong>Would I want to be friends with this person?</strong></li>
<li>An exceptional person - smart and effective</li>
<li><strong>If you have hesitancy towards someone, listen to it.</strong></li>
<li><strong>Find people you have things in common with</strong> - shared interests, hobbies and values</li>
<li>The person is more important than the idea</li>
</ol>
<ul>
<li><strong>The median time between when the founders joined the site and when they matched with the person they ultimately ended up working with is 100 days</strong></li>
</ul>
</li>
<li><a href="https://medium.com/entrepreneur-first/the-co-founder-checklist-2f5be597e887">EF advice</a>
<ul>
<li>Both co-founders are technical, or technical with a domain expert</li>
<li>Both co-founders are involved in everything</li>
<li>No âpity foundingâ - we're mates!</li>
<li>A co-founder isnât a âmust-haveâ, a generic role that needs to be filled... they are a vital value-add</li>
<li>"Each co-founder needs to have a strong relationship with each co-founder. In our experience, in a three this often isnât the case. Someone is usually making a compromise and although that may seem ok in the short term, this doesnât lead to harmonious founding teams in the long term."</li>
<li>you have, or can quickly acquire the skills needed to build the product you want to build</li>
</ul>
</li>
<li><a href="http://www.paulgraham.com/really.html">Paul Graham</a>
<ul>
<li>"You haven't seen someone's true colors unless you've worked with them on a startup."</li>
<li>"the relationship between founders was more important than ability"</li>
<li>Share <a href="https://www.startupschool.org/faq">startupschool FAQs</a> with them - aim to pick a trial project to work on together</li>
</ul>
</li>
<li>More questions to ask each other from <a href="https://docs.google.com/document/d/1L4wY9qRk75K7B5y9_unueE4Tw0YLRarA9vaCTwfwL-4/edit#heading=h.xhucswxvxook">YCs doc here</a></li>
</ul>
<hr class="footnotes-sep" />
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn1" class="footnote-item"><p>This is a good opportunity to stress test the relationship too. Really question the idea, poke holes, debate it and see how they respond. What would it take to work on something else? You probably need to pivot in the future, which is a difficult conversation if you don't arrive at it at the same time. <a href="https://lukesingham.com/how-to-find-a-co-founder/#fnref1" class="footnote-backref">âŠď¸</a></p>
</li>
</ol>
</section>
An introduction to Monzoâs data stack2021-10-25T00:00:00Zhttps://lukesingham.com/intro-monzo-data-stack/<p><em>Originally posted on <a href="https://medium.com/data-monzo/an-introduction-to-monzos-data-stack-827ae531bc99">Data@Monzo Medium</a></em></p>
<p>The modern data stack is a collection of rapidly evolving technologies that together provide a platform for analytics.</p>
<p>While we at Monzo already have quite a modern data stack, there is always more to learn and room for improvement. This post summarises the core components of our modern data platform, some of the challenges we currently face, and what we want to change.</p>
<p>We follow two principles to guide our choices on the data platform and how we structure our data team:</p>
<ul>
<li>Centralised data management</li>
<li>Decentralised value creation</li>
</ul>
<p>Centralised data management allows companies to have a 360 degree view of their customers and their business. It unlocks data-driven decision making which is key to competing in todayâs markets. Next to having all the data in one place, centralised data management also enables organisational efficiencies. Many data operations and data governance problems become a lot easier to solve if everyone in a company treats data in the same way.</p>
<p>Decentralised value creation means we have a discipline of data folks that work embedded across the entire company. This is sometimes known as a <a href="https://monzo.com/blog/2021/08/10/spinning-up-an-analytics-engineering-team">hub and spoke model</a>. The teams closest to the problem space are the ones creating their data products.</p>
<p><strong>An overview of our data architecture</strong><br />
<img src="https://lukesingham.com/content/images/2021/monzo-data-platform-architecture.png" alt="A diagram of Monzoâs data architecture where data flows from the Monzo application flows through to BigQuery where data pipelines are created and load analytics into a data visualisation tool" /></p>
<p>At a very high level, most of our data is in the form of event data being produced in our backend services. These events are streamed via âthe firehoseâ (using Kafka and NSQ) to specific append-only projects in BigQuery.</p>
<p>Data developers create data pipelines from these events using <a href="https://www.getdbt.com/">dbt</a> (by dbt Labs, formerly Fishtown Analytics). We use dbt to incorporate software engineering best practices into our data work.</p>
<p>Once a data developer has created a pipeline, they also create dashboards and get insights in Looker. Itâs worth expanding on what we mean by âdata developerâ here. Most of the data work is done by the data discipline, however anyone in the company can jump into the data pipelines and make dashboards. For a more detailed view of how we empower people to use data check out <a href="https://monzo.com/blog/2021/08/10/spinning-up-an-analytics-engineering-team">Borjaâs post</a>.</p>
<p>Letâs dive a little deeper into each component and where weâre going with it.</p>
<h1 id="data-loaders">Data loaders <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#data-loaders">#</a></h1>
<p>Analytics Event Processor and Shipper is a tool we built that takes events from backend services posted onto the firehose, and analytics events from the app or internal tooling. There are two microservices involved:</p>
<ol>
<li>Analytics Event Processor consumes two NSQ topics <code>firehose</code> and <code>client_analytics</code>. The Processor then enriches the payload of the event. For example, an event containing <code>account_id</code> is expanded into an account object. Then the Processor sanitises the payload to remove personally identifiable information (PII).</li>
<li>Analytics Event Shipper is responsible for writing both sanitised and raw events to BigQuery event tables. It creates BigQuery tables as necessary.</li>
</ol>
<p>Itâs worth noting that the event tables in BigQuery have a column for the payload. This allows the schema to change. A data developer can then extract columns using <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/functions-and-operators#json_functions">JSON_EXTRACT functions</a> in BigQuery.</p>
<p>We use Fivetran for loading data from several systems, including for our accounting and web analytics data.</p>
<p>We also load a few miscellaneous bits of data, `or example data dumps from third parties directly loaded into BigQuery or Google Cloud Storage. However, we aim to keep bespoke approaches to a minimum.</p>
<p>A challenge we face, particularly with our events data is ownership. Frequently, teams that generate the event data in our production stack are not the teams consuming the data. When a data developer picks that event data up in BigQuery, they may not have all of the domain context and the data can have many issues, for example all the data needed for particular analysis needs. Not knowing who to contact about the data slows us down. We will solve this by having clear ownership of all our data, from our production stack to our analytics stack (see the data discovery and metadata management section later in this post đ). Where possible we also include a data developer within data-generating teams.</p>
<h1 id="data-build-tool-(dbt)">Data Build Tool (dbt) <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#data-build-tool-(dbt)">#</a></h1>
<p>dbt, made by dbt Labs, has become central to the way we interact with data at Monzo. dbt allows users to run models in a <a href="https://docs.getdbt.com/docs/guides/managing-environments#how-do-i-maintain-different-environments-with-dbt">separate development environment</a>. This is huge when working with data; to know you can quickly iterate and test in a sandpit environment without impacting production data. You can tell dbt is a successful part of our platform when not only our data team is using the tool, but also our software engineers and business analysts too. It hasnât been a seamless experience though. Using anything at scale usually brings its fair share of technical challenges.</p>
<p>Speed of running dbt has been a real issue. Currently we have over 4700 models in our dbt project with ~600k lines of SQL. When you have engineers using something slow, inevitably you will end up with workarounds, like our <a href="https://github.com/monzo/ddbt">go-lang re-write of the dbt command line tool</a>. However, having multiple competing tools isnât ideal either and we donât plan to rival dbt Labs any time soon!</p>
<p>We forked dbt in 2019 and placed the setup in a Docker container. At the time this seemed like a good idea for the speed at which we wanted to get security and productivity features into our data workflows. Docker containers are great for consistency between local and production environments and help new Monzonauts get to grips with Monzo data.</p>
<p>Weâve built a few custom features including <code>dbt upstream prod</code> and <code>indirect_ref</code>.</p>
<p><code>dbt upstream prod</code> improves speed and saves money. When running dbt it is common to run with the upstream model operator <code>dbt run -m +tableD</code>. This command runs all of the data pipeline in your development environment up to the common source tables used in production. In the example below you can see this runs 5 models.</p>
<p><img src="https://lukesingham.com/content/images/2021/dbt-upstream-operator.png" alt="" /></p>
<p>With our upstream production feature, the dbt run will read upstream dependencies from production tables. In this case, the same aim is achieved by running 1 model.</p>
<p><img src="https://lukesingham.com/content/images/2021/dbt-upstream-prod.png" alt="" /></p>
<p>When a data developer is working on two tables of the data pipeline, our upstream feature intelligently figures out what part of the pipeline should be built in the development environment and which tables can be read from production. In the example diagram below, the developer has made edits to tableB and tableD. At the command line the developer enters <code>upstream prod -m tableB tableD</code>. Our upstream feature figures out tableC should also be run in the development environment because one of its upstream dependencies, tableB, has been modified.</p>
<p><img src="https://lukesingham.com/content/images/2021/dbt-upstream-prod-multiple-models.png" alt="" /></p>
<p><code>indirect_ref</code> helps us make sure Monzonauts only access the data they need. Some teams work with separate BigQuery projects where only a few people need to have access to that data. But how do we use aggregates or anonymous versions of that data? <code>indirect_ref</code> solves this problem by introducing the concept of 'interface tables' which sit at the end of a sensitive projectâs data pipeline and act like an API dataset. The interface table is the only node that downstream consumers can access. Additionally, the interface table explicitly declares its downstream consumers. So if a downstream team uses the interface model, they will modify the dbt config in the interface model.</p>
<p>For example:</p>
<p><img src="https://lukesingham.com/content/images/2021/interface-model-config.png" alt="" /></p>
<p>This explicitly forms a contract between the interface and consumer. As the interface model is owned by the team working with sensitive data, <a href="https://github.blog/2017-07-06-introducing-code-owners/">GitHubâs code owners automatically tags</a> the producing team in the pull request.</p>
<p>Recently we did some benchmarking of how we could speed up dbt by a significant amount. We found:</p>
<ul>
<li>using an M1, the new Apple laptops with Appleâs own CPUs improved speed by 3x</li>
<li>upgrading from dbt 0.15.0 -> 0.20.0 resulted in another <a href="https://discourse.getdbt.com/t/release-v0-20-0-margaret-mead/2621#performance-5">3x speed increase</a></li>
<li>moving dbt out of a container resulted in a 2x speed increase for those using a Intel CPU MacBook Pro</li>
</ul>
<p>Combining these will mean a 9x speed increase for Monzonauts with an M1 and an upgrade to dbt đ </p>
<p>These benchmark results give us a good set of actions for our next project on dbt. We will move away from using Docker containers and refactor our customisations into plugins. Weâd love to raise these as PRs on the dbt project or open source them as separate tools (đ dbt!). Once weâve refactored, weâll upgrade to the latest version of dbt to gain the speed improvements and many features that have been added since v0.15.0.</p>
<h1 id="git-ci%2Fcd">Git CI/CD <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#git-ci%2Fcd">#</a></h1>
<p>All our code is version controlled and we use GitHub for <a href="https://github.com/monzo/">our code repositories</a>. We use a mono-repo approach, which ensures consistency of tooling, CI checks and increases discoverability. Data developers test the code, but parts of the testing are also automated as part of CI so we can merge changes with confidence.</p>
<p>Our main challenge here using a mono repo has been the speed of CI checks. In the last year we reduced CI checks from ~30mins down to ~5mins. However as the complexity of our pipeline and number of models grow we will likely need to return to this as a problem. We also want to use CI/CD to start raising data standards. For example by setting minimum thresholds in our CI checks we can raise documentation, testing and metadata standards across Monzo.</p>
<h1 id="schedule-orchestration">Schedule orchestration <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#schedule-orchestration">#</a></h1>
<p>We use Airflow to schedule, run and monitor the refresh of our data warehouse. Most of our models run on a nightly schedule, with some tables being ânear real timeâ, rebuilding every 15 minutes. We use dbt to compile the <a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed Acyclic Graphs (DAG)</a> of data models, we then automatically convert this DAG to the format Airflow expects, <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html">a DAG of tasks</a>.</p>
<h2 id="notifications">Notifications <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#notifications">#</a></h2>
<p>If everything ran smoothly all the time that would be great, but it doesnât! We make use of <a href="https://docs.getdbt.com/reference/model-configs#apply-configurations-to-one-model-only">dbt model tags</a> and a bit of Python glue work to post model run statuses to a #data-monitoring Slack channel. Hereâs what that looks like in one of our model configs:</p>
<p><img src="https://lukesingham.com/content/images/2021/slack-notification-dbt-config.png" alt="" /></p>
<p>The Slack post tags the person who last modified the code and provides a few handy links to help resolve the issue.</p>
<p><img src="https://lukesingham.com/content/images/2021/dbt-slack-alert.png" alt="" /></p>
<p>With a penchant for automating the boring things, we have quite a few bots. Other notable examples are the core models bot that helps the data team keep track of the daily run of our most business critical tables. We also have a stale tables bot which scans for tables that havenât been updated in a given period of time, then archives them and ultimately deletes them.</p>
<p>Airflow is a well-supported and widely-used Apache project that has served us well. However, managing a self-hosted instance is a single point of failure and regularly requires maintenance. To reduce the work with self-hosting we will likely move to using <a href="https://cloud.google.com/composer/">Google Cloud Composer</a> in 2022.</p>
<h1 id="data-discovery-and-metadata-management">Data discovery and metadata management <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#data-discovery-and-metadata-management">#</a></h1>
<p>Data discovery is a more nascent component of the modern data stack. Weâve been using dbt docs â however with >4700 models the manifest.json and catalog.json is >300mbs, so the user experience isnât great, particularly in the remote environment where some users may have slow internet. Part of our data governance is to have accountability for datasets: who should I contact regarding this dataset?</p>
<p>The Data Platform Engineering team is currently working on a solution to collect, store and surface metadata in an accessible way. We want to:</p>
<ul>
<li>make our end to end data discoverable â our event data sits outside of dbt docs until it gets consumed into a data pipeline, but how do you discover the events in the first place?</li>
<li>have a single interface to consume the metadata, for example Amundsen, Marquez or DataHub</li>
<li>make sure metadata doesnât only apply to assets in our data warehouse â it also includes our backend database (CassandraDB) and our visualisation assets (charts and dashboards)</li>
</ul>
<h1 id="cloud-compute-%2F-data-warehouse">Cloud compute / data warehouse <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#cloud-compute-%2F-data-warehouse">#</a></h1>
<p>A data platform needs a database. Google Cloud Platform (GCP) is a crucial part of Monzoâs data platform. BigQuery, is GCPâs serverless, highly scalable, and cost-effective multi-cloud data warehouse. At our scale, we pay a significant amount to Google every month so we are constantly working on ways to reduce our bill. Sometimes these are as easy as a configuration change for a model to execute in a different BigQuery project, or helpful tooling that detects these opportunities and notifies the data team of them. Other times it can involve a complete analytics engineering rethink of our data models (some are ~60 terabytes rebuilt daily).</p>
<p>For more information, check out this <a href="https://cloud.google.com/customers/monzo/">case study from Google on how we use BigQuery</a> for analytics.</p>
<h1 id="data-visualisation">Data visualisation <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#data-visualisation">#</a></h1>
<p>We use Looker for all of our dashboards. With automations like reports being sent to slack channels when particular conditions are met, Looker helps make Monzo a data-driven organisation. What struck me when joining Monzo was that the main company Key Performance Indicators (KPIs) were sent out to every employee every week. Iâd never seen such transparency with a companyâs performance. One of Monzoâs values is to <a href="https://monzo.com/blog/2017/03/10/transparent-by-default">default to transparency</a> and with this transparency brings a focus on the key outcomes we want to achieve. By default, everyone who works at Monzo has access to Looker and over 80% of our staff are active users.</p>
<p>A small amount of custom data visualisation occurs in Google Sheets and in Google Colab (Googleâs hosted Jupyter notebook solution).</p>
<p>The main challenge has been the responsiveness of Looker at our scale. Despite Lookerâs importance, nobody in the company owns it explicitly. This has resulted in continual degradation of cleanliness, structure and speed over the last 4 years. To date we have run a couple of ad-hoc projects to optimise Looker cleanliness and loading for end users. Weâve also created automation scripts to clean up broken dashboards, charts and detect columns in LookML that no longer exist.</p>
<h1 id="machine-learning-platform">Machine Learning platform <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#machine-learning-platform">#</a></h1>
<p>The Machine Learning (ML) platform deserves a separate Machine Learning Infrastructure blog post â watch this space! Briefly, our ML stack broadly sits at the intersection of Analytics Infrastructure (GCP), where we house and analyse our data, and the Backend (AWS), where we deploy our business logic. Our long-term strategy is to enable data scientists and backend engineers alike to safely and easily use machine learning as a tool, where appropriate, across the entire spectrum of problems we are tackling at Monzo.</p>
<p>Some organisations have data scientists make predictive models and once happy with them, software engineers then need to reimplement them in the production code base. At Monzo we decided this would slow the deployment of ML too much. So we built scripts that template a Python ML microservice using Sanic as the API webserver that our data scientists and ML engineers can use to deploy to production. We also use CoLab for quick and dirty exploration.</p>
<p>Influenced by <a href="https://docs.feast.dev/">Feastâs architectural overview</a>, the Feature Store is a common component for production ML that automates the journey of shipping features between our analytics (BigQuery) and production (Cassandra) databases. For a detailed writeup on <a href="http://nlathia.github.io/2020/12/Building-a-feature-store.html">this component see Nealâs post on the topic</a>.</p>
<p>For more information on ML at Monzo see these posts:</p>
<ul>
<li><a href="https://neal-lathia.medium.com/machine-learning-faster-ce404dc4cd2">Machine Learning Faster</a></li>
<li><a href="https://medium.com/monzo-bank/machine-learning-at-monzo-d83a7d1a71e1">Machine Learning at Monzo</a></li>
<li><a href="http://nlathia.github.io/2020/12/Building-a-feature-store.html">Building a feature store</a></li>
</ul>
<h1 id="recommendations">Recommendations <a class="direct-link" href="https://lukesingham.com/intro-monzo-data-stack/#recommendations">#</a></h1>
<p>Finally, if youâre interested in reading more on the topic, we love the thought leadership on modern analytics tooling from <a href="https://roundup.getdbt.com/">Tristan Handy in the analytics roundup</a> and accompanying podcast, <a href="https://benn.substack.com/">Benn Stencilâs substack</a>, <a href="https://martinfowler.com/">Martin Fowler</a> (in particular the <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">Data Mesh post by Zhamak Dehghani</a>), and the <a href="https://www.dataengineeringpodcast.com/">Data Engineering Podcast</a> to mention a few.</p>
<p>If youâre interested in working for the data team at Monzo, <a href="https://monzo.com/careers/">weâre hiring!</a></p>
8 Reasons Why Signal is Better than Telegram2021-01-16T00:00:00Zhttps://lukesingham.com/signal-is-better-than-telegram/<h2 id="1.-signal-is-a-not-for-profit-%F0%9F%98%87">1. Signal Is A Not-For-Profit đ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#1.-signal-is-a-not-for-profit-%F0%9F%98%87">#</a></h2>
<p>Telegram is for profit đ°. I want the reassurance that the profit motive isn't there for my communications platform - <a href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%98%88-the-faustian-pact">because we've seen where that leads to before</a>.</p>
<h2 id="2.-telegram-collects-more-of-your-data-%F0%9F%97%84">2. Telegram Collects More Of Your Data đ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#2.-telegram-collects-more-of-your-data-%F0%9F%97%84">#</a></h2>
<p><a href="https://mybroadband.co.za/news/security/382036-whatsapp-vs-signal-vs-telegram-vs-facebook-the-data-apps-collect-about-you.html">Telegram collects more data on you</a>. Whereas Signal, even under the full force of an <a href="https://arstechnica.com/tech-policy/2016/10/fbi-demands-signal-user-data-but-theres-not-much-to-hand-over/">FBI subpoena - which they declared publicly - could only share account creation date</a>, because that is all they had.</p>
<p><img src="https://lukesingham.com/content/images/2021/data-collected-and-linked-to-you.png" alt="" /></p>
<h2 id="3.-signal's-code-is-completely-open-source-%F0%9F%94%8D">3. Signal's Code Is Completely Open Source đ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#3.-signal's-code-is-completely-open-source-%F0%9F%94%8D">#</a></h2>
<p><a href="https://github.com/signalapp">Here is Signal's code</a> - the fact that it is publicly available is important for scrutiny to occur. Telegram's code is only partially open source. Most importantly, Telegram's server software - where your valuable messages get sent - is not open source. How can we trust what we can't see?</p>
<h2 id="4.-signals-e2e-is-the-best-%F0%9F%A5%87">4. Signals E2E Is The Best đĽ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#4.-signals-e2e-is-the-best-%F0%9F%A5%87">#</a></h2>
<p>Signal's end-to-end encryption protocol is <a href="https://www.wired.com/story/signal-encryption-protocol-hacker-lexicon/">regarded as best in class</a>, and is widely regarded as far superior to Telegram's. The <a href="https://security.stackexchange.com/a/49802/115711">security industry has heavily criticised Telegram's encryption model</a>.</p>
<h2 id="5.-signal-protects-you-by-default-%F0%9F%94%92">5. Signal Protects You By Default đ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#5.-signal-protects-you-by-default-%F0%9F%94%92">#</a></h2>
<p>All of Signal's messages and content are encrypted by default. Not the case for Telegram.</p>
<h2 id="6.-signal-is-independently-verified-%E2%9C%85">6. Signal Is Independently Verified â <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#6.-signal-is-independently-verified-%E2%9C%85">#</a></h2>
<p>Telegram does not have a great reputation when it comes to independent audits. In contrast, Signal's protocol has been <a href="https://threatpost.com/signal-audit-reveals-protocol-cryptographically-sound/121892/">inspected by industry experts and given the thumbs-up</a> đ.</p>
<h2 id="7.-signal's-growth-is-outpacing-the-alternatives-%F0%9F%93%88">7. Signal's Growth Is Outpacing The Alternatives đ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#7.-signal's-growth-is-outpacing-the-alternatives-%F0%9F%93%88">#</a></h2>
<p>The <a href="https://en.wikipedia.org/wiki/Network_effect">network effect</a> is incredibly important for a messaging app. In terms of the absolute number of users, I'd like to say that Signal wins, but it's unknown exactly which is larger. Signal doesn't publish its user numbers, maybe because it doesn't have a profit motive to do so. As far as I can see on the app stores I've looked at, <a href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%92%AC-but-my-network%2Fgroup-chats-are-on-whatsapp">Signal was downloaded more than Telegram</a>. Signal '<a href="https://techcrunch.com/2021/01/12/signal-brian-acton-talks-about-exploding-growth-monetization-and-whatsapp-data-sharing-outrage/">ranks at the top on App Store in 40 countries and on Google Play Store in 18 countries</a>'. The pace of download growth is staggering; +5,923% - any tech team would struggle to handle that explosion in growth in that amount of time đ¤Ż.</p>
<p><img src="https://lukesingham.com/content/images/2021/messaging-app-downloads.png" alt="" /></p>
<h2 id="8.-signal's-group-chats-are-limited-in-size-%F0%9F%97%A3">8. Signal's Group Chats Are Limited In Size đŁ <a class="direct-link" href="https://lukesingham.com/signal-is-better-than-telegram/#8.-signal's-group-chats-are-limited-in-size-%F0%9F%97%A3">#</a></h2>
<p>Ok this sounds like a reason against Signal - hear me out! Signal's <a href="https://support.signal.org/hc/en-us/articles/360007319331-Group-chats">group chats are limited to 1000</a>, Telegram's are not. I can't think of a group I'd need to be directly messaging that had anywhere close to a thousand participants. Signal is not trying to be social media. I want my primary messaging app to do one thing, and do it well. The ability to communicate with more than 1000 people can help the growth of adverse political groups, for <a href="https://t.me/s/proudboysusa">example things like Proud Boys on Telegram đ¤Śââď¸</a> - I'd rather not see another assault on American democracy or feel ethically challenged because my messaging app enables such things.</p>
<p><img src="https://lukesingham.com/content/images/2021/proud-boys-on-telegram.png" alt="" /></p>
Goodbye Whatsapp - Hello Signal2021-01-11T00:00:00Zhttps://lukesingham.com/goodbye-whatsapp/<p><img src="https://lukesingham.com/content/images/2021/data-linked-to-you.png" alt="" /></p>
<p>Anyone who uses Whatsapp would have seen the ultimatum Facebook is handing its users. The summary is, <em>give us your data or don't use our platform, you have until the 8th of Feb 2021</em>. The choice is very simple for me; Whatsapp has been something I have reluctantly used ever since it was bought by Facebook. However, not everyone in my network pays attention to privacy or values it as much as I value it. Reasonably my friends and family ask 'but why should I stop using Whatsapp?' or 'but everyone I know is on Whatsapp'. These are great objections. In this post, I'll explain why I'm moving and hopefully tackle some of the common objections I've had.</p>
<h1 id="%F0%9F%91%A8%E2%80%8D%E2%9A%96%EF%B8%8F-the-case-for-signal">đ¨ââď¸ The case for Signal <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%91%A8%E2%80%8D%E2%9A%96%EF%B8%8F-the-case-for-signal">#</a></h1>
<h2 id="%F0%9F%98%88-the-faustian-pact">đ The faustian pact <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%98%88-the-faustian-pact">#</a></h2>
<blockquote>
<p>if you are not paying for the product, you are the product</p>
</blockquote>
<p>In 2014 <a href="https://www.bbc.com/news/business-26266689">Facebook bought Whatsapp for $19 billion</a>. This was not out of charity. The intention was always to <a href="https://mashable.com/article/facebook-data-merge-privacy-concerns/">integrate the valuable data</a> on you and monetise this huge purchase.</p>
<p>Facebook's primary business is collecting and storing as much data about you as possible, to build complex models that most people don't understand - including the data scientists working at these companies. But that doesn't matter as long as they can optimise keeping your eyeballs on their products, they can sell your attention to whoever wants to buy it, from consumer product companies to people trying to manipulate election outcomes.</p>
<p>The worst result of Facebook's giant panopticon on the world is the undermining of democracy. <a href="https://theconversation.com/fact-check-us-what-is-the-impact-of-russian-interference-in-the-us-presidential-election-146711">Russian influence on the US 2016 election</a> was confirmed by intelligence agencies. Within the US the Trump campaign utilised <a href="https://www.bbc.com/news/technology-43581892">Cambridge Analytica to profile American voters using Facebook</a> - providing them the information on who to target i.e. which Facebook bubbles. From here they launched a wholesale disinformation campaign using Facebook. In Brasil at the inauguration of Bolsonaro <a href="https://www.vox.com/policy-and-politics/2019/1/22/18177076/social-media-facebook-far-right-authoritarian-populism">the crowd cheered 'Facebook Facebook Facebook' and 'Whatsapp Whatsapp Whatsapp'</a> - they were crediting these platforms for the win, which had a wealth consortium funding a massive fake news campaign. The examples to list are too long. Facebook empowers authoritarian movements. <a href="https://www.theguardian.com/technology/2019/feb/18/a-digital-gangster-destroying-democracy-the-damning-verdict-on-facebook">An analysis of a UK government inquiry</a>, put it well</p>
<blockquote>
<p>Facebook is an out-of-control train wreck that is destroying democracy and must be brought under control</p>
</blockquote>
<p>One way to do that is by not using its platform or the platforms it has purchased, like Whatsapp, to further increase its power. But, in typical Orwellian fashion, it will try everything to stop you from leaving, from <a href="https://twitter.com/signalapp/status/1348079223701794819/">paying for ads on the app store to be the first result when you search for 'Signal'</a>, to texting you to come back.</p>
<p><img src="https://lukesingham.com/content/images/2021/fb-please-come-back.png" alt="" /></p>
<h2 id="%E2%9D%93-why-signal-and-not-something-else%3F">â Why Signal and not something else? <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%E2%9D%93-why-signal-and-not-something-else%3F">#</a></h2>
<p><a href="https://teddit.net/r/technology/comments/kt91qk/signal_private_messenger_team_here_we_support_an/">Here's what Signal</a> had to say:</p>
<blockquote>
<p>Some quick facts about us: we are an <strong>open-sourced nonprofit organization</strong> whose <strong>mission is to bring private and secure communication</strong> to anyone and everyone. One of the reasons we opted for organizing as a nonprofit is that it aligned with our want to create a business model for a technology that wasnât predicated on the need for personal data in any way.</p>
<p>As an organization we work very hard to <strong>not know anything about you all. There arenât analytics in the app, we use end to end encryption for everything</strong> from your messages and calls/video as well as all your metadata so we have no idea who you talk to or what you talk about</p>
</blockquote>
<h2 id="%F0%9F%95%B5%EF%B8%8F%E2%80%8D%E2%99%82%EF%B8%8F-facebook-is-not-about-privacy">đľď¸ââď¸ Facebook is not about privacy <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%95%B5%EF%B8%8F%E2%80%8D%E2%99%82%EF%B8%8F-facebook-is-not-about-privacy">#</a></h2>
<p>Despite what Zuckerberg might say publicly, it is deeply embedded into Facebook's DNA to get your data and sell it - it is not there to offer you privacy or the best private messaging service; that's an afterthought. As with this update to the t&cs they are demonstrating that they can change the rules whenever they like to what they most prefer - more of your data. <a href="https://twitter.com/Shiftreduce/status/1347546599384346624">A tweet</a> revealed a subtle change of Whatsapp's end-to-end encryption paper, which is a little concerning:</p>
<p><img src="https://lukesingham.com/content/images/2021/whatsapp-policy-diff.png" alt="" /></p>
<h2 id="%F0%9F%92%B0-whatsapp-founder-gave-%2450-mil-to-signal">đ° Whatsapp founder gave $50 mil to Signal <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%92%B0-whatsapp-founder-gave-%2450-mil-to-signal">#</a></h2>
<p>Yes that's right, the co-founder of Whatsapp <a href="https://signal.org/blog/signal-foundation/">gave $50 million dollars to Signal</a>, its competitor. Why? He regrets selling to Facebook; when the <a href="https://www.bbc.com/news/technology-54722362">Facebook Cambridge Analytica scandal</a> happened several years back, he tweeted <a href="https://twitter.com/brianacton/status/976231995846963201">'It is time. #deletefacebook'</a>. Both Founders ended up leaving over disputes of monetisation. And now it's time to delete Whatsapp too.</p>
<h2 id="%F0%9F%9A%80-elon-musk-says-so!">đ Elon Musk says so! <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%9A%80-elon-musk-says-so!">#</a></h2>
<p>Ok, this is more to relay a funny anecdote. The following tweet,</p>
<p><img src="https://lukesingham.com/content/images/2021/elon-use-signal.png" alt="" /></p>
<p><a href="https://www.cnbc.com/2021/01/08/elon-musk-boosts-signal-app-signal-advance-stock-jumps-1100percent.html">Led to a 1,100% surge in unrelated stock</a> with a similar name. But in all seriousness, Elon does have a knack for doubling down on what the future technologies are going to be.</p>
<h1 id="%E2%9C%8B-the-objections">â The objections <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%E2%9C%8B-the-objections">#</a></h1>
<h2 id="%F0%9F%92%AC-but-my-network%2Fgroup-chats-are-on-whatsapp">đŹ But my network/group chats are on Whatsapp <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%92%AC-but-my-network%2Fgroup-chats-are-on-whatsapp">#</a></h2>
<p>And people are moving in droves - just have a quick search on the net. The top downloaded app in several countries right now, is Signal.</p>
<p><img src="https://lukesingham.com/content/images/2021/signal-top-downloaded-app.png" alt="" /></p>
<p>We've all done this before. In my lifetime, I've moved through the following for messaging:</p>
<p>SMS âĄď¸ MSN âĄď¸ Myspace âĄď¸ FB âĄď¸ Hangouts âĄď¸ Whatsapp âĄď¸ Signal â </p>
<h2 id="%F0%9F%93%B1-signal-takes-up-too-much-room-on-my-phone">đą Signal takes up too much room on my phone <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%93%B1-signal-takes-up-too-much-room-on-my-phone">#</a></h2>
<p>This is the same for Whatsapp. The chunky content you receive like files, videos and voice messages can be cleared out on either app.</p>
<h2 id="%F0%9F%94%8E-i've-got-nothing-to-hide%2C-facebook-can-have-my-data">đ I've got nothing to hide, Facebook can have my data <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%94%8E-i've-got-nothing-to-hide%2C-facebook-can-have-my-data">#</a></h2>
<p><a href="https://en.wikipedia.org/wiki/Nothing_to_hide_argument#Criticism">Edward Snowden counters</a> this argument very well:</p>
<blockquote>
<p>Arguing that you don't care about the right to privacy because you have nothing to hide is no different than saying you don't care about free speech because you have nothing to say." "When you say, âI have nothing to hide,â youâre saying, âI donât care about this right.â Youâre saying, âI donât have this right, because Iâve got to the point where I have to justify it.â The way rights work is, the government has to justify its intrusion into your rights.</p>
</blockquote>
<p>Handing over our data to this giant profiling machine is aiding the corrosion of our democratic societies.</p>
<h2 id="%F0%9F%8F%9B-companies-like-facebook-are-killing-democracy">đ Companies like Facebook are killing democracy <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%8F%9B-companies-like-facebook-are-killing-democracy">#</a></h2>
<p>The huge swaths of data allow for incredible levels of targeting. The more privacy and data you give up, the more targetable and manipulable you are. 'But <em>I'm</em> not influenced by ads' - sadly you may not be aware of it. 93% of Americans believe they are in the top 50% of drivers - this is <a href="https://en.wikipedia.org/wiki/Illusory_superiority">illusory superiority</a>. And based on the analytics, you and I very much are influenceable. Even if you are unmovable, frequently it's our friends unintentionally giving up our data; uploading photos of us, tagging us, and merely being a part of your network allows data scientists to make highly accurate guesses about who you are.</p>
<blockquote>
<p>'We have created tools that are ripping apart the social fabric of how society works'<br />
-Chamath Palihapitiya, former head of Facebookâs user growth team</p>
</blockquote>
<h2 id="%F0%9F%98%90-i-won't-make-a-difference-if-i-move%2C-i'm-just-one-person">đ I won't make a difference if I move, I'm just one person <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%98%90-i-won't-make-a-difference-if-i-move%2C-i'm-just-one-person">#</a></h2>
<p>This a <a href="https://en.wikipedia.org/wiki/Paradox_of_voting#:~:text=The%20paradox%20of%20voting%2C%20also,normally%20exceed%20the%20expected%20benefits.&text=Moreover%2C%20the%20expected%20benefits%20are,will%20never%20be%20certainly%20pivotal.">paradox of voting</a> and an <a href="https://en.wikipedia.org/wiki/Voter_apathy#:~:text=In%20political%20science%2C%20voter%20apathy,vote%20where%20voting%20is%20compulsory.">apathy towards action</a>. The argument is made when it comes to voting and democracy. If everyone believes it won't make a difference and doesn't vote, then the one dictator who wants to take power can walk down to the ballot box and cast that one vote and take control. An extreme hypothetical, but the point is, you might not feel like you make that much of a difference but collectively we make an impact.</p>
<p><em>Update 16/01/2021 - <a href="https://lukesingham.com/signal-is-better-than-telegram/#6.-signal's-growth-is-outpacing-the-alternatives-%F0%9F%93%88">Signal's growth of downloads is truly mindboggling</a></em></p>
<h2 id="%F0%9F%99%83-i-don't-want-another-app-on-my-phone">đ I don't want another app on my phone <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%99%83-i-don't-want-another-app-on-my-phone">#</a></h2>
<p>This argument is pretty lazy but I have had it. We are downloading apps all the time, whether it's to pay for parking, collect coffee loyalty points, play games or get in on the latest hip social media thing the cool kids are using. This is an ethical win for the price of moving your digits on that mini screen your eyes are glued to.</p>
<h2 id="%F0%9F%A4%94-ok%2C-you've-piqued-my-interest%2C-where-can-i-learn-even-more%3F">đ¤ Ok, you've piqued my interest, where can I learn even more? <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%A4%94-ok%2C-you've-piqued-my-interest%2C-where-can-i-learn-even-more%3F">#</a></h2>
<p>The following Netflix documentaries:</p>
<ul>
<li><a href="https://www.imdb.com/title/tt4736550/?ref_=wl_li_tt">The Great Hack</a></li>
<li><a href="https://www.imdb.com/title/tt11464826/?ref_=vp_back">The Social Dilemma</a></li>
</ul>
<p>Or đ <a href="https://amzn.to/3i7zyCk">The People Vs Tech</a></p>
<h2 id="%F0%9F%9A%80-i'm-onboard%2C-what-can-i-do%3F">đ I'm onboard, what can I do? <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%F0%9F%9A%80-i'm-onboard%2C-what-can-i-do%3F">#</a></h2>
<ul>
<li>
<p>â <a href="https://signal.org/download/">Download Signal</a>!</p>
</li>
<li>
<blockquote>
<p>Be mindful that if you only delete WhatsApp from your device without using our in-app delete my account feature, your information will be stored with us for a longer period.<br />
<a href="https://www.whatsapp.com/legal/updates/privacy-policy?eea=0#privacy-policy-updates-how-we-work-with-other-facebook-companie">https://www.whatsapp.com/legal/updates/privacy-policy?eea=0#privacy-policy-updates-how-we-work-with-other-facebook-companie</a></p>
</blockquote>
</li>
<li>
<p>Replace your Whatsapp display photo with the Signal logo.</p>
</li>
<li>
<p>Update your 'bio' message to notify your contacts you won't be on Whatsapp after the 8th of Feb.</p>
</li>
<li>
<p>Leave Whatsapp a 1 star review on the app store.</p>
</li>
<li>
<p>For easier transition of group chats, <a href="https://support.signal.org/hc/en-us/articles/360051086971-Group-Link-or-QR-code">set up your Signal group and send a link</a> to your whatsapp group to help them migrate over.</p>
</li>
<li>
<p>If you found this post useful, feel free to share it.</p>
</li>
<li>
<p>Signal is a Not-For-Profit and relies on donations. You can <a href="https://signal.org/donate/">donate directly</a>. If you use Amazon Smile, you can set up the donations to go to the "Signal Technology Foundation".</p>
</li>
</ul>
<p>đ See you on Signal!</p>
<h1 id="%E2%84%B9%EF%B8%8F-other-tidbits">âšď¸ Other tidbits <a class="direct-link" href="https://lukesingham.com/goodbye-whatsapp/#%E2%84%B9%EF%B8%8F-other-tidbits">#</a></h1>
<ul>
<li>If your groups don't move, then a possible solution is to setup a WhatsApp<->Signal bridge using a separate phone+disposable sim using <a href="https://matrix.org/bridges/">Matrix bridges</a></li>
<li><a href="https://danielbmarkham.com/the-platform-is-the-enemy/">Interesting article</a>, not on Whatsapp but social media generally.</li>
<li>AMA on reddit - <a href="https://teddit.net/r/technology/comments/kt91qk/signal_private_messenger_team_here_we_support_an/">here's the teddit link</a>.</li>
<li><a href="https://officechai.com/stories/whatsapps-original-founder-brian-actons-new-app-signal-becomes-top-free-app-in-india/">Apparently</a> 'Signal has become Indiaâs top free app on the App store. More than 100,000 users installed Signal across the app stores of Apple and Google in the last two days'</li>
</ul>
Grokking Algorithms - Summary2021-01-04T00:00:00Zhttps://lukesingham.com/grokking-algorithms/<p>đ Book: <a href="https://amzn.to/2MqMMyg">Grokking Algorithms</a></p>
<p>âď¸ Rating: 5/5 I don't think you could manage to produce a more friendly introduction to algorithms.</p>
<p><img src="https://lukesingham.com/content/images/2020/grokking-algorithms-frontcover.png#thumbnail" alt="" /></p>
<p>I highly recommend this book for anyone who hasn't studied algorithms. You do not need to be a good programmer or to remember all of your high school maths you have probably forgotten. <em>Grokking Algorithms</em> is a beautifully formatted book that explains complex material simply using pictures, analogies and high level practical explanations.</p>
<p>The only criticism I have of the book is the lightness of the last two chapters. Chapter 10 seems to detour into machine learning. And chapter 11 gives a very very quick skim over a few other algorithms not covered. I would have preferred to have seen more of the previous chapters - additional algorithms could have been covered to similar depth of the other chapters.</p>
<p>For anyone who has done a university level course in Algorithms - this book is not deep enough. Although it might help as a light refresher that you can pick up and put down easily, as opposed to getting out an in-depth (hard)core algorithms book like <a href="https://amzn.to/35aXK1h">Introduction to Algorithms</a>.</p>
<h1 id="summary">Summary <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#summary">#</a></h1>
<p>This is my personal summary of <em>Grokking Algorithms</em>. The headings match that of the book, so if a section needs fleshing out you should find the relevant section of the book. Italics reflect direct quotations. Not all sections are included as I felt not all needed summarising.</p>
<h1 id="1.-introduction-to-algorithms">1. Introduction to Algorithms <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#1.-introduction-to-algorithms">#</a></h1>
<p><em>An algorithm is a set of instructions for accomplishing a task.</em></p>
<h2 id="binary-search">Binary Search <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#binary-search">#</a></h2>
<ul>
<li>Data: List</li>
<li>Requirement: List is alpha sorted</li>
<li>Most efficient alg: Binary Search</li>
<li>Analogy: Searching through an address/phone book</li>
</ul>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">binary_search</span><span class="token punctuation">(</span><span class="token builtin">list</span><span class="token punctuation">,</span> item<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token comment"># keep track of what part of the list your searching</span><br /> <span class="token comment"># initialised with the full list</span><br /> low <span class="token operator">=</span> <span class="token number">0</span><br /> high <span class="token operator">=</span> <span class="token builtin">len</span><span class="token punctuation">(</span><span class="token builtin">list</span><span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token number">1</span><br /><br /> <span class="token keyword">while</span> low <span class="token operator"><=</span> high<span class="token punctuation">:</span><br /> <span class="token comment"># check the middle element</span><br /> mid <span class="token operator">=</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token punctuation">(</span>low <span class="token operator">+</span> high<span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">2</span><span class="token punctuation">)</span><br /> guess <span class="token operator">=</span> <span class="token builtin">list</span><span class="token punctuation">[</span>mid<span class="token punctuation">]</span><br /> <span class="token keyword">if</span> guess <span class="token operator">==</span> item<span class="token punctuation">:</span><br /> <span class="token keyword">return</span> mid<br /> <span class="token keyword">if</span> guess <span class="token operator">></span> item<span class="token punctuation">:</span><br /> high <span class="token operator">=</span> mid <span class="token operator">-</span> <span class="token number">1</span><br /> <span class="token keyword">else</span><span class="token punctuation">:</span><br /> low <span class="token operator">=</span> mid <span class="token operator">+</span> <span class="token number">1</span><br /> <span class="token keyword">return</span> <span class="token boolean">None</span><br /><br />my_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">9</span><span class="token punctuation">]</span><br /><br /><span class="token comment"># Find the index of 3 in the list</span><br /><span class="token keyword">print</span><span class="token punctuation">(</span>binary_search<span class="token punctuation">(</span>my_list<span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment"># 1</span><br /><br /><span class="token comment"># Find the index of -1 in the list</span><br /><span class="token keyword">print</span><span class="token punctuation">(</span>binary_search<span class="token punctuation">(</span>my_list<span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment"># None</span></code></pre>
<h2 id="running-time">Running Time <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#running-time">#</a></h2>
<ul>
<li>Let, list = 4 billion</li>
<li>Linear time - Simple Search, starting at the beginning of the list until you find the right element, could take 4 billion steps</li>
<li>Logarithmic time - Finding the right element takes log 4 billion = 32 (rounded)</li>
</ul>
<p>Lesson: Algorithm run times grow at different rates. Timing two different algs is not enough because it doesn't reflect how the run time will grow at different scales.</p>
<h2 id="big-o-notation">Big O Notation <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#big-o-notation">#</a></h2>
<blockquote>
<p>It's called Big O notation because you put a âbig Oâ in front of the number of operations (it sounds like a joke, but itâs true!)<br />
<em>Big O notation is about the worst-case scenario.</em></p>
</blockquote>
<h3 id="some-common-big-o-run-times">Some Common Big O Run Times <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#some-common-big-o-run-times">#</a></h3>
<ul>
<li><strong>O(log n)</strong> <em>log time</em>. E.g. Binary search.</li>
<li><strong>O(n)</strong> <em>linear time</em>. E.g. Simple search.</li>
<li><strong>O(n * log n)</strong> <em>log linear time</em>. E.g. A fast sorting algorithm, like quicksort (see <a href="https://lukesingham.com/grokking-algorithms/#4.-quicksort">chapter 4</a>).</li>
<li><strong>O(n<sup>2</sup>)</strong> <em>quadratic time</em> E.g. A slow sorting algorithm, like selection sort (see <a href="https://lukesingham.com/grokking-algorithms/#2.-selection-sort">chapter 2</a>).</li>
<li><strong>O(n!)</strong> <em>factorial time</em> E.g. A really slow algorithm, like the traveling salesperson (see <a href="https://lukesingham.com/grokking-algorithms/#the-traveling-salesperson">next!</a>).</li>
</ul>
<p>Note: O(1) is constant time e.g. Is a number odd or even?</p>
<p><img src="https://lukesingham.com/content/images/2020/bigO_complexity_graphs.svg" alt="" /></p>
<h2 id="the-traveling-salesperson">The Traveling Salesperson <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-traveling-salesperson">#</a></h2>
<p>Is a classic example of a task that takes O(n!) to complete. To find the shortest route possible for a travelling salesperson covering <em>n</em> cities. All possible combinations of cities must be calculated to find the combination with the shortest distance.</p>
<h1 id="2.-selection-sort">2. Selection Sort <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#2.-selection-sort">#</a></h1>
<blockquote>
<p>Each time you want to store an item in memory, you ask the computer for some space, and it gives you an address where you can store your item. If you want to store multiple items, there are two basic ways to do so: arrays and lists</p>
</blockquote>
<h4 id="memory%3A">Memory: <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#memory%3A">#</a></h4>
<ul>
<li><strong>Arrays</strong> - require contiguous memory. I.e. require slots in memory that are side by side.</li>
<li><strong>Linked Lists</strong> - Do not require contiguous memory, instead each element contains an address to the next element in the list.</li>
</ul>
<h4 id="adding-more-elements-to-a-lists%3A">Adding more elements to a lists: <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#adding-more-elements-to-a-lists%3A">#</a></h4>
<ul>
<li><strong>Arrays</strong> - will need to find a slot in memory large enough for all elements or have been defined initially reserving extra space (trade off being a potential waste of memory).</li>
<li><strong>Linked Lists</strong> - update easily without having to reserve or pre-allocate memory.</li>
</ul>
<h4 id="reading%3A">Reading: <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#reading%3A">#</a></h4>
<ul>
<li><strong>Arrays</strong> are very fast at reading any element since the address is logically deduced using simple maths e.g. If the first element is at memory address 0 then where is element 5? It's at address 4.</li>
<li><strong>Linked lists</strong> are slower since you have to read each element's address to find the next elements address.</li>
</ul>
<p><em>Terminology:</em> The position in memory is known as it's 'index'.</p>
<h4 id="inserts%3A">Inserts: <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#inserts%3A">#</a></h4>
<p>If you want to insert an item into the middle of a list:</p>
<ul>
<li><strong>Arrays</strong> - you will need to move the location of all downstream elements to new memory locations. At worse, you will need to find an entire new memory slot to cater for the contiguous memory need.</li>
<li><strong>Linked Lists</strong> - only need one repoint of memory.</li>
</ul>
<p><strong>Deletions:</strong></p>
<blockquote>
<p>Unlike insertions, deletions will always work. Insertions can fail sometimes when thereâs no space leĹżt in memory. But you can always delete an element... There are two different types of access: random access and sequential access. Sequential access means reading the elements one by one, starting at the first element. Linked lists can only do sequential access.</p>
</blockquote>
<table>
<thead>
<tr>
<th></th>
<th>Arrays</th>
<th>Lists</th>
</tr>
</thead>
<tbody>
<tr>
<td>Reading</td>
<td>O(1)</td>
<td>O(n)</td>
</tr>
<tr>
<td>Insertion</td>
<td>O(n)</td>
<td>O(1)</td>
</tr>
<tr>
<td>Deletion</td>
<td>O(n)</td>
<td>O(1)</td>
</tr>
</tbody>
</table>
<h2 id="selection-sort">Selection sort <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#selection-sort">#</a></h2>
<p>One method of sorting a list e.g.</p>
<table>
<thead>
<tr>
<th style="text-align:right">artist</th>
<th>play count</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right">kooks</td>
<td>10</td>
</tr>
<tr>
<td style="text-align:right">Beatles</td>
<td>15</td>
</tr>
<tr>
<td style="text-align:right">...</td>
<td>...</td>
</tr>
</tbody>
</table>
<p>Would be to iterate n (number of artists) times through the list and create a new list. The first search would be for the 1st highest playcount, then the 2nd and so on. Each search would be O(n) for O(n) artists, i.e. O(n<sup>2</sup>).</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">findSmallest</span><span class="token punctuation">(</span>arr<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> smallest <span class="token operator">=</span> arr<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><br /> smallest_index <span class="token operator">=</span> <span class="token number">0</span><br /> <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token builtin">len</span><span class="token punctuation">(</span>arr<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token keyword">if</span> arr<span class="token punctuation">[</span>i<span class="token punctuation">]</span> <span class="token operator"><</span> smallest<span class="token punctuation">:</span><br /> smallest <span class="token operator">=</span> arr<span class="token punctuation">[</span>i<span class="token punctuation">]</span><br /> smallest_index <span class="token operator">=</span> i<br /> <span class="token keyword">return</span> smallest_index<br /><br /><span class="token keyword">def</span> <span class="token function">selectionSort</span><span class="token punctuation">(</span>arr<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> newArr <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br /> <span class="token keyword">for</span> i <span class="token keyword">in</span> <span class="token builtin">range</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>arr<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">:</span><br /> smallest <span class="token operator">=</span> findSmallest<span class="token punctuation">(</span>arr<span class="token punctuation">)</span><br /> newArr<span class="token punctuation">.</span>append<span class="token punctuation">(</span>arr<span class="token punctuation">.</span>pop<span class="token punctuation">(</span>smallest<span class="token punctuation">)</span><span class="token punctuation">)</span><br /> <span class="token keyword">return</span> newArr<br /><br /><span class="token keyword">print</span><span class="token punctuation">(</span>selectionSort<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">6</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<h1 id="3.-recursion">3. Recursion <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#3.-recursion">#</a></h1>
<blockquote>
<p>âLoops may achieve a performance gain for your program. Recursion may achieve a performance gain for your programmer. Choose which is more important in your situation!â</p>
</blockquote>
<h2 id="base-case-and-recursive-case">Base case and recursive case <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#base-case-and-recursive-case">#</a></h2>
<blockquote>
<p>Because a recursive function calls itself, itâs easy to write a function incorrectly that ends up in an infinite loop.</p>
</blockquote>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">countdown</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token keyword">print</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><br /> countdown<span class="token punctuation">(</span>i<span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span></code></pre>
<p>The above code will run indefinitely. Like a <code>while</code> loop without a clause.</p>
<blockquote>
<p>... every recursive function has two parts: the base case, and the recursive case</p>
</blockquote>
<p>Re-written with a base case:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">countdown</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token keyword">print</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><br /> <span class="token keyword">if</span> i <span class="token operator"><=</span> <span class="token number">0</span><span class="token punctuation">:</span> <span class="token comment"># base case</span><br /> <span class="token keyword">return</span><br /> <span class="token keyword">else</span><span class="token punctuation">:</span> <span class="token comment"># Recursive case</span><br /> countdown<span class="token punctuation">(</span>i<span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span></code></pre>
<h2 id="the-stack">The Stack <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-stack">#</a></h2>
<p>The <em>call stack</em> of computer is conceptually like stacking sticky notes on top of each other. This has it's own simple data structure, <em>the stack</em>. If a function, <code>greet()</code>, calls another function <code>how_are_you()</code>, the computer first allocates a slot in memory for the function call, then places variables used in that slot e.g. <code>Rowan</code>. Therefore, a call to <code>greet()</code> creates the first 'empty sticky note' in the call stack. It then get's filled with <code>Rowan</code>. The next sticky note to go onto the stack is when the call to <code>how_are_you()</code> is made.</p>
<blockquote>
<p>when you call a function from another function, the calling function is paused in a partially completed state.</p>
</blockquote>
<h2 id="the-call-stack-with-recursion">The Call Stack With Recursion <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-call-stack-with-recursion">#</a></h2>
<p>Below is a recursive function to calculate a factorial.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">fact</span><span class="token punctuation">(</span>x<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token keyword">if</span> x <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">:</span><br /> <span class="token keyword">return</span> <span class="token number">1</span><br /> <span class="token keyword">else</span><span class="token punctuation">:</span><br /> <span class="token keyword">return</span> x <span class="token operator">*</span> fact<span class="token punctuation">(</span>x<span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">)</span></code></pre>
<p>If you call <code>fact(3)</code>, each time the base case is not met, <code>fact()</code> will call itself, pausing the previous function call on the stack. The subsequent call of <code>fact()</code> until the base case is met. At which point each function called in the stack get's popped off i.e. resolves all the way back down the stack to the original call of <code>fact(3)</code>.</p>
<p>Stacks have two operations:</p>
<ul>
<li><strong><code>push()</code></strong> adds an element to the collection, and</li>
<li><strong><code>pop()</code></strong> removes the most recently added element that has not yet been removed</li>
</ul>
<p>In the case where <code>fact(3)</code> is called. The first sticky note in the stack is <code>fact(3)</code>, the last is <code>fact(1)</code>. Before the final result are display the each result of <code>fact()</code> is multiplied by each of the copies of the function in the call stack with the variable x (per the logic of the function). E.g. 3x2x1 = 6.</p>
<p>Using the call stack is convenient but can take lots of memory if the stack becomes too tall. Alternatively, you could rewrite your code as a loop or use tail recursion (not covered by book).</p>
<h1 id="4.-quicksort">4. Quicksort <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#4.-quicksort">#</a></h1>
<h2 id="divide-and-conquer-(d%26c)">Divide and Conquer (D&C) <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#divide-and-conquer-(d%26c)">#</a></h2>
<p>D&C algorithms are recursive. The two steps to solving a problem using D&C:</p>
<ol>
<li>What is the base case? (Simplest possible)</li>
<li>Divide the problem until it becomes the base case.</li>
</ol>
<p>Example: Say you have to divide up a piece of land into the largest possible squares and all squares have to be the same size. The plot of land is 400 x 240m. The largest possible square is 240. But there is a left over land, 240 x 160m. Now apply the same technique to the remainder, the largest possible square is 160m. The remainder becomes 160 x 80m - 80m is the base case as any further division would leave no remainder. See <a href="https://en.wikipedia.org/wiki/Euclidean_algorithm">Euclid's Algorithm</a></p>
<blockquote>
<p>D&C isnât a simple algorithm that you can apply to a problem. Instead, itâs a way to think about a problem.</p>
</blockquote>
<h2 id="for-loop-vs-recursion-to-sum-an-array">For Loop vs Recursion To Sum An Array <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#for-loop-vs-recursion-to-sum-an-array">#</a></h2>
<h3 id="for-loop">For Loop <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#for-loop">#</a></h3>
<p>For each item in the array, increment the total by the value of the item:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">sum</span><span class="token punctuation">(</span>array<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> total <span class="token operator">=</span> <span class="token number">0</span><br /> <span class="token keyword">for</span> item <span class="token keyword">in</span> array<span class="token punctuation">:</span><br /> total <span class="token operator">+=</span> item</code></pre>
<h3 id="recursion">Recursion <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#recursion">#</a></h3>
<p>What is the simplest 'base case' array? One with either 1 or 0 elements.<br />
Divide the problem up.</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">sum</span><span class="token punctuation">(</span>array<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token comment"># Base case</span><br /> <span class="token keyword">if</span> array <span class="token operator">==</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">:</span><br /> <span class="token keyword">return</span> <span class="token number">0</span><br /> <span class="token keyword">else</span><span class="token punctuation">:</span><br /> <span class="token comment"># divide the problem up</span><br /> <span class="token keyword">return</span> array<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">+</span> <span class="token builtin">sum</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<h2 id="quicksort">Quicksort <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#quicksort">#</a></h2>
<ul>
<li>Is faster than selection sort.</li>
<li>Also uses a divide and conquer strategy.</li>
</ul>
<p>Let's say you have an array of <code>[33,15,10]</code>. Quicksort will:</p>
<ol>
<li>
<p>Pick a number <em>the pivot</em> e.g. 33</p>
</li>
<li>
<p>Partition into two subarrays:<br />
a. all elements greater than the pivot<br />
b. all elements less than the pivot<br />
Such that you will have the pivot <code>[33]</code>, an array of number/s smaller than the pivot <code>[15,10]</code> and an array for numbers larger than the pivot <code>[]</code>.</p>
</li>
<li>
<p>Call quicksort recursively on the two subarrays</p>
</li>
</ol>
<h2 id="aside%3A-inductive-proofs">Aside: Inductive Proofs <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#aside%3A-inductive-proofs">#</a></h2>
<p>Is a method to test and prove an algorithm works. An inductive proof contains two steps:</p>
<ol>
<li>The base case.</li>
<li>The inductive case.</li>
</ol>
<p>E.g. You want to climb a ladder, the inductive case is that given a rung you can go to the next rung.<br />
The base case is that your legs are on rung 1 of the ladder.</p>
<h4 id="python-quicksort">Python Quicksort <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#python-quicksort">#</a></h4>
<pre class="language-python"><code class="language-python"><span class="token keyword">def</span> <span class="token function">quicksort</span><span class="token punctuation">(</span>array<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token comment"># base case an with 1 or less elements.</span><br /> <span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>array<span class="token punctuation">)</span> <span class="token operator"><</span><span class="token number">2</span><span class="token punctuation">:</span><br /> <span class="token keyword">return</span> array<br /> <span class="token keyword">else</span><span class="token punctuation">:</span><br /> <span class="token comment"># recursive case</span><br /> pivot <span class="token operator">=</span> array<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><br /> less <span class="token operator">=</span> <span class="token punctuation">[</span>i <span class="token keyword">for</span> i <span class="token keyword">in</span> array<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">]</span> <span class="token keyword">if</span> i <span class="token operator"><=</span> pivot<span class="token punctuation">]</span><br /> greater <span class="token operator">=</span> <span class="token punctuation">[</span>i <span class="token keyword">for</span> i <span class="token keyword">in</span> array<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token punctuation">]</span> <span class="token keyword">if</span> i <span class="token operator">></span> pivot<span class="token punctuation">]</span><br /><br /> <span class="token keyword">return</span> quicksort<span class="token punctuation">(</span>less<span class="token punctuation">)</span> <span class="token operator">+</span> pivot <span class="token operator">+</span> quicksort<span class="token punctuation">(</span>greater<span class="token punctuation">)</span></code></pre>
<h2 id="big-o-notation-2">Big O Notation <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#big-o-notation-2">#</a></h2>
<ul>
<li>Selection Sort is O(n<sup>2</sup>)</li>
<li>Quick Sort is O(n<sup>2</sup>)
<ul>
<li>Worst case O(n<sup>2</sup>)</li>
<li>Avg. case O(n log n)</li>
</ul>
</li>
<li>Merge Sort is O(n log n)</li>
</ul>
<p>If the avg. case of quicksort is O(n log n) why not use merge sort?</p>
<h2 id="merge-sort-vs-quicksort">Merge sort vs quicksort <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#merge-sort-vs-quicksort">#</a></h2>
<p>If you have two algorithms that print a list of numbers in a <code>for</code> loop where one sleeps for a second between each print operation. Both algorithms will be O(n) time. However we know in practice one is faster than the other. This is because really Big 0 notation is <code>c x n</code> where <code>c</code> is some fixed amount of time. Usually the constant doesn't matter. e.g. simple search vs binary search. Let's say the constant is 10ms for simple search and we penalise binary search with a constant of 1sec. If we had to search 4 billion records simple search would take 463 days vs binary taking 32 seconds.</p>
<p>However, sometimes, like in the case of mergesort vs quicksort. Quicksort has a smaller constant than mergesort. Therefore, if quicksort is O(n log n) time then quicksort is faster.</p>
<h2 id="average-case-vs-worst-case">Average case vs worst case <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#average-case-vs-worst-case">#</a></h2>
<p>The performance of quicksort depends on the chosen pivot. E.g. if the first element is always chosen and the array is already sorted it would take a call stack of 8 to sort the array O(n). On the other hand, picking the middle of the array each time would result in a call stack of 4 to sort the array O(log n). Both of these involve touching every element O(n). Therefore, the worst case is O(n) * O(n) = O(n<sup>2</sup>) and the best case (average case) is O(n) * O(log n) = O(n log n). Therefore, if you choose the pivot randomly the runtime of quicksort is O(n log n).</p>
<h1 id="5.-hash-tables">5. Hash Tables <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#5.-hash-tables">#</a></h1>
<p>A hash function returns the index of a value removing the need to perform a search. For this to work a hash function must:</p>
<ul>
<li>consistently map an input to the same index</li>
<li>map different strings to different indexes</li>
<li>not return indexes outside the size of the array</li>
</ul>
<p>A hash function combined with an array is a hash table. Hash tables are a complex data structure also known as maps, dictionaries and associative arrays.</p>
<h2 id="use-cases">Use cases <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#use-cases">#</a></h2>
<p>DNS resolution - mapping domain names to IP addresses<br />
Caching - Frequently requested pages are commonly cached in server memory.</p>
<h2 id="collisions">Collisions <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#collisions">#</a></h2>
<p>It's extremely difficult to write a hash function that doesn't experience collisions. E.g. a grocery store with an alphabetical array with 26 slots. Apples goes to the first slot, then bananas in the second slot. But what if the store now needs to store a price for avocados? It would need to be stored at the slot where apples is. There are many ways to deal with collisions, however having a linked list at the slot is a common solution. If the linked list is small, then the performance hit is negligible. If a store only sold products starting with A, an alphabetical index would perform very poorly.<br />
<em>Take away</em>: you want an index that distributes the items as evenly as possible.</p>
<h2 id="performance">Performance <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#performance">#</a></h2>
<p>Average case: O(1) constant time<br />
Worst case: 0(n) linear time</p>
<p>To avoid worst case performance, collisions must be avoided. To avoid collisions:</p>
<ul>
<li>Have a low load factor (number of items / total number of slots). When the items grow and hence the load factor, you should <em>resize</em> the hash table and increase the slots. Rule of thumb, when LF > .07</li>
<li>a good hash function</li>
</ul>
<h1 id="6.-breadth-first-search-(bfs)">6. Breadth First Search (BFS) <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#6.-breadth-first-search-(bfs)">#</a></h1>
<p>BFS is great at Shortest Path problems, examples of which include:</p>
<ul>
<li>fewest moves to victory in a checkers game</li>
<li>fewest edits from a misspelt word to a correct word</li>
<li>find the closest doctor</li>
</ul>
<p>To solve these problems, first model them as a graph then apply BFS.</p>
<h2 id="what-is-a-graph%3F">What is a graph? <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#what-is-a-graph%3F">#</a></h2>
<p>Graphs are ways to model how different things are connected to each other, they have nodes (things) and edges (how things are connected/related to each other).</p>
<h2 id="bfs">BFS <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#bfs">#</a></h2>
<p>Can answer:</p>
<ul>
<li>is there a path between A and B?</li>
<li>what is the shortest path between A and B?</li>
</ul>
<p>First degree nodes will be searched first before second degree nodes. Therefore, nodes will be queued in a FIFO manner.</p>
<h2 id="implementing-the-graph">Implementing the graph <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#implementing-the-graph">#</a></h2>
<p>e.g. You have 3 friends, this can be implemented in a hash table.</p>
<pre class="language-python"><code class="language-python">graph <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />graph<span class="token punctuation">[</span><span class="token string">'you'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'alice'</span><span class="token punctuation">,</span> <span class="token string">'bob'</span><span class="token punctuation">,</span> <span class="token string">'claire'</span><span class="token punctuation">]</span></code></pre>
<p>Where your friends are contained in an array. Extending this a little:</p>
<pre class="language-python"><code class="language-python">graph<span class="token punctuation">[</span><span class="token string">'alice'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'peggy'</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'peggy'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'bob'</span><span class="token punctuation">]</span></code></pre>
<p>The ordering of which you add to the graph doesn't matter as hash tables do not have ordering. This is an example of a Directed Graph, where two nodes can have 'directed' relationships, e.g. A -> B. Alice points to Peggy, but Peggy doesn't point back to Alice.</p>
<h2 id="implementing-the-algorithm">Implementing the algorithm <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#implementing-the-algorithm">#</a></h2>
<ol>
<li>Keep a queue of items</li>
<li>Dequeue an item</li>
<li>Check if the item meets the condition</li>
<li>Does it meet the connection?<br />
a. meets condition <code>END</code><br />
b. Add all connected nodes to the condition</li>
<li>Loop</li>
<li>If the queue is empty then <code>END</code>.</li>
</ol>
<pre class="language-py"><code class="language-py"><span class="token keyword">from</span> collections <span class="token keyword">import</span> deque<br /><br /><span class="token comment"># example graph</span><br />graph <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />graph<span class="token punctuation">[</span><span class="token string">'you'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'alice'</span><span class="token punctuation">,</span> <span class="token string">'bob'</span><span class="token punctuation">,</span> <span class="token string">'claire'</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'bob'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'anuj'</span><span class="token punctuation">,</span> <span class="token string">'peggy'</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'alice'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'peggy'</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'claire'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'thom'</span><span class="token punctuation">,</span> <span class="token string">'jonny'</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'anuj'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'peggy'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'thom'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br />graph<span class="token punctuation">[</span><span class="token string">'jonny'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br /><br /><span class="token keyword">def</span> <span class="token function">person_is_seller</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token triple-quoted-string string">'''Silly example - If the name ends w. 'm' then seller'''</span><br /> <span class="token keyword">return</span> name<span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token string">'m'</span><br /><br /><span class="token keyword">def</span> <span class="token function">bfs_search</span><span class="token punctuation">(</span>name<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> search_queue <span class="token operator">=</span> deque<span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token comment"># create queue</span><br /> search_queue <span class="token operator">+=</span> graph<span class="token punctuation">[</span><span class="token string">"you"</span><span class="token punctuation">]</span><br /> searched <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br /><br /> <span class="token comment"># while the queue isn't empty, get the first item check</span><br /> <span class="token keyword">while</span> search_queue<span class="token punctuation">:</span><br /> person <span class="token operator">=</span> search_queue<span class="token punctuation">.</span>popleft<span class="token punctuation">(</span><span class="token punctuation">)</span><br /> <span class="token keyword">if</span> <span class="token keyword">not</span> person <span class="token keyword">in</span> searched<span class="token punctuation">:</span><br /> <span class="token keyword">if</span> person_is_seller<span class="token punctuation">(</span>person<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token keyword">print</span><span class="token punctuation">(</span>person <span class="token operator">+</span> <span class="token string">" is a mango seller!"</span><span class="token punctuation">)</span><br /> <span class="token keyword">return</span> <span class="token boolean">True</span><br /> <span class="token keyword">else</span><span class="token punctuation">:</span><br /> search_queue <span class="token operator">+=</span> graph<span class="token punctuation">[</span>person<span class="token punctuation">]</span><br /> <span class="token keyword">return</span> <span class="token boolean">False</span><br /><br />bfs_search<span class="token punctuation">(</span><span class="token string">'you'</span><span class="token punctuation">)</span></code></pre>
<h2 id="run-time">Run Time <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#run-time">#</a></h2>
<p>Searching each vertice (V) plus adding a person (Edge) to a queue will take O(V+E).</p>
<h1 id="7.-dijkstra's-algorithm">7. Dijkstra's Algorithm <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#7.-dijkstra's-algorithm">#</a></h1>
<p>In contrast to BFS which finds the fewest vertices between two nodes. Dijkstra's algorithm uses graphs that have weights (e.g. travel time between two points) finds the smallest total weight between two nodes. Dijkstra's Algorithm has 4 steps:</p>
<ol>
<li>Find the cheapest node</li>
<li>Check if there's a cheaper path to the neighbours of the cheapest nodes, if so update their costs.</li>
<li>Repeat for all nodes in the graph.</li>
<li>Calculate final path.</li>
</ol>
<p>N.B. - Dijkstra's algorithm only works with directed acyclic graph (DAGs).</p>
<ul>
<li>You can only use positive weights for negatively weighted graphs see the <a href="https://en.wikipedia.org/wiki/Bellman%E2%80%93Ford_algorithm">Bellman-Ford algorithm</a>.</li>
</ul>
<h2 id="implementation">Implementation <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#implementation">#</a></h2>
<pre class="language-py"><code class="language-py"><span class="token comment"># wieghted graph</span><br />graph<span class="token punctuation">[</span><span class="token string">'start'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />graph<span class="token punctuation">[</span><span class="token string">'start'</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'a'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">6</span><br />graph<span class="token punctuation">[</span><span class="token string">'start'</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'b'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">2</span><br />graph<span class="token punctuation">[</span><span class="token string">'a'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />graph<span class="token punctuation">[</span><span class="token string">'a'</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'fin'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">1</span><br />graph<span class="token punctuation">[</span><span class="token string">'b'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />graph<span class="token punctuation">[</span><span class="token string">'b'</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'a'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">3</span><br />graph<span class="token punctuation">[</span><span class="token string">'b'</span><span class="token punctuation">]</span><span class="token punctuation">[</span><span class="token string">'fin'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">5</span><br />graph<span class="token punctuation">[</span><span class="token string">'fin'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br /><br /><span class="token comment"># graph for updating costs</span><br />infinity <span class="token operator">=</span> <span class="token builtin">float</span><span class="token punctuation">(</span><span class="token string">'inf'</span><span class="token punctuation">)</span><br />costs <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />costs<span class="token punctuation">[</span><span class="token string">'a'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">6</span><br />costs<span class="token punctuation">[</span><span class="token string">'b'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token number">2</span><br />costs<span class="token punctuation">[</span><span class="token string">'fin'</span><span class="token punctuation">]</span> <span class="token operator">=</span> infinity<br /><br /><span class="token comment"># hash table for parents</span><br />parents <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />parents<span class="token punctuation">[</span><span class="token string">'a'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">'start'</span><br />parents<span class="token punctuation">[</span><span class="token string">'b'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">'start'</span><br />parents<span class="token punctuation">[</span><span class="token string">'fin'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token boolean">None</span><br /><br /><span class="token comment"># array to track processed nodes</span><br />processed <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><br /><br /><span class="token keyword">def</span> <span class="token function">find_lowest_cost_node</span><span class="token punctuation">(</span>costs<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> lowest_cost <span class="token operator">=</span> <span class="token builtin">float</span><span class="token punctuation">(</span><span class="token string">'inf'</span><span class="token punctuation">)</span><br /> lowest_cost_node <span class="token operator">=</span> <span class="token boolean">None</span><br /> <span class="token comment"># Go through each node.</span><br /> <span class="token keyword">for</span> node <span class="token keyword">in</span> costs<span class="token punctuation">:</span><br /> cost <span class="token operator">=</span> costs<span class="token punctuation">[</span>node<span class="token punctuation">]</span><br /> <span class="token comment"># If itâs the lowest cost so far and hasnât been processed</span><br /> <span class="token keyword">if</span> cost <span class="token operator"><</span> lowest_cost <span class="token keyword">and</span> node <span class="token keyword">not</span> <span class="token keyword">in</span> processed<span class="token punctuation">:</span><br /> <span class="token comment"># set it as the new lowest-cost node.</span><br /> lowest_cost <span class="token operator">=</span> cost<br /> lowest_cost_node <span class="token operator">=</span> node<br /> <span class="token keyword">return</span> lowest_cost_node<br /><br /><span class="token comment"># find the lowest cost node that you havenât processed yet.</span><br />node <span class="token operator">=</span> find_lowest_cost_node<span class="token punctuation">(</span>costs<span class="token punctuation">)</span><br /><br /><span class="token keyword">while</span> node <span class="token keyword">is</span> <span class="token keyword">not</span> <span class="token boolean">None</span><span class="token punctuation">:</span><br /> cost <span class="token operator">=</span> costs<span class="token punctuation">[</span>node<span class="token punctuation">]</span><br /> neighbors <span class="token operator">=</span> graph<span class="token punctuation">[</span>node<span class="token punctuation">]</span><br /> <span class="token comment"># go through all the neighbors of this node.</span><br /> <span class="token keyword">for</span> n <span class="token keyword">in</span> neighbors<span class="token punctuation">.</span>keys<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span><br /> new_cost <span class="token operator">=</span> cost <span class="token operator">+</span> neighbors<span class="token punctuation">[</span>n<span class="token punctuation">]</span><br /> <span class="token comment"># If itâs cheaper to get to this neighbor by going through this node</span><br /> <span class="token comment"># update the cost for this node.</span><br /> <span class="token keyword">if</span> costs<span class="token punctuation">[</span>n<span class="token punctuation">]</span> <span class="token operator">></span> new_cost<span class="token punctuation">:</span><br /> costs<span class="token punctuation">[</span>n<span class="token punctuation">]</span> <span class="token operator">=</span> new_cost<br /> <span class="token comment"># this node becomes the new parent for this neighbor</span><br /> parents<span class="token punctuation">[</span>n<span class="token punctuation">]</span> <span class="token operator">=</span> node<br /> <span class="token comment"># mark the node as processed.</span><br /> processed<span class="token punctuation">.</span>append<span class="token punctuation">(</span>node<span class="token punctuation">)</span><br /> <span class="token comment"># Find the next node to process, and loop.</span><br /> node <span class="token operator">=</span> find_lowest_cost_node<span class="token punctuation">(</span>costs<span class="token punctuation">)</span></code></pre>
<h1 id="8.-greedy-algorithms">8. Greedy Algorithms <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#8.-greedy-algorithms">#</a></h1>
<blockquote>
<p>Greedy algorithms optimize locally, hoping to end up with a global optimum. Greedy algorithms are easy to write and fast to run, so they make good approximation algorithms.</p>
</blockquote>
<h2 id="the-classroom-scheduling-problem">The Classroom Scheduling problem <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-classroom-scheduling-problem">#</a></h2>
<p>With overlapping classroom run times how does one maximise the amount of classes to be held? The algorithm is very simple:</p>
<ol>
<li>Pick the class that ends the soonest, this will be the first class.</li>
<li>Now pick a class that starts after the first class and ends the soonest. Repeat.</li>
</ol>
<h2 id="the-knapsack-problem">The Knapsack problem <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-knapsack-problem">#</a></h2>
<p>You are trying to maximise the value of the items one can place in a small bag 35lbs. The items are:</p>
<ul>
<li>Stereo $3000 30lbs</li>
<li>Laptop $2000 20lbs</li>
<li>Guitar $1500 15lbs</li>
</ul>
<p>The greedy strategy is to take the most expensive item. Buut the stereo leaves room for nothing else. $3000 get's pretty close to the optimal solution of $3500 (Laptop + Guitar).</p>
<h2 id="the-set-covering-problem">The Set Covering Problem <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-set-covering-problem">#</a></h2>
<p>You want to broadcast a radio show in the USA and reach listeners in all states. To minimise cost you want to minimise the number of stations you have to pay whilst maximising the states they broadcast to.</p>
<ol>
<li>To list every possible subset of stations is called the <em>power set</em>.</li>
<li>From this pick the set with the smallest number of stations that covers all 50 states.</li>
</ol>
<p>Calculating the power set is a O(2<sup>n</sup>) problem. If there are only 32 stations it would take 13.6 years to calculate at 10 subsets a second.</p>
<h2 id="approximation-algorithms">Approximation Algorithms <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#approximation-algorithms">#</a></h2>
<p>A greedy algorithm will get close to the optimal solution with much less computation, e.g.:</p>
<ol>
<li>Pick a station that covers the most stations.</li>
<li>Repeat until all states are covered.<br />
N.b some overlap will occur.</li>
</ol>
<p>An approximation algorithm is judged on speed and how close the optimal solution they get.</p>
<h3 id="python-set-operations">Python set operations <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#python-set-operations">#</a></h3>
<pre class="language-py"><code class="language-py"><span class="token operator">>></span><span class="token operator">></span> fruits <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span>âavocadoâ<span class="token punctuation">,</span> âtomatoâ<span class="token punctuation">,</span> âbananaâ<span class="token punctuation">]</span><span class="token punctuation">)</span><br /><span class="token operator">>></span><span class="token operator">></span> vegetables <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span>âbeetsâ<span class="token punctuation">,</span> âcarrotsâ<span class="token punctuation">,</span> âtomatoâ<span class="token punctuation">]</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># This is a set union.</span><br /><span class="token operator">>></span><span class="token operator">></span> fruits <span class="token operator">|</span> vegetables<br /><span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span>âavocadoâ<span class="token punctuation">,</span> âbeetsâ<span class="token punctuation">,</span> âcarrotsâ<span class="token punctuation">,</span> âtomatoâ<span class="token punctuation">,</span> âbananaâ<span class="token punctuation">]</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># This is a set intersection.</span><br /><span class="token operator">>></span><span class="token operator">></span> fruits <span class="token operator">&</span> vegetables<br /><span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span>âtomatoâ<span class="token punctuation">]</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># This is a set difference.</span><br /><span class="token operator">>></span><span class="token operator">></span> fruits â vegetables<br /><span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span>âavocadoâ<span class="token punctuation">,</span> âbananaâ<span class="token punctuation">]</span><span class="token punctuation">)</span></code></pre>
<p>The greedy solution to the state problem:</p>
<pre class="language-py"><code class="language-py">states_needed <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'mt'</span><span class="token punctuation">,</span> <span class="token string">'wa'</span><span class="token punctuation">,</span> <span class="token string">'or'</span><span class="token punctuation">,</span> <span class="token string">'id'</span><span class="token punctuation">,</span> <span class="token string">'nv'</span><span class="token punctuation">,</span> <span class="token string">'ut'</span><span class="token punctuation">,</span> <span class="token string">'ca'</span><span class="token punctuation">,</span> <span class="token string">'az'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /><br />stations <span class="token operator">=</span> <span class="token punctuation">{</span><span class="token punctuation">}</span><br />stations<span class="token punctuation">[</span><span class="token string">'kone'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'id'</span><span class="token punctuation">,</span> <span class="token string">'nv'</span><span class="token punctuation">,</span> <span class="token string">'ut'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />stations<span class="token punctuation">[</span><span class="token string">'ktwo'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'wa'</span><span class="token punctuation">,</span> <span class="token string">'id'</span><span class="token punctuation">,</span> <span class="token string">'mt'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />stations<span class="token punctuation">[</span><span class="token string">'kthree'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'or'</span><span class="token punctuation">,</span> <span class="token string">'nv'</span><span class="token punctuation">,</span> <span class="token string">'ca'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />stations<span class="token punctuation">[</span><span class="token string">'kfour'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'nv'</span><span class="token punctuation">,</span> <span class="token string">'ut'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />stations<span class="token punctuation">[</span><span class="token string">'kfive'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'ca'</span><span class="token punctuation">,</span> <span class="token string">'az'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /><br />final_stations <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">)</span><br /><br /><span class="token keyword">while</span> states_needed<span class="token punctuation">:</span><br /> best_station <span class="token operator">=</span> <span class="token boolean">None</span><br /> states_covered <span class="token operator">=</span> <span class="token builtin">set</span><span class="token punctuation">(</span><span class="token punctuation">)</span><br /><br /> <span class="token keyword">for</span> station<span class="token punctuation">,</span> states_for_station <span class="token keyword">in</span> stations<span class="token punctuation">.</span>items<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token comment"># states that in both needed and for_station</span><br /> covered <span class="token operator">=</span> states_needed <span class="token operator">&</span> states_for_station<br /><br /> <span class="token comment"># check this station covers more than the current best station if so, assign as the new best station</span><br /> <span class="token keyword">if</span> <span class="token builtin">len</span><span class="token punctuation">(</span>covered<span class="token punctuation">)</span> <span class="token operator">></span> <span class="token builtin">len</span><span class="token punctuation">(</span>states_covered<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> best_station <span class="token operator">=</span> station<br /> states_covered <span class="token operator">=</span> covered<br /><br /> final_stations<span class="token punctuation">.</span>add<span class="token punctuation">(</span>best_station<span class="token punctuation">)</span> <span class="token comment"># add to list of stations</span><br /> states_needed <span class="token operator">-=</span> states_covered <span class="token comment"># rm states covered from those needed</span><br /><br /><span class="token keyword">print</span><span class="token punctuation">(</span>final_stations<span class="token punctuation">)</span><br /><span class="token comment"># {'kfive', 'kone', 'kthree', 'ktwo'}</span></code></pre>
<h2 id="np-complete-problems">NP-Complete Problems <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#np-complete-problems">#</a></h2>
<h3 id="traveling-salesperson">Traveling Salesperson <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#traveling-salesperson">#</a></h3>
<p>Problem: A salesperson has to find the shortest distance when traveling to say 5 cities.</p>
<p>For each additional cities the number goes up in a factorial fashion, here's an an example:</p>
<blockquote>
<p>1 start city * 1 route = 1 total route<br />
2 start cities * 1 route = 2 total routes<br />
3 start cities * 2 route = 6 total routes<br />
4 start cities * 6 route = 24 total routes<br />
5 start cities * 24 route = 120 total routes<br />
6 start cities * 120 route = 720 total routes<br />
...<br />
10 start cities * 362,880 route = 3,628,800 total routes</p>
</blockquote>
<p>A good approximation algorithm would be to pick an arbitrary starting city and pick the closest non-visited city next.</p>
<h3 id="how-to-tell-if-a-problem-is-np-complete%3F">How to tell if a problem is NP complete? <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#how-to-tell-if-a-problem-is-np-complete%3F">#</a></h3>
<p>There's no easy way! Here's some clues:</p>
<ul>
<li>Runs fine with small n, but slow with big n</li>
<li>"All combinations of X" / can't be broken into smaller problems</li>
<li>Problem that involve sequences e.g. cities</li>
<li>Problem that involve sets e.g. radio stations</li>
</ul>
<p>In short:</p>
<ul>
<li>Greedy algorithms optimise locally, hopefully it will be a global optimum</li>
<li>NP complete problems have no known fast solution</li>
<li>Approximation algorithms including greedy algorithms are used to solve NP complete problems</li>
</ul>
<h1 id="9.-dynamic-programming">9. Dynamic Programming <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#9.-dynamic-programming">#</a></h1>
<h2 id="the-knapsack-problem-(revisited)">The Knapsack Problem (revisited) <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#the-knapsack-problem-(revisited)">#</a></h2>
<p>You're a thief with Knapsack capacity of 4kgs. There are 3 items to steal:</p>
<ul>
<li>Stereo $3000 4kgs</li>
<li>Laptop $2000 3kgs</li>
<li>Guitar $1500 1kg</li>
</ul>
<p>The simplest algorithm would be to calculate every possible set of goods. However this doesn't scale, O(2<sup>n</sup>). The optimal solution can be calculated using dynamic programming.</p>
<h4 id="dynamic-programming-steps%3A">Dynamic Programming Steps: <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#dynamic-programming-steps%3A">#</a></h4>
<ul>
<li>start by solving subproblems and builds up to solving the big problem</li>
<li>start with a grid</li>
<li>fill each row out</li>
<li>for rows beyond the first row the cell above is the current max for that column. Check the current row's item + remaining space is > than the previous max. e.g. at cell [stereo][4kg] previous max is $1500. Placing the Stereo in the bag is greater than the previous max so this is the solution for this cell.</li>
</ul>
<h4 id="returning-to-the-knapsack-problem.">Returning to the knapsack problem. <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#returning-to-the-knapsack-problem.">#</a></h4>
<p>Make a grid where items go down one side and knapsack weights go down the other</p>
<p>If we start with the Guitar and ignore the rest of the products we can fill out the top row with the most value we can steal quite easily. This becomes our current best guess.</p>
<div class="twrap">
<table>
<thead>
<tr>
<th style="text-align:left"></th>
<th style="text-align:left">1kg</th>
<th style="text-align:left">2kg</th>
<th style="text-align:left">3kg</th>
<th style="text-align:left">4kg</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">Guitar $1500 1kg</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
</tr>
<tr>
<td style="text-align:left">Stereo $3000 4kgs</td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
</tr>
<tr>
<td style="text-align:left">Laptop $2000 3kgs</td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
</tr>
</tbody>
</table>
</div>
<p>With the Stereo, it can't fit in any bag smaller than 4kg. Therefore, we can take the value from the cell immediately above for columns <4kg. In the 4kg column, the value is $3000 as that is much better than taking the Guitar at half the Stereo's value.</p>
<div class="twrap">
<table>
<thead>
<tr>
<th style="text-align:left"></th>
<th style="text-align:left">1kg</th>
<th style="text-align:left">2kg</th>
<th style="text-align:left">3kg</th>
<th style="text-align:left">4kg</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">Guitar $1500 1kg</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
</tr>
<tr>
<td style="text-align:left">Stereo $3000 4kgs</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$3000</td>
</tr>
<tr>
<td style="text-align:left">Laptop $2000 3kgs</td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
<td style="text-align:left"></td>
</tr>
</tbody>
</table>
</div>
<p>For the Laptop, it can't fit in any bag <3kg. So we drop down the values from the cell immediately above. At 3kg, taking the Laptop at $2000 is better than the cell value immediately above. Lastly, at 4kg <em>only</em> taking the laptop would be less than the value above of $3000. However, if we took the Laptop wit hthe 4kg bag, we would have 1kg left over. What's the best possible thing we could take at 1kg - a Guitar. Therefore, the optimal solution for 4kg is a Laptop + Guitar.</p>
<div class="twrap">
<table>
<thead>
<tr>
<th style="text-align:left"></th>
<th style="text-align:left">1kg</th>
<th style="text-align:left">2kg</th>
<th style="text-align:left">3kg</th>
<th style="text-align:left">4kg</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left">Guitar $1500 1kg</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
</tr>
<tr>
<td style="text-align:left">Stereo $3000 4kgs</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$3000</td>
</tr>
<tr>
<td style="text-align:left">Laptop $2000 3kgs</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$1500</td>
<td style="text-align:left">$2000</td>
<td style="text-align:left">$3500</td>
</tr>
</tbody>
</table>
</div>
<h2 id="dynamic-programming-faqs">Dynamic Programming FAQs <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#dynamic-programming-faqs">#</a></h2>
<h4 id="what-happens-if-you-add-another-item%3F">What happens if you add another item? <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#what-happens-if-you-add-another-item%3F">#</a></h4>
<p>All previously calculated values stay the same. You just work out the new rows values.</p>
<h4 id="what-happens-if-you-add-a-smaller-item-e.g.-necklace-0.5kg%3F">What happens if you add a smaller item e.g. necklace 0.5kg? <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#what-happens-if-you-add-a-smaller-item-e.g.-necklace-0.5kg%3F">#</a></h4>
<p>For original problem, the column increments were set on the smallest product weight, the Guitar at 1kg. To include a necklace, the columns would need to increment by 0.5kg starting at 0.5kg and ending at the max bag capacity 4kg.</p>
<h4 id="can-you-steal-fractions-of-an-item%3F">Can you steal fractions of an item? <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#can-you-steal-fractions-of-an-item%3F">#</a></h4>
<p>Using dynamic programming, no. But you could with a greedy algorithm.</p>
<h4 id="can-dp-handle-items-that-depend-on-each-other%3F">Can DP handle items that depend on each other? <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#can-dp-handle-items-that-depend-on-each-other%3F">#</a></h4>
<p>No.</p>
<blockquote>
<p>Dynamic programming only works when each subproblem is discrete - when it doesn't depend on other subproblems.</p>
</blockquote>
<h2 id="longest-common-substring">Longest common substring <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#longest-common-substring">#</a></h2>
<p>Suppose you need to create a search functionality on a website, and you want to have search suggestions when someone makes a typo. For example, someone types <code>hish</code> when they probably mean <code>fish</code>. How would you use DP to solve the problem? <em>There is no formula, but DP should provide a framework to develop your idea.</em></p>
<table>
<thead>
<tr>
<th style="text-align:left"></th>
<th style="text-align:left">H</th>
<th style="text-align:left">I</th>
<th style="text-align:left">S</th>
<th style="text-align:left">H</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left"><strong>F</strong></td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
</tr>
<tr>
<td style="text-align:left"><strong>I</strong></td>
<td style="text-align:left">0</td>
<td style="text-align:left">1</td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
</tr>
<tr>
<td style="text-align:left"><strong>S</strong></td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
<td style="text-align:left">2</td>
<td style="text-align:left">0</td>
</tr>
<tr>
<td style="text-align:left"><strong>H</strong></td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
<td style="text-align:left">0</td>
<td style="text-align:left">3</td>
</tr>
</tbody>
</table>
<ol>
<li>If the letters don't match, then 0</li>
<li>If they do match, then the value is the top-left neighbor + 1</li>
</ol>
<p>DP is used in the following applications:</p>
<ul>
<li>DNA sequencing, <em>how similar are two animals DNA?</em></li>
<li><code>git diff</code></li>
<li>string similarity problems, <em>Levenshtein distance</em> applied to spell check to determining the amount of copyright content is in something uploaded.</li>
</ul>
<h2 id="dp-takeaways">DP Takeaways <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#dp-takeaways">#</a></h2>
<ul>
<li>DP is useful when you're trying to optimise something given a constraint</li>
<li>DP is usable when the problem can be divided into discrete subproblems</li>
<li>DP solutions always involve a grid</li>
<li>each cell is a subproblem</li>
<li>there is no single formula for DP solutions</li>
</ul>
<h1 id="10.-k-nearest-neighbours">10. K-Nearest Neighbours <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#10.-k-nearest-neighbours">#</a></h1>
<p>You need to classify citrus fruit into oranges or grapefruits.</p>
<p><img src="https://lukesingham.com/content/images/2020/knn.svg" alt="" /></p>
<p>To make a decision, you could see what the 3 nearest neighbours were, and which ever has the majority vote will classify the mystery fruit.</p>
<p>This can be generalised to any number of variables (i.e. features), in the above example there was only size and colour. So figuring out the distance between two points would be done by:</p>
<p><math xmlns="http://www.w3.org/1998/Math/MathML"><mstyle displaystyle="true"><mi>d</mi><mo>=</mo><msqrt><mrow><msup><mrow><mo>(</mo><msub><mi>x</mi><mn>1</mn></msub><mo>-</mo><msub><mi>x</mi><mn>2</mn></msub><mo>)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><msup><mrow><mo>(</mo><msub><mi>y</mi><mn>1</mn></msub><mo>-</mo><msub><mi>y</mi><mn>2</mn></msub><mo>)</mo></mrow><mrow><mn>2</mn></mrow></msup></mrow></msqrt></mstyle></math></p>
<p>This expands to as many variables as required.</p>
<p><math xmlns="http://www.w3.org/1998/Math/MathML"><mstyle displaystyle="true"><mi>d</mi><mo>=</mo><msqrt><mrow><msup><mrow><mo>(</mo><msub><mi>x</mi><mn>1</mn></msub><mo>-</mo><msub><mi>x</mi><mn>2</mn></msub><mo>)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><msup><mrow><mo>(</mo><msub><mi>y</mi><mn>1</mn></msub><mo>-</mo><msub><mi>y</mi><mn>2</mn></msub><mo>)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><msup><mrow><mo>(</mo><msub><mi>z</mi><mn>1</mn></msub><mo>-</mo><msub><mi>z</mi><mn>2</mn></msub><mo>)</mo></mrow><mrow><mn>2</mn></mrow></msup><mo>+</mo><mo>...</mo><mi>e</mi><mi>t</mi><mi>c</mi></mrow></msqrt></mstyle></math></p>
<p>N.B. There were two more sections to this chapter <em>Regression</em> and <em>Intro to ML</em> which were very basic and not worth summarising.</p>
<h1 id="11.-where-to-go-next">11. Where To Go Next <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#11.-where-to-go-next">#</a></h1>
<p>This chapter does a very basic introduction to Binary Search Trees, Inverted Indexes, Fourier Transform, Parallel Algorithms, Map Reduce, Bloom Filters and a few others. They are summaries themselves.</p>
<h2 id="other-references">Other references <a class="direct-link" href="https://lukesingham.com/grokking-algorithms/#other-references">#</a></h2>
<ul>
<li><a href="https://github.com/trekhleb/javascript-algorithms">js Algorithms</a></li>
<li><a href="https://www.youtube.com/watch?v=oo_sb4luiPo">Animation of Algs from the book publisher</a></li>
</ul>
The Pragmatic Programmer - Review2020-12-28T00:00:00Zhttps://lukesingham.com/pragmatic-programmer-review/<p>đ Book: <a href="https://www.goodreads.com/book/show/4099.The_Pragmatic_Programmer">The Pragmatic Programmer</a></p>
<p>âď¸ Rating: 4/5 Sagely advice that is written in an engaging and highly consumable form.</p>
<p><img src="https://lukesingham.com/content/images/2020/pragmatic-programmer-frontcover.png#thumbnail" alt="" /></p>
<p>The Pragmatic Programmer is a very good book full of practical advice. It's written with little <em>Tip</em> boxes which are the tl:dr; of a particular section. Plus, there's a healthy sprinkling of Dad jokes throughout the book đ.</p>
<p>Unlike, a <a href="https://lukesingham.com/mythical-man-month-review/">Mythical Man Month</a>, I would recommend this to seasoned professionals as well as newbies. Though this also has a lot of content for the seasoned professional, it is much easier to parse. Instead of having long anecdotes about long since dead technology to gleam the lesson, this book is to the point, with helpful tip boxes, exercises and the examples are mostly relevant to today.</p>
<h2 id="%E2%9D%97%EF%B8%8F-some-criticisms-of-the-book%3A">âď¸ Some criticisms of the book: <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%E2%9D%97%EF%B8%8F-some-criticisms-of-the-book%3A">#</a></h2>
<h4 id="tip-17%3A-program-close-to-the-problem-domain">Tip 17: Program close to the problem domain <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#tip-17%3A-program-close-to-the-problem-domain">#</a></h4>
<p>Whilst I agree with the principle, the accompanying pages advocate for the creation of <em>mini languages</em> in order to get closer to the domain. The project manager senses in me (I am not a project manager) can see this quickly becoming a rabbit hole distraction where developers end up on an academic pursuit of these mini languages, the necessary language parsers, documentation etc. The suggestion to create languages for all makes me cringe,</p>
<blockquote>
<p>There's the end user, who understands the business rules and the requried outputs. There are also secondary users: operations staff, configuration and test managers, support and maintenance programmers, and future generations of developers. Each of these users has their own problem domain and you can generate mini-environments and languages for all of them.</p>
</blockquote>
<p>I've not observed anyone creating a mini-language during a project. What I think is more practical and occurs in real-world projects is creating packages of functions whose scope is limited to a particular domain. Such that the ontological representation, the digital recreation of real world concepts happens with a set of classes and/or functions that map to the domain.</p>
<h4 id="chp-3---the-basic-tools">Chp 3 - The Basic Tools <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#chp-3---the-basic-tools">#</a></h4>
<p>This chapter discusses tools that a pragmatic programmer should have in their tool kit. Inevitably the technological tools have changed in the intervening years. Well, eMacs and Vi are still around, but the recommended plugins etc. are out of date. If you don't have a good toolset already, I'd recommend going through MIT's <a href="https://missing.csail.mit.edu/">The Missing Semester of Your CS Education.</a></p>
<h2 id="%E2%9C%85-major-(useful)-points-of-the-book">â Major (Useful) Points of the Book <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%E2%9C%85-major-(useful)-points-of-the-book">#</a></h2>
<h3 id="%E2%9A%96%EF%B8%8F-involve-your-users-in-the-trade-off">âď¸ Involve Your Users in the Trade-Off <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%E2%9A%96%EF%B8%8F-involve-your-users-in-the-trade-off">#</a></h3>
<p>It's quite a common occurrence for technical people when asked by a stakeholder to feel anxious or pressured to provide an estimate of when something can be done on the spot. For a particular data request, this might mean involving the stakeholder in the tradeoff between how accurate vs how quick the deliverable's turnaround is.</p>
<h3 id="%F0%9F%8D%B8-dry">đ¸ DRY <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%8D%B8-dry">#</a></h3>
<p>Don't repeat yourself (DRY) is a commonly held maxim of programming. The Pragramatic Programmer exposits the common causes for duplication. This can arise from multiple places, project requirements, multiple target platforms, multiple programming languages being used in one system, laziness etc. A programmer may have to duplicate the schema of a database table to represent it as a class in their program. Documentation in code as comments and as a separate accompanying document. But these can be duplicative of each other.</p>
<blockquote>
<p>good code has lots of comments</p>
</blockquote>
<p>âď¸ A slight digression from DRY, but this go-to book for the industry is in favour of lots of comments in code. I've come across a few developers who like to argue the code <em>is</em> documentation. But code will never tell you <em>why</em> the code has been written this way, or the <em>trade offs</em> considered in choosing a particular approach. I've never heard anyone complain <em>'there are just too many comments in this codebase'</em>.</p>
<p>With all rules there are <a href="https://orbifold.xyz/dry-trade-off.html">exceptions and trade-offs with DRY</a>.</p>
<h3 id="%F0%9F%93%90-orthogonality">đ Orthogonality <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%93%90-orthogonality">#</a></h3>
<blockquote>
<p>Tip 13: Eliminate effects between unrelated things</p>
</blockquote>
<p>I know orthogonality as <em>modularity</em> or <em>decoupling</em>. A great example of modularity is the way unix command line was designed with lots of modular tools that you can chain together using <code>|</code> pipes.</p>
<blockquote>
<p>There are two main benefits of orthogonality increased productivity and reduced risk.</p>
</blockquote>
<h3 id="%F0%9F%8E%AF-tracer-bullets">đŻ Tracer Bullets <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%8E%AF-tracer-bullets">#</a></h3>
<p>Better known these days as 'agile development', 'MVP' and 'iteration'. <em>Tracer Bullets</em> are discussed over several pages. Most people could probably skip this as the doctrine of Agile has mostly replaced Waterfall.</p>
<h3 id="%F0%9F%A4%94-estimates">đ¤ Estimates <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%A4%94-estimates">#</a></h3>
<blockquote>
<p>Tip 18: Estimate to avoid surprises</p>
</blockquote>
<p><em>Involve your users</em> and ask if they need high accuracy or a ballpark figure for the estimate. First understand the problem space, this will involve working with your user to understand not just what they are asking, but what their <em>actual problem</em> is. Then whiteboard what the system is you will build to solve it. If this is large enough a system, then estimate how long each component could take to build. Keep track of this estimate so you can track the accuracy of your estimates. As you go through each project iteration (sprint) update the estimate based on any change in delivery schedule that may now be apparent.</p>
<p>A fun fact from a footnote:</p>
<blockquote>
<p>What is the value of Ď? "3"... if you are a legislator. In 1987, Indiana State Legislature House Bill No. 246 attempted to decree that henceforth Ď should have the value of "3".</p>
</blockquote>
<p>Lastly, when asked for an estimate <em>you say, "I'll get back to you"</em>.</p>
<h3 id="%F0%9F%90%9B-debugging">đ Debugging <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%90%9B-debugging">#</a></h3>
<p>This is a good chapter and great if you're new to tech. <em>The first rule of debugging</em>:</p>
<blockquote>
<p>Tip 25: Don't Panic</p>
</blockquote>
<p>I've done this, and seen others do it. But it really is counterproductive. Take a deep breathe đ§ââď¸, and approach it with the Zen of a master. This will help avoid the myopia of jumping on fixes for the symptoms that you immediately see rather than taking a step back to assess the bigger picture here. This is something I usually call, <em>developing hypotheses</em> of the root causes or, what are the possible <em>solution paths</em> we could go down - it's best to get a range of options before jumping down one that could prove fruitless. Instead, rank the hypotheses in order of most likely, then proceed with the highest ranked hypothesis.</p>
<h4 id="various-debugging-tips">Various debugging tips <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#various-debugging-tips">#</a></h4>
<ul>
<li><em>Reproduce the bug</em> on your machine or a dev environment.</li>
<li>Visualise your data. Log it to a console, use code breaks.</li>
<li>đĽ <a href="https://en.wikipedia.org/wiki/Rubber_duck_debugging">Rubber Duck</a> with someone else - if they have the time, doing this at the outset can help develop more hypotheses as to what is causing the problem and get a better assessment on which is most likely.</li>
<li>If there's not an obvious place to start, apply the principle of binary search.</li>
<li><em>Tip 27: Don't assume it, prove it</em>, this can help avoid putting fixes in place that aren't really fixes. This can happen because your dev environment is different to prod. Or you accidentally were on the wrong branch of code.</li>
</ul>
<h3 id="%F0%9F%93%83-design-by-contract">đ Design by Contract <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%93%83-design-by-contract">#</a></h3>
<p>The book talks about Java Object Oriented examples. However the principle applies to all code. Does your code return the expected data with the expected types? Does the logic do what it promises to do? Include assertions, tests and document assumptions. An example of the principle applied to a data pipeline where each model is some Spark or SQL code <code>A -> B -> C</code>. The final select of model <code>A</code> uses <code>SELECT *</code> or <code>SELECT *EXCEPT(x,y,z)</code> this is not enforcing model <code>A</code>'s side of the contract. When model <code>B</code> consumes from model <code>A</code>, if similar shorthand <code>SELECT *</code> is applied, you can end up with a minor code fix that introduces a new column, changing the schema of tables flowing right through this simple pipeline all the way to model <code>C</code>. This can be easily avoided by explicitly naming the columns to be returned such that downstream models won't accidentally have their schemas changed by an upstream model. Even better, enforce the contract when you select columns from an upstream data model.</p>
<blockquote>
<p>Tip 33: If it can't happen, use assertions[or tests] to ensure that it won't</p>
</blockquote>
<blockquote>
<p>Tip 36: Minimise coupling between modules</p>
</blockquote>
<h3 id="%F0%9F%8E%9B-design-for-concurrency">đ Design for Concurrency <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%8E%9B-design-for-concurrency">#</a></h3>
<p>The book uses the example of making a cocktail and all the tasks that are required from 'open blender' to 'get pink umbrellas'. If you analyse all of these tasks, some are dependent on others to complete. But many tasks are able to happen in parallel to another task. Diagrams can help design for concurrency.</p>
<p><img src="https://lukesingham.com/content/images/2020/design-for-concurrency.svg" alt="" /></p>
<h3 id="%F0%9F%A7%B9-refactoring">𧚠Refactoring <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%A7%B9-refactoring">#</a></h3>
<p>Code needs to evolve. It is not static. And unfortunately many people perceive software as <em>done</em> once it is delivered. Software is more like gardening - you might start a gardening project with a plan, plant seedlings according to the plan, but later some plants didn't like the soil others have overgrown. You monitor and regularly tend to the garden to keep it at its best.</p>
<h4 id="when-to-refactor%3F">When to refactor? <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#when-to-refactor%3F">#</a></h4>
<p>These are common qualifiers:</p>
<ul>
<li>Duplication</li>
<li>Highly coupled components</li>
<li>Outdated knowledge i.e. does the code still reflect the real world problem</li>
<li>Performance</li>
</ul>
<blockquote>
<p>Tip 47: Refactor Early, Refactor Often</p>
</blockquote>
<p>If time pressure is used as an excuse from management, explain technical debt is like a cancer that grows and is riskier and more time consuming to remove the longer you leave it.</p>
<h4 id="how-to-refactor%3F">How to refactor? <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#how-to-refactor%3F">#</a></h4>
<ul>
<li>Don't try to refactor and add functionality. Leave TODOs if you must.</li>
<li>Make sure you have good tests before starting to refactor, otherwise you may not know when you have broken something.</li>
<li>Take short deliberate changes and test them. Rather than large sweeping changes all at once.</li>
</ul>
<h3 id="%F0%9F%93%9A-requirements">đ Requirements <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%93%9A-requirements">#</a></h3>
<p>To understand what the user needs:</p>
<blockquote>
<p>Tip 52: Work with a user to think like a user</p>
</blockquote>
<p>If you have the problem of too many requirements, then point out how much each suggested requirement is going to increase project delivery by.</p>
<h3 id="%F0%9F%A7%AA-testing">𧪠Testing <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#%F0%9F%A7%AA-testing">#</a></h3>
<blockquote>
<p>Test Early. Test Often. Test Automatically.</p>
</blockquote>
<ul>
<li><strong>Unit Tests</strong> - is code that exercise a particular piece of code in isolation.</li>
<li><strong>Integration Tests</strong> - shows that the major subsystems that make up the project work and play well with each other.</li>
<li><strong>User Testing</strong> - does it meet the users expectations?</li>
<li><strong>Performance Testing / Stress Testing</strong> - does it scale? Is it going to crash as soon as 100 users are using it at the same time?</li>
<li><strong>Usability Testing</strong> - This is about UX. It might pass <em>user testing</em> but does it feel like the extension of the users hand? Does it feel more like an Apple product or another product that technically does the same but just feels awkward.</li>
</ul>
<h2 id="notes">Notes <a class="direct-link" href="https://lukesingham.com/pragmatic-programmer-review/#notes">#</a></h2>
<ul>
<li>I'd recommend the entirety of chapter 6 <em>While you are coding</em> - particularly the 'How to program deliberately section'</li>
<li>One of the simplest explainations of O notation is in this book.</li>
<li>There's a great section in Chp 8 Pragmatic Projects on team organisation advocating for teams to be organised around functionality which also reflects the decoupled principle.</li>
</ul>
Go Tour Summary2020-12-07T00:00:00Zhttps://lukesingham.com/go-tour-summary/<p>These are my summary notes of <a href="https://tour.golang.org/">A Tour of Go</a> - which is meant for people who are familiar with programming to have a quick tour of the Go language. I'm most familiar with dynamic languages (<code>R</code>, <code>Python</code>, <code>JavaScript</code>) so there are some go-lang features that naturally feel new by the nature of Go being a statically typed language. A subsequent post will expand upon some of the more advanced features like pointers, concurrency and polymorphism.</p>
<h2 id="named-return-values">Named Return Values <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#named-return-values">#</a></h2>
<p>A return statement without arguments returns the named return values. This is known as a "naked" return.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">split</span><span class="token punctuation">(</span>mynumber <span class="token builtin">int</span><span class="token punctuation">)</span> <span class="token punctuation">(</span>x<span class="token punctuation">,</span> y <span class="token builtin">int</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> x <span class="token operator">=</span> mynumber <span class="token operator">*</span> <span class="token number">4</span><br /> y <span class="token operator">=</span> mynumber <span class="token operator">-</span> x<br /> <span class="token keyword">return</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token number">17</span><span class="token punctuation">)</span><span class="token punctuation">)</span>myt<br /><span class="token punctuation">}</span><br /><span class="token comment">// This will print x and y:</span><br /><span class="token comment">// 68 -51</span></code></pre>
<h2 id="variables">Variables <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#variables">#</a></h2>
<p>The <code>var</code> statement declares a list of variables; as in function argument lists, the type is last.</p>
<h3 id="variables-with-initializers">Variables with initializers <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#variables-with-initializers">#</a></h3>
<p>A <code>var</code> declaration can include initializers, one per variable.</p>
<p>If an initializer is present, the type can be omitted; the variable will take the type of the initializer.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">var</span> a <span class="token builtin">int</span> <span class="token comment">// will have a value of 0 as a placeholder</span><br /><span class="token keyword">var</span> f <span class="token builtin">float64</span> <span class="token comment">// 0</span><br /><span class="token keyword">var</span> b <span class="token builtin">bool</span> <span class="token comment">// false</span><br /><span class="token keyword">var</span> s <span class="token builtin">string</span> <span class="token comment">// ""</span><br /><br /><span class="token keyword">var</span> i<span class="token punctuation">,</span> j <span class="token builtin">int</span> <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span> <span class="token comment">// explicit</span><br /><br /><span class="token keyword">var</span> k<span class="token punctuation">,</span> l <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span> <span class="token comment">// implicit</span><br /></code></pre>
<h2 id="%3A%3D"><code>:=</code> <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#%3A%3D">#</a></h2>
<p>Inside a function, the <code>:=</code> short assignment statement can be used in place of a var declaration with implicit type.</p>
<p>Outside a function, every statement begins with a keyword (var, func, and so on) and so the <code>:=</code> construct is not available.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token comment">// := only within fn</span><br /> k <span class="token operator">:=</span> <span class="token number">3</span><br /><span class="token punctuation">}</span></code></pre>
<h2 id="types-in-go">Types in go <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#types-in-go">#</a></h2>
<pre class="language-go"><code class="language-go"><span class="token builtin">bool</span><br /><br /><span class="token builtin">string</span><br /><br /><span class="token builtin">int</span> <span class="token builtin">int8</span> <span class="token builtin">int16</span> <span class="token builtin">int32</span> <span class="token builtin">int64</span><br /><span class="token builtin">uint</span> <span class="token builtin">uint8</span> <span class="token builtin">uint16</span> <span class="token builtin">uint32</span> <span class="token builtin">uint64</span> <span class="token builtin">uintptr</span><br /><br /><span class="token builtin">byte</span> <span class="token comment">// alias for uint8</span><br /><br /><span class="token builtin">rune</span> <span class="token comment">// alias for int32</span><br /> <span class="token comment">// represents a Unicode code point</span><br /><br /><span class="token builtin">float32</span> <span class="token builtin">float64</span><br /><br /><span class="token builtin">complex64</span> <span class="token builtin">complex128</span></code></pre>
<h2 id="type-conversions">Type conversions <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#type-conversions">#</a></h2>
<pre class="language-go"><code class="language-go"><span class="token keyword">var</span> i <span class="token builtin">int</span> <span class="token operator">=</span> <span class="token number">42</span><br /><span class="token keyword">var</span> f <span class="token builtin">float64</span> <span class="token operator">=</span> <span class="token function">float64</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><br /><br /><span class="token comment">//or</span><br />i <span class="token operator">:=</span> <span class="token number">42</span><br />f <span class="token operator">:=</span> <span class="token function">float64</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span></code></pre>
<h2 id="constants">Constants <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#constants">#</a></h2>
<p>Constants are declared like variables, but with the const keyword.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">const</span> Pi <span class="token operator">=</span> <span class="token number">3.14</span></code></pre>
<h2 id="for">For <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#for">#</a></h2>
<p>Go has only one looping construct, the <code>for</code> loop.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> sum <span class="token operator">:=</span> <span class="token number">0</span><br /> <span class="token keyword">for</span> i <span class="token operator">:=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator"><</span> <span class="token number">10</span><span class="token punctuation">;</span> i<span class="token operator">++</span> <span class="token punctuation">{</span><br /> sum <span class="token operator">+=</span> i<br /> <span class="token punctuation">}</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>sum<span class="token punctuation">)</span><br /><span class="token punctuation">}</span></code></pre>
<p>The init and post statements are optional.</p>
<pre class="language-go"><code class="language-go">sum <span class="token operator">:=</span> <span class="token number">1</span><br /><span class="token keyword">for</span> <span class="token punctuation">;</span> sum <span class="token operator"><</span> <span class="token number">1000</span><span class="token punctuation">;</span> <span class="token punctuation">{</span><br /> sum <span class="token operator">+=</span> sum<br /><span class="token punctuation">}</span></code></pre>
<h2 id="if">If <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#if">#</a></h2>
<p>Go's <code>if</code> statements are like its <code>for</code> loops; the expression need not be surrounded by parentheses ( ) but the braces { } are required.</p>
<h3 id="if-with-a-short-statement">If with a short statement <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#if-with-a-short-statement">#</a></h3>
<p>Like the <code>for</code> loop, the <code>if</code> statement can start with a short statement to execute before the condition.</p>
<p>Variables declared by the statement are only in scope until the end of the if.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">pow</span><span class="token punctuation">(</span>x<span class="token punctuation">,</span> n<span class="token punctuation">,</span> lim <span class="token builtin">float64</span><span class="token punctuation">)</span> <span class="token builtin">float64</span> <span class="token punctuation">{</span><br /> <span class="token comment">// v doesn't exist outside of the if statement scope</span><br /> <span class="token keyword">if</span> v <span class="token operator">:=</span> math<span class="token punctuation">.</span><span class="token function">Pow</span><span class="token punctuation">(</span>x<span class="token punctuation">,</span> n<span class="token punctuation">)</span><span class="token punctuation">;</span> v <span class="token operator"><</span> lim <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> v<br /> <span class="token punctuation">}</span><br /> <span class="token keyword">return</span> lim<br /><span class="token punctuation">}</span></code></pre>
<h2 id="switch">Switch <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#switch">#</a></h2>
<p>A switch statement is a shorter way to write a sequence of <code>if</code> <code>else</code> statements. It runs the first case whose value is equal to the condition expression.</p>
<p>Go's switch is like the one in <code>C</code>, <code>C++</code>, <code>Java</code>, <code>JavaScript</code> and <code>PHP</code>, except that Go only runs the selected case, not all the cases that follow. In effect, the break statement that is needed at the end of each case in those languages is provided automatically in Go. Another important difference is that Go's switch cases need not be constants, and the values involved need not be integers.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">switch</span> <span class="token number">1</span> <span class="token punctuation">{</span><br /> <span class="token keyword">case</span> <span class="token number">1</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"matches"</span><span class="token punctuation">)</span><br /> <span class="token keyword">case</span> <span class="token number">1</span> <span class="token operator">+</span> <span class="token number">1</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"doesnt match"</span><span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// matches</span></code></pre>
<h3 id="switch-with-no-condition">Switch with no condition <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#switch-with-no-condition">#</a></h3>
<p>Switch without a condition is the same as switch true. This construct can be a clean way to write long if-then-else chains.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> t <span class="token operator">:=</span> time<span class="token punctuation">.</span><span class="token function">Now</span><span class="token punctuation">(</span><span class="token punctuation">)</span><br /> <span class="token keyword">switch</span> <span class="token punctuation">{</span><br /> <span class="token keyword">case</span> t<span class="token punctuation">.</span><span class="token function">Hour</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator"><</span> <span class="token number">12</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"Good morning!"</span><span class="token punctuation">)</span><br /> <span class="token keyword">case</span> t<span class="token punctuation">.</span><span class="token function">Hour</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token operator"><</span> <span class="token number">17</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"Good afternoon."</span><span class="token punctuation">)</span><br /> <span class="token keyword">default</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"Good evening."</span><span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span></code></pre>
<h2 id="defer">Defer <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#defer">#</a></h2>
<p>Defer evaluates but doesn't execute</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"hello1"</span><span class="token punctuation">)</span><br /> <span class="token keyword">defer</span> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"world"</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"hello2"</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// hello1</span><br /><span class="token comment">// hello2</span><br /><span class="token comment">// world</span></code></pre>
<h3 id="stacking-defers">Stacking defers <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#stacking-defers">#</a></h3>
<p>Deferred function calls are pushed onto a stack. When a function returns, its deferred calls are executed in last-in-first-out order.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"counting"</span><span class="token punctuation">)</span><br /> <span class="token keyword">for</span> i <span class="token operator">:=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator"><</span> <span class="token number">4</span><span class="token punctuation">;</span> i<span class="token operator">++</span> <span class="token punctuation">{</span><br /> <span class="token keyword">defer</span> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"done"</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// counting</span><br /><span class="token comment">// done</span><br /><span class="token comment">// 3</span><br /><span class="token comment">// 2</span><br /><span class="token comment">// 1</span><br /><span class="token comment">// 0</span></code></pre>
<h2 id="pointers">Pointers <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#pointers">#</a></h2>
<p>Pointers are useful if you have read something into memory and you don't want to duplicate it. So you can just point to the memory address. Useful for optimising your code.</p>
<p>So if you want to change a variable stored in memory, you can 'mutate' it by using pointers.</p>
<p>The type <code>*T</code> is a pointer to a <code>T</code> value. Its zero value is nil.</p>
<p><code>var p *int</code></p>
<p>The <code>&</code> operator generates a pointer to its operand.<br />
n.b. In 3+6, 3 and 6 are operands. And the + is the operator.</p>
<pre class="language-go"><code class="language-go">i <span class="token operator">:=</span> <span class="token number">42</span><br />p <span class="token operator">:=</span> <span class="token operator">&</span>i <span class="token comment">// a pointer p=0xc00002c008</span><br />z <span class="token operator">:=</span> p <span class="token comment">// 'dereferences the pointer. z=42</span><br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"i"</span><span class="token punctuation">,</span>i<span class="token punctuation">,</span><span class="token string">"p"</span><span class="token punctuation">,</span>p<span class="token punctuation">,</span> <span class="token string">"z"</span><span class="token punctuation">,</span> z<span class="token punctuation">)</span><br /><span class="token comment">// i 42 p 0xc00002c008 z 42</span></code></pre>
<p>The <code>*</code> operator denotes the pointer's underlying value.</p>
<pre class="language-go"><code class="language-go">fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token operator">*</span>p<span class="token punctuation">)</span> <span class="token comment">// read i through the pointer p</span><br /><span class="token comment">// 42</span><br /><span class="token operator">*</span>p <span class="token operator">=</span> <span class="token number">21</span> <span class="token comment">// set i through the pointer p</span><br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><br /><span class="token comment">// 21</span></code></pre>
<p>This is known as "dereferencing" or "indirecting".</p>
<h2 id="structs">Structs <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#structs">#</a></h2>
<p>A struct is a collection of fields.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> Vertex <span class="token keyword">struct</span> <span class="token punctuation">{</span><br /> X <span class="token builtin">int</span><br /> Y <span class="token builtin">int</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>Vertex<span class="token punctuation">{</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">}</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// {1 2}</span></code></pre>
<h2 id="pointers-to-structs">Pointers to structs <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#pointers-to-structs">#</a></h2>
<p>Struct fields can be accessed through a struct pointer.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> Vertex <span class="token keyword">struct</span> <span class="token punctuation">{</span><br /> X <span class="token builtin">int</span><br /> Y <span class="token builtin">int</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v <span class="token operator">:=</span> Vertex<span class="token punctuation">{</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">}</span><br /> p <span class="token operator">:=</span> <span class="token operator">&</span>v<br /> p<span class="token punctuation">.</span>X <span class="token operator">=</span> <span class="token number">10</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// {10 2}</span></code></pre>
<h2 id="struct-literals">Struct Literals <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#struct-literals">#</a></h2>
<p>A struct literal denotes a newly allocated struct value by listing the values of its fields.</p>
<p>You can list just a subset of fields by using the Name: syntax. The order of named fields is irrelevant.</p>
<p>The special prefix <code>&</code> returns a pointer to the struct value.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> Vertex <span class="token keyword">struct</span> <span class="token punctuation">{</span><br /> X<span class="token punctuation">,</span> Y <span class="token builtin">int</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">var</span> <span class="token punctuation">(</span><br /> v1 <span class="token operator">=</span> Vertex<span class="token punctuation">{</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">}</span> <span class="token comment">// has type Vertex</span><br /> v2 <span class="token operator">=</span> Vertex<span class="token punctuation">{</span>X<span class="token punctuation">:</span> <span class="token number">1</span><span class="token punctuation">}</span> <span class="token comment">// Y:0 is implicit</span><br /> v3 <span class="token operator">=</span> Vertex<span class="token punctuation">{</span><span class="token punctuation">}</span> <span class="token comment">// X:0 and Y:0</span><br /> p <span class="token operator">=</span> <span class="token operator">&</span>Vertex<span class="token punctuation">{</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">}</span> <span class="token comment">// has type *Vertex</span><br /><span class="token punctuation">)</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>v1<span class="token punctuation">,</span> p<span class="token punctuation">,</span> v2<span class="token punctuation">,</span> v3<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// {1 2} &{1 2} {1 0} {0 0}</span></code></pre>
<h2 id="arrays">Arrays <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#arrays">#</a></h2>
<p>The type <code>[n]T</code> is an array of <code>n</code> values of type <code>T</code>. The expression<br />
<code>var a [10]int</code><br />
declares a variable <code>a</code> as an array of ten integers. <strong>An array's length is part of its type, so arrays cannot be resized.</strong></p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">var</span> a <span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span><span class="token builtin">string</span><br /> a<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">"Hello"</span><br /> a<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">"World"</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>a<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">,</span> a<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>a<span class="token punctuation">)</span><br /><br /> primes <span class="token operator">:=</span> <span class="token punctuation">[</span><span class="token number">6</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">{</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">11</span><span class="token punctuation">,</span> <span class="token number">13</span><span class="token punctuation">}</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>primes<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// Hello World</span><br /><span class="token comment">// [Hello World]</span><br /><span class="token comment">// [2 3 5 7 11 13]</span></code></pre>
<h2 id="slices">Slices <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#slices">#</a></h2>
<p>An array has a fixed size. A slice, on the other hand, is a dynamically-sized, flexible view into the elements of an array</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> primes <span class="token operator">:=</span> <span class="token punctuation">[</span><span class="token number">6</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">{</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">11</span><span class="token punctuation">,</span> <span class="token number">13</span><span class="token punctuation">}</span><br /><br /> <span class="token keyword">var</span> sliced <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span> <span class="token operator">=</span> primes<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token number">4</span><span class="token punctuation">]</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>sliced<span class="token punctuation">)</span><br /><span class="token punctuation">}</span></code></pre>
<p>A slice does not store any data, it just describes a section of an underlying array. Changing the elements of a slice modifies the corresponding elements of its underlying array. Other slices that share the same underlying array will see those changes.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> names <span class="token operator">:=</span> <span class="token punctuation">[</span><span class="token number">4</span><span class="token punctuation">]</span><span class="token builtin">string</span><span class="token punctuation">{</span><br /> <span class="token string">"John"</span><span class="token punctuation">,</span><br /> <span class="token string">"Paul"</span><span class="token punctuation">,</span><br /> <span class="token string">"George"</span><span class="token punctuation">,</span><br /> <span class="token string">"Ringo"</span><span class="token punctuation">,</span><br /> <span class="token punctuation">}</span><br /><br /> a <span class="token operator">:=</span> names<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">:</span><span class="token number">2</span><span class="token punctuation">]</span><br /> b <span class="token operator">:=</span> names<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">:</span><span class="token number">3</span><span class="token punctuation">]</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><br /><br /> b<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">"XXX"</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>a<span class="token punctuation">,</span> b<span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>names<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// [John Paul] [Paul George]</span><br /><span class="token comment">// [John XXX] [XXX George]</span><br /><span class="token comment">// [John XXX George Ringo]</span></code></pre>
<h2 id="slice-literals">Slice literals <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#slice-literals">#</a></h2>
<p>A slice literal is like an array literal without the length.<br />
This is an array literal:<br />
<code>[3]bool{true, true, false}</code></p>
<p>And this creates the same array as above, then builds a slice that references it:<br />
<code>[]bool{true, true, false}</code></p>
<pre class="language-go"><code class="language-go"> q <span class="token operator">:=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">{</span><span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">,</span> <span class="token number">7</span><span class="token punctuation">,</span> <span class="token number">11</span><span class="token punctuation">,</span> <span class="token number">13</span><span class="token punctuation">}</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>q<span class="token punctuation">)</span><br /> <span class="token comment">// [2 3 5 7 11 13]</span></code></pre>
<h2 id="slice-length-and-capacity">Slice length and capacity <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#slice-length-and-capacity">#</a></h2>
<p>The length of a slice is the number of elements it contains.<br />
The capacity of a slice is the number of elements in the underlying array, counting from the first element in the slice.<br />
The length and capacity of a slice <code>s</code> can be obtained using the expressions <code>len(s)</code> and <code>cap(s)</code>.</p>
<h2 id="nil-slices">Nil slices <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#nil-slices">#</a></h2>
<p>The zero value of a slice is <code>nil</code>.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">var</span> s <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>s<span class="token punctuation">,</span> <span class="token function">len</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function">cap</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">)</span><br /> <span class="token keyword">if</span> s <span class="token operator">==</span> <span class="token boolean">nil</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"nil!"</span><span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// [] 0 0</span><br /><span class="token comment">// nil!</span></code></pre>
<h2 id="creating-a-slice-with-make">Creating a slice with <code>make</code> <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#creating-a-slice-with-make">#</a></h2>
<p>Slices can be created with the built-in <code>make</code> function; this is how you create dynamically-sized arrays. The make function allocates a zeroed array and returns a slice that refers to that array:</p>
<pre class="language-go"><code class="language-go">a <span class="token operator">:=</span> <span class="token function">make</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">)</span> <span class="token comment">// len(a)=5</span><br /><br /><span class="token comment">// To specify a capacity, pass a third argument to make:</span><br /><br />b <span class="token operator">:=</span> <span class="token function">make</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">,</span> <span class="token number">5</span><span class="token punctuation">)</span> <span class="token comment">// len(b)=0, cap(b)=5</span></code></pre>
<h2 id="appending-to-a-slice">Appending to a slice <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#appending-to-a-slice">#</a></h2>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">var</span> s <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><br /> <span class="token function">printSlice</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><br /><br /> <span class="token comment">// append works on nil slices.</span><br /> s <span class="token operator">=</span> <span class="token function">append</span><span class="token punctuation">(</span>s<span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><br /> <span class="token function">printSlice</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><br /><br /> <span class="token comment">// The slice grows as needed.</span><br /> s <span class="token operator">=</span> <span class="token function">append</span><span class="token punctuation">(</span>s<span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><br /> <span class="token function">printSlice</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><br /><br /> <span class="token comment">// We can add more than one element at a time.</span><br /> s <span class="token operator">=</span> <span class="token function">append</span><span class="token punctuation">(</span>s<span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><br /> <span class="token function">printSlice</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">printSlice</span><span class="token punctuation">(</span>s <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Printf</span><span class="token punctuation">(</span><span class="token string">"len=%d cap=%d %v\n"</span><span class="token punctuation">,</span> <span class="token function">len</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function">cap</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><span class="token punctuation">,</span> s<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// len=0 cap=0 []</span><br /><span class="token comment">// len=1 cap=1 [0]</span><br /><span class="token comment">// len=2 cap=2 [0 1]</span><br /><span class="token comment">// len=5 cap=6 [0 1 2 3 4]</span></code></pre>
<h2 id="range">Range <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#range">#</a></h2>
<p>The <code>range</code> form of the <code>for</code> loop iterates over a slice or map.<br />
When ranging over a slice, two values are returned for each iteration. The first is the index, and the second is a copy of the element at that index.</p>
<p><strong>n.b.</strong> <a href="https://golang.org/pkg/fmt/#hdr-Printing">verbs in string templates.</a></p>
<pre class="language-go"><code class="language-go"><span class="token keyword">var</span> pow <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token builtin">int</span><span class="token punctuation">{</span><span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">2</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">8</span><span class="token punctuation">,</span> <span class="token number">16</span><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">for</span> i<span class="token punctuation">,</span> v <span class="token operator">:=</span> <span class="token keyword">range</span> pow <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Printf</span><span class="token punctuation">(</span><span class="token string">"2**%d = %d\n"</span><span class="token punctuation">,</span> i<span class="token punctuation">,</span> v<span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// 2**0 = 1</span><br /><span class="token comment">// 2**1 = 2</span><br /><span class="token comment">// 2**2 = 4</span><br /><span class="token comment">// 2**3 = 8</span><br /><span class="token comment">// 2**4 = 16</span></code></pre>
<p>You can skip the index or value by assigning to _.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">for</span> i<span class="token punctuation">,</span> <span class="token boolean">_</span> <span class="token operator">:=</span> <span class="token keyword">range</span> pow<br /><span class="token keyword">for</span> <span class="token boolean">_</span><span class="token punctuation">,</span> value <span class="token operator">:=</span> <span class="token keyword">range</span> pow</code></pre>
<h2 id="map-literals">Map literals <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#map-literals">#</a></h2>
<p>Map literals are like struct literals, but the keys are required.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> Vertex <span class="token keyword">struct</span> <span class="token punctuation">{</span><br /> Lat<span class="token punctuation">,</span> Long <span class="token builtin">float64</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">var</span> m <span class="token operator">=</span> <span class="token keyword">map</span><span class="token punctuation">[</span><span class="token builtin">string</span><span class="token punctuation">]</span>Vertex<span class="token punctuation">{</span><br /> <span class="token string">"Bell Labs"</span><span class="token punctuation">:</span> Vertex<span class="token punctuation">{</span><br /> <span class="token number">40.68433</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">74.39967</span><span class="token punctuation">,</span><br /> <span class="token punctuation">}</span><span class="token punctuation">,</span><br /> <span class="token string">"Google"</span><span class="token punctuation">:</span> Vertex<span class="token punctuation">{</span><br /> <span class="token number">37.42202</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">122.08408</span><span class="token punctuation">,</span><br /> <span class="token punctuation">}</span><span class="token punctuation">,</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">// m can also be expressed</span><br /><span class="token keyword">var</span> m <span class="token operator">=</span> <span class="token keyword">map</span><span class="token punctuation">[</span><span class="token builtin">string</span><span class="token punctuation">]</span>Vertex<span class="token punctuation">{</span><br /> <span class="token string">"Bell Labs"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span><span class="token number">40.68433</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">74.39967</span><span class="token punctuation">}</span><span class="token punctuation">,</span><br /> <span class="token string">"Google"</span><span class="token punctuation">:</span> <span class="token punctuation">{</span><span class="token number">37.42202</span><span class="token punctuation">,</span> <span class="token operator">-</span><span class="token number">122.08408</span><span class="token punctuation">}</span><span class="token punctuation">,</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>m<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// map[Bell Labs:{40.68433 -74.39967} Google:{37.42202 -122.08408}]</span></code></pre>
<h2 id="mutating-maps">Mutating Maps <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#mutating-maps">#</a></h2>
<pre class="language-go"><code class="language-go"><span class="token comment">// Insert or update an element in map m:</span><br />m<span class="token punctuation">[</span>key<span class="token punctuation">]</span> <span class="token operator">=</span> elem<br /><br /><span class="token comment">// Retrieve an element:</span><br />elem <span class="token operator">=</span> m<span class="token punctuation">[</span>key<span class="token punctuation">]</span><br /><br /><span class="token comment">// Delete an element:</span><br /><span class="token function">delete</span><span class="token punctuation">(</span>m<span class="token punctuation">,</span> key<span class="token punctuation">)</span><br /><br /><span class="token comment">// Test that a key is present with a two-value assignment:</span><br />elem<span class="token punctuation">,</span> ok <span class="token operator">=</span> m<span class="token punctuation">[</span>key<span class="token punctuation">]</span></code></pre>
<h2 id="function-values">Function values <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#function-values">#</a></h2>
<p>Functions are values too. They can be passed around just like other values. Function values may be used as function arguments and return values.</p>
<h2 id="function-closures">Function closures <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#function-closures">#</a></h2>
<p>Go functions may be closures. A closure is a function value that references variables from outside its body. The function may access and assign to the referenced variables; in this sense the function is "bound" to the variables.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">adder</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token keyword">func</span><span class="token punctuation">(</span><span class="token builtin">int</span><span class="token punctuation">)</span> <span class="token builtin">int</span> <span class="token punctuation">{</span><br /> sum <span class="token operator">:=</span> <span class="token number">0</span><br /> <span class="token keyword">return</span> <span class="token keyword">func</span><span class="token punctuation">(</span>x <span class="token builtin">int</span><span class="token punctuation">)</span> <span class="token builtin">int</span> <span class="token punctuation">{</span><br /> sum <span class="token operator">+=</span> x<br /> <span class="token keyword">return</span> sum<br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> pos<span class="token punctuation">,</span> neg <span class="token operator">:=</span> <span class="token function">adder</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token function">adder</span><span class="token punctuation">(</span><span class="token punctuation">)</span><br /> <span class="token keyword">for</span> i <span class="token operator">:=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator"><</span> <span class="token number">10</span><span class="token punctuation">;</span> i<span class="token operator">++</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><br /> <span class="token function">pos</span><span class="token punctuation">(</span>i<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> <span class="token function">neg</span><span class="token punctuation">(</span><span class="token operator">-</span><span class="token number">2</span><span class="token operator">*</span>i<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> <span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span></code></pre>
<h2 id="methods">Methods <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#methods">#</a></h2>
<p>Go does not have classes. However, you can define methods on types.<br />
A method is a function with a special receiver argument.<br />
The receiver appears in its own argument list between the func keyword and the method name.<br />
In this example, the <code>Abs</code> method has a receiver of type <code>Vertex</code> named <code>v</code>.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> Vertex <span class="token keyword">struct</span> <span class="token punctuation">{</span><br /> X<span class="token punctuation">,</span> Y <span class="token builtin">float64</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">// as a method</span><br /><span class="token keyword">func</span> <span class="token punctuation">(</span>v Vertex<span class="token punctuation">)</span> <span class="token function">Abs</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token builtin">float64</span> <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> math<span class="token punctuation">.</span><span class="token function">Sqrt</span><span class="token punctuation">(</span>v<span class="token punctuation">.</span>X<span class="token operator">*</span>v<span class="token punctuation">.</span>X <span class="token operator">+</span> v<span class="token punctuation">.</span>Y<span class="token operator">*</span>v<span class="token punctuation">.</span>Y<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v <span class="token operator">:=</span> Vertex<span class="token punctuation">{</span><span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">}</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>v<span class="token punctuation">.</span><span class="token function">Abs</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">// as a normal function</span><br /><span class="token keyword">func</span> <span class="token function">Abs</span><span class="token punctuation">(</span>v Vertex<span class="token punctuation">)</span> <span class="token builtin">float64</span> <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> math<span class="token punctuation">.</span><span class="token function">Sqrt</span><span class="token punctuation">(</span>v<span class="token punctuation">.</span>X<span class="token operator">*</span>v<span class="token punctuation">.</span>X <span class="token operator">+</span> v<span class="token punctuation">.</span>Y<span class="token operator">*</span>v<span class="token punctuation">.</span>Y<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v <span class="token operator">:=</span> Vertex<span class="token punctuation">{</span><span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">}</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token function">Abs</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /></code></pre>
<h2 id="pointer-receivers">Pointer receivers <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#pointer-receivers">#</a></h2>
<p>Methods with pointer receivers can modify the value to which the receiver points (as <code>Scale</code> does here đ). Since methods often need to modify their receiver, pointer receivers are more common than value receivers.</p>
<p>With a value receiver, the Scale method operates on a copy of the original Vertex value. (This is the same behavior as for any other function argument.) The Scale method must have a pointer receiver to change the Vertex value declared in the main function.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> Vertex <span class="token keyword">struct</span> <span class="token punctuation">{</span><br /> X<span class="token punctuation">,</span> Y <span class="token builtin">float64</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token punctuation">(</span>v Vertex<span class="token punctuation">)</span> <span class="token function">Abs</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token builtin">float64</span> <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> math<span class="token punctuation">.</span><span class="token function">Sqrt</span><span class="token punctuation">(</span>v<span class="token punctuation">.</span>X<span class="token operator">*</span>v<span class="token punctuation">.</span>X <span class="token operator">+</span> v<span class="token punctuation">.</span>Y<span class="token operator">*</span>v<span class="token punctuation">.</span>Y<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">// method on a pointer</span><br /><span class="token keyword">func</span> <span class="token punctuation">(</span>v <span class="token operator">*</span>Vertex<span class="token punctuation">)</span> <span class="token function">Scale</span><span class="token punctuation">(</span>f <span class="token builtin">float64</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v<span class="token punctuation">.</span>X <span class="token operator">=</span> v<span class="token punctuation">.</span>X <span class="token operator">*</span> f<br /> v<span class="token punctuation">.</span>Y <span class="token operator">=</span> v<span class="token punctuation">.</span>Y <span class="token operator">*</span> f<br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v <span class="token operator">:=</span> Vertex<span class="token punctuation">{</span><span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">}</span><br /> v<span class="token punctuation">.</span><span class="token function">Scale</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// {30 40}</span><br /><br /><span class="token comment">// method on value receiver</span><br /><span class="token keyword">func</span> <span class="token punctuation">(</span>v Vertex<span class="token punctuation">)</span> <span class="token function">Scale</span><span class="token punctuation">(</span>f <span class="token builtin">float64</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v<span class="token punctuation">.</span>X <span class="token operator">=</span> v<span class="token punctuation">.</span>X <span class="token operator">*</span> f<br /> v<span class="token punctuation">.</span>Y <span class="token operator">=</span> v<span class="token punctuation">.</span>Y <span class="token operator">*</span> f<br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> v <span class="token operator">:=</span> Vertex<span class="token punctuation">{</span><span class="token number">3</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">}</span><br /> v<span class="token punctuation">.</span><span class="token function">Scale</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// {3 4}</span><br /><span class="token comment">// the result {3 4} is because the the original values are not changed. A copy is made on execution. As opposed to the pointer receiver, which modifies the original values.</span></code></pre>
<h2 id="methods-and-pointer-indirection-(2)">Methods and pointer indirection (2) <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#methods-and-pointer-indirection-(2)">#</a></h2>
<p>Functions that take a value argument must take a value of that specific type:</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">var</span> v Vertex<br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token function">AbsFunc</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// OK</span><br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token function">AbsFunc</span><span class="token punctuation">(</span><span class="token operator">&</span>v<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// Compile error!</span></code></pre>
<p>Methods with value receivers take either a value or a pointer as the receiver when they are called:</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">var</span> v Vertex<br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>v<span class="token punctuation">.</span><span class="token function">Abs</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// OK</span><br />p <span class="token operator">:=</span> <span class="token operator">&</span>v<br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>p<span class="token punctuation">.</span><span class="token function">Abs</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token comment">// OK</span></code></pre>
<h2 id="interfaces">Interfaces <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#interfaces">#</a></h2>
<p>An interface type is defined as a set of method signatures.<br />
A value of interface type can hold any value that implements those methods.</p>
<p>A type implements an interface by implementing its methods. There is no explicit declaration of intent, no "implements" keyword</p>
<h3 id="interface-values">Interface values <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#interface-values">#</a></h3>
<p>Under the hood, interface values can be thought of as a tuple of a value and a concrete type:<br />
<code>(value, type)</code><br />
An interface value holds a value of a specific underlying concrete type.</p>
<h3 id="interface-values-with-nil-underlying-values">Interface values with <code>nil</code> underlying values <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#interface-values-with-nil-underlying-values">#</a></h3>
<p>If the concrete value inside the interface itself is nil, the method will be called with a <code>nil</code> receiver.<br />
In some languages this would trigger a null pointer exception, but in Go it is common to write methods that gracefully handle being called with a <code>nil</code> receiver.</p>
<h3 id="the-empty-interface">The empty interface <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#the-empty-interface">#</a></h3>
<p>The interface type that specifies zero methods is known as the empty interface:<br />
<code>interface{}</code><br />
An empty interface may hold values of any type. Every type implements at least zero methods. Empty interfaces are used by code that handles values of unknown type. For example, <code>fmt.Print</code> takes any number of arguments of type interface{}.</p>
<h3 id="type-assertions">Type assertions <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#type-assertions">#</a></h3>
<p>A type assertion provides access to an interface value's underlying concrete value.<br />
<code>t := i.(T)</code><br />
This statement asserts that the interface value <code>i</code> holds the concrete type <code>T</code> and assigns the underlying <code>T</code> value to the variable <code>t</code>.<br />
If <code>i</code> does not hold a <code>T</code>, the statement will trigger a panic.<br />
To test whether an interface value holds a specific type, a type assertion can return two values: the underlying value and a boolean value that reports whether the assertion succeeded.<br />
<code>t, ok := i.(T)</code></p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">var</span> i <span class="token keyword">interface</span><span class="token punctuation">{</span><span class="token punctuation">}</span> <span class="token operator">=</span> <span class="token string">"hello"</span> <span class="token comment">// implicit string type</span><br /><br /> s <span class="token operator">:=</span> i<span class="token punctuation">.</span><span class="token punctuation">(</span><span class="token builtin">string</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span> <span class="token comment">//"hello"</span><br /><br /> s<span class="token punctuation">,</span> ok <span class="token operator">:=</span> i<span class="token punctuation">.</span><span class="token punctuation">(</span><span class="token builtin">string</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>s<span class="token punctuation">,</span> ok<span class="token punctuation">)</span> <span class="token comment">//"hello", true</span><br /><br /> f<span class="token punctuation">,</span> ok <span class="token operator">:=</span> i<span class="token punctuation">.</span><span class="token punctuation">(</span><span class="token builtin">float64</span><span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>f<span class="token punctuation">,</span> ok<span class="token punctuation">)</span> <span class="token comment">//0, false</span><br /><br /> f <span class="token operator">=</span> i<span class="token punctuation">.</span><span class="token punctuation">(</span><span class="token builtin">float64</span><span class="token punctuation">)</span> <span class="token comment">// panic</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>f<span class="token punctuation">)</span> <span class="token comment">//panic: interface conversion: interface {} is string, not float64</span><br /><span class="token punctuation">}</span></code></pre>
<h3 id="type-switches">Type switches <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#type-switches">#</a></h3>
<p>A type switch is a construct that permits several type assertions in series.<br />
A type switch is like a regular switch statement, but the cases in a type switch specify types (not values), and those values are compared against the type of the value held by the given interface value.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">do</span><span class="token punctuation">(</span>i <span class="token keyword">interface</span><span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">switch</span> v <span class="token operator">:=</span> i<span class="token punctuation">.</span><span class="token punctuation">(</span><span class="token keyword">type</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">case</span> <span class="token builtin">int</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Printf</span><span class="token punctuation">(</span><span class="token string">"Twice %v is %v\n"</span><span class="token punctuation">,</span> v<span class="token punctuation">,</span> v<span class="token operator">*</span><span class="token number">2</span><span class="token punctuation">)</span><br /> <span class="token keyword">case</span> <span class="token builtin">string</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Printf</span><span class="token punctuation">(</span><span class="token string">"%q is %v bytes long\n"</span><span class="token punctuation">,</span> v<span class="token punctuation">,</span> <span class="token function">len</span><span class="token punctuation">(</span>v<span class="token punctuation">)</span><span class="token punctuation">)</span><br /> <span class="token keyword">default</span><span class="token punctuation">:</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Printf</span><span class="token punctuation">(</span><span class="token string">"I don't know about type %T!\n"</span><span class="token punctuation">,</span> v<span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token function">do</span><span class="token punctuation">(</span><span class="token number">21</span><span class="token punctuation">)</span><br /> <span class="token function">do</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><br /> <span class="token function">do</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// Twice 21 is 42</span><br /><span class="token comment">// "hello" is 5 bytes long</span><br /><span class="token comment">// I don't know about type bool!</span></code></pre>
<h2 id="errors">Errors <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#errors">#</a></h2>
<p>Go programs express error state with error values. The error type is a built-in interface similar to <code>fmt.Stringer</code>:</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">type</span> <span class="token builtin">error</span> <span class="token keyword">interface</span> <span class="token punctuation">{</span><br /> <span class="token function">Error</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token builtin">string</span><br /><span class="token punctuation">}</span></code></pre>
<p>As with <code>fmt.Stringer</code>, the fmt package looks for the error interface when printing values.<br />
Functions often return an error value, and calling code should handle errors by testing whether the error equals nil.</p>
<pre class="language-go"><code class="language-go">i<span class="token punctuation">,</span> err <span class="token operator">:=</span> strconv<span class="token punctuation">.</span><span class="token function">Atoi</span><span class="token punctuation">(</span><span class="token string">"42"</span><span class="token punctuation">)</span><br /><span class="token keyword">if</span> err <span class="token operator">!=</span> <span class="token boolean">nil</span> <span class="token punctuation">{</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Printf</span><span class="token punctuation">(</span><span class="token string">"couldn't convert number: %v\n"</span><span class="token punctuation">,</span> err<span class="token punctuation">)</span><br /> <span class="token keyword">return</span><br /><span class="token punctuation">}</span><br />fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span><span class="token string">"Converted integer:"</span><span class="token punctuation">,</span> i<span class="token punctuation">)</span></code></pre>
<h2 id="goroutines">Goroutines <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#goroutines">#</a></h2>
<p>A goroutine is a lightweight thread managed by the Go runtime.</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">func</span> <span class="token function">say</span><span class="token punctuation">(</span>s <span class="token builtin">string</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">for</span> i <span class="token operator">:=</span> <span class="token number">0</span><span class="token punctuation">;</span> i <span class="token operator"><</span> <span class="token number">3</span><span class="token punctuation">;</span> i<span class="token operator">++</span> <span class="token punctuation">{</span><br /> time<span class="token punctuation">.</span><span class="token function">Sleep</span><span class="token punctuation">(</span><span class="token number">100</span> <span class="token operator">*</span> time<span class="token punctuation">.</span>Millisecond<span class="token punctuation">)</span><br /> fmt<span class="token punctuation">.</span><span class="token function">Println</span><span class="token punctuation">(</span>s<span class="token punctuation">)</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">func</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">go</span> <span class="token function">say</span><span class="token punctuation">(</span><span class="token string">"world"</span><span class="token punctuation">)</span><br /> <span class="token function">say</span><span class="token punctuation">(</span><span class="token string">"hello"</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><span class="token comment">// hello</span><br /><span class="token comment">// world</span><br /><span class="token comment">// world</span><br /><span class="token comment">// hello</span><br /><span class="token comment">// hello</span></code></pre>
<h3 id="-------------------------------------------------------------------------------------------------------">------------------------------------------------------------------------------------------------------- <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#-------------------------------------------------------------------------------------------------------">#</a></h3>
<h3 id="%F0%9F%9A%A7-todo---continue-on-here-https%3A%2F%2Ftour.golang.org%2Fconcurrency%2F2">đ§ TODO - continue on here <a href="https://tour.golang.org/concurrency/2">https://tour.golang.org/concurrency/2</a> <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#%F0%9F%9A%A7-todo---continue-on-here-https%3A%2F%2Ftour.golang.org%2Fconcurrency%2F2">#</a></h3>
<h3 id="---------------------------------------------------------------------------------------------------------">--------------------------------------------------------------------------------------------------------- <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#---------------------------------------------------------------------------------------------------------">#</a></h3>
<h1 id="observations">Observations <a class="direct-link" href="https://lukesingham.com/go-tour-summary/#observations">#</a></h1>
<ul>
<li>
<p>unused variables are not allowed by the compiler - very cool</p>
</li>
<li>
<p>hard concepts (read more on)</p>
<ul>
<li>pointer indirection
<ul>
<li>performance</li>
<li>modify in place</li>
<li>when optional, pointer to a string... no empty strings in go. But the pointer to the string will be <code>nil</code></li>
</ul>
</li>
<li>interfaces and polymorphism (types)</li>
<li>concurrency</li>
<li>stringer, but why? Pretty printing? <a href="https://tour.golang.org/methods/17">https://tour.golang.org/methods/17</a></li>
</ul>
</li>
<li>
<p>skipped</p>
<ul>
<li>exercises</li>
<li>reader</li>
<li>images</li>
<li>routines, channels, buffered channels, range and close, select,sync.Mutex</li>
</ul>
</li>
<li>
<p>typing a particular concept into youtube usual has a human to explain it.</p>
<ul>
<li>sentdex is pretty good at explaining in 5mins</li>
</ul>
</li>
<li>
<p>is go functional? Nope and it's not OO. It's procedural. Very explicit.</p>
</li>
</ul>
The Mythical Man Month - Review2020-11-18T00:00:00Zhttps://lukesingham.com/mythical-man-month-review/<p>đ Book: <a href="https://amzn.to/38P7XBK">Mythical Man Month</a></p>
<p>âď¸ Rating: 3/5 (4/5 if you haven't had experience in tech)</p>
<p><img src="https://lukesingham.com/content/images/2020/mythical-man-month-frontcover.png#thumbnail" alt="" /></p>
<p>The Mythical Man Month (MMM) is a collection of essays that encapsulate wisdom accumulated by the author Fred Brooks. If, like me, you have been in the tech industry for a number of years you are unlikely to gain much value from this book - as you have either read articles or experienced first hand these lessons.</p>
<h3 id="some-criticisms-of-the-book%3A">Some criticisms of the book: <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#some-criticisms-of-the-book%3A">#</a></h3>
<ul>
<li>đ¤Śââď¸ There's not a female pronoun in sight - one of the few mentions of a female, is for an analogy where the project is a pregnancy, adding more women to the project will not decrease the nine months it will take for the birth to happen. This is to be expected of a book originally written in 1975 and with a title like <em>Mythical Man Month</em>. I read the 1995 edition. I would hope any subsequent editions endeavour to have more gender inclusive language.</li>
<li>đź There's quite a lot of God mentions, more mentions of God than Women.</li>
<li>đ Many of the examples are very out of date. And the details on these examples of e.g. IBM 360 are superfluous to the modern reader.</li>
</ul>
<h2 id="major-(useful)-points-of-the-book">Major (Useful) Points of the Book <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#major-(useful)-points-of-the-book">#</a></h2>
<h3 id="%F0%9F%92%A3-combinatorial-explosion">đŁ Combinatorial Explosion <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#%F0%9F%92%A3-combinatorial-explosion">#</a></h3>
<p>Where the title of the book comes from. Many simpler types of work scale linearly with additional resources. For example, if you were a Farmer of 20 fields employing labour to til those fields. You might imagine doubling labour from 2 to 4 would double results. This is true, as there is little requirement for coordination of the manual labour. Labourers each get an allocation of fields to independently til. However, with work that requires much coordination, alignment of corresponding components of a greater system like software, adding more labour is frequently counterproductive. This leads to an assertion that <em>adding engineers to a late project will make it later</em> because of the communication overhead and the need to onboard.</p>
<p>The effect is known as <a href="https://en.wikipedia.org/wiki/Combinatorial_explosion">combinatorial explosion</a>. Where each node (worker) that needs to be aligned with the rest of the team adds many more communication links.<br />
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mrow><mtext>links</mtext></mrow><mo>=</mo><mfrac><mrow><msup><mi>n</mi><mn>2</mn></msup><mo>-</mo><mi>n</mi></mrow><mn>2</mn></mfrac></math></p>
<p>A few examples demonstrate this:</p>
<ul>
<li>when <code>n=2</code>, there is 1 link</li>
<li>when <code>n=4</code>, there are 6 links</li>
<li>when <code>n=8</code>, there are 28 links</li>
</ul>
<p>This rule is also known as the '<a href="https://www.theguardian.com/technology/2018/apr/24/the-two-pizza-rule-and-the-secret-of-amazons-success">two pizza rule</a>' đ - teams shouldn't be larger than can be fed with two pizzas. Not helpful if you eat an entire pizza to yourself đ¤¤. However, 5-7 people is what is meant.</p>
<h3 id="%F0%9F%A7%A0-conceptual-integrity">đ§ Conceptual Integrity <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#%F0%9F%A7%A0-conceptual-integrity">#</a></h3>
<blockquote>
<p>... conceptual integrity is the most important consideration in system design</p>
</blockquote>
<p>Brooks talks a lot about <em>Conceptual Integrity</em> of software systems. In data analytics this is known by various terms e.g. ontology, entity models. If you have worked with any system with multiple contributors you have likely witnessed the core concepts of a system drift over time. Here Brooks suggests a solution,</p>
<blockquote>
<p>If a system is to have conceptual integrity, someone must control the concepts. That is an aristocracy that needs no apology.</p>
</blockquote>
<h3 id="%F0%9F%93%88-10x-programmers">đ 10x programmers <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#%F0%9F%93%88-10x-programmers">#</a></h3>
<p>To maintain <em>conceptual integrity</em> smaller teams should be preferred.</p>
<blockquote>
<p>one wants the system to be built by as few minds as possible</p>
</blockquote>
<p>The type of team you should favour is one of a few senior engineers rather than one that is larger with junior engineers. Brooks cites a study measuring programmer productivity which is probably where the <em>10x engineers</em> terminology originated. I thought this might be an exaggeration, but I've seen this play out in a workplace. A leader of a project thought that software was a sweatshop and if they just got more cheap juniors fresh out of code bootcamps that software would happen. It didn't. However, hiring one senior and one mid-level engineer saw delivery sky rocket. And yes, the senior was an order of magnitude more productive than juniors and for less than 3x the price!</p>
<h3 id="%F0%9F%93%84-documentation">đ Documentation <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#%F0%9F%93%84-documentation">#</a></h3>
<p>Write a spec/proposal document prior to commencing on the code. This outlines a number of things, the <em>why</em>, the requirements of a solution, the range of solutions considered and the proposed solution to be implemented. This helps in many ways, the writing process helps expose gaps in thinking, communicates to others, helps align the team and provides a record of what is being done or has been done.</p>
<h3 id="%F0%9F%9A%A7-maintenance">đ§ Maintenance <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#%F0%9F%9A%A7-maintenance">#</a></h3>
<blockquote>
<p>A program doesn't stop changing when it is delivered for customer usage</p>
</blockquote>
<p>Brooks estimates that maintenance is '40 percent or more' than the initial cost of developing a system. The more users you have the greater the cost as more bugs are found by more users.</p>
<h2 id="resources">Resources <a class="direct-link" href="https://lukesingham.com/mythical-man-month-review/#resources">#</a></h2>
<ul>
<li><a href="http://asciimath.org/">asciimath.org</a> - used to generate mathML mathjax forumula</li>
</ul>
The Best Self-Hosted RSS Feed Readers2020-05-31T00:00:00Zhttps://lukesingham.com/rss-feed-reader/<h2 id="what-is-rss%3F">What is <a href="https://en.wikipedia.org/wiki/RSS">RSS</a>? <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#what-is-rss%3F">#</a></h2>
<p>It aims to be an open standard for the open web, where applications can get updates from websites. This runs against the current closed off portions of the web like Facebook that keep content behind their account login - i.e. the internet's <a href="https://en.wikipedia.org/wiki/Closed_platform">'walled gardens'</a>. For a while, a lot of people used the popular <a href="https://en.wikipedia.org/wiki/Google_Reader">Google Reader</a> RSS application - unfortunately, as with many loved Google projects it got killed off because of declining usage and it didn't generate revenue.</p>
<p>So why get one now?</p>
<ul>
<li>Ideals of an open web. I don't have a Facebook account. I believe there are interesting blogs out there that I don't necessarily see unless they are surfaced through the front page of search results or aggregators like Hacker News or Reddit.</li>
<li>Solve the problem of - <em>huh this blog/site was really interesting. How do I keep track of new content without signing up to a thousand newsletters or entering a walled garden?</em></li>
</ul>
<h2 id="my-starting-point">My Starting Point <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#my-starting-point">#</a></h2>
<p>There's a bunch of open source Feed Readers listed on the <a href="https://github.com/awesome-selfhosted/awesome-selfhosted#feed-readers">Feed Reader section of Awesome Self-Hosted</a>. My criteria to filter the list were:</p>
<ul>
<li>Minimum 1000 Github Stars</li>
<li>Commit in the last 6 months</li>
</ul>
<p>This is what I was left with. Which I then filtered based on my UI and tech preferences i.e. don't be ugly and don't use PHP/Java.</p>
<h2 id="%F0%9F%91%A8%E2%80%8D%F0%9F%91%A9%E2%80%8D%F0%9F%91%A7%E2%80%8D%F0%9F%91%A7-the-contenders">đ¨âđŠâđ§âđ§ The Contenders <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#%F0%9F%91%A8%E2%80%8D%F0%9F%91%A9%E2%80%8D%F0%9F%91%A7%E2%80%8D%F0%9F%91%A7-the-contenders">#</a></h2>
<h3 id="winds"><a href="https://getstream.io/winds/">Winds</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#winds">#</a></h3>
<p><em>The kitchen sink reader</em><br />
Github Stars: 7.5k<br />
Built With: JavaScript (React + Express)<br />
Test Website: No, but free account creation<br />
Features:</p>
<ul>
<li>Recommender service via external service</li>
<li>Very nice UI</li>
<li>'Listen and read' i.e. podcast player too</li>
</ul>
<h3 id="stringer"><a href="https://github.com/swanson/stringer">Stringer</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#stringer">#</a></h3>
<p><em>Stringer is simple... 'has no external dependencies, no social recommendations/sharing, and no fancy machine learning algorithms.'</em><br />
Github Stars: 3.2k<br />
Built With: Ruby<br />
Test Website: No, but has a click to deploy to Heroku option<br />
Features:</p>
<ul>
<li>Simple and nice UI</li>
<li>Keyboard shortcuts (I use <a href="https://vimium.github.io/">Vimium</a> so not sold on this)</li>
</ul>
<h3 id="selfoss"><a href="https://selfoss.aditu.de/">selfoss</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#selfoss">#</a></h3>
<p><em>Didn't look further at this one PHP+ugly =</em> đąđââď¸.<br />
Github Stars: 1.9k<br />
Built With: PHP<br />
Test Website: No</p>
<h3 id="miniflux"><a href="https://github.com/miniflux/miniflux">Miniflux</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#miniflux">#</a></h3>
<p><em>Ultra simplicity in terms of stack and UI. Slightly ugly, but I agree with the software philosophy of keeping it simple.</em><br />
Github Stars: 1.9k<br />
Built With: Plain go-lang+minimal vanilla JavaScript<br />
Test Website: No<br />
Features:</p>
<ul>
<li>Support multiple enclosures/attachments (Podcasts, videos, music, and images)</li>
<li>Play videos from YouTube channels directly inside Miniflux</li>
<li>Categories</li>
<li>Bookmarks</li>
<li>Fetch website icons (favicons)</li>
<li>Save articles to third-party services e.g. Pocket</li>
<li>Dark theme (otherwise kind of ugly)</li>
<li>Optional keyboard shortcuts</li>
</ul>
<h3 id="freshrss"><a href="https://github.com/FreshRSS/FreshRSS">FreshRSS</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#freshrss">#</a></h3>
<p><em>Like Selfoss - Didn't look further at this one PHP+ugly =</em> đąđââď¸.<br />
Github Stars: 2.3k<br />
Built With: PHP<br />
Test website: <a href="https://demo.freshrss.org/i/">Yes</a></p>
<h3 id="feedbin"><a href="https://github.com/feedbin/feedbin">feedbin</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#feedbin">#</a></h3>
<p><em>Didn't look further as the app tech + ui wasn't appealing to me</em>.<br />
Github Stars: 2.5k<br />
Built With: Ruby on Rails<br />
Test Website: No - but free trial</p>
<h3 id="commafeed"><a href="https://github.com/Athou/commafeed">Commafeed</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#commafeed">#</a></h3>
<p><em>Github at the time had a <strong>đbuild error</strong> + Java turned me away</em><br />
Github Stars: 1.7k<br />
Built With: Java đą at the time of reviewing had a 'build: error'<br />
Test Website: no - but free trial</p>
<p><strong>Final round contenders: Miniflux, Stringer and Winds</strong></p>
<h2 id="%F0%9F%8F%81-top-3-contenders---install-and-review">đ Top 3 contenders - Install and Review <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#%F0%9F%8F%81-top-3-contenders---install-and-review">#</a></h2>
<p>N.B. I'm running these locally to assess before deploying to my server.</p>
<h3 id="stringer-2"><a href="https://github.com/swanson/stringer">Stringer</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#stringer-2">#</a></h3>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">docker</span> run <span class="token parameter variable">--rm</span> <span class="token parameter variable">-it</span> <span class="token parameter variable">-e</span> <span class="token assign-left variable">DATABASE_URL</span><span class="token operator">=</span><span class="token string">"sqlite3:':memory:'"</span> <span class="token parameter variable">-p</span> <span class="token number">8080</span>:8080 mdswanson/stringer</code></pre>
<p>Opened on <code>localhost:8080</code> - Simple as that. UI feels nice and simple. Unfortunately, the only import methods are from the long defunct Google Reader or with one URL at a time.<br />
<img src="https://lukesingham.com/content/images/2020/05/image.png" alt="" /><br />
I tried importing the OPML file through this method but that seemed to crash the application - poor error handling?<br />
<img src="https://lukesingham.com/content/images/2020/05/image-1.png" alt="" /><br />
Once you do manually load some feeds the <code>/feeds</code> interface is nice and clean.<br />
<img src="https://lukesingham.com/content/images/2020/05/image-2.png" alt="" /><br />
The <code>/news</code> view is well laid out and the fonts used are a nice reading experience.<br />
<img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-31-at-09.41.46.png" alt="" /><br />
Lastly, whilst poking around the Github repo I found this comment on <a href="https://github.com/swanson/stringer/issues/512">this recent issue</a>:</p>
<blockquote>
<p>... I don't think anyone is currently developing new features. I'm available to review code and merge it if you want to submit a patch.</p>
</blockquote>
<p>So whilst the UI is nice and the setup is incredibly easy. The active development of  Miniflux and Wind have started to look more appealing.</p>
<p>âď¸ 6.5 / 10 - Easy install, nice UI, functionality is low and active development has slowed.</p>
<h3 id="miniflux-2"><a href="https://github.com/miniflux/miniflux">Miniflux</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#miniflux-2">#</a></h3>
<ol>
<li>You need postgres which I had not setup on my mac. I recommend doing this the easy way with <a href="https://postgresapp.com/">postgres.app</a>.</li>
<li>The <a href="https://miniflux.app/docs/installation.html#docker">instructions </a>here are a little hairy. I went with saving the following to a <code>docker-compose.yml</code>. I couldn't get it working on port 80 so switched the port to 8050.</li>
</ol>
<pre class="language-docker"><code class="language-docker">version: '3'<br />services:<br /> miniflux:<br /> image: miniflux/miniflux:latest<br /> ports:<br /> - "8050:8080"<br /> depends_on:<br /> - db<br /> environment:<br /> - DATABASE_URL=postgres://miniflux:secret@db/miniflux?sslmode=disable<br /> - RUN_MIGRATIONS=1<br /> - CREATE_ADMIN=1<br /> - ADMIN_USERNAME=admin<br /> - ADMIN_PASSWORD=test123<br /> db:<br /> image: postgres:latest<br /> environment:<br /> - POSTGRES_USER=miniflux<br /> - POSTGRES_PASSWORD=secret<br /> volumes:<br /> - miniflux-db:/var/lib/postgresql/data<br />volumes:<br /> miniflux-db:</code></pre>
<p>Then run the following commands and go to <code>localhost:8050</code> once the containers have finished booting:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">docker-compose</span> up <span class="token parameter variable">-d</span> db<br />$ <span class="token function">docker-compose</span> up miniflux</code></pre>
<p>Firstly, the initial UI is <em>very</em> simple, whilst incredibly plain is very functional. Navigating the app is explicit with text whereas Stringer you had to poke around using icons without text descriptions.<br />
<img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-31-at-09.46.28.png" alt="" /><br />
There are several integrations to apps like Pocket, Instapaper and Wallabag. And after changing the theme. the reader is quite nice. Albeit, I do notice the entire page refresh from the server side oriented app - but I have to tell myself that the simplicity philosophy trumps the complexity of introducing a frontend framework.<br />
<img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-31-at-13.36.10.png" alt="" /><br />
âď¸ 8.5 / 10 - Whilst being minimalistic has all the features you need, dark theme and the most active development of them all with a clear software philosophy.</p>
<h2 id="winds-2"><a href="https://getstream.io/winds/">Winds</a> <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#winds-2">#</a></h2>
<p>The install is relatively long particularly with the external dependencies which include:</p>
<ul>
<li><a href="https://getstream.io/">Stream</a> - an API for building activity feeds + handle personalization (machine learning).</li>
<li><a href="https://www.algolia.com/">Algolia</a> - for search</li>
<li><a href="https://www.mongodb.com/cloud/atlas">MongoDB Atlas</a> - a DB as a service (DBaaS)</li>
</ul>
<p>At this point I opt to sign up for a free account to try it out. I get the <a href="https://apps.apple.com/us/app/winds-by-getstream-io/id1381446741">app from the apple store</a>. Upon opening Winds, you are first confronted with a selection of interests like 'Programming', 'News' etc. I pick a couple and click next. I then have to create an account đ which I do. Then once logged in I have a very populated set of feeds and podcasts.<br />
<img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-31-at-14.02.22.png" alt="" /><br />
I then want to try and remove all of these and start fresh, which no amount of clicking around can I figure out how to do other than clicking individually on each feed to remove it. I could create a new account and then add my OPML file - however at this point I'm already missing the bloat free approach of Miniflux.</p>
<p>âď¸ 7 / 10  - for a non self-hosted piece of software I can see this is actually really nice. However major downsides of the installation complexity for self-hosted installation combined with features I don't particularly want. For instance, I don't want a recommender system - I can find my own content and probably consume too much as it is! If I want reinforcing recommender systems I'd go into a walled garden like Facebook.</p>
<h2 id="%F0%9F%8E%89-winner---miniflux-%F0%9F%8E%89">đ Winner - Miniflux đ <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#%F0%9F%8E%89-winner---miniflux-%F0%9F%8E%89">#</a></h2>
<h2 id="rss-tips">RSS Tips <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#rss-tips">#</a></h2>
<p>Do you have a list of websites you want to follow?</p>
<ul>
<li>A lot of sites you can just <code>/rss</code> e.g <code>domainname.com/rss</code></li>
<li>For medium sites add <code>/feed/</code> e.g. <a href="https://medium.com/feed/@cjolowicz/"><code>https://medium.com/feed/@verygoodblogger/</code></a></li>
<li>I used <a href="https://opml-gen.ovh/">this site</a> to convert a list feeds into an OPML file I could mass import to the various RSS feed reader apps.</li>
<li>About a decade ago I used blogspot/blogger. It had a feed reader, but no export facility. Nostalgically, I thought I'd get the list from there. Here's a hack to export the list. First navigate to '<a href="https://www.blogger.com/manage-blogs-following.g">Manage blogs I'm following</a>', then open your browser dev tools and run the following JavaScript to get a list of the links.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> urls <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'.blogUrl'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> url_list <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span><br /><br />urls<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">url</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> url_list<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>url<span class="token punctuation">.</span>innerText<span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>url_list<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="related-interesting-projects">Related Interesting Projects <a class="direct-link" href="https://lukesingham.com/rss-feed-reader/#related-interesting-projects">#</a></h2>
<ul>
<li><a href="https://github.com/RSS-Bridge/rss-bridge">RSS-Bridge</a> - Create RSS feeds for websites without a feed.</li>
<li><a href="https://github.com/wallabag/wallabag">Wallabag</a> - Save articles from RSS feeds.</li>
<li><a href="https://archivebox.io/">Archivebox</a> - Save all content e.g. videos, entire web pages and configure that save in multiple format. Solves link rot issues.</li>
</ul>
How to Setup a Ubiquiti Access Point2020-05-17T00:00:00Zhttps://lukesingham.com/ubiquiti-ap-setup/<p><strong>Aim</strong>: Have strong WiFi across a property. In other words elimate the deadzone.</p>
<p><strong>Solution:</strong> Keep existing ISP provided router and plugin a Ubiquiti Long Range Access Point.</p>
<p><strong>tl:dr</strong> - â Solved - No deadzone. But, setting up up commercial grade wifi was not user friendly.</p>
<p><strong>Ingredients:</strong></p>
<ul>
<li><a href="https://amzn.to/2KSuLIK">Ubiquiti Networks UAP-AC-LR 175.7 x 43.2 mm 2.4-5 GHz 802.11ac Dual-Radio Long Range Access Point - White</a></li>
<li>2m Ethernet Cable (+ later a 10m)</li>
</ul>
<p>There are various ways to solve internet problems. I've tried stronger routers, stronger receivers, 30m ethernet cables, TP-Link powerlines. Except for the 30m cable, all the others were disappointments. But a cable doesn't solve the WiFi. So some reading on network topologies led me to evaluating mesh networks, boosters and access points. With a trusted friends recommendation I was swayed to go with a commercial grade access point. There were many failures along the way.</p>
<h2 id="%F0%9F%A4%A6%E2%80%8D%E2%99%80%EF%B8%8F-unboxing...">đ¤Śââď¸ Unboxing... <a class="direct-link" href="https://lukesingham.com/ubiquiti-ap-setup/#%F0%9F%A4%A6%E2%80%8D%E2%99%80%EF%B8%8F-unboxing...">#</a></h2>
<p>đ Mistake 1 - Doesn't come with an ethernet cable.<br />
Yes this ÂŁ100 solution doesn't come with even the shortest of ethernet cables to connect it to my router. I also had no power adaptor space. Amazon Prime to the rescue.</p>
<h2 id="%F0%9F%A4%94-all-plugged-in%3A-now-how-do-i-connect-it-up%3F">đ¤ All plugged in: Now how do I connect it up? <a class="direct-link" href="https://lukesingham.com/ubiquiti-ap-setup/#%F0%9F%A4%94-all-plugged-in%3A-now-how-do-i-connect-it-up%3F">#</a></h2>
<p>The instructions that come with it don't help. Next stop, interwebs. I start to hit a lot of content that is sending me wayward.</p>
<p>đ Mistake 2 - This is not the <a href="https://play.google.com/store/apps/details?id=com.ubnt.unifi.edu">UniFi software controller</a> you are looking for. This android app was the first UniFi app to come up and was a pure dead end.</p>
<p>đ Mistake 3 - Next attempt was the <a href="https://www.ui.com/download/unifi/">OSX software controller</a>. Run for the hills if you see this error:<br />
<img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-17-at-22.02.26.png" alt="" /><br />
I messed around with some open JDKs but ultimately was entering a rabbit hole. Seems like <a href="https://tongfamily.com/2019/10/08/ubiquiti-unifi-controller-4-11-47-and-java-runtime-nightmare/">others have shared this 'nightmare'</a> and didn't come up with much in the way of solution.</p>
<p>I started to think I needed a physical controller since most of the docs are oriented around his setup. However, I revisted the Android options and downloaded the <a href="https://play.google.com/store/apps/details?id=com.ubnt.easyunifi&hl=en">Unifi Network app</a>. Upon opening you get the following screen:<br />
<img src="https://lukesingham.com/content/images/2020/05/Screenshot_20200517-223735-1.png" alt="" /><br />
Again with the controller emphasis. Unintuitively, if you click on account you can then access setting up 'Standalone Devices'.<br />
<img src="https://lukesingham.com/content/images/2020/05/Screenshot_20200517-223744-1.png" alt="" /><br />
It's at this point you can search and add your new Unifi device. Once added, you can then tinker with the configuration. What's really nice is that the 2G and 5G by default share the same SSID - i.e. unlike lots of homes that have a network for each, the UniFi device optimises the network based on the strength of connection with a device on the network. đ<br />
<img src="https://lukesingham.com/content/images/2020/05/Screenshot_20200517-221603-4.png" alt="" /></p>
<h2 id="results">Results <a class="direct-link" href="https://lukesingham.com/ubiquiti-ap-setup/#results">#</a></h2>
<p>I had installed the access point right next to the router. I experienced immediate results as I walked through the deadzone. My mobile switched automatically to the Unifi network once a little way down the garden. And all the way to the garden shed. What I didn't test was inside the shed - it's amazing how buildings kill WiFi. Patch at best to no signal inside. đ¤Ż<br />
I decided to run a 10m cable between the router and the Unifi AP such that the Unifi device sits right at the window onlooking the garden. Now getting very strong signal across the garden - significant drop off ~80% once inside the shed, but at least it was a consistent connection. Based on several speed tests, I was getting 3mbs - 13mbs. That'll do pig. đ</p>
<h3 id="semi-helpful-docs">Semi-helpful docs <a class="direct-link" href="https://lukesingham.com/ubiquiti-ap-setup/#semi-helpful-docs">#</a></h3>
<ul>
<li><a href="https://lukesingham.com/ubiquiti-ap-setup/help.ui.com/hc/en-us/articles/204909754">UniFi - Device Adoption Methods for Remote UniFi Controllers</a></li>
</ul>
A Review and Summary of the Javascript30 Course2020-05-06T00:00:00Zhttps://lukesingham.com/summary-of-javascript30/<p>đCourse: <a href="https://javascript30.com/">Javascript30</a><br />
đ°Price: Free<br />
âď¸Rating: 8.5/10</p>
<h2 id="review">Review <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#review">#</a></h2>
<p>What could be better? Not a lot. I think Wes (the instructor) could have thought harder about prompting learners to pause the video and trying solo for themselves. He does once or twice somewhere in the first 10 lessons. But didn't do it after, which meant I tended to follow along waiting for the prompt. There are a couple of mistakes here and there - one was on a resource Wes was hosting returning a <code>404</code> for one of the lessons, but after emailing Wes fixed it about a week later.</p>
<p>I thought some items could have been explained better e.g. CSS query selectors and the different syntax to use. For someone with some programming experience and understanding who wants to focus on improving JavaScript skills, working with the APIs and developing with the browser this is a fantastic course. A couple of lessons are out-of-date however the finished solutions available in the repo are mostly up-to-date. I really like the honesty in the mistakes and even debugging his lesson code on video - it gives a much more honest representation of what programming is like. If a JavaScript jedi like Wes makes mistakes and spends time debugging so too will you.</p>
<h2 id="01---javascript-drum-kit">01 - JavaScript Drum Kit <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#01---javascript-drum-kit">#</a></h2>
<ul>
<li>There's an <code><audio></code> div <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio">tag for embedded audio.</a></li>
<li>Adding event listeners is easy, in this example :</li>
</ul>
<pre class="language-js"><code class="language-js">window<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'keydown'</span><span class="token punctuation">,</span> <span class="token keyword">function</span> <span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> audio <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">audio[data-key="</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>e<span class="token punctuation">.</span>keyCode<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">"]</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Learn/HTML/Howto/Use_data_attributes">HTML data attributes</a> - store additional info on html elements using <code>data-</code> prefix:</li>
<li>The data of the elements can then be accessed using <code>dataset</code>:</li>
</ul>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>spacing<span class="token punctuation">"</span></span><span class="token attr-name">...</span> <span class="token attr-name">data-sizing</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>px<span class="token punctuation">"</span></span><span class="token punctuation">></span></span></code></pre>
<pre class="language-js"><code class="language-js"><span class="token keyword">function</span> <span class="token function">handleUpdate</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> suffix <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">.</span>dataset<span class="token punctuation">;</span><br /><span class="token punctuation">}</span><br /><br />inputs<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">node</span> <span class="token operator">=></span> node<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'change'</span><span class="token punctuation">,</span> handleUpdate<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="02---clock">02 - Clock <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#02---clock">#</a></h2>
<ul>
<li>Select any element with <code>document.querySelector('.second-hand');</code>. Where <code>.second-hand</code> is a class name. Known as <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors">CSS selectors</a>.</li>
<li>Use <code>setInterval(yourfn, 1000)</code> to run a function on a repeated time interval.</li>
</ul>
<h2 id="03---css-variables">03 - CSS Variables <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#03---css-variables">#</a></h2>
<ul>
<li>Pattern, get all elements you want to manipulate, create a function defining that change, create trigger for that function.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> inputs <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'.controls input'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br /><span class="token keyword">function</span> <span class="token function">handleUpdate</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> suffix <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">.</span>dataset<span class="token punctuation">.</span>sizing <span class="token operator">||</span> <span class="token string">''</span><span class="token punctuation">;</span><br /> document<span class="token punctuation">.</span>documentElement<span class="token punctuation">.</span>style<span class="token punctuation">.</span><span class="token function">setProperty</span><span class="token punctuation">(</span><span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">--</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span><span class="token keyword">this</span><span class="token punctuation">.</span>name<span class="token interpolation-punctuation punctuation">}</span></span><span class="token template-punctuation string">`</span></span><span class="token punctuation">,</span> <span class="token keyword">this</span><span class="token punctuation">.</span>value<span class="token operator">+</span>suffix<span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><br /><br />inputs<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">node</span> <span class="token operator">=></span> node<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'change'</span><span class="token punctuation">,</span> handleUpdate<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />inputs<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">node</span> <span class="token operator">=></span> node<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'mousemove'</span><span class="token punctuation">,</span> handleUpdate<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="04---array-cardio-day-1">04 - array cardio day 1 <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#04---array-cardio-day-1">#</a></h2>
<p><em>Array methods map(), reduce(), reduce()  and  filter() are introduced.</em></p>
<ul>
<li>Log a table to the console, i.e. a list of objects</li>
</ul>
<pre class="language-js"><code class="language-js">console<span class="token punctuation">.</span><span class="token function">table</span><span class="token punctuation">(</span>sorted<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h1 id="05---flex-panels">05 - Flex panels <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#05---flex-panels">#</a></h1>
<ul>
<li>Light touch on some flex panels (pretty foreign though, without much intro)</li>
<li>Pattern of applying some transforms and transitions css using js</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// grab relevant elements</span><br /><span class="token keyword">var</span> panels <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'.panel'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br /><span class="token comment">// function to toggle</span><br /><span class="token keyword">function</span> <span class="token function">toggleOpen</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">this</span><span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">toggle</span><span class="token punctuation">(</span><span class="token string">'open'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">// event listener to toggle</span><br />panels<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> x<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'click'</span><span class="token punctuation">,</span> toggleOpen<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="06---type-ahead">06 - Type Ahead <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#06---type-ahead">#</a></h2>
<p>Implement a search box against json data.</p>
<ul>
<li>Get data via a promise:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> cities <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span><br /><span class="token function">fetch</span><span class="token punctuation">(</span>endpoint<span class="token punctuation">)</span><br /> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">blob</span> <span class="token operator">=></span> blob<span class="token punctuation">.</span><span class="token function">json</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /> <span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">data</span> <span class="token operator">=></span> cities<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span><span class="token operator">...</span>data<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<ul>
<li>Regex can be used like this:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// (g)lobal, (i)nsensitive</span><br /><span class="token keyword">const</span> regex <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">RegExp</span><span class="token punctuation">(</span>wordToMatch<span class="token punctuation">,</span> <span class="token string">'gi'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">return</span> place<span class="token punctuation">.</span>city<span class="token punctuation">.</span><span class="token function">match</span><span class="token punctuation">(</span>regex<span class="token punctuation">)</span></code></pre>
<h2 id="07---array-cardio-day-2">07 - array cardio day 2 <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#07---array-cardio-day-2">#</a></h2>
<p><em>Some extra array methods</em></p>
<ul>
<li><code>Array.prototype.some()</code> - checks if at least one element meets a condition is met. Returns Boolean.</li>
<li><code>Array.prototype.every()</code> - Â - checks if all elements meet a condition is met. Returns Boolean.</li>
<li><code>Array.prototype.find()</code> - returns element from array that meets the specified condition.</li>
<li><code>Array.prototype.findIndex()</code> - like <code>find()</code> but returns the index.</li>
<li><code>Array.prototype.splice()</code> - delete the element at a given index.</li>
</ul>
<h2 id="08---fun-with-html5-canvas">08 - Fun with HTML5 Canvas <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#08---fun-with-html5-canvas">#</a></h2>
<p><em>Could have provided more context on the why of canvas vs alternatives of divs, could have been more structured as well. What is canvas? What are its features? etc</em></p>
<ul>
<li>Canvas has a context, where you can update it's properties:</li>
</ul>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>body</span><span class="token punctuation">></span></span><br /><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>canvas</span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>draw<span class="token punctuation">"</span></span> <span class="token attr-name">width</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>800<span class="token punctuation">"</span></span> <span class="token attr-name">height</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>800<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>canvas</span><span class="token punctuation">></span></span><br /><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span><span class="token punctuation">></span></span><span class="token script"><span class="token language-javascript"><br /><span class="token keyword">const</span> canvas <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">"#draw"</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> ctx <span class="token operator">=</span> canvas<span class="token punctuation">.</span><span class="token function">getContext</span><span class="token punctuation">(</span><span class="token string">'2d'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />canvas<span class="token punctuation">.</span>width <span class="token operator">=</span> window<span class="token punctuation">.</span>innerWidth<span class="token punctuation">;</span><br />canvas<span class="token punctuation">.</span>height <span class="token operator">=</span> window<span class="token punctuation">.</span>innerHeight<span class="token punctuation">;</span><br />ctx<span class="token punctuation">.</span>strokeStyle <span class="token operator">=</span> <span class="token string">'#BBDA55'</span><span class="token punctuation">;</span><br /></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span></code></pre>
<ul>
<li>Instead of <code>rbg()</code> or hex colors you can set color using hsl(): <code>style="color: hsl(200, 100%, 50%)"</code></li>
</ul>
<h2 id="09---dev-tools-domination">09 - Dev Tools Domination <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#09---dev-tools-domination">#</a></h2>
<p><em>Depending on how familiar one is with the dev tools, some very good tips to be had</em></p>
<ul>
<li>Various methods of <code>console</code>, e.g. <code>console.warn()</code> <code>console.info()</code></li>
<li>Break on a paticular element action, inspect the element, right click on element in dev tools and add a 'break on' -> '<em>choose an option</em>'</li>
<li><code>console.groupCollapsed()</code> group a bunch console logs and display them by default as collapsed.</li>
</ul>
<pre class="language-js"><code class="language-js">console<span class="token punctuation">.</span><span class="token function">groupCollapsed</span><span class="token punctuation">(</span><span class="token string">'my group'</span><span class="token punctuation">)</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">'part1 collapsed'</span><span class="token punctuation">)</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">'part2 collapsed'</span><span class="token punctuation">)</span><br />console<span class="token punctuation">.</span><span class="token function">groupEnd</span><span class="token punctuation">(</span><span class="token string">'end of my group'</span><span class="token punctuation">)</span></code></pre>
<ul>
<li><code>console.count</code> will print the contents and also the count of how many times that content has been logged</li>
<li><code>console.time('X operation took:)</code> put your code you want to time and end it with <code>console.timeEnd(X operation took:)</code></li>
<li><em>Covered in earlier episodes but very handy</em> display in the console in tabular form an array of objects <code>console.table(myArrayOfObjects)</code></li>
</ul>
<h2 id="10---hold-shift-and-check-checkboxes">10 - Hold Shift and Check Checkboxes <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#10---hold-shift-and-check-checkboxes">#</a></h2>
<p><em>I found this one hard - and then returned to it after a 7mth break đą</em></p>
<ul>
<li>No new techniques in this lesson, but a harder application of the pattern of, select DOM elements, define callback function</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// select the divs</span><br /><span class="token keyword">const</span> checkboxes <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'.inbox input[type="checkbox"]'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br /><span class="token keyword">function</span> <span class="token function">handleCheck</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token comment">// check if shiftkey pressed AND box is checked</span><br /> <span class="token keyword">let</span> inBetween <span class="token operator">=</span> <span class="token boolean">false</span><span class="token punctuation">;</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>e<span class="token punctuation">.</span>shiftKey <span class="token operator">&&</span> <span class="token keyword">this</span><span class="token punctuation">.</span>checked<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token comment">// loop through each checkbox</span><br /> checkboxes<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">checkbox</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> <span class="token comment">// these are the outer bounds of the selection</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>checkbox <span class="token operator">===</span> <span class="token keyword">this</span> <span class="token operator">||</span> checkbox <span class="token operator">===</span> lastChecked<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> inBetween <span class="token operator">=</span> <span class="token operator">!</span>inBetween<span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><br /> <span class="token comment">// whilst in between is true - check those boxes</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>inBetween <span class="token operator">===</span> <span class="token boolean">true</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> checkbox<span class="token punctuation">.</span>checked <span class="token operator">=</span> <span class="token boolean">true</span><span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><br /> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><br /> lastChecked <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">// add event listener to handle clicks</span><br />checkboxes<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">checkbox</span> <span class="token operator">=></span> checkbox<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'click'</span><span class="token punctuation">,</span> handleCheck<span class="token punctuation">)</span> <span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="11---custom-video-player">11 - Custom Video Player <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#11---custom-video-player">#</a></h2>
<ul>
<li>I found this easier to fly solo on but also a good few hours of work, you get a video in a page, but the buttons to play/pause, skip etc don't work. The first thing I needed was some documentation on what events I could use on the <code>video</code>. W3 Schools was the first link, but not too helpful. Mozilla to the rescue with a nice list of <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#Events">video events</a>.</li>
<li>You'll also need to know what the <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video#Attributes">video attributes are</a>.</li>
</ul>
<h2 id="12---key-sequence-detection">12 - Key Sequence Detection <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#12---key-sequence-detection">#</a></h2>
<ul>
<li>This was a half hour one, much better after day 11. The aim is to do some action on the detection of some sequence of keystrokes, which TIL is called a <a href="https://en.wikipedia.org/wiki/Konami_Code">Konami_Code</a>.</li>
<li>Solution involved adding an eventlistener to the event <code>keydown</code>, push the keys in an array, and checking for a sequence matching the konami code.</li>
<li>And another amazing tidbit</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// Add this script tag</span><br /><span class="token operator"><</span>script type<span class="token operator">=</span><span class="token string">"text/javascript"</span> src<span class="token operator">=</span><span class="token string">"https://www.cornify.com/js/cornify.js"</span><span class="token operator">></span><span class="token operator"><</span><span class="token operator">/</span>script<span class="token operator">></span><br /><span class="token operator">...</span><br /><span class="token comment">// Then call this function to put corny shit on your webpage</span><br /><span class="token function">cornify_add</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<h2 id="13---slide-in-on-scroll">13 - Slide in on Scroll <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#13---slide-in-on-scroll">#</a></h2>
<p><em>Tip</em> - At this point I'm getting a little tired of the <code>CMD+S</code> to save in VS Code, <code>CMD+TAB</code> to switch applications to the browser and then <code>CMD+SHIFT+R</code> to hard refresh. The <a href="https://github.com/ritwickdey/vscode-live-server">Live Server extension</a> solves that problem - as soon as you save, the page reloads in the browser you launch with it.</p>
<ul>
<li>For the interest of keeping JS30 plain js Wes copies in the <a href="https://lodash.com/docs/4.17.15#debounce"><code>debounce</code> function</a> from the <a href="https://lodash.com/"><code>lodash</code> library</a>. I personally think importing this library and a brief explainer on why lodash is such a widely popular library would have been good.</li>
<li>Calculating how far someone has scrolled on a web page:</li>
<li><code>window.scrollY</code> - how far you have scrolled measured at the top of the window <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/scrollY">(mdn ref)</a>.</li>
<li><code>window.innerHeight</code> - the height of the window, measured using the viewport, i.e. what is visible <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/innerHeight">(mdn ref)</a>.</li>
<li><code>HTMLElement.offsetTop</code> - the distance of the element relative to the offsetParent node, in this tutorial that is the body <a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLelement/offsetParent">(mdn ref)</a>.</li>
<li><code>Element.classList</code> - '... class attributes of the element. This can then be used to manipulate the class list.' <a href="https://developer.mozilla.org/en-US/docs/Web/API/Element/classList">(mdn ref)</a>.</li>
<li>crux of the lesson is utilising the above properties to work out where on a page the user has currently scrolled and to conditionally apply CSS already written for you.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">if</span> <span class="token punctuation">(</span>conditions<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> image<span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token string">'active'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span><br /> image<span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">remove</span><span class="token punctuation">(</span><span class="token string">'active'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span></code></pre>
<p>Simplified CSS</p>
<pre class="language-css"><code class="language-css"><span class="token comment">/* take half a second to transition, initial set to invisible*/</span><br /><span class="token selector">.slide-in</span> <span class="token punctuation">{</span><br /> <span class="token property">opacity</span><span class="token punctuation">:</span> 0<span class="token punctuation">;</span><br /> <span class="token property">transition</span><span class="token punctuation">:</span> all .5s<span class="token punctuation">;</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment">/* on active, set to invisible*/</span><br /><span class="token selector">.slide-in.active</span> <span class="token punctuation">{</span><br /> <span class="token property">opacity</span><span class="token punctuation">:</span> 1<span class="token punctuation">;</span><br /><span class="token punctuation">}</span></code></pre>
<h2 id="14---javascript-references-vs-copying">14 - JavaScript References VS Copying <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#14---javascript-references-vs-copying">#</a></h2>
<ul>
<li>This covers some js idiosyncrasies on reassignment and copying variables, arrays and objects.</li>
<li>Starting off with variables, variables can be reassigned. <code>const</code> is not covered which behaves differently. <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Invalid_const_assignment">mdn ref</a></li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// const</span><br /><span class="token keyword">const</span> age <span class="token operator">=</span> <span class="token number">100</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> age2 <span class="token operator">=</span> age<span class="token punctuation">;</span><br />age2 <span class="token operator">=</span> <span class="token string">"new"</span><span class="token punctuation">;</span> <span class="token comment">// đ TypeError: invalid assignment to const</span><br /><br /><span class="token comment">// both var + let behave the same</span><br /><span class="token keyword">let</span> num <span class="token operator">=</span> <span class="token number">100</span><span class="token punctuation">;</span><br /><span class="token keyword">let</span> num2 <span class="token operator">=</span> num<span class="token punctuation">;</span><br />num2 <span class="token operator">=</span> <span class="token number">200</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>num<span class="token punctuation">,</span> num2<span class="token punctuation">)</span> <span class="token comment">// đ 100 200 </span></code></pre>
<ul>
<li>Arrays are different:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// Make a copy of an array</span><br /><span class="token keyword">const</span> players <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'Wes'</span><span class="token punctuation">,</span> <span class="token string">'Sarah'</span><span class="token punctuation">,</span> <span class="token string">'Ryan'</span><span class="token punctuation">,</span> <span class="token string">'Poppy'</span><span class="token punctuation">]</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> team <span class="token operator">=</span> players<span class="token punctuation">;</span><br />team<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">"new"</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>players<span class="token punctuation">,</span> team<span class="token punctuation">)</span><br /><span class="token comment">// [ "Wes", "new", "Ryan", "Poppy" ]</span><br /><span class="token comment">// [ "Wes", "new", "Ryan", "Poppy" ]</span><br /><span class="token comment">// đ changes both `team` and `players` because `team` is just a pointer to `players`</span><br /><br /><span class="token comment">// solution</span><br /><span class="token keyword">const</span> players <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'Wes'</span><span class="token punctuation">,</span> <span class="token string">'Sarah'</span><span class="token punctuation">,</span> <span class="token string">'Ryan'</span><span class="token punctuation">,</span> <span class="token string">'Poppy'</span><span class="token punctuation">]</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> team <span class="token operator">=</span> players<span class="token punctuation">.</span><span class="token function">slice</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />team<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">"new"</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>players<span class="token punctuation">,</span> team<span class="token punctuation">)</span><br /><span class="token comment">// [ "Wes", "Sarah", "Ryan", "Poppy" ]</span><br /><span class="token comment">// [ "Wes", "new", "Ryan", "Poppy" ]</span><br /><span class="token comment">// đ could also do `const team = [...players];`</span></code></pre>
<ul>
<li>Objects also need a special method to copy, however, the following will only do one level deep of the object.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// with Objects</span><br /><span class="token keyword">const</span> person <span class="token operator">=</span> <span class="token punctuation">{</span><br /> <span class="token literal-property property">name</span><span class="token operator">:</span> <span class="token string">'Wes Bos'</span><span class="token punctuation">,</span><br /> <span class="token literal-property property">age</span><span class="token operator">:</span> <span class="token number">80</span><br /><span class="token punctuation">}</span><span class="token punctuation">;</span><br /><br /><span class="token comment">// attempt to make a copy:</span><br /><span class="token keyword">const</span> captain <span class="token operator">=</span> person<span class="token punctuation">;</span><br />captain<span class="token punctuation">.</span>number <span class="token operator">=</span> <span class="token number">99</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>person<span class="token punctuation">)</span><br /><span class="token comment">// Object { name: "Wes Bos", age: 80, number: 99 }</span><br /><span class="token comment">// đ this has modified the original</span><br /><br /><span class="token comment">// solution</span><br /><span class="token keyword">const</span> captain2 <span class="token operator">=</span> Object<span class="token punctuation">.</span><span class="token function">assign</span><span class="token punctuation">(</span><span class="token punctuation">{</span><span class="token punctuation">}</span><span class="token punctuation">,</span> person<span class="token punctuation">)</span><br />captain2<span class="token punctuation">.</span>number <span class="token operator">=</span> <span class="token number">100</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>person<span class="token punctuation">)</span><br /><span class="token comment">// Object { name: "Wes Bos", age: 80, number: 99 } đ</span></code></pre>
<ul>
<li>lodash has a <a href="https://lodash.com/docs/4.17.15#cloneDeep">cloneDeep method</a> to do a full object copy</li>
</ul>
<h2 id="15---local-storage-and-event-delegation">15 - Local Storage and Event Delegation <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#15---local-storage-and-event-delegation">#</a></h2>
<p>N.b - this one is big, video is over 30mins.</p>
<ul>
<li>This covers topics that seem like a precursor to react, storing state and using parent elements to instruct child elements.</li>
<li>When working with forms and wanting to perform actions upon form submission, the appropriate event to listen for is <code>submit</code>. By default the submit form event will reload the page. To prevent this use <code>e.preventDefault()</code> in the handler.</li>
<li>Persist data on the client by using local storage, a key value store, where your value needs to be a string</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// args are key value</span><br />localStorage<span class="token punctuation">.</span><span class="token function">setItem</span><span class="token punctuation">(</span><span class="token string">'items'</span><span class="token punctuation">,</span> <span class="token constant">JSON</span><span class="token punctuation">.</span><span class="token function">stringify</span><span class="token punctuation">(</span>myObject<span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<ul>
<li>If your data is not what you expect and you see <code>[object Object]</code> instead - you most likely have tried to convert an object to a string <code>Object().toString()</code>. There are <a href="https://stackoverflow.com/a/25419538/3691003">other variations of this behaviour</a> - because everything in javascript is an object.</li>
</ul>
<h2 id="16---css-text-shadow-mouse-move-effect">16 - CSS Text Shadow Mouse Move Effect <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#16---css-text-shadow-mouse-move-effect">#</a></h2>
<ul>
<li>Wes takes a weird approach of grabbing the hero elements offset x and y values and the event's offset x and y values. When the mousemove event occurs over another element on the page the x and  y values are then calculated from <a href="https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/offsetX"><em>the the padding edge of the target node</em></a>. A much simpler approach is taking the clientX and clientY values. Wes approach:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> <span class="token punctuation">{</span> <span class="token literal-property property">offsetWidth</span><span class="token operator">:</span> width<span class="token punctuation">,</span> <span class="token literal-property property">offsetHeight</span><span class="token operator">:</span> height<span class="token punctuation">}</span> <span class="token operator">=</span> hero<br /><span class="token keyword">let</span> <span class="token punctuation">{</span> <span class="token literal-property property">offsetX</span><span class="token operator">:</span> x<span class="token punctuation">,</span> <span class="token literal-property property">offsetY</span><span class="token operator">:</span> y <span class="token punctuation">}</span> <span class="token operator">=</span> e<span class="token punctuation">;</span><br /><br /><span class="token comment">// uses this if statement to handle when the target is the whole screen, to then add on the screen offset</span><br /><span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token keyword">this</span> <span class="token operator">!==</span> e<span class="token punctuation">.</span>target<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> x <span class="token operator">=</span> x <span class="token operator">+</span> e<span class="token punctuation">.</span>target<span class="token punctuation">.</span>offsetLeft<span class="token punctuation">;</span><br /> y <span class="token operator">=</span> y <span class="token operator">+</span> e<span class="token punctuation">.</span>target<span class="token punctuation">.</span>offsetTop<span class="token punctuation">;</span><br /><span class="token punctuation">}</span></code></pre>
<p>Much easier to use the following:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">let</span> <span class="token punctuation">{</span> <span class="token literal-property property">clientX</span><span class="token operator">:</span> x<span class="token punctuation">,</span> <span class="token literal-property property">clientY</span><span class="token operator">:</span> y<span class="token punctuation">}</span> <span class="token operator">=</span> e<span class="token punctuation">;</span></code></pre>
<ul>
<li>A graphic effect is applied using the following</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> walk <span class="token operator">=</span> <span class="token number">100</span> <span class="token comment">//px</span><br /><span class="token keyword">const</span> xWalk <span class="token operator">=</span> <span class="token punctuation">(</span>x <span class="token operator">/</span> width <span class="token operator">*</span> walk<span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token punctuation">(</span>walk <span class="token operator">/</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> yWalk <span class="token operator">=</span> <span class="token punctuation">(</span>y <span class="token operator">/</span> height <span class="token operator">*</span> walk<span class="token punctuation">)</span> <span class="token operator">-</span> <span class="token punctuation">(</span>walk <span class="token operator">/</span> <span class="token number">2</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br />text<span class="token punctuation">.</span>style<span class="token punctuation">.</span>textShadow <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>xWalk<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px </span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>yWalk<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px 0 rgba(255,0,255,0.7)</span><span class="token template-punctuation string">`</span></span></code></pre>
<h2 id="17---sorting-band-names-without-articles">17 - Sorting Band Names without articles <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#17---sorting-band-names-without-articles">#</a></h2>
<p>N.b. An 'article', being one of the grammatical articles. In this coding example, The definite article 'the' or the indefinite articles 'an' or 'a'.</p>
<ul>
<li>Wes takes a really nice and succinct approach here utilising the <a href="https://lukesingham.com/summary-of-javascript30/Array.prototype.sort()">Array.prototype.sort()</a>. However I prefer the mdn docs approach which is a little more verbose:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">let</span> sorted <span class="token operator">=</span> bandsArray<span class="token punctuation">.</span><span class="token function">sort</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">a<span class="token punctuation">,</span>b</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">strip</span><span class="token punctuation">(</span>a<span class="token punctuation">)</span> <span class="token operator">></span> <span class="token function">strip</span><span class="token punctuation">(</span>b<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> <span class="token number">1</span><br /> <span class="token punctuation">}</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token function">strip</span><span class="token punctuation">(</span>a<span class="token punctuation">)</span> <span class="token operator"><</span> <span class="token function">strip</span><span class="token punctuation">(</span>b<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> <span class="token operator">-</span><span class="token number">1</span><br /> <span class="token punctuation">}</span><br /> <span class="token keyword">return</span> <span class="token number">0</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<ul>
<li>To append this to a div, e.g. <a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/ul">the unordered list <code><ul></code></a>:</li>
</ul>
<pre class="language-js"><code class="language-js">document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">'#ulDivId'</span><span class="token punctuation">)</span><span class="token punctuation">.</span>innerHTML <span class="token operator">=</span><br /> sortedArray<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">item</span> <span class="token operator">=></span><br /> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string"><li></span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>item<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string"></li></span><span class="token template-punctuation string">`</span></span><br /> <span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="18---tally-string-times-with-reduce">18 - Tally String Times with Reduce <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#18---tally-string-times-with-reduce">#</a></h2>
<p>N.b. simple if you have some knowledge of map <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/Reduce">reduce functions</a>.<br />
My answer pretty similar to Wes's except instead of using a forEach to append the values into an array, he used a much nicer <code>const timeNodes = Array.from(document.querySelectorAll('[data-time]'));</code>. My answer:</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> videos <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'li'</span><span class="token punctuation">)</span><br /><span class="token keyword">const</span> times <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token punctuation">]</span><span class="token punctuation">;</span><br /><br />videos<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">vid</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> times<span class="token punctuation">.</span><span class="token function">push</span><span class="token punctuation">(</span>vid<span class="token punctuation">.</span>dataset<span class="token punctuation">.</span>time<span class="token punctuation">)</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br /><span class="token keyword">const</span> totalTime <span class="token operator">=</span> times<span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">x</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> <span class="token punctuation">[</span>min<span class="token punctuation">,</span> sec<span class="token punctuation">]</span> <span class="token operator">=</span> x<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token string">':'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token keyword">return</span> <span class="token function">Number</span><span class="token punctuation">(</span>min<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">60</span> <span class="token operator">+</span> <span class="token function">Number</span><span class="token punctuation">(</span>sec<span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">reduce</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">acc<span class="token punctuation">,</span> val</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> <span class="token keyword">return</span> acc <span class="token operator">+</span> val<span class="token punctuation">;</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><br /><br /><span class="token keyword">const</span> mins <span class="token operator">=</span> Math<span class="token punctuation">.</span><span class="token function">trunc</span><span class="token punctuation">(</span>totalTime <span class="token operator">/</span> <span class="token number">60</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> secs <span class="token operator">=</span> totalTime <span class="token operator">%</span> <span class="token number">60</span><span class="token punctuation">;</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>mins<span class="token punctuation">,</span> <span class="token string">':'</span><span class="token punctuation">,</span> secs<span class="token punctuation">)</span><br /><span class="token comment">// 298 : 58 - I should handle hours but meh đ¤ˇâ</span></code></pre>
<h2 id="19---unreal-webcam-fun">19 - Unreal Webcam Fun <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#19---unreal-webcam-fun">#</a></h2>
<p><em>N.b. This was a slower start with a few issues to iron out cover in the first 3 bullet points...</em></p>
<ul>
<li>The files so far have been read locally into the browser, such that in the address bar you'll find the file path <code>file://</code> instead of <code>https://</code>. However the use of the webcam requires a server due to accessing the computer's camera and the security requirement that the website is a secure origin, <code>https</code> or <code>localhost</code>. Therefore we need to spin up a local server. If developing with <a href="https://lukesingham.com/summary-of-javascript30/#13-slide-in-on-scroll">VSCode with the Live Server extension</a> this has already been done for you. I've used Python's <code>http.server</code> (used to be called <code>SimpleHTTPServer</code>). <a href="https://www.browsersync.io/">Browsersync</a> Â seems like the JavaScript equilavent.</li>
<li>I got some intitial <code>404</code> errors where the html template pointed to a file that no longer exists. I fixed this by downloading a camera sound locally.</li>
</ul>
<pre class="language-html"><code class="language-html"><span class="token comment"><!-- 404 --></span><br /><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>audio</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>snap<span class="token punctuation">"</span></span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>https://wesbos.com/demos/photobooth/snap.mp3<span class="token punctuation">"</span></span> <span class="token attr-name">hidden</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>audio</span><span class="token punctuation">></span></span><br /><span class="token comment"><!-- replace with a local sound file --></span><br /><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>audio</span> <span class="token attr-name">class</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>snap<span class="token punctuation">"</span></span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>camera_sound.mp3<span class="token punctuation">"</span></span> <span class="token attr-name">hidden</span><span class="token punctuation">></span></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>audio</span><span class="token punctuation">></span></span></code></pre>
<ul>
<li>The video tutorial for this instructs you to use <a href="https://developer.mozilla.org/en-US/docs/Web/API/URL/createObjectURL"><code>createObjectURL</code> which has now been deprecated</a>. I wasted some time here, checking I had done things correctly, searching for the error until I saw a note in the solution on this to use <code>[srcObject](https://developer.mozilla.org/en-US/docs/Web/API/HTMLMediaElement/srcObject)</code>.</li>
<li>To access a user's web camera use <a href="https://developer.mozilla.org/en-US/docs/Web/API/Navigator/mediaDevices/getUserMedia"><code>[navigator.mediaDevices.getUserMedia()](https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia)</code></a>. This returns a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise"><code>Promise</code></a> that resolves to a <a href="https://developer.mozilla.org/en-US/docs/Web/API/MediaStream"><code>MediaStream</code></a> object.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">function</span> <span class="token function">getVideo</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> navigator<span class="token punctuation">.</span>mediaDevices<span class="token punctuation">.</span><span class="token function">getUserMedia</span><span class="token punctuation">(</span><span class="token punctuation">{</span><br /> <span class="token literal-property property">video</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span><br /> <span class="token literal-property property">audio</span><span class="token operator">:</span> <span class="token boolean">false</span><br /> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">then</span><span class="token punctuation">(</span><span class="token parameter">localMediaStream</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> video<span class="token punctuation">.</span>srcObject <span class="token operator">=</span> localMediaStream<span class="token punctuation">;</span><br /> video<span class="token punctuation">.</span><span class="token function">play</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><span class="token punctuation">)</span><br /> <span class="token punctuation">.</span><span class="token function">catch</span><span class="token punctuation">(</span><br /> <span class="token parameter">err</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> console<span class="token punctuation">.</span><span class="token function">error</span><span class="token punctuation">(</span><span class="token string">"Whoops Error: "</span><span class="token punctuation">,</span> err<span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span></code></pre>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Navigator">The Navigator interface</a> enables you to query some interesting properties of the user e.g. <code>[geolocation](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/geolocation)</code>, <code>[connection](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/connection)</code> Â and <code>[webdriver](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/webdriver)</code>.</li>
<li>If you want to pause the execution and enter the debugger at a particular point of your code, just write <code>debugger</code>.</li>
</ul>
<pre class="language-js"><code class="language-js">console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>pixels<span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">debugger</span></code></pre>
<h2 id="20---native-speech-recognition">20 - Native Speech Recognition <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#20---native-speech-recognition">#</a></h2>
<p>N.b. This was definitely a frustrating first 5mins of the tutorial, where if you're following along you don't get anything like what Wes is getting. This is because he is demonstrating the finished app on the left half of the screen whilst coding from scratch the app. It's not until 5mins when switches that he realises that it's not working.</p>
<ul>
<li>This only works on Chrome, not Firefox.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> recognition <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">SpeechRecognition</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />recognition<span class="token punctuation">.</span>interimResults <span class="token operator">=</span> <span class="token boolean">true</span><span class="token punctuation">;</span><br />recognition<span class="token punctuation">.</span>lang <span class="token operator">=</span> <span class="token string">'en-US'</span><span class="token punctuation">;</span><br /><br /><span class="token keyword">let</span> p <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">createElement</span><span class="token punctuation">(</span><span class="token string">'p'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token keyword">const</span> words <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">'.words'</span><span class="token punctuation">)</span><br />words<span class="token punctuation">.</span><span class="token function">appendChild</span><span class="token punctuation">(</span>p<span class="token punctuation">)</span><br /><br />recognition<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'result'</span><span class="token punctuation">,</span> <span class="token parameter">e</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> transcript <span class="token operator">=</span> Array<span class="token punctuation">.</span><span class="token function">from</span><span class="token punctuation">(</span>e<span class="token punctuation">.</span>results<span class="token punctuation">)</span><br /> <span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /> <span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token parameter">result</span> <span class="token operator">=></span> result<span class="token punctuation">.</span>transcript<span class="token punctuation">)</span><br /> <span class="token punctuation">.</span><span class="token function">join</span><span class="token punctuation">(</span><span class="token string">''</span><span class="token punctuation">)</span><br /><br /> p<span class="token punctuation">.</span>textContent <span class="token operator">=</span> transcript<span class="token punctuation">;</span><br /> <span class="token comment">// create a new paragraph if the speaking has finished</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>e<span class="token punctuation">.</span>results<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">.</span>isFinal<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> p <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">createElement</span><span class="token punctuation">(</span><span class="token string">'p'</span><span class="token punctuation">)</span><br /> words<span class="token punctuation">.</span><span class="token function">appendChild</span><span class="token punctuation">(</span>p<span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><br /> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>transcript<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// the results</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><br /><br />recognition<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'end'</span><span class="token punctuation">,</span> recognition<span class="token punctuation">.</span>start<span class="token punctuation">)</span><br />recognition<span class="token punctuation">.</span><span class="token function">start</span><span class="token punctuation">(</span><span class="token punctuation">)</span></code></pre>
<h2 id="21---geolocation-based-speedometer-and-compass">21 - Geolocation based Speedometer and Compass <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#21---geolocation-based-speedometer-and-compass">#</a></h2>
<ul>
<li>This requires using Xcode to simulate someone using there phone whilst moving around.</li>
</ul>
<p><img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-09-at-20.19.21.png" alt="" /></p>
<ul>
<li>Once the simulator boots you'll have an iphone on screen. Then turn on a 'City Run'.</li>
</ul>
<p><img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-09-at-20.20.36.png" alt="" /></p>
<ul>
<li>Then open Safari to connect the dev tools to the simulator iphone</li>
</ul>
<p><img src="https://lukesingham.com/content/images/2020/05/Screen-Shot-2020-05-09-at-20.35.50.png" alt="" /></p>
<ul>
<li>And to access the geo data:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// the request to use the geolocation of the client will ask for permissions from user</span><br />navigator<span class="token punctuation">.</span>geolocation<span class="token punctuation">.</span><span class="token function">watchPosition</span><span class="token punctuation">(</span><span class="token punctuation">(</span><span class="token parameter">data</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>data<span class="token punctuation">)</span> <span class="token comment">// see the data object</span><br /><br /> <span class="token comment">// update dom elements with the data</span><br /> speed<span class="token punctuation">.</span>textContent <span class="token operator">=</span> data<span class="token punctuation">.</span>coords<span class="token punctuation">.</span>speed<span class="token punctuation">;</span><br /> arrow<span class="token punctuation">.</span>style<span class="token punctuation">.</span>transform <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">rotate(</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>data<span class="token punctuation">.</span>coords<span class="token punctuation">.</span>heading<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">deg)</span><span class="token template-punctuation string">`</span></span><br /><span class="token punctuation">}</span><span class="token punctuation">,</span> <span class="token punctuation">(</span><span class="token parameter">err</span><span class="token punctuation">)</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> <span class="token comment">// if they reject the permissions</span><br /> console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span>err<span class="token punctuation">)</span><br /> <span class="token function">alert</span><span class="token punctuation">(</span><span class="token string">"you privacy nut"</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span></code></pre>
<h2 id="22---follow-along-links">22 - Follow Along Links <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#22---follow-along-links">#</a></h2>
<p><em>N.b. lesson applies an element that hovers over the link the mouse is currently positioned over.</em></p>
<ul>
<li>Method: Create a 'highlight' DOM element, and transition it based on the <code>mouseenter</code> event on any <code><a></code> divs.</li>
<li>Create event listener</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> triggers <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'a'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />triggers<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">a</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> a<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'mouseenter'</span><span class="token punctuation">,</span> addHighlight<span class="token punctuation">)</span><br /><span class="token punctuation">}</span></code></pre>
<ul>
<li>To get the coodinates and size of the current <code>a</code> div, use <code>[getBoundingClientRect()](https://developer.mozilla.org/en-US/docs/Web/API/Element/getBoundingClientRect)</code> in the callback.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">function</span> <span class="token function">addHighlight</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> domDimension <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">getBoundingClientRect</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token punctuation">}</span></code></pre>
<ul>
<li><code>[translate(Xpx, Ypx)](https://developer.mozilla.org/en-US/docs/Web/CSS/transform-function/translate)</code> - CSS function that repositions an element. Use this in the call back to transition the highlight element. Wes uses the <code>top</code> and <code>left</code> properties, but X and Y works just the same and makes more sense to me.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> highlight <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">createElement</span><span class="token punctuation">(</span><span class="token string">'span'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />highlight<span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token string">'highlight'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />document<span class="token punctuation">.</span>body<span class="token punctuation">.</span><span class="token function">appendChild</span><span class="token punctuation">(</span>highlight<span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br /><span class="token keyword">function</span> <span class="token function">addHighlight</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token operator">...</span><br /> highlight<span class="token punctuation">.</span>style<span class="token punctuation">.</span>width <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>domDimension<span class="token punctuation">.</span>width<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px</span><span class="token template-punctuation string">`</span></span><br /> highlight<span class="token punctuation">.</span>style<span class="token punctuation">.</span>height <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>domDimension<span class="token punctuation">.</span>height<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px</span><span class="token template-punctuation string">`</span></span><br /> highlight<span class="token punctuation">.</span>style<span class="token punctuation">.</span>transform <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token string">translate(</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>domDimension<span class="token punctuation">.</span>x<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px,</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>domDimension<span class="token punctuation">.</span>y<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px)</span><span class="token template-punctuation string">`</span></span><br /><span class="token punctuation">}</span></code></pre>
<h2 id="23---speech-synthesis">23 - Speech Synthesis <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#23---speech-synthesis">#</a></h2>
<p>N.b. - Firefox wasn't returning the voice objects from <code>speechSynthesis.getVoices()</code> at page load - switched to Chrome for this one.</p>
<ul>
<li>Use the experimental <a href="https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis">SpeechSynthesis API</a>.</li>
<li>When filtering an array based on a singular user selection, I naturally wanted to <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/filter"><code>filter()</code></a> however this returns an array which you will have to select an element from. However, <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/find"><code>find()</code></a> will return the first element that meets the match critera.</li>
</ul>
<pre class="language-js"><code class="language-js">msg<span class="token punctuation">.</span>voice <span class="token operator">=</span> voices<span class="token punctuation">.</span><span class="token function">filter</span><span class="token punctuation">(</span><span class="token parameter">v</span> <span class="token operator">=></span> v<span class="token punctuation">.</span>name <span class="token operator">===</span> <span class="token keyword">this</span><span class="token punctuation">.</span>value<span class="token punctuation">)</span><span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><br /><span class="token comment">// vs find</span><br />msg<span class="token punctuation">.</span>voice <span class="token operator">=</span> voices<span class="token punctuation">.</span><span class="token function">find</span><span class="token punctuation">(</span><span class="token parameter">v</span> <span class="token operator">=></span> v<span class="token punctuation">.</span>name <span class="token operator">===</span> <span class="token keyword">this</span><span class="token punctuation">.</span>value<span class="token punctuation">)</span></code></pre>
<h2 id="24---sticky-nav">24 - Sticky Nav <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#24---sticky-nav">#</a></h2>
<p><em>Fix a navbar to the top as you scroll down.</em></p>
<ul>
<li>Add an event listenr on the navbar and for the callback place a control statement to detect when the <em>fixing</em> of the nav should apply.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> navbar <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">'#main'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token comment">// top of navbar location</span><br /><span class="token keyword">const</span> navHeight <span class="token operator">=</span> navbar<span class="token punctuation">.</span>offsetTop<span class="token punctuation">;</span><br /><br /><span class="token keyword">function</span> <span class="token function">moveNavBar</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">const</span> scrollAmt <span class="token operator">=</span> window<span class="token punctuation">.</span>scrollY<span class="token punctuation">;</span><br /><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>scrollAmt <span class="token operator">>=</span> navHeight<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> document<span class="token punctuation">.</span>body<span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token string">'fixed-nav'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> document<span class="token punctuation">.</span>body<span class="token punctuation">.</span>style<span class="token punctuation">.</span>paddingTop <span class="token operator">=</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>navbar<span class="token punctuation">.</span>offsetHeight<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px</span><span class="token template-punctuation string">`</span></span><br /><br /> <span class="token punctuation">}</span> <span class="token keyword">else</span> <span class="token punctuation">{</span><br /> document<span class="token punctuation">.</span>body<span class="token punctuation">.</span>style<span class="token punctuation">.</span>paddingTop <span class="token operator">=</span> <span class="token number">0</span><br /> document<span class="token punctuation">.</span>body<span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">remove</span><span class="token punctuation">(</span><span class="token string">'fixed-nav'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><br />document<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'scroll'</span><span class="token punctuation">,</span> _<span class="token punctuation">.</span><span class="token function">debounce</span><span class="token punctuation">(</span>moveNavBar<span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<ul>
<li>To find the hight of a DOM element there looks to be to properties you could use. Here's the <a href="https://stackoverflow.com/a/4106585/3691003">difference between clientHeight vs offsetHeight,</a> go with offsetHeight.</li>
</ul>
<h2 id="25---event-capture%2C-propagation%2C-bubbling-and-once">25 - Event Capture, Propagation, Bubbling and Once <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#25---event-capture%2C-propagation%2C-bubbling-and-once">#</a></h2>
<p><em>This is a very quick practical on the concept of bubbling up and propagation. Requires some extra research if new to the concept.</em></p>
<ul>
<li><a href="https://www.sitepoint.com/event-bubbling-javascript/">Event propagation</a> - is the blanket term for both event bubbling and event capturing.</li>
<li>If you have 3 divs nested and place an event listener on those divs. Then load the page and click on the inner most div and log which div has been clicked on. The browser will show all divs from inner most to outer most <code><body></code> as having been clicked on - this is <em>bubbling up.</em></li>
<li>Behind the scenes the browser is first capturing the event by 'trickling down' the DOM hierarchy until the target on which the event occurred i.e. the click, is reached.</li>
<li>To make use of the third argument to an event listener:</li>
</ul>
<pre class="language-js"><code class="language-js">divs<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">div</span> <span class="token operator">=></span> <span class="token punctuation">{</span><br /> div<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'click'</span><span class="token punctuation">,</span> logText<span class="token punctuation">,</span> <span class="token punctuation">{</span><br /> <span class="token literal-property property">capture</span><span class="token operator">:</span> <span class="token boolean">true</span><span class="token punctuation">,</span> <span class="token comment">// 'trickle' down events</span><br /> <span class="token literal-property property">once</span><span class="token operator">:</span> <span class="token boolean">true</span> <span class="token comment">// removeEventLister - unbind once event occurs once</span><br /> <span class="token punctuation">}</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="26---stripe-follow-along-dropdown">26 - Stripe Follow Along Dropdown <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#26---stripe-follow-along-dropdown">#</a></h2>
<p><em>Extension of 22.</em></p>
<ul>
<li>I was trying to access the child nodes of <code>this</code> within the context of a callback by traversing the tree e.g. <code>this.children.item(0)</code>. But of course, you can just <code>this.querySelector</code> to query the tree of <code>this</code>.</li>
<li>The crux of the logic relied on adding and removing CSS classes. The pared down version:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> triggers <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelectorAll</span><span class="token punctuation">(</span><span class="token string">'.cool > li'</span><span class="token punctuation">)</span><br /><span class="token keyword">const</span> background <span class="token operator">=</span> document<span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">'.dropdownBackground'</span><span class="token punctuation">)</span><br /><br /><span class="token keyword">function</span> <span class="token function">handleEnter</span><span class="token punctuation">(</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> background<span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token string">'open'</span><span class="token punctuation">)</span><br /> <span class="token keyword">this</span><span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token string">'trigger-enter'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token keyword">this</span><span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">add</span><span class="token punctuation">(</span><span class="token string">'trigger-enter-active'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /><br /> <span class="token comment">// don't forget you can querySelector on `this`</span><br /> <span class="token keyword">const</span> dropdown <span class="token operator">=</span> <span class="token keyword">this</span><span class="token punctuation">.</span><span class="token function">querySelector</span><span class="token punctuation">(</span><span class="token string">'.dropdown'</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> <span class="token keyword">const</span> dropdownDim <span class="token operator">=</span> dropdown<span class="token punctuation">.</span><span class="token function">getBoundingClientRect</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br /> background<span class="token punctuation">.</span>style<span class="token punctuation">.</span><span class="token function">setProperty</span><span class="token punctuation">(</span><span class="token string">'width'</span><span class="token punctuation">,</span> <span class="token template-string"><span class="token template-punctuation string">`</span><span class="token interpolation"><span class="token interpolation-punctuation punctuation">${</span>dropdownDim<span class="token punctuation">.</span>width<span class="token interpolation-punctuation punctuation">}</span></span><span class="token string">px</span><span class="token template-punctuation string">`</span></span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token keyword">function</span> <span class="token function">handleLeave</span><span class="token punctuation">(</span><span class="token parameter">params</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token keyword">this</span><span class="token punctuation">.</span>classList<span class="token punctuation">.</span><span class="token function">remove</span><span class="token punctuation">(</span><span class="token string">'trigger-enter'</span><span class="token punctuation">,</span> <span class="token string">'trigger-enter-active'</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br />triggers<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">trigger</span> <span class="token operator">=></span> trigger<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'mouseenter'</span><span class="token punctuation">,</span> handleEnter<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span><br />triggers<span class="token punctuation">.</span><span class="token function">forEach</span><span class="token punctuation">(</span><span class="token parameter">trigger</span> <span class="token operator">=></span> trigger<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'mouseleave'</span><span class="token punctuation">,</span> handleLeave<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h2 id="27---click-and-drag-to-scroll">27 - Click and Drag to Scroll <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#27---click-and-drag-to-scroll">#</a></h2>
<ul>
<li>Lesson centres on making use of 4 mouse events <code>mousedown</code>, <code>mouseleave</code>, <code>mouseup</code> and <code>mousemove</code> to build the controls around clicking and dragging.</li>
<li>As flexible as you think <code>let</code> is, you will encounter errors if you try to redeclare a variable created with <code>let</code>.</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token comment">// đ errors</span><br /><span class="token keyword">let</span> yourVar <span class="token operator">=</span> <span class="token boolean">true</span><span class="token punctuation">;</span><br /><span class="token keyword">let</span> yourVar <span class="token operator">=</span> <span class="token boolean">false</span><span class="token punctuation">;</span><br /><br /><span class="token comment">// â happy days</span><br /><span class="token keyword">let</span> yourVar <span class="token operator">=</span> <span class="token boolean">true</span><span class="token punctuation">;</span><br />yourVar <span class="token operator">=</span> <span class="token boolean">false</span><span class="token punctuation">;</span></code></pre>
<ul>
<li>You can <code>console.log</code> the var name and value at the same time by wrapping it in an object:</li>
</ul>
<pre class="language-js"><code class="language-js"><span class="token keyword">const</span> yourVarName <span class="token operator">=</span> <span class="token number">10</span><span class="token punctuation">;</span><br /><span class="token comment">// slow way</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token string">'yourVarName'</span><span class="token punctuation">,</span> yourVarName<span class="token punctuation">)</span><span class="token punctuation">;</span><br /><span class="token comment">// fast way</span><br />console<span class="token punctuation">.</span><span class="token function">log</span><span class="token punctuation">(</span><span class="token punctuation">{</span>yourVarName<span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/MouseEvent/pageX">MouseEvent.pageX</a> - "returns the X (horizontal) coordinate (in pixels) at which the mouse was clicked, relative to the left edge of the entire document. This includes any portion of the document not currently visible."</li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/offsetLeft">HTMLElement.offsetLeft</a> - "The <strong><code>HTMLElement.offsetLeft</code></strong> read-only property returns the number of pixels that the <em>upper left corner</em> of the current element is offset to the left within the <a href="https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement/offsetParent"><code>HTMLElement.offsetParent</code></a> node."</li>
</ul>
<h2 id="28---video-speed-controller-ui">28 - Video Speed Controller UI <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#28---video-speed-controller-ui">#</a></h2>
<p><em>This is a small lesson at ~9mins and doesn't introduce any new content. Â Mostly a simpler version of Lesson 11.</em></p>
<h2 id="29---countdown-clock">29 - Countdown Clock <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#29---countdown-clock">#</a></h2>
<p><em>Not a lot new in this one - but it is a fun use of the skills learnt to date in making a countdown clock.</em></p>
<ul>
<li><code>setInteval</code> will continue to run. So, if using it within a function that gets called multiple times, you may need to have a <code>clearInterval()</code> call to clear any existing process.</li>
<li>If an html element has a name attribute, it can be accessed directly from the document.</li>
</ul>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>form</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>customForm<span class="token punctuation">"</span></span> <span class="token attr-name">id</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>custom<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><br /> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>input</span> <span class="token attr-name">type</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>text<span class="token punctuation">"</span></span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>minutes<span class="token punctuation">"</span></span> <span class="token attr-name">placeholder</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>Enter Minutes<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><br /><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>form</span><span class="token punctuation">></span></span></code></pre>
<pre class="language-js"><code class="language-js">document<span class="token punctuation">.</span>customForm<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'submit'</span><span class="token punctuation">,</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span><span class="token punctuation">{</span><span class="token operator">...</span><span class="token punctuation">}</span><span class="token punctuation">)</span><br /><span class="token comment">// Nested elements can be accessed too</span><br />document<span class="token punctuation">.</span>customForm<span class="token punctuation">.</span>minutes<span class="token punctuation">.</span><span class="token function">addEventListener</span><span class="token punctuation">(</span><span class="token string">'onhover'</span><span class="token punctuation">,</span> <span class="token keyword">function</span><span class="token punctuation">(</span><span class="token parameter">e</span><span class="token punctuation">)</span><span class="token punctuation">{</span><span class="token operator">...</span><span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<h2 id="30---whack-a-mole-game">30 - Whack A Mole Game <a class="direct-link" href="https://lukesingham.com/summary-of-javascript30/#30---whack-a-mole-game">#</a></h2>
<p><em>As the title suggests this is a bit of a fun simple game that is using the concepts learnt in previous lessons - a good way to end the course.</em></p>
<ul>
<li>On event's <a href="https://developer.mozilla.org/en-US/docs/Web/API/Event/isTrusted">there is a <code>isTrusted</code> boolean property</a> that can be utilised to ensure the action on the web page is user generated.</li>
</ul>
Setting up Pi-Hole on a Raspberry Pi Zero W using SSH2020-04-26T00:00:00Zhttps://lukesingham.com/pi-hole/<p><strong>Aim:</strong> đ Block all advertisements for all devices on my home WiFi network</p>
<p><strong>Estimated time to complete:</strong> Best part of a morning â° depending on how well things go for you i.e. 1.5-3hrs</p>
<p>When the Raspberry Pi Zero came out it hit front pages as the <a href="https://money.cnn.com/2015/11/27/technology/raspberry-pi-zero/index.html">$5 computer and sold out instantly</a>. Here are the specs for the Raspberry Pi Zero:</p>
<ul>
<li>1GHz, single-core CPU</li>
<li>512MB RAM</li>
<li>Mini HDMI and USB On-The-Go ports</li>
<li>Micro USB power</li>
<li>HAT-compatible 40-pin header</li>
<li>Composite video and reset headers</li>
<li>CSI camera connector</li>
</ul>
<p>Additional with the Raspberry Pi Zero W:</p>
<ul>
<li>802.11 b/g/n wireless LAN</li>
<li>Bluetooth 4.1</li>
<li>Bluetooth Low Energy (BLE)</li>
</ul>
<p>It genuinely is something to be excited about đ. I have the slightly more expensive 'W' version - it's twice the price but I'd certainly recommend it for connectivity reasons, otherwise you'll be a little stuck for internet!</p>
<h2 id="step-1%3A-the-ingredients-%F0%9F%A4%A4">Step 1: The Ingredients 𤤠<a class="direct-link" href="https://lukesingham.com/pi-hole/#step-1%3A-the-ingredients-%F0%9F%A4%A4">#</a></h2>
<p><img src="https://lukesingham.com/content/images/2020/04/raspberry-pi-zero-w-annotated-2.jpg" alt="" /><br />
<strong>Hardware</strong></p>
<ul>
<li><a href="https://amzn.to/3aZjGjz">R Pi 0 W - $18 USD</a></li>
<li><a href="https://amzn.to/3o1qoJH">R Pi Power Supply - $10 USD)</a></li>
<li><a href="https://amzn.to/3rOdUav">NOOBs 32GB micro sd card - $18 USD</a></li>
</ul>
<p>Total = $46 (~ÂŁ26)</p>
<p><strong>Software</strong></p>
<ul>
<li><a href="https://www.raspbian.org/">Raspbian</a></li>
<li><a href="https://pi-hole.net/">Pi-Hole</a></li>
</ul>
<h2 id="step-2%3A-setting-up-the-micro-sd-with-the-operating-system-and-connectivity-%F0%9F%91%A8%E2%80%8D%F0%9F%92%BB">Step 2: Setting up the Micro SD with the operating system and connectivity đ¨âđť <a class="direct-link" href="https://lukesingham.com/pi-hole/#step-2%3A-setting-up-the-micro-sd-with-the-operating-system-and-connectivity-%F0%9F%91%A8%E2%80%8D%F0%9F%92%BB">#</a></h2>
<p>The NOOBs micro sd card comes preloaded with raspbian - if you have a blank SD you'll have to find instructions to flash it with Raspbian or a suitable alternative unix flavour. (Re)Connect the flashed SD card to your computer - you should see that it has been recognised as <code>BOOT</code>. In here are a range of files. If you can't already see a <code>wpa_supplicant.conf</code> file, create one in the root of the sd card i.e.<code>BOOT/wpa_supplicant.conf</code>and enter the following updating the country and wifi details:</p>
<pre class="language-bash"><code class="language-bash"><span class="token assign-left variable">country</span><span class="token operator">=</span>GB <span class="token comment">#your country's 2digit isocode</span><br /><span class="token assign-left variable">ctrl_interface</span><span class="token operator">=</span>DIR<span class="token operator">=</span>/var/run/wpa_supplicant <span class="token assign-left variable">GROUP</span><span class="token operator">=</span>netdev<br /><span class="token assign-left variable">update_config</span><span class="token operator">=</span><span class="token number">1</span><br /><br /><span class="token assign-left variable">network</span><span class="token operator">=</span><span class="token punctuation">{</span><br /> <span class="token assign-left variable">ssid</span><span class="token operator">=</span><span class="token string">"yourWiFiNetworkName"</span><br /> <span class="token assign-left variable">psk</span><span class="token operator">=</span><span class="token string">"yourWiFiPassword"</span><br /><span class="token punctuation">}</span></code></pre>
<p>You can add multiple WiFi networks by repeating the above config with the relevant network credentials.</p>
<p>In the same directory create a blank file <code>ssh</code>, ensuring that there's no extension.</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">touch</span> <span class="token function">ssh</span></code></pre>
<h2 id="step-3%3A-%F0%9F%9F%A2-turn-on-the-r-pi-connect-to-it-and-update">Step 3: đ˘ Turn on the R Pi connect to it and update <a class="direct-link" href="https://lukesingham.com/pi-hole/#step-3%3A-%F0%9F%9F%A2-turn-on-the-r-pi-connect-to-it-and-update">#</a></h2>
<p>When you first power up the R Pi it will immediately start flashing a small green LED. However, it may take a little while to be able to ssh into the R Pi - allow up to 2 minutes. Of course if you mistyped your WiFi credentials you won't be able to ssh in đ¤Śââď¸. Your network will assign an IP address to the PI but the default hostname is nicer.</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">ssh</span> pi@raspberrypi.local</code></pre>
<p>Press enter / answer yes to the prompts. Â When asked for a password, Â <code>raspberry</code> is the default.</p>
<p>Run the following command to update the R Pi - this can take around 30mins so take a break at this point đĽą:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">sudo</span> <span class="token function">apt-get</span> update <span class="token operator">&&</span> <span class="token function">sudo</span> <span class="token function">apt-get</span> upgrade <span class="token parameter variable">-y</span></code></pre>
<p>Then change password:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">passwd</span></code></pre>
<h2 id="step-4%3A-optional-what's-the-fastest-upstream-dns-to-select-%F0%9F%8F%81">Step 4: <em>Optional</em> what's the fastest Upstream DNS to select đ <a class="direct-link" href="https://lukesingham.com/pi-hole/#step-4%3A-optional-what's-the-fastest-upstream-dns-to-select-%F0%9F%8F%81">#</a></h2>
<p>You need to select an upstream DNS as part of the Pi-Hole install. I wanted to know which one was going to give me the best performance. I found <a href="https://github.com/mwiora/NAMEinator">this tool</a> a fork from a broken Google tool that did the job. If you don't have <code>go-lang</code> installed skip this step and choose a DNS provider you're comfortable with - I'd recommend Cloudflare.</p>
<pre class="language-bash"><code class="language-bash">$ go get github.com/mwiora/NAMEinator<br />$ go get github.com/miekg/dns<br />$ <span class="token builtin class-name">cd</span> ~/go/src/github.com/mwiora/NAMEinator/<br />$ go build<br />$ ./NAMEinator</code></pre>
<p>After <code>NAMEinator</code> finishes testing the DNSs for responses you should get a report print out:</p>
<pre class="language-bash"><code class="language-bash">LETS GO - each dot is a completed domain request against all nameservers<br /><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><br />finished - presenting results:<br /><br /><span class="token number">208.67</span>.222.222:<br />Avg. <span class="token punctuation">[</span><span class="token number">88</span>.618011ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">606</span>.061151ms<span class="token punctuation">]</span><br /><br /><span class="token number">156.154</span>.71.1:<br />Avg. <span class="token punctuation">[</span><span class="token number">101</span>.25658ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">2</span>.004470267s<span class="token punctuation">]</span><br /><br /><span class="token number">216.146</span>.35.35:<br />Avg. <span class="token punctuation">[</span><span class="token number">117</span>.992645ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">1</span>.402119851s<span class="token punctuation">]</span><br /><br /><span class="token number">1.0</span>.0.1:<br />Avg. <span class="token punctuation">[</span><span class="token number">50</span>.555672ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">615</span>.732289ms<span class="token punctuation">]</span><br /><br /><span class="token number">1.1</span>.1.1:<br />Avg. <span class="token punctuation">[</span><span class="token number">28</span>.486788ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">329</span>.542844ms<span class="token punctuation">]</span><br /><br /><span class="token number">8.8</span>.8.8:<br />Avg. <span class="token punctuation">[</span><span class="token number">79</span>.401147ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">2</span>.003982489s<span class="token punctuation">]</span><br /><br /><span class="token number">8.8</span>.4.4:<br />Avg. <span class="token punctuation">[</span><span class="token number">65</span>.633078ms<span class="token punctuation">]</span>, Min. <span class="token punctuation">[</span>10ms<span class="token punctuation">]</span>, Max. <span class="token punctuation">[</span><span class="token number">491</span>.458848ms<span class="token punctuation">]</span><br /><br />Au revoir<span class="token operator">!</span></code></pre>
<p>I ran the test a couple of times, <a href="https://www.cloudflare.com/en-gb/">Cloudflare <code>1.1.1.1</code></a> came out with the lowest response times both times.</p>
<h2 id="step-5%3A-install-pi-hole-%F0%9F%95%B3">Step 5: Install Pi-Hole đł <a class="direct-link" href="https://lukesingham.com/pi-hole/#step-5%3A-install-pi-hole-%F0%9F%95%B3">#</a></h2>
<p>You can run Pi-Hole in a container, however I chose to directly install this for two reasons, I suspect the overhead of Docker would cause performance issues and I won't be utilising the reasons for Docker i.e. I'm unlikely to be putting other software on this R Pi. I went with the simplest option:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">curl</span> <span class="token parameter variable">-sSL</span> https://install.pi-hole.net <span class="token operator">|</span> <span class="token function">bash</span></code></pre>
<p>This will take you through a series of prompts that you mostly will press enter/yes to, select your upstream DNS and once you pop out the other end you will have a print out that looks like:<br />
<img src="https://lukesingham.com/content/images/2020/04/Screen-Shot-2020-04-27-at-09.39.50.png" alt="" /><br />
Take note of the IP address at the bottom of the print out next to<code>Pi-hole DNS (IPv4)</code> and the <code>Web Interface password</code>.</p>
<h2 id="step-6%3A-configure-your-router-%F0%9F%92%BB">Step 6: Configure Your Router đť <a class="direct-link" href="https://lukesingham.com/pi-hole/#step-6%3A-configure-your-router-%F0%9F%92%BB">#</a></h2>
<p>If you haven't done much tinkering with routers or consumed a bunch networking content this can be a strange place full of foreign acronyms. Here's a simple explaination of the two most important for this exercise:</p>
<p><strong>IP address</strong> - a device's unique identifier on a network.</p>
<p><strong>Domain Name System (DNS)</strong> - Resolves domain names into IP addresses e.g. <code>google.com</code> -> <code>172.217.169.78</code>.</p>
<p><strong>Dynamic Host Configuration Protocol (DHCP)</strong> - a process that occurs on your WiFi network to assign unique IPs. <a href="https://www.youtube.com/watch?v=S43CFcpOZSI">5 min video explainer</a>.</p>
<p>There are 3 options to get Pi-Hole working outlined in <a href="https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245">more detail here</a>.</p>
<ol>
<li>**Configure the DNS to the static IP of the R Pi **- Â I'd recommend putting in a secondary option of Cloudflare in case your R Pi doesn't work. This option means anyone who connects to your device will be ad free without thinking about it. However no amount of rebooting routers and devices, or forgetting my WiFi network and re-adding it managed to get this option to work for me. It should be as simple as setting this option:</li>
</ol>
<p><img src="https://lukesingham.com/content/images/2020/04/Screen-Shot-2020-04-26-at-22.18.25-1.png" alt="" /><br />
2. **Let Pi-Hole takeover the DHCP side of things **- this worked easily. Turn off the DHCP setting on your router.<br />
<img src="https://lukesingham.com/content/images/2020/04/Screen-Shot-2020-04-26-at-22.36.07.png" alt="" /><br />
Then turn it on in Settings in the Pi-Hole web interface.<br />
<img src="https://lukesingham.com/content/images/2020/04/Screen-Shot-2020-04-26-at-22.38.08-4.png" alt="" /><br />
You will also need to assign a static IP for the R Pi. However, if your R Pi stops working, so too will your internet - so I went for the option I least wanted, option 3.</p>
<p>**3. Manually enter your Pi-Hole DNS per device 'opt-in' **- It means that instead of everyone benefitting automatically without thinking about DNS config when they sign up to your WiFi - you have to manually put in the DNS config. For OSX that looks like this:<br />
<img src="https://lukesingham.com/content/images/2020/04/osx-enter-custom-dns.png" alt="" /><br />
For Android. Go to WiFi networks > select 'Modify Network' > 'Advanced options' > Change 'IP settings' to 'Static' > Enter your R Pi IP address.</p>
<p>Pretty instantly I ended up seeing lovely blocked results in myPi-Hole web interface.<br />
<img src="https://lukesingham.com/content/images/2020/04/Screen-Shot-2020-04-27-at-00.00.33.png" alt="" /></p>
<h2 id="reference-%F0%9F%93%9A">Reference đ <a class="direct-link" href="https://lukesingham.com/pi-hole/#reference-%F0%9F%93%9A">#</a></h2>
<ul>
<li><a href="https://learn.pimoroni.com/tutorial/sandyj/setting-up-a-headless-pi">Pimoroni where I've bought all my R Pi related gear so far and they have great tutorials like this one.</a></li>
<li><a href="https://www.losant.com/blog/getting-started-with-the-raspberry-pi-zero-w-without-a-monitor?q=%20&hPP=20&idx=production_BLOG&p=0&is_v=1">Helpful guide for the initial micro sd card setup</a></li>
<li><a href="https://www.reddit.com/r/pihole/comments/61an37/pihole_refusing_to_work_over_wifi/">Reddit thread on all the rebooting advice - didn't solve it for me.</a></li>
</ul>
How to Make a Churn Model in R2017-11-21T00:00:00Zhttps://lukesingham.com/how-to-make-a-churn-model-in-r/<p>The following post details how to make a churn model in R. It was part of an interview process for which a take home assignment was one of the stages. The company stated this should take 2hrs, which is entirely unrealistic. To minimise the time cost, my analysis is very succinct and short on the exploratory analysis and amount of models compared. The churn model got me to the final stage, however little in the way of feedback was offered. There is considerable debate in the tech industry as to whether take home exams are a fair assessment or even a reasonable ask on an individual. My assessment is that this is a lazy way to interview and a very high cost on the interviewee. If you really want to work for a particular company, then sure you might be prepared to do what it takes. I doubt I will do another take home assignment.</p>
<p>The analysis was done in <a href="http://rmarkdown.rstudio.com/">Rmarkdown</a> and the output copied into this blog post following <a href="https://notes.innovea.tech/how-to-write-latex-formula-in-ghosts-post/">these helpful pointers</a>. If you would like to play with the data and full unabridged code please visit <a href="https://github.com/ucg8j/how-to-make-churn-model-in-R">this github repo</a>.</p>
<h2 id="aim">Aim <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#aim">#</a></h2>
<ol>
<li>Please create a model that predicts which businesses are likely to churn at the start of 2015 based on the <code>vertical</code> and <code>incorporation_date</code>.</li>
<li>We have an inkling that a dropoff in the number of mandates added might be an advance indicator of someone churning. Please can you assess whether this might be true, and if you think it is useful, incorporate it into your model from part (1).</li>
</ol>
<pre class="language-r"><code class="language-r"><span class="token comment"># Load packages</span><br />suppressPackageStartupMessages<span class="token punctuation">(</span><span class="token punctuation">{</span><br /> library<span class="token punctuation">(</span>data.table<span class="token punctuation">)</span> <span class="token comment"># Fast I/O</span><br /> library<span class="token punctuation">(</span>dplyr<span class="token punctuation">)</span> <span class="token comment"># Data munging</span><br /> library<span class="token punctuation">(</span>tidyr<span class="token punctuation">)</span> <span class="token comment"># Data munging</span><br /> library<span class="token punctuation">(</span>lubridate<span class="token punctuation">)</span> <span class="token comment"># Makes dates easy</span><br /> library<span class="token punctuation">(</span>plotly<span class="token punctuation">)</span> <span class="token comment"># Interactive charts</span><br /> library<span class="token punctuation">(</span>magrittr<span class="token punctuation">)</span> <span class="token comment"># pipe operators</span><br /> library<span class="token punctuation">(</span>caret<span class="token punctuation">)</span> <span class="token comment"># Handy ML functions</span><br /> library<span class="token punctuation">(</span>rpart<span class="token punctuation">)</span> <span class="token comment"># Decision Trees</span><br /> library<span class="token punctuation">(</span>rpart.plot<span class="token punctuation">)</span> <span class="token comment"># Pretty tree plots</span><br /> library<span class="token punctuation">(</span>ROCR<span class="token punctuation">)</span> <span class="token comment"># ML evaluation</span><br /> library<span class="token punctuation">(</span>e1071<span class="token punctuation">)</span> <span class="token comment"># Misc stat fns</span><br /> library<span class="token punctuation">(</span>randomForest<span class="token punctuation">)</span> <span class="token comment"># rf</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><br /><br />set.seed<span class="token punctuation">(</span><span class="token number">42</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Read data and drop row number column</span><br />df <span class="token operator"><-</span> fread<span class="token punctuation">(</span><span class="token string">"monthly_data_(2)_(2).csv"</span><span class="token punctuation">,</span> drop <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Have a glimpse of the data</span><br />glimpse<span class="token punctuation">(</span>df<span class="token punctuation">)</span></code></pre>
<pre class="language-r"><code class="language-r"><span class="token comment">## Observations: 902</span><br /><span class="token comment">## Variables: 27</span><br /><span class="token comment">## $ company_id <int> 4, 5, 6, 7, 8, 10, 11, 12, 13, 14...</span><br /><span class="token comment">## $ 2014-01-01_payments <dbl> 8, 0, 2, 3, 0, 0, 0, 0, 0, 0, 4, ...</span><br /><span class="token comment">## $ 2014-02-01_payments <dbl> 4, 0, 2, 3, 0, 0, 0, 2, 0, 11, 1,...</span><br /><span class="token comment">## $ 2014-03-01_payments <dbl> 7, 39, 1, 1, 6, 0, 0, 0, 8, 0, 1,...</span><br /><span class="token comment">## $ 2014-04-01_payments <dbl> 7, 0, 3, 7, 50, 1, 0, 0, 2, 0, 0,...</span><br /><span class="token comment">## $ 2014-05-01_payments <dbl> 1, 54, 1, 4, 119, 0, 1, 0, 0, 0, ...</span><br /><span class="token comment">## $ 2014-06-01_payments <dbl> 2, 0, 2, 1, 151, 3, 0, 0, 0, 0, 2...</span><br /><span class="token comment">## $ 2014-07-01_payments <dbl> 2, 0, 2, 7, 182, 0, 0, 0, 3, 0, 0...</span><br /><span class="token comment">## $ 2014-08-01_payments <dbl> 4, 22, 1, 2, 167, 0, 0, 0, 2, 0, ...</span><br /><span class="token comment">## $ 2014-09-01_payments <dbl> 3, 0, 1, 5, 180, 0, 0, 0, 0, 9, 5...</span><br /><span class="token comment">## $ 2014-10-01_payments <dbl> 5, 0, 2, 8, 157, 1, 0, 0, 0, 2, 1...</span><br /><span class="token comment">## $ 2014-11-01_payments <dbl> 5, 0, 1, 2, 105, 0, 0, 0, 0, 0, 0...</span><br /><span class="token comment">## $ 2014-12-01_payments <dbl> 9, 0, 3, 8, 57, 0, 0, 0, 0, 0, 0,...</span><br /><span class="token comment">## $ 2014-01-01_mandates <dbl> 0, 4, 0, 0, 0, 4, 4, 0, 0, 22, 0,...</span><br /><span class="token comment">## $ 2014-02-01_mandates <dbl> 0, 31, 1, 0, 2, 3, 8, 0, 0, 20, 1...</span><br /><span class="token comment">## $ 2014-03-01_mandates <dbl> 0, 24, 0, 0, 0, 5, 19, 0, 0, 11, ...</span><br /><span class="token comment">## $ 2014-04-01_mandates <dbl> 53, 18, 0, 1, 0, 0, 0, 0, 1, 11, ...</span><br /><span class="token comment">## $ 2014-05-01_mandates <dbl> 0, 8, 0, 0, 0, 0, 0, 0, 1, 15, 0,...</span><br /><span class="token comment">## $ 2014-06-01_mandates <dbl> 0, 7, 0, 0, 0, 0, 0, 0, 0, 13, 5,...</span><br /><span class="token comment">## $ 2014-07-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 16...</span><br /><span class="token comment">## $ 2014-08-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 14, 8,...</span><br /><span class="token comment">## $ 2014-09-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 16, 2,...</span><br /><span class="token comment">## $ 2014-10-01_mandates <dbl> 0, 0, 0, 0, 1, 0, 0, 0, 0, 16, 1,...</span><br /><span class="token comment">## $ 2014-11-01_mandates <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 0, ...</span><br /><span class="token comment">## $ 2014-12-01_mandates <dbl> 0, 0, 0, 3, 0, 0, 0, 0, 0, 11, 0,...</span><br /><span class="token comment">## $ vertical <chr> "gym/fitness", "freelance d...</span><br /><span class="token comment">## $ incorporation_date <chr> "2003-09-25", "2008-10-22", ...</span></code></pre>
<h2 id="data-munging">Data Munging <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#data-munging">#</a></h2>
<p>Some data munging needs to occur for our binary classifiers to make use of the data within. This includes handling dates.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Reshape data and create new columns</span><br />df <span class="token percent-operator operator">%<>%</span><br /> gather<span class="token punctuation">(</span>key <span class="token operator">=</span> date<span class="token punctuation">,</span> value <span class="token operator">=</span> quantity<span class="token punctuation">,</span> starts_with<span class="token punctuation">(</span><span class="token string">"20"</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> separate<span class="token punctuation">(</span>date<span class="token punctuation">,</span> c<span class="token punctuation">(</span><span class="token string">"date"</span><span class="token punctuation">,</span><span class="token string">"paymentMandate"</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token string">"_"</span><span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> spread<span class="token punctuation">(</span>paymentMandate<span class="token punctuation">,</span> quantity<span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> mutate<span class="token punctuation">(</span><br /> incorporation_date <span class="token operator">=</span> as.Date<span class="token punctuation">(</span>incorporation_date<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> date <span class="token operator">=</span> as.Date<span class="token punctuation">(</span>date<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> incorporation_time <span class="token operator">=</span> round<span class="token punctuation">(</span><br /> as.numeric<span class="token punctuation">(</span>difftime<span class="token punctuation">(</span>as.Date<span class="token punctuation">(</span><span class="token string">"2014-12-01"</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br /> as.Date<span class="token punctuation">(</span>incorporation_date<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> unit<span class="token operator">=</span><span class="token string">"weeks"</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token number">52.25</span><span class="token punctuation">,</span><br /> digits <span class="token operator">=</span> <span class="token number">1</span><span class="token punctuation">)</span><br /> <span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> arrange<span class="token punctuation">(</span>date<span class="token punctuation">)</span><br /><br /><span class="token comment"># What does the new data look like?</span><br />glimpse<span class="token punctuation">(</span>df<span class="token punctuation">)</span></code></pre>
<pre class="language-r"><code class="language-r"><span class="token comment">## Observations: 10,824</span><br /><span class="token comment">## Variables: 7</span><br /><span class="token comment">## $ company_id <int> 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14,...</span><br /><span class="token comment">## $ vertical <chr> "gym/fitness", "gym/fitness", "freelance de...</span><br /><span class="token comment">## $ incorporation_date <date> 2013-05-30, 2003-09-25, 2008-10-22, 2005-0...</span><br /><span class="token comment">## $ date <date> 2014-01-01, 2014-01-01, 2014-01-01, 2014-0...</span><br /><span class="token comment">## $ mandates <dbl> 1, 0, 0, 0, 4, 0, 0, 0, 4, 4, 0, 0, 22, 0, ...</span><br /><span class="token comment">## $ payments <dbl> 0, 1, 6, 8, 0, 2, 3, 0, 0, 0, 0, 0, 0, 4, 4...</span><br /><span class="token comment">## $ incorporation_time <dbl> 1.5, 11.2, 6.1, 9.4, 1.2, 9.1, 4.0, 2.9, 1....</span></code></pre>
<h3 id="what-is-churn">What is Churn <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#what-is-churn">#</a></h3>
<p>For the purposes of this project I have defined âchurnâ as zero mandates and payments for the first instance of that after the last mandate/payment made.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Create binary 'churn' column</span><br />df<span class="token operator">$</span>churn <span class="token operator"><-</span> <span class="token number">0</span><br /><br /><span class="token comment"># For use in for loop - upper bound of data</span><br />max.date <span class="token operator"><-</span> max<span class="token punctuation">(</span>df<span class="token operator">$</span>date<span class="token punctuation">)</span><br /><br /><span class="token comment"># Mark all companies as churned in the month immediately their last activity</span><br /><span class="token keyword">for</span> <span class="token punctuation">(</span>company <span class="token keyword">in</span> unique<span class="token punctuation">(</span>df<span class="token operator">$</span>company_id<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /><br /> <span class="token comment"># Subset data to company</span><br /> df.sub <span class="token operator"><-</span> subset<span class="token punctuation">(</span>df<span class="token punctuation">,</span> df<span class="token operator">$</span>company_id <span class="token operator">==</span> company<span class="token punctuation">)</span><br /><br /> <span class="token comment"># Index of last positive mandate OR payment</span><br /> last.pos.idx <span class="token operator"><-</span> tail<span class="token punctuation">(</span>which<span class="token punctuation">(</span>df.sub<span class="token operator">$</span>mandates <span class="token operator">!=</span> <span class="token number">0</span> <span class="token operator">|</span> df.sub<span class="token operator">$</span>payments <span class="token operator">!=</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><br /><br /> <span class="token comment"># Get date of last activity</span><br /> last.activity.date <span class="token operator"><-</span> df.sub<span class="token operator">$</span>date<span class="token punctuation">[</span>last.pos.idx<span class="token punctuation">]</span><br /><br /> <span class="token comment"># If less than max.date of dataset mark churn ELSE do nothing i.e. positive at end of period</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>last.activity.date <span class="token operator"><</span> max.date<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /><br /> <span class="token comment"># Get churn date (last positive month plus 1mth)</span><br /> churn.date <span class="token operator"><-</span> last.activity.date <span class="token percent-operator operator">%m+%</span> months<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /> <span class="token comment"># Mark month of churn as 1</span><br /> df<span class="token punctuation">[</span>df<span class="token operator">$</span>date <span class="token operator">==</span> churn.date <span class="token operator">&</span> df<span class="token operator">$</span>company_id <span class="token operator">==</span> company<span class="token punctuation">,</span> <span class="token punctuation">]</span><span class="token operator">$</span>churn <span class="token operator"><-</span> <span class="token number">1</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment"># Multiple rows per company, filter for last month or churn month values...</span><br /><span class="token comment"># Get churners</span><br />df <span class="token percent-operator operator">%>%</span> filter<span class="token punctuation">(</span>churn <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token operator">-></span> churners<br /><br /><span class="token comment"># Get max date row of remainers (non-churners)</span><br />df <span class="token percent-operator operator">%>%</span><br /> filter<span class="token punctuation">(</span>churn <span class="token operator">==</span> <span class="token number">0</span> <span class="token operator">&</span> <span class="token operator">!</span><span class="token punctuation">(</span>company_id <span class="token percent-operator operator">%in%</span> churners<span class="token operator">$</span>company_id<span class="token punctuation">)</span> <span class="token operator">&</span> date <span class="token operator">==</span> max<span class="token punctuation">(</span>date<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">-></span> remainers<br /><br /><span class="token comment"># Combine and variables coded ready for modelling</span><br />churners <span class="token percent-operator operator">%>%</span><br /> rbind<span class="token punctuation">(</span>remainers<span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> mutate<span class="token punctuation">(</span>vertical <span class="token operator">=</span> as.factor<span class="token punctuation">(</span>vertical<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> churn <span class="token operator">=</span> as.factor<span class="token punctuation">(</span>churn<span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">-></span> model.df</code></pre>
<h3 id="churns-over-the-year">Churns over the year <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#churns-over-the-year">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># Plot churners</span><br />model.df <span class="token percent-operator operator">%>%</span><br /> filter<span class="token punctuation">(</span>churn <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> group_by<span class="token punctuation">(</span>date<span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> summarise<span class="token punctuation">(</span>n <span class="token operator">=</span> n<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> plot_ly<span class="token punctuation">(</span> x <span class="token operator">=</span> <span class="token operator">~</span>date<span class="token punctuation">,</span> y <span class="token operator">=</span> <span class="token operator">~</span>n<span class="token punctuation">,</span> type <span class="token operator">=</span> <span class="token string">'scatter'</span><span class="token punctuation">,</span> mode <span class="token operator">=</span> <span class="token string">'lines'</span><span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/11/churns-over-year.png" alt="" /></p>
<p>Could do more on exploratory but this is the easiest to cut back for this report.</p>
<h3 id="balance-of-the-data">Balance of the data <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#balance-of-the-data">#</a></h3>
<p>The proportion of churn is 32.04%. Representing an imbalanced dataset. Accuracy is an inappropriate measure (I could get 67.96% accuracy predicting no businesses leave), so I will focus on recall and accuracy.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Loyal vs Churn</span><br />table<span class="token punctuation">(</span>model.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br /><span class="token comment">##</span><br /><span class="token comment">## 0 1</span><br /><span class="token comment">## 613 289</span></code></pre>
<h2 id="model">Model <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#model">#</a></h2>
<p>Survival models and binary classifiers are common approaches to âChurnâ models. I will approach the model using the latter, though if I had more time I would investigate other classifiers and a survival model. I will limit the range of models to a logistic, a decision tree and an ensemble.</p>
<h3 id="split-data">Split Data <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#split-data">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># 80/20 train test split</span><br />index <span class="token operator"><-</span> createDataPartition<span class="token punctuation">(</span>model.df<span class="token operator">$</span>churn<span class="token punctuation">,</span> p <span class="token operator">=</span> <span class="token number">0.8</span><span class="token punctuation">,</span> list <span class="token operator">=</span> <span class="token boolean">FALSE</span><span class="token punctuation">)</span><br />train.df <span class="token operator"><-</span> model.df<span class="token punctuation">[</span>index<span class="token punctuation">,</span> <span class="token punctuation">]</span><br />test.df <span class="token operator"><-</span> model.df<span class="token punctuation">[</span><span class="token operator">-</span>index<span class="token punctuation">,</span> <span class="token punctuation">]</span><br /><br /><span class="token comment"># Check balance of the training split</span><br />table<span class="token punctuation">(</span>train.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br /><span class="token comment">##</span><br /><span class="token comment">## 0 1</span><br /><span class="token comment">## 491 232</span></code></pre>
<pre class="language-r"><code class="language-r"><span class="token comment"># Check balance of the test split</span><br />table<span class="token punctuation">(</span>test.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br /><span class="token comment">##</span><br /><span class="token comment">## 0 1</span><br /><span class="token comment">## 122 57</span></code></pre>
<h3 id="logistic">Logistic <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#logistic">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># Run model</span><br />Logistic.model <span class="token operator"><-</span> glm<span class="token punctuation">(</span>churn <span class="token operator">~</span> incorporation_time <span class="token operator">+</span> vertical<span class="token punctuation">,</span><br /> data <span class="token operator">=</span> train.df<span class="token punctuation">,</span><br /> family <span class="token operator">=</span> binomial<span class="token punctuation">(</span>link <span class="token operator">=</span> <span class="token string">'logit'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Predict</span><br />log.pred <span class="token operator"><-</span> predict<span class="token punctuation">(</span>Logistic.model<span class="token punctuation">,</span> newdata <span class="token operator">=</span> test.df<span class="token punctuation">,</span> type <span class="token operator">=</span> <span class="token string">'response'</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Convert probs to binary</span><br />log.pred <span class="token operator"><-</span> as.factor<span class="token punctuation">(</span>ifelse<span class="token punctuation">(</span>log.pred <span class="token operator">></span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Evaluation Metrics</span><br />log.result <span class="token operator"><-</span> confusionMatrix<span class="token punctuation">(</span>data <span class="token operator">=</span> log.pred<span class="token punctuation">,</span> test.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br />log.precision <span class="token operator"><-</span> log.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Pos Pred Value'</span><span class="token punctuation">]</span><br />log.recall <span class="token operator"><-</span> log.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Sensitivity'</span><span class="token punctuation">]</span><br />log.F1 <span class="token operator"><-</span> log.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'F1'</span><span class="token punctuation">]</span></code></pre>
<h3 id="decision-tree">Decision Tree <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#decision-tree">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># Train model</span><br />tree.model <span class="token operator"><-</span> rpart<span class="token punctuation">(</span>churn <span class="token operator">~</span> incorporation_time <span class="token operator">+</span> vertical<span class="token punctuation">,</span><br /> data <span class="token operator">=</span> train.df<span class="token punctuation">,</span><br /> method <span class="token operator">=</span> <span class="token string">"class"</span><span class="token punctuation">,</span><br /> control <span class="token operator">=</span> rpart.control<span class="token punctuation">(</span>xval <span class="token operator">=</span> <span class="token number">10</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Plot</span><br />rpart.plot<span class="token punctuation">(</span>tree.model<span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/11/decision-tree.png" alt="" /></p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Evaluation metrics</span><br />tree.pred <span class="token operator"><-</span> predict<span class="token punctuation">(</span>tree.model<span class="token punctuation">,</span> newdata <span class="token operator">=</span> test.df<span class="token punctuation">,</span> type <span class="token operator">=</span> <span class="token string">"class"</span><span class="token punctuation">)</span><br />tree.result <span class="token operator"><-</span> confusionMatrix<span class="token punctuation">(</span>data <span class="token operator">=</span> tree.pred<span class="token punctuation">,</span> test.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br />tree.precision <span class="token operator"><-</span> tree.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Pos Pred Value'</span><span class="token punctuation">]</span><br />tree.recall <span class="token operator"><-</span> tree.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Sensitivity'</span><span class="token punctuation">]</span><br />tree.F1 <span class="token operator"><-</span> tree.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'F1'</span><span class="token punctuation">]</span></code></pre>
<h3 id="random-forest-(ensemble)">Random Forest (Ensemble) <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#random-forest-(ensemble)">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># Train model</span><br />forest.model <span class="token operator"><-</span> randomForest<span class="token punctuation">(</span>churn <span class="token operator">~</span> incorporation_time <span class="token operator">+</span> vertical<span class="token punctuation">,</span><br /> data <span class="token operator">=</span> train.df<span class="token punctuation">,</span><br /> ntree<span class="token operator">=</span><span class="token number">200</span><span class="token punctuation">,</span><br /> type<span class="token operator">=</span><span class="token string">"classification"</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># See error reduction with number of trees ( not much gained beyond ~25 trees)</span><br />plot<span class="token punctuation">(</span>forest.model<span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/11/random-forest-error-reduction.png" alt="" /></p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Look at the variable Importance from the random forest</span><br />varImpPlot<span class="token punctuation">(</span>forest.model<span class="token punctuation">,</span> sort <span class="token operator">=</span> T<span class="token punctuation">,</span> main<span class="token operator">=</span><span class="token string">"Variable Importance"</span><span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/11/random-forest-variable-importance.png" alt="" /></p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Evaluation metrics</span><br />forest.pred <span class="token operator"><-</span> predict<span class="token punctuation">(</span>forest.model<span class="token punctuation">,</span> newdata <span class="token operator">=</span> test.df<span class="token punctuation">,</span> type <span class="token operator">=</span> <span class="token string">"class"</span><span class="token punctuation">)</span><br />forest.result <span class="token operator"><-</span> confusionMatrix<span class="token punctuation">(</span>data <span class="token operator">=</span> forest.pred<span class="token punctuation">,</span> test.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br />forest.precision <span class="token operator"><-</span> forest.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Pos Pred Value'</span><span class="token punctuation">]</span><br />forest.recall <span class="token operator"><-</span> forest.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Sensitivity'</span><span class="token punctuation">]</span><br />forest.F1 <span class="token operator"><-</span> forest.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'F1'</span><span class="token punctuation">]</span></code></pre>
<h3 id="evaluation-metrics">Evaluation Metrics <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#evaluation-metrics">#</a></h3>
<pre class="language-r"><code class="language-r">log.precision<span class="token punctuation">;</span><br /><span class="token comment">## Pos Pred Value</span><br /><span class="token comment">## 0.8333333</span><br />tree.precision<span class="token punctuation">;</span><br /><span class="token comment">## Pos Pred Value</span><br /><span class="token comment">## 0.8088235</span><br />forest.precision<span class="token punctuation">;</span><br /><span class="token comment">## Pos Pred Value</span><br /><span class="token comment">## 0.8134328</span></code></pre>
<pre class="language-r"><code class="language-r">log.recall<span class="token punctuation">;</span><br /><span class="token comment">## Sensitivity</span><br /><span class="token comment">## 0.9016393</span><br />tree.recall<span class="token punctuation">;</span><br /><span class="token comment">## Sensitivity</span><br /><span class="token comment">## 0.9016393</span><br />forest.recall<span class="token punctuation">;</span><br /><span class="token comment">## Sensitivity</span><br /><span class="token comment">## 0.8934426</span></code></pre>
<pre class="language-r"><code class="language-r">log.F1<span class="token punctuation">;</span><br /><span class="token comment">## F1</span><br /><span class="token comment">## 0.8661417</span><br />tree.F1<span class="token punctuation">;</span><br /><span class="token comment">## F1</span><br /><span class="token comment">## 0.8527132</span><br />forest.F1<span class="token punctuation">;</span><br /><span class="token comment">## F1</span><br /><span class="token comment">## 0.8515625</span></code></pre>
<p>Surprisingly, the logistic regression model performs the best, with the top precision score and equal recall score with that of the decision tree. With more time, Iâd see if tweaks to the decision tree and random forest models could change this. Its also possible over/undersampling could help. From here on I will use the logistic regression model.</p>
<h2 id="examine-the-incorporation-of-time-information">Examine the incorporation of time information <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#examine-the-incorporation-of-time-information">#</a></h2>
<p>Per the second aim, examine the inclusion of time information. This requires more data munging. Taking code used from above, I will extend to include the derivation of a very basic time period variable.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Create binary 'leading_indicator' column</span><br />model.df<span class="token operator">$</span>leading_indicator <span class="token operator"><-</span> <span class="token number">0</span><br /><br /><span class="token comment"># Min date for which a leading_indicator can be calculated (lower limit of data)</span><br />min.date <span class="token operator"><-</span> min<span class="token punctuation">(</span>df<span class="token operator">$</span>date<span class="token punctuation">)</span><br /><br /><span class="token comment"># If month before 'churn' (churn-1), is below the level of mandates of the month 2 months prior (churn-2) then</span><br /><span class="token comment"># make leading_indicator == 1</span><br /><span class="token keyword">for</span> <span class="token punctuation">(</span>company <span class="token keyword">in</span> churners<span class="token operator">$</span>company_id<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /><br /> <span class="token comment"># Subset data to company</span><br /> df.sub <span class="token operator"><-</span> subset<span class="token punctuation">(</span>df<span class="token punctuation">,</span> df<span class="token operator">$</span>company_id <span class="token operator">==</span> company<span class="token punctuation">)</span><br /><br /> <span class="token comment"># Get month prior to churn</span><br /> month.prior <span class="token operator"><-</span> df.sub<span class="token operator">$</span>date<span class="token punctuation">[</span>df.sub<span class="token operator">$</span>churn <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">]</span> <span class="token percent-operator operator">%m-%</span> months<span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /> <span class="token comment"># Get two months prior to churn</span><br /> two.month.prior <span class="token operator"><-</span> df.sub<span class="token operator">$</span>date<span class="token punctuation">[</span>df.sub<span class="token operator">$</span>churn <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">]</span> <span class="token percent-operator operator">%m-%</span> months<span class="token punctuation">(</span><span class="token number">2</span><span class="token punctuation">)</span><br /><br /> <span class="token comment"># If two months prior is within dataset date range and level of mandates is greater than 0</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span><span class="token punctuation">(</span>two.month.prior <span class="token operator">></span> min.date<span class="token punctuation">)</span> <span class="token operator">&&</span> <span class="token punctuation">(</span>df.sub<span class="token operator">$</span>mandates<span class="token punctuation">[</span>df.sub<span class="token operator">$</span>date <span class="token operator">==</span> two.month.prior<span class="token punctuation">]</span> <span class="token operator">></span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /><br /> <span class="token comment"># Compare number of mandates 1 month prior to 2 months prior, if less, mark 'leading_indicator' as '1'</span><br /> <span class="token keyword">if</span> <span class="token punctuation">(</span>df.sub<span class="token operator">$</span>mandates<span class="token punctuation">[</span>df.sub<span class="token operator">$</span>date <span class="token operator">==</span> month.prior<span class="token punctuation">]</span> <span class="token operator"><</span> df.sub<span class="token operator">$</span>mandates<span class="token punctuation">[</span>df.sub<span class="token operator">$</span>date <span class="token operator">==</span> two.month.prior<span class="token punctuation">]</span><span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> model.df<span class="token punctuation">[</span>model.df<span class="token operator">$</span>company_id <span class="token operator">==</span> company<span class="token punctuation">,</span> <span class="token punctuation">]</span><span class="token operator">$</span>leading_indicator <span class="token operator"><-</span> <span class="token number">1</span><br /> <span class="token punctuation">}</span><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span></code></pre>
<h3 id="of-the-churners%2C-how-many-have-a-leading-indicator%3F">Of the churners, how many have a leading indicator? <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#of-the-churners%2C-how-many-have-a-leading-indicator%3F">#</a></h3>
<pre class="language-r"><code class="language-r">model.df <span class="token percent-operator operator">%>%</span><br /> filter<span class="token punctuation">(</span>churn <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> group_by<span class="token punctuation">(</span>leading_indicator<span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> summarise<span class="token punctuation">(</span>n<span class="token operator">=</span>n<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token percent-operator operator">%>%</span><br /> mutate<span class="token punctuation">(</span>percent <span class="token operator">=</span> round<span class="token punctuation">(</span>n <span class="token operator">/</span> sum<span class="token punctuation">(</span>n<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">100</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<pre class="language-r"><code class="language-r"><span class="token comment">## # A tibble: 2 x 3</span><br /><span class="token comment">## leading_indicator n percent</span><br /><span class="token comment">## <dbl> <int> <dbl></span><br /><span class="token comment">## 1 0 264 91.3</span><br /><span class="token comment">## 2 1 25 8.7</span></code></pre>
<h3 id="re-train-model-and-evaluate">Re-train model and evaluate <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#re-train-model-and-evaluate">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># Re-split data so 'leading_indicator' is in columns (the index remains the same)</span><br />train.df <span class="token operator"><-</span> model.df<span class="token punctuation">[</span>index<span class="token punctuation">,</span> <span class="token punctuation">]</span><br />test.df <span class="token operator"><-</span> model.df<span class="token punctuation">[</span><span class="token operator">-</span>index<span class="token punctuation">,</span> <span class="token punctuation">]</span></code></pre>
<h3 id="logistic-2">Logistic <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#logistic-2">#</a></h3>
<pre class="language-r"><code class="language-r"><span class="token comment"># Run model</span><br />lead.logistic.model <span class="token operator"><-</span> glm<span class="token punctuation">(</span>churn <span class="token operator">~</span> incorporation_time <span class="token operator">+</span> vertical <span class="token operator">+</span> leading_indicator<span class="token punctuation">,</span><br /> data <span class="token operator">=</span> train.df<span class="token punctuation">,</span><br /> family <span class="token operator">=</span> binomial<span class="token punctuation">(</span>link <span class="token operator">=</span> <span class="token string">'logit'</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Predict</span><br />log.pred <span class="token operator"><-</span> predict<span class="token punctuation">(</span>lead.logistic.model<span class="token punctuation">,</span> newdata <span class="token operator">=</span> test.df<span class="token punctuation">,</span> type <span class="token operator">=</span> <span class="token string">'response'</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Convert probs to binary</span><br />log.pred <span class="token operator"><-</span> as.factor<span class="token punctuation">(</span>ifelse<span class="token punctuation">(</span>log.pred <span class="token operator">></span> <span class="token number">0.5</span><span class="token punctuation">,</span> <span class="token number">1</span><span class="token punctuation">,</span> <span class="token number">0</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Evaluation Metrics</span><br />lead.log.result <span class="token operator"><-</span> confusionMatrix<span class="token punctuation">(</span>data <span class="token operator">=</span> log.pred<span class="token punctuation">,</span> test.df<span class="token operator">$</span>churn<span class="token punctuation">)</span><br />lead.log.precision <span class="token operator"><-</span> log.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Pos Pred Value'</span><span class="token punctuation">]</span><br />lead.log.recall <span class="token operator"><-</span> log.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'Sensitivity'</span><span class="token punctuation">]</span><br />lead.log.F1 <span class="token operator"><-</span> log.result<span class="token operator">$</span>byClass<span class="token punctuation">[</span><span class="token string">'F1'</span><span class="token punctuation">]</span></code></pre>
<h3 id="evaluation-metrics-2">Evaluation Metrics <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#evaluation-metrics-2">#</a></h3>
<p>Compare the precision and recall of the logistic model with and without the <code>lead_indicator</code>.</p>
<pre class="language-r"><code class="language-r">log.precision<br /><span class="token comment">## Pos Pred Value</span><br /><span class="token comment">## 0.8333333</span></code></pre>
<pre class="language-r"><code class="language-r">lead.log.precision<br /><span class="token comment">## Pos Pred Value</span><br /><span class="token comment">## 0.8333333</span></code></pre>
<pre class="language-r"><code class="language-r">log.recall<br /><span class="token comment">## Sensitivity</span><br /><span class="token comment">## 0.9016393</span></code></pre>
<pre class="language-r"><code class="language-r">lead.log.recall<br /><span class="token comment">## Sensitivity</span><br /><span class="token comment">## 0.9016393</span></code></pre>
<pre class="language-r"><code class="language-r"><span class="token comment"># Have a look at the model coefficients and p-values</span><br />summary<span class="token punctuation">(</span>lead.logistic.model<span class="token punctuation">)</span><br /><br /><span class="token comment">## Call:</span><br /><span class="token comment">## glm(formula = churn ~ incorporation_time + vertical + leading_indicator,</span><br /><span class="token comment">## family = binomial(link = "logit"), data = train.df)</span><br /><span class="token comment">##</span><br /><span class="token comment">## Deviance Residuals:</span><br /><span class="token comment">## Min 1Q Median 3Q Max </span><br /><span class="token comment">## -1.4429 -0.8252 -0.4957 0.9938 2.5564 </span><br /><span class="token comment">##</span><br /><span class="token comment">## Coefficients:</span><br /><span class="token comment">## Estimate Std. Error z value Pr(>|z|)</span><br /><span class="token comment">## (Intercept) 0.74066 0.20007 3.702 0.000214 ***</span><br /><span class="token comment">## incorporation_time -0.26140 0.03045 -8.584 < 2e-16 ***</span><br /><span class="token comment">## verticalfreelance developer 0.12614 0.21737 0.580 0.561714</span><br /><span class="token comment">## verticalgym/fitness -0.91099 0.22342 -4.077 4.55e-05 ***</span><br /><span class="token comment">## leading_indicator 18.06219 482.97671 0.037 0.970168</span><br /><span class="token comment">##</span><br /><span class="token comment">## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1</span><br /><span class="token comment">##</span><br /><span class="token comment">## (Dispersion parameter for binomial family taken to be 1)</span><br /><span class="token comment">##</span><br /><span class="token comment">## Null deviance: 907.42 on 722 degrees of freedom</span><br /><span class="token comment">## Residual deviance: 736.18 on 718 degrees of freedom</span><br /><span class="token comment">## AIC: 746.18</span><br /><span class="token comment">##</span><br /><span class="token comment">## Number of Fisher Scoring iterations: 15</span></code></pre>
<p>Generally, it is best to minimise the <code>AIC</code>. The <code>Logistic.model</code> model AIC is 800.5 compared with the <code>lead.logistic.model</code> incorporating the <code>lead_indicator</code> is 746.18. The <code>lead_indicator</code> has not changed the predictive power on this dataset, but since the <code>AIC</code> favours a better model fit whilst penalising for additional predictors, the model to choose is the <code>lead.logistic.model</code>.</p>
<h4 id="re-train-model-and-save">Re-train model and save <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#re-train-model-and-save">#</a></h4>
<pre class="language-r"><code class="language-r"><span class="token comment"># Re-train on all data</span><br />final.model <span class="token operator"><-</span> glm<span class="token punctuation">(</span>churn <span class="token operator">~</span> incorporation_time <span class="token operator">+</span> vertical <span class="token operator">+</span> leading_indicator<span class="token punctuation">,</span><br /> data <span class="token operator">=</span> model.df<span class="token punctuation">,</span><br /> family <span class="token operator">=</span> binomial<span class="token punctuation">(</span>link <span class="token operator">=</span> <span class="token string">'logit'</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<h2 id="recommendations-for-further-investigation%2Fcomments">Recommendations for further investigation/Comments <a class="direct-link" href="https://lukesingham.com/how-to-make-a-churn-model-in-r/#recommendations-for-further-investigation%2Fcomments">#</a></h2>
<ul>
<li>The <code>lead_indicator</code> was a quick and dirty test, Iâd spend more time looking at a better construction (e.g. moving avg)</li>
<li>If the cost of rentention activities vs losing a customer was known then an the optimal trade-off in terms of business cost could be found.</li>
<li>Incorporating additional data, e.g. CRM data showing interactions. Potentially assessing sales/account staff.</li>
<li>If it were a production model prompting staff to do retention calls, evaluate the impact of such calls through modelling e.g. a/b testing</li>
<li>Test model performance with a change to balancing the classes e.g. under/oversampling, boostrap samples. This may explain the relative underperformance of tree based models in this exercise.</li>
<li>Try other binary classifier models.</li>
</ul>
How to Deploy Plotly's Dash using Shinyproxy2017-11-14T00:00:00Zhttps://lukesingham.com/how-to-deploy-plotlys-dash-using-shinyproxy/<p>In a <a href="https://lukesingham.com/shiny-containers-with-shinyproxy/#addingadditionalnonshinyapps">previous post</a> I established that I could easily deploy a 'Hello World' <a href="http://flask.pocoo.org/">flask.py</a> web application using <a href="https://www.shinyproxy.io/">Shinyproxy</a>. Therefore, I thought it would be straightforward to deploy a <a href="https://plot.ly/dash/">Dash</a> app which is built on top of <a href="http://flask.pocoo.org/">flask.py</a>. However, it proved to be a little more difficult than that. This blog post runs through the errors and eventual solution to deploying a Dash app on Shinyproxy.</p>
<h3 id="the-issue">The Issue <a class="direct-link" href="https://lukesingham.com/how-to-deploy-plotlys-dash-using-shinyproxy/#the-issue">#</a></h3>
<p>Dash apps, like most web apps, load additional static resources, but upon deploying on Shinyproxy these aren't being delivered to the client (browser) as the URL path is wrong. This results in the all too familiar <strong>404 Not Found</strong>. <em>Disclaimer</em>: I haven't programmed in Java or SpringBoot (yet!). However, as far as I can see from looking at <a href="https://developer.chrome.com/devtools">Chrome dev tools</a> there are two ways Shinyproxy delivers static files to the client. I'm taking a look first at how it works with a <a href="https://shiny.rstudio.com/">Shiny</a> app.</p>
<ol>
<li>
<p><strong>Webjars</strong> - I can see that the <a href="https://github.com/openanalytics/shinyproxy/blob/master/src/main/resources/templates/app.html">main template of shinyproxy</a> loads the bootstrap CSS which also gets utilised by each Shiny app.</p>
</li>
<li>
<p><strong>Load from the container</strong> - Additional static content for shiny apps is requested by using the container name, in this case <code>peaceful_jepsen</code> e.g.:</p>
</li>
</ol>
<pre class="language-bash"><code class="language-bash">Request URL:http://<span class="token operator"><</span>my-ip-address<span class="token operator">></span>/peaceful_jepsen/dt-core-1.10.12/css/jquery.dataTables.min.css</code></pre>
<p>Now let's move on to it not working with Dash...</p>
<h2 id="deploying-a-dash-app-on-shinyproxy">Deploying a Dash App on Shinyproxy <a class="direct-link" href="https://lukesingham.com/how-to-deploy-plotlys-dash-using-shinyproxy/#deploying-a-dash-app-on-shinyproxy">#</a></h2>
<p>Using <code>react.min.js</code> as an example, I'm going to contrast how the app fetches static content on my local machine in a container vs Shinyproxy on a remote server. I'll do this by inspecting the path for the <code>GET</code> requests using <a href="https://developer.chrome.com/devtools">Chrome dev tools</a>. Firstly, what is the default behaviour of Dash.</p>
<blockquote>
<p>By default, dash serves the JavaScript and CSS resources from the online CDNs.<br />
Source: <a href="https://plot.ly/dash/external-resources">Dash Docs</a></p>
</blockquote>
<p><strong>Dash Container on my Local Machine</strong><br />
<img src="https://lukesingham.com/content/images/2017/11/local-dash-cdn-get.png" alt="local-dash-cdn-get" /></p>
<p>All resources loaded and the application displayed correctly.</p>
<p><strong>Dash Container on Shinyproxy</strong><br />
<img src="https://lukesingham.com/content/images/2017/11/shinyproxy-dash-cdn-get.png" alt="shinyproxy-dash-cdn-get" /></p>
<p>On Shinyproxy <code>react.min.js</code> gets loaded however <code>_dash-layout</code> and <code>_dash-dependencies</code> fail to GET the necessary resources from the <code>Request URL: http://<my-ip-address>/_dash-layout</code>. As we know from the Shiny app, the container name should be in there e.g. <code>Request URL: http://<my-ip-address>/<container_name>/_dash-layout</code>. This is how Dash is supposed to behave, looking at <a href="https://github.com/plotly/dash/blob/4ee769d3593d5297602c2bb6faca0eb63884d480/dash/dash.py#L74">the source code</a> we can see a configurable <code>routes_pathname_prefix</code> (I'll come back to this later in the post). So a possible solution is to get the Dash app to obtain the container name from which it is running.</p>
<p>The <a href="https://plot.ly/dash/external-resources">Dash Docs</a> have the following lines of code to serve content locally.</p>
<pre class="language-python"><code class="language-python">app<span class="token punctuation">.</span>css<span class="token punctuation">.</span>config<span class="token punctuation">.</span>serve_locally <span class="token operator">=</span> <span class="token boolean">True</span><br />app<span class="token punctuation">.</span>scripts<span class="token punctuation">.</span>config<span class="token punctuation">.</span>serve_locally <span class="token operator">=</span> <span class="token boolean">True</span></code></pre>
<p>Let's see how that changes the behaviour.</p>
<p><strong>Dash Container on my Local Machine</strong><br />
<img src="https://lukesingham.com/content/images/2017/11/local-dash-serve-local-get.png" alt="local-dash-serve-local-get" /><br />
Since all static content is now requested from Dash this is probably going to prevent all content from being loaded on Shinyproxy.</p>
<p><strong>Dash Container on Shinyproxy</strong><br />
<img src="https://lukesingham.com/content/images/2017/11/shinyproxy-dash-serve-local-get.png" alt="shinyproxy-dash-serve-local-get" /><br />
Yup! A whole lot of red 404s.</p>
<p>Essentially it doesn't matter if you configure Dash to serve static files locally, the underlying issue is the relative URL route prefix. We know from the <a href="https://github.com/plotly/dash/blob/master/dash/dash.py#L50">Dash source code</a> how we could prefix the URL route. However, Shinyproxy is already doing this for Shiny apps, what's different for the requests from Dash?</p>
<p>I started scouring the Shinyproxy Java code base. I found the construction of the <code>containerPath</code> in <a href="https://github.com/openanalytics/shinyproxy/blob/f934f108573f1ed1d24a719d9e0815012240e11f/src/main/java/eu/openanalytics/controllers/AppController.java#L74">Shinyproxy's AppController.java</a>. But since I am not a Java developer I decided that this would probably not be the best allocation of my efforts. However, what the <code>containerPath</code> does show is the construction of the path begins with <code>'/'</code>.</p>
<p>Another approach I thought of was obtaining the container name from within the Dash application and appending that to the GET URL requests initiated Dash. Alas, Docker doesn't allow that. You can access the ContainerID but not the ContainerName. There is a <a href="https://github.com/moby/moby/issues/8427">3 year old open issue regarding this</a>. I then started looking for an easy way to pass into the container the containerName but I couldn't find a non-hacky way to do that either.</p>
<p><strong>Sidenote</strong> - I was wondering what the logic was on the container name generation e.g. <code>adorin_raman</code>. If you look at the docker source code you will find <a href="https://github.com/moby/moby/blob/master/pkg/namesgenerator/names-generator.go">some go-lang code called <code>names-generator.go</code>.</a> This contains two lists, a list of adjectives and a list of "notable scientists and hackers". The function that randomly combines these into a container name formatted as "adjective_surname" has one funny exception:</p>
<pre class="language-go"><code class="language-go"><span class="token keyword">if</span> name <span class="token operator">==</span> <span class="token string">"boring_wozniak"</span> <span class="token comment">/* Steve Wozniak is not boring */</span> <span class="token punctuation">{</span><br /> <span class="token keyword">goto</span> begin<br /><span class="token punctuation">}</span></code></pre>
<p>Two other approaches that weren't very appealing would have been managing the resources as <a href="https://www.webjars.org/">webjars</a> or an <a href="https://nginx.org/en/">Nginx</a> redirect.</p>
<h2 id="the-final-solution">The Final Solution <a class="direct-link" href="https://lukesingham.com/how-to-deploy-plotlys-dash-using-shinyproxy/#the-final-solution">#</a></h2>
<p>I posted <a href="https://support.openanalytics.eu/t/what-is-the-best-way-of-delivering-static-assets-to-the-client-for-custom-apps/363/5">a question to the shinyproxy forum</a>, <a href="http://www.fcm-consulting.be/resume.html">Frederick</a> one of the main authors responded with:</p>
<blockquote>
<p>Then that means the href being used was <code>/_dash-layout</code> and not <code>_dash-layout</code>.<br />
Do you control the hrefs? If so, removing the slash may solve your issue. If not, it becomes trickier⌠you could use javascript and extract the correct URL from window.location.</p>
</blockquote>
<p>If we dig into the <a href="https://github.com/plotly/dash/blob/master/dash/dash.py">dash.py code</a> We can see the construction of the URL paths for <code>_dash-layout</code> <code>_dash-dependencies</code> occurs on lines 73-80:</p>
<pre class="language-python"><code class="language-python">add_url<span class="token punctuation">(</span><br /> <span class="token string">'{}_dash-layout'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>config<span class="token punctuation">[</span><span class="token string">'routes_pathname_prefix'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br /> self<span class="token punctuation">.</span>serve_layout<span class="token punctuation">)</span><br /><br />add_url<span class="token punctuation">(</span><br /> <span class="token string">'{}_dash-dependencies'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span>self<span class="token punctuation">.</span>config<span class="token punctuation">[</span><span class="token string">'routes_pathname_prefix'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">,</span> self<span class="token punctuation">.</span>dependencies<span class="token punctuation">)</span></code></pre>
<p>The next question is, where is <code>routes_pathname_prefix</code> defined? On lines 46-51 <code>routes_pathname_prefix</code> is mapped to <code>url_base_pathname</code>:</p>
<pre class="language-python"><code class="language-python">self<span class="token punctuation">.</span>url_base_pathname <span class="token operator">=</span> url_base_pathname<br />self<span class="token punctuation">.</span>config <span class="token operator">=</span> _AttributeDict<span class="token punctuation">(</span><span class="token punctuation">{</span><br /> <span class="token string">'suppress_callback_exceptions'</span><span class="token punctuation">:</span> <span class="token boolean">False</span><span class="token punctuation">,</span><br /> <span class="token string">'routes_pathname_prefix'</span><span class="token punctuation">:</span> url_base_pathname<span class="token punctuation">,</span><br /> <span class="token string">'requests_pathname_prefix'</span><span class="token punctuation">:</span> url_base_pathname<br /><span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
<p>Which begs the question, where is url <code>url_base_pathname</code> defined... on lines 20-28:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">class</span> <span class="token class-name">Dash</span><span class="token punctuation">(</span><span class="token builtin">object</span><span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token keyword">def</span> <span class="token function">__init__</span><span class="token punctuation">(</span><br /> self<span class="token punctuation">,</span><br /> name<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">,</span><br /> server<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">,</span><br /> static_folder<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">,</span><br /> url_base_pathname<span class="token operator">=</span><span class="token string">'/'</span><span class="token punctuation">,</span><br /> <span class="token operator">**</span>kwargs<br /> <span class="token punctuation">)</span><span class="token punctuation">:</span></code></pre>
<p>As Frederick said, the request should not have the <code>'/'</code> prefix in the GET request. Unsurprisingly, having <code>'//'</code> in your request paths is not good practice (see this <a href="https://stackoverflow.com/a/20524044/3691003">SO answer</a> and <a href="https://stackoverflow.com/a/10161264/3691003">this one</a>) as it can cause problems depending on how requests are handled. The <a href="https://lukesingham.com/shiny-containers-with-shinyproxy/#post-content">software layers of Shinyproxy</a> are reasonably complex with requests being handled my multiple technologies. The best solution after this trouble-shooting exercise is to use the following code in your Dash app.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># In order to work on shinyproxy (and perhaps other middleware)</span><br />app<span class="token punctuation">.</span>config<span class="token punctuation">.</span>supress_callback_exceptions <span class="token operator">=</span> <span class="token boolean">True</span><br />app<span class="token punctuation">.</span>config<span class="token punctuation">.</span>update<span class="token punctuation">(</span><span class="token punctuation">{</span><br /> <span class="token comment"># remove the default of '/'</span><br /> <span class="token string">'routes_pathname_prefix'</span><span class="token punctuation">:</span> <span class="token string">''</span><span class="token punctuation">,</span><br /><br /> <span class="token comment"># remove the default of '/'</span><br /> <span class="token string">'requests_pathname_prefix'</span><span class="token punctuation">:</span> <span class="token string">''</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span></code></pre>
How to Use Shiny Containers with Shinyproxy2017-10-11T00:00:00Zhttps://lukesingham.com/shiny-containers-with-shinyproxy/<p><strong>Aim</strong>: Setup <a href="https://www.shinyproxy.io/">shinyproxy</a> as a production level way of deploying multiple containerised shiny apps with authentication. Additionally I'll demonstrate how to incorporate containerised python web apps and deploy this behind a reverse proxy using Nginx.</p>
<p><strong>Conceptual Overview</strong><br />
<img src="https://lukesingham.com/content/images/2017/10/overview.png" alt="" /></p>
<p>This is a pretty significant piece of software. It renders <a href="https://www.rstudio.com/products/shiny-server-pro/">Shiny Server Pro</a> redundant. As this solution, provides authentication, scalability, decreases maintenance ( containers) and enables any web app to be running behind the scenes. So you could have a <a href="https://shiny.rstudio.com/gallery/">shiny</a> app, a <a href="https://www.djangoproject.com/">Django</a> app, a <a href="http://flask.pocoo.org/">flask</a> app, <a href="https://vuejs.org/">Vue.js</a>... and the end-user will simply see a website, whilst the backend has a whole range of different web applications spinning up on demand.</p>
<p>I ran into some hurdles with <a href="https://www.shinyproxy.io/getting-started/#prerequisites">these instructions</a>. So I'm documenting what worked for me. <strong>My setup:</strong> I set this up once on a Dell XPS 13 with a fresh install of Ubuntu 16.04.3 LTS and a Ubuntu 16.04.03 Azure VM.</p>
<h2 id="pre-requisites-%2F-assumptions">Pre-requisites / Assumptions <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#pre-requisites-%2F-assumptions">#</a></h2>
<h3 id="docker">Docker <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#docker">#</a></h3>
<p>If you don't have <a href="https://www.docker.com/">Docker</a> installed yet, I highly recommend running through <a href="https://docs.docker.com/get-started/">Part 1: Orientation and setup</a> to <a href="https://docs.docker.com/get-started/part3/">Part 3: Services</a> of <a href="https://docs.docker.com/get-started/">Docker's Getting Started Documentation</a>.</p>
<p>If running the following commands return similar console prints you should be fine to continue on.</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">docker</span> <span class="token parameter variable">-v</span><br /><span class="token comment"># Docker version 17.06.2-ce, build cec0b72</span><br /><br />$ <span class="token function">docker</span> run hello-world<br /><span class="token comment"># Hello from Docker!</span><br /><span class="token comment"># This message shows that your installation appears to be working correctly.</span></code></pre>
<h4 id="docker-problems-encountered-for-ubuntu-16">Docker Problems Encountered for Ubuntu 16 <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#docker-problems-encountered-for-ubuntu-16">#</a></h4>
<p>The most time consuming part of the setup were <a href="https://support.openanalytics.eu/t/shinyproxy-on-ubuntu-16-04/38/4">the instructions for Ubuntu 16</a> on the <a href="https://www.shinyproxy.io/">shinyproxy site</a> guiding modification of <code>/lib/systemd/system/docker.service</code>. Per the documentation on Shinyproxy:</p>
<p><img src="https://lukesingham.com/content/images/2017/09/shinyproxy-ubuntu16-docker-config.png" alt="shinyproxy-docker-instructions" /></p>
<p>Running this configuration produced the following error:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">sudo</span> systemctl restart <span class="token function">docker</span><br /><span class="token comment"># Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.</span></code></pre>
<h4 id="solution">Solution <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#solution">#</a></h4>
<p>Instead, edit <code>/lib/systemd/system/docker.service</code> and set <code>ExecStart</code> to:</p>
<pre class="language-docker"><code class="language-docker">ExecStart=/usr/bin/dockerd -H fd:// -D -H tcp://127.0.0.1:2375</code></pre>
<p>Why? Shinyproxy will look to talk to the docker daemon on <code>tcp://127.0.0.1:2375</code>. If it can't, then the application won't work.</p>
<p>Ensure that you restart docker after editing the config with the following commands:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">sudo</span> systemctl daemon-reload<br />$ <span class="token function">sudo</span> systemctl restart <span class="token function">docker</span></code></pre>
<h3 id="java8">Java8 <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#java8">#</a></h3>
<p>Shinyproxy requires a minimum of <a href="http://www.oracle.com/technetwork/java/javase/overview/java8-2100321.html">Java 8</a>. I followed the advice per <a href="https://www.shinyproxy.io/getting-started/">shinyproxy.io</a> and installed the OpenJDK <a href="http://zulu.org/">Zulu</a>. <a href="https://www.quora.com/What-is-it-like-to-use-Azul-Systems-Zulu-JVM/answer/Matt-Schuetze">This Quora answer from the product manager of Zulu</a> gives some context behind this JDK.</p>
<p>Run the following at the command line:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">java</span> <span class="token parameter variable">-version</span><br /><span class="token comment"># openjdk version "1.8.0_144"</span><br /><span class="token comment"># OpenJDK Runtime Environment (Zulu 8.23.0.3-linux64) (build 1.8.0_144-b01)</span><br /><span class="token comment"># OpenJDK 64-Bit Server VM (Zulu 8.23.0.3-linux64) (build 25.144-b01, mixed mode)</span></code></pre>
<p>If you have Java 8 then continue. N.B. Zulu 8.xx.x.x is Java 8. Otherwise, run the following commands to install:</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">sudo</span> apt-key adv <span class="token parameter variable">--keyserver</span> hkp://keyserver.ubuntu.com:80 --recv-keys 0x219BD9C9<br />$ <span class="token assign-left variable">apt_source</span><span class="token operator">=</span><span class="token string">'deb http://repos.azulsystems.com/debian stable main'</span><br />$ <span class="token assign-left variable">apt_list</span><span class="token operator">=</span><span class="token string">'/etc/apt/sources.list.d/zulu.list'</span><br />$ <span class="token builtin class-name">echo</span> <span class="token string">"<span class="token variable">$apt_source</span>"</span> <span class="token operator">|</span> <span class="token function">sudo</span> <span class="token function">tee</span> <span class="token string">"<span class="token variable">$apt_list</span>"</span> <span class="token operator">></span> /dev/null<br />$ <span class="token function">sudo</span> <span class="token function">apt-get</span> update<br />$ <span class="token function">sudo</span> <span class="token function">apt-get</span> <span class="token function">install</span> zulu-8</code></pre>
<p><strong>N.B.</strong> I did experience a maven build fail when running <code>mvn -U clean install</code> on a Mac with a Java9 runtime. Uninstalling Java 9 and running <code>brew cask install java8</code> fixed the problem.</p>
<h2 id="download-and-compile-shiny-proxy">Download and Compile Shiny Proxy <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#download-and-compile-shiny-proxy">#</a></h2>
<p>There are two options for downloading the shinyproxy software.</p>
<p><strong>Option 1</strong> - I cloned the <a href="https://github.com/openanalytics/shinyproxy/">shinyproxy repo</a>. I did not have <a href="https://maven.apache.org/">apache maven</a> installed so I needed to <a href="https://maven.apache.org/install.html">run through the installation of maven first</a>.</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Clone the repo</span><br />$ <span class="token function">git</span> clone https://github.com/openanalytics/shinyproxy.git<br /><br /><span class="token comment"># Change directory</span><br />$ <span class="token builtin class-name">cd</span> shinyproxy/<br /><br /><span class="token comment"># Compile the program</span><br />$ mvn <span class="token parameter variable">-U</span> clean <span class="token function">install</span></code></pre>
<p><strong>Option 2</strong> - You can also go to the <a href="https://www.shinyproxy.io/downloads/">downloads page</a> and select the latest file for your particular operating system.</p>
<h2 id="pull-the-shiny-app-containers">Pull the shiny app containers <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#pull-the-shiny-app-containers">#</a></h2>
<p>We now have shinyproxy ready to run containerised applications. However, we don't have the images that shinyproxy is expecting. We need to pull those first:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Pull the shiny app images</span><br />$ <span class="token function">sudo</span> <span class="token function">docker</span> pull openanalytics/shinyproxy-demo<br /><br /><span class="token comment"># Check they are now in the list of docker images</span><br />$ <span class="token function">docker</span> images <span class="token operator">|</span> <span class="token function">grep</span> shinyproxy</code></pre>
<h2 id="run-shiny-proxy">Run Shiny Proxy <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#run-shiny-proxy">#</a></h2>
<p>Run the following commands:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Ensure you are in the right directory</span><br />$ <span class="token builtin class-name">cd</span> ~/shinyproxy/target/<br /><br /><span class="token comment"># Run shinyproxy</span><br />$ <span class="token function">java</span> <span class="token parameter variable">-jar</span> shinyproxy-1.0.0.jar</code></pre>
<p>At this point you should now be able to navigate to <a href="http://localhost:8080/">http://localhost:8080</a> and see the sign in page and login with 'tesla' and 'password' as username and password respectively. If you are doing this on a remote server, test the application is up and running by running:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Print the html of the site to the terminal</span><br /><span class="token function">curl</span> http://localhost:8080</code></pre>
<p><img src="https://lukesingham.com/content/images/2017/09/shinyproxy-login.png" alt="shinyproxy-login" /></p>
<p>Once logged in launch an application.</p>
<p><img src="https://lukesingham.com/content/images/2017/09/shinyproxy-select-app.png" alt="shinyproxy-select-app" /></p>
<p>And bingo we have a containerised and authenticated shiny environment.</p>
<p><img src="https://lukesingham.com/content/images/2017/09/shinyproxy-shiny-app.png" alt="" /></p>
<h2 id="adding-additional-shiny-apps">Adding Additional Shiny Apps <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#adding-additional-shiny-apps">#</a></h2>
<p><a href="https://www.shinyproxy.io/deploying-apps/">Shinyproxy docs</a> provide a ready to go shiny application with a <a href="https://docs.docker.com/engine/reference/builder/">dockerfile</a> to build the image. Of course, to use shinyproxy for your own custom shiny applications the dockerfile in the <a href="https://github.com/openanalytics/shinyproxy-template">shinyproxy-template repo</a> acts as a template to follow. I'm going to go through the process of adding shinyproxy's pre-built shiny app.</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Clone the shiny app repo</span><br />$ <span class="token function">git</span> clone https://github.com/openanalytics/shinyproxy-template.git<br /><br /><span class="token comment"># Change into the repo</span><br />$ <span class="token builtin class-name">cd</span> shinyproxy-template/<br /><br /><span class="token comment"># Build the image</span><br />$ <span class="token function">docker</span> build <span class="token parameter variable">-t</span> openanalytics/shinyproxy-template <span class="token builtin class-name">.</span><br /><br /><span class="token comment"># Check the image is now available in docker</span><br />$ <span class="token function">docker</span> images <span class="token operator">|</span> <span class="token function">grep</span> <span class="token string">"shinyproxy-template\|REPOSITORY"</span><br /><span class="token comment"># REPOSITORY TAG IMAGE ID CREATED SIZE</span><br /><span class="token comment"># openanalytics/shinyproxy-template latest 16e8c49e2261 25 minutes ago 851MB</span><br /><br /><span class="token comment"># Check the shiny app runs normally outside shinyproxy</span><br />$ <span class="token function">docker</span> run <span class="token parameter variable">-it</span> <span class="token parameter variable">-p</span> <span class="token number">3838</span>:3838 openanalytics/shinyproxy-template</code></pre>
<p>The new shiny app should now be running on <a href="http://0.0.0.0:3838/">http://0.0.0.0:3838/</a>.</p>
<p><img src="https://lukesingham.com/content/images/2017/09/additional-shiny-app.png" alt="additional-shiny-app" /></p>
<p>Now we need to tell shinyproxy to include this additional shiny app by editing creating and editing a <code>application.yml</code> file.</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Ensure you are in the right directory</span><br />$ <span class="token builtin class-name">cd</span> ~/shinyproxy/target/<br /><br /><span class="token comment"># Create application.yml from repo template</span><br />$ <span class="token function">curl</span> https://raw.githubusercontent.com/openanalytics/shinyproxy/master/src/main/resources/application-demo.yml <span class="token operator">></span> application.yml</code></pre>
<p>We now need to add our new shiny app to the <code>application.yml</code> file.</p>
<pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">apps</span><span class="token punctuation">:</span><br /> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> euler<br /> <span class="token key atrule">display-name</span><span class="token punctuation">:</span> Euler's number<br /> <span class="token key atrule">description</span><span class="token punctuation">:</span> Adding another app to shinyproxy<br /> <span class="token key atrule">docker-cmd</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"R"</span><span class="token punctuation">,</span> <span class="token string">"-e shiny::runApp('/root/euler')"</span><span class="token punctuation">]</span><br /> <span class="token key atrule">docker-image</span><span class="token punctuation">:</span> openanalytics/shinyproxy<span class="token punctuation">-</span>template<br /> <span class="token key atrule">groups</span><span class="token punctuation">:</span> scientists</code></pre>
<p>Now I'm going to run shinyproxy again to confirm the change has been successful.</p>
<p><img src="https://lukesingham.com/content/images/2017/09/add-another-shiny-app.png" alt="add-another-shiny-app" /></p>
<p>And does the application launch</p>
<p><img src="https://lukesingham.com/content/images/2017/09/euler-app-running.png" alt="" /></p>
<p>Success!</p>
<h2 id="adding-additional-non-shiny-apps">Adding Additional non-Shiny Apps <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#adding-additional-non-shiny-apps">#</a></h2>
<p>I'm going with a very basic 'hello world' flask app. The main issue here is configuring the ports for shinyproxy to pick up the app. Since shiny by default runs on port 3838, I mapped the containerised flask app to run on port 3838 too. I pushed <a href="https://github.com/ucg8j/flask-port3838-container">a flask repo</a> to github where I adjusted the 'hello world' <a href="https://github.com/ucg8j/flask-port3838-container/blob/master/app.py">flask app</a> on line 28 to run on port 3838 (inside the container) and the <a href="https://github.com/ucg8j/flask-port3838-container/blob/master/Dockerfile">dockerfile</a> to expose the container's port 3838 (line 14).</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Clone repo</span><br />$ <span class="token function">git</span> clone https://github.com/ucg8j/flask-port3838-container.git<br /><br /><span class="token comment"># Change directory</span><br />$ <span class="token builtin class-name">cd</span> flask-port3838-container/<br /><br /><span class="token comment"># Build image</span><br />$ <span class="token function">docker</span> build <span class="token parameter variable">-t</span> flaskapp <span class="token builtin class-name">.</span><br /><br /><span class="token comment"># Test it runs as a docker container</span><br />$ <span class="token function">docker</span> run <span class="token parameter variable">-p</span> <span class="token number">4000</span>:3838 flaskapp</code></pre>
<p>You'll note that I am passing the publish argument <code>-p 4000:3838</code>. This allows you to map a container's port to the port of your host in the format of <code>hostport:containerport</code>. Since I've configured the flask docker container to port 3838 to mimic a shiny app I'll see whether I can manually map it to my computer's localhost of port 4000. That way I'll know that shinyproxy <em>should</em> be able to do the same.</p>
<p><img src="https://lukesingham.com/content/images/2017/09/flask-hello-world.png" alt="flask-hello-world" /></p>
<p>Now to add the flask hello world image to shinyproxy by appending it to the app similar to the way we did with the additional shiny app:</p>
<pre class="language-yaml"><code class="language-yaml"><span class="token key atrule">apps</span><span class="token punctuation">:</span><br /> <span class="token punctuation">-</span> <span class="token key atrule">name</span><span class="token punctuation">:</span> flaskapp<br /> <span class="token key atrule">display-name</span><span class="token punctuation">:</span> Demo Flask app<br /> <span class="token key atrule">description</span><span class="token punctuation">:</span> Adding another non shiny app to shinyproxy<br /> <span class="token key atrule">docker-cmd</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">"python"</span><span class="token punctuation">,</span> <span class="token string">"app.py"</span><span class="token punctuation">]</span><br /> <span class="token key atrule">docker-image</span><span class="token punctuation">:</span> flaskapp<br /> <span class="token key atrule">groups</span><span class="token punctuation">:</span> scientists</code></pre>
<p>Now (re)start shiny proxy <code>java -jar shinyproxy-1.0.0.jar</code>.</p>
<p><img src="https://lukesingham.com/content/images/2017/10/add-flask-app.gif" alt="add-flask-app" /></p>
<h3 id="auth">Auth <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#auth">#</a></h3>
<p>By default the authentication is setup to use a demo ldap server <code>ldap://ldap.forumsys.com:389/dc=example,dc=com</code> for more details visit <a href="http://www.forumsys.com/tutorials/integration-how-to/ldap/online-ldap-test-server/">this page</a>. I'm going to switch this to use the <code>simple</code> authentication. I'm also going to change one of the passwords to be confident these changes are taking effect. The top half of my <code>application.yml</code> file now looks like this:</p>
<p><img src="https://lukesingham.com/content/images/2017/09/application-switch-auth-to-simple.png" alt="application-switch-auth-to-simple" /></p>
<h2 id="deploy-on-a-cloud-service-(nginx-config)">Deploy on a cloud service (nginx config) <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#deploy-on-a-cloud-service-(nginx-config)">#</a></h2>
<p>At this point it would be nice to have this on the cloud. Using <a href="https://nginx.org/en/">nginx</a> the following config worked for me which was an adaption of the config listed on the <a href="https://www.shinyproxy.io/security/">shinyproxy site</a>. The box I was using was an <a href="https://azure.microsoft.com/en-gb/services/virtual-machines/">Azure VM</a>.</p>
<pre class="language-nginx"><code class="language-nginx"><span class="token comment"># Navigate into the nginx config</span><br />$ <span class="token directive"><span class="token keyword">cd</span> /etc/nginx/sites-available<br /><span class="token comment"># print the config that works for me</span><br />$ cat default<br /><span class="token comment"># Default server configuration</span><br />server</span> <span class="token punctuation">{</span><br /> <span class="token directive"><span class="token keyword">listen</span> <span class="token number">80</span> default_server</span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">listen</span> [::]:80 default_server</span><span class="token punctuation">;</span><br /><br /> <span class="token directive"><span class="token keyword">root</span> /var/www/html</span><span class="token punctuation">;</span><br /><br /> <span class="token comment"># Add index.php to the list if you are using PHP</span><br /> <span class="token directive"><span class="token keyword">index</span> index.html index.htm index.nginx-debian.html</span><span class="token punctuation">;</span><br /><br /> <span class="token directive"><span class="token keyword">server_name</span> **put your dns or domain here**</span><span class="token punctuation">;</span><br /><br /> <span class="token directive"><span class="token keyword">location</span> /</span> <span class="token punctuation">{</span><br /><br /> <span class="token directive"><span class="token keyword">proxy_pass</span> http://127.0.0.1:8090/</span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_http_version</span> 1.1</span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_set_header</span> Upgrade <span class="token variable">$http_upgrade</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_set_header</span> Connection <span class="token string">"upgrade"</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_read_timeout</span> <span class="token number">600s</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_redirect</span> <span class="token boolean">off</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_set_header</span> Host <span class="token variable">$http_host</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_set_header</span> X-Real-IP <span class="token variable">$remote_addr</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_set_header</span> X-Forwarded-For <span class="token variable">$proxy_add_x_forwarded_for</span></span><span class="token punctuation">;</span><br /> <span class="token directive"><span class="token keyword">proxy_set_header</span> X-Forwarded-Protocol <span class="token variable">$scheme</span></span><span class="token punctuation">;</span><br /><br /> <span class="token punctuation">}</span><br /><span class="token punctuation">}</span></code></pre>
<p>Once you have shinyproxy running on a cloud server, you may want to leave the application running without having a terminal open. To run leave shinyproxy running in the background run the following:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">nohup</span> <span class="token function">java</span> <span class="token parameter variable">-jar</span> shinyproxy-1.0.0.jar <span class="token operator">&</span></code></pre>
<p><code>nohup</code> is not a permanent solution. For instance, if your server goes down, shinyproxy will not automatically reboot. <code>nohup</code> did allow me to quickly get the app running in the background. Check the app, discover issues I needed to address then find the process by running <code>ps aux | grep java</code> kill that process... rinse and repeat until complete. To leave this permanently on a Ubuntu 16 server the <a href="https://spring.io/docs">Spring documentation</a> has <a href="https://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html#deployment-systemd-service">a configuration guide for setting this as a system service</a>. Additionally, <a href="https://stackoverflow.com/a/22121547/3691003">this SO post</a> is also very helpful.</p>
<h2 id="conclusion">Conclusion <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#conclusion">#</a></h2>
<p>Shinyproxy is a great contribution to the R/Shiny community by the team at <a href="https://www.openanalytics.eu/">openanalytics</a>. So a big thank you to them! <em>Productionising</em> shiny apps can be a bit of a pain considering the open source shiny server limitations (no authentication, no concurrency and no resource monitoring). Even accepting those limitations, <a href="https://lukesingham.com/using-package-management-in-r/">dependency management of R applications</a> is not widely practiced. Even when R applications use <a href="https://rstudio.github.io/packrat/">packrat</a>, language level dependency management doesn't prevent your software breaking due to different system level requirements. Hence, the generalisation of running a website that proxies to containerised applications <em>should</em> be an incredibly reliable architecture.</p>
<h2 id="resources">Resources <a class="direct-link" href="https://lukesingham.com/shiny-containers-with-shinyproxy/#resources">#</a></h2>
<ul>
<li><a href="https://lukesingham.com/how-to-deploy-plotlys-dash-using-shinyproxy/">How to deploy Dash apps on Shinyproxy</a></li>
<li><a href="https://stackoverflow.com/a/40187306/3691003">Helpful SO post</a></li>
<li><a href="https://www.linode.com/docs/web-servers/nginx/how-to-configure-nginx">Configuring nginx</a></li>
<li><a href="https://docs.spring.io/spring-boot/docs/current/reference/html/common-application-properties.html">Spring configuration options i.e. <code>application.yml</code></a></li>
</ul>
Dual Booting Ubuntu and Windows 10 on a Dell XPS 132017-09-20T00:00:00Zhttps://lukesingham.com/dual-booting-ubuntu-and-windows-10-on-a-dell-xps/<p><strong>My Setup</strong>: <a href="http://www.dell.com/en-uk/shop/laptop-and-2-in-1-pcs/xps-13/spd/xps-13-9360-laptop">Dell XPS 13-9360</a> + Windows 10</p>
<p><strong>Aim</strong> - Get Ubuntu on the machine.</p>
<p>Essentially, follow <a href="https://askubuntu.com/a/868294/424657">these instructions</a> with two key modifications.</p>
<ol>
<li>You should be able to get away with switching from <a href="https://en.wikipedia.org/wiki/RAID">RAID</a> to <a href="https://en.wikipedia.org/wiki/Advanced_Host_Controller_Interface">AHCI</a> without re-imaging windows using <a href="http://triplescomputers.com/blog/uncategorized/solution-switch-windows-10-from-raidide-to-ahci-operation/">these instuctions</a>.</li>
<li>Disable <a href="https://en.wikipedia.org/wiki/RAID">Bitlocker</a> - this is crucial! The first time I ran this... I didn't complete this step which led me to screwing up my Windows 10 installation.</li>
</ol>
<p>I neglected Step 2. If the boot priority is Ubuntu first (i.e. Grub) then Ubuntu loads fine. However, windows doesn't load as the Bitlocker detects a change to the BIOS and will require a code to unlock it which, if you're like me, you probably don't have.</p>
<p>I did try running through the <a href="https://help.ubuntu.com/community/Boot-Repair">boot-repair tool</a> from Ubuntu but kept facing the <code>GPT detected...</code> error.</p>
<p><strong>Solution:</strong> I use the BIOS boot priority to change the operating system. Now both Win10 and Ubuntu are running on my machine.</p>
<h2 id="resources">Resources <a class="direct-link" href="https://lukesingham.com/dual-booting-ubuntu-and-windows-10-on-a-dell-xps/#resources">#</a></h2>
<ul>
<li><a href="https://askubuntu.com/a/868294/424657">Main instuctions</a></li>
<li><a href="http://triplescomputers.com/blog/uncategorized/solution-switch-windows-10-from-raidide-to-ahci-operation/">Switching from RAID to AHCI without re-imaging Win10</a></li>
<li><a href="http://triplescomputers.com/blog/uncategorized/solution-switch-windows-10-from-raidide-to-ahci-operation/">Turn off Bitlocker</a></li>
<li><a href="https://www.reddit.com/r/Dell/comments/5xw27t/dell_xps_15_9560_dualboot_windows_10_ubuntu_lts/">Reddit thread</a></li>
</ul>
Anonymous Functions in R and Python2017-09-13T00:00:00Zhttps://lukesingham.com/anonymous-functions-in-r-python/<blockquote>
<p>What's in a name? That which we call a rose by any other name would smell as sweet.</p>
</blockquote>
<h2 id="normal-functions">Normal functions <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#normal-functions">#</a></h2>
<p>Before moving to anonymous functions, let's start with what normal functions look like.</p>
<h4 id="in-r">In R <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#in-r">#</a></h4>
<pre class="language-r"><code class="language-r"><span class="token comment"># Define a function</span><br />functionName <span class="token operator"><-</span> <span class="token keyword">function</span><span class="token punctuation">(</span>variables<span class="token punctuation">)</span> <span class="token punctuation">{</span><br /> <span class="token comment"># Function definition... do some stuff</span><br /> print<span class="token punctuation">(</span>paste<span class="token punctuation">(</span>variables<span class="token punctuation">,</span> <span class="token string">"doing stuff"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token punctuation">}</span><br /><br /><span class="token comment"># Call the function</span><br /><span class="token operator">></span> functionName<span class="token punctuation">(</span><span class="token string">"boring strings"</span><span class="token punctuation">)</span><br /><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token string">"boring strings doing stuff"</span></code></pre>
<h4 id="in-python">In Python <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#in-python">#</a></h4>
<pre class="language-python"><code class="language-python"><span class="token comment"># Define function</span><br /><span class="token keyword">def</span> <span class="token function">functionName</span><span class="token punctuation">(</span>variables<span class="token punctuation">)</span><span class="token punctuation">:</span><br /> <span class="token comment"># Do some stuff</span><br /> <span class="token keyword">print</span> variables<span class="token punctuation">,</span> <span class="token string">"doing stuff"</span><br /><br /><span class="token comment"># Call the function</span><br /><span class="token operator">>></span><span class="token operator">></span> functionName<span class="token punctuation">(</span><span class="token string">"boring strings"</span><span class="token punctuation">)</span><br />boring strings doing stuff</code></pre>
<h2 id="anonymous-functions">Anonymous Functions <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#anonymous-functions">#</a></h2>
<p>Have no identity, no name, but still do stuff! They will not live in the global environment. Like a person without a name, you would not be able to look the person up in the address book.</p>
<p>The anonymous function can be called like a normal function <code>functionName()</code>, except the functionName is switched for logic contained within parentheses <code>(fn logic goes here)()</code>.</p>
<h4 id="in-r-2">In R <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#in-r-2">#</a></h4>
<pre class="language-r"><code class="language-r"><span class="token comment"># Doing the same stuff anonymously</span><br /><span class="token operator">></span> <span class="token punctuation">(</span><span class="token keyword">function</span><span class="token punctuation">(</span>variables<span class="token punctuation">)</span> print<span class="token punctuation">(</span>paste<span class="token punctuation">(</span>variables<span class="token punctuation">,</span> <span class="token string">"doing stuff"</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token string">"code"</span><span class="token punctuation">)</span><br /><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token string">"code doing stuff"</span></code></pre>
<h4 id="in-python-2">In Python <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#in-python-2">#</a></h4>
<p>Python introduces the <code>lambda</code> keyword for anonymous functions, in contrast to R which sticks with the <code>function</code> keyword.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Doing the same stuff anonymously</span><br /><span class="token operator">>></span><span class="token operator">></span> <span class="token punctuation">(</span><span class="token keyword">lambda</span> variable<span class="token punctuation">:</span> variable <span class="token operator">+</span> <span class="token string">" doing stuff"</span><span class="token punctuation">)</span><span class="token punctuation">(</span><span class="token string">"code"</span><span class="token punctuation">)</span><br /><span class="token string">'code doing stuff'</span></code></pre>
<h3 id="r-convention">R Convention <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#r-convention">#</a></h3>
<p>The most common convention is to use anonymous functions when using the <code>*apply</code> family of functions. For example, you might want to do an operation across a set of columns in a dataset.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Create a dataset</span><br />df <span class="token operator"><-</span> data.frame<span class="token punctuation">(</span><br /> col1 <span class="token operator">=</span> c<span class="token punctuation">(</span><span class="token string">"element1"</span><span class="token punctuation">,</span> <span class="token string">"element2"</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br /> col2 <span class="token operator">=</span> c<span class="token punctuation">(</span><span class="token string">"element1"</span><span class="token punctuation">,</span> <span class="token string">"element2"</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br /> stringsAsFactors <span class="token operator">=</span> <span class="token boolean">FALSE</span><br /><span class="token punctuation">)</span><br /><br /><span class="token comment"># lapply an anonymous function to the columns of the dataset</span><br /><span class="token operator">></span> lapply<span class="token punctuation">(</span>df<span class="token punctuation">,</span> <span class="token keyword">function</span><span class="token punctuation">(</span>x<span class="token punctuation">)</span> paste<span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token string">"doing stuff"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token operator">$</span>col1<br /><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token string">"element1 doing stuff"</span> <span class="token string">"element2 doing stuff"</span><br /><br /><span class="token operator">$</span>col2<br /><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token string">"element1 doing stuff"</span> <span class="token string">"element2 doing stuff"</span></code></pre>
<h3 id="python-convention">Python Convention <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#python-convention">#</a></h3>
<p>Doing the exact operation as above in python:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> pandas <span class="token keyword">as</span> pd<br /><br /><span class="token comment"># Create DataFrame</span><br />df <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span><br /> <span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token string">"element1"</span><span class="token punctuation">,</span> <span class="token string">"element1"</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token string">"element2"</span><span class="token punctuation">,</span> <span class="token string">"element2"</span><span class="token punctuation">]</span><span class="token punctuation">]</span><span class="token punctuation">,</span><br /> columns<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'col1'</span><span class="token punctuation">,</span> <span class="token string">'col2'</span><span class="token punctuation">]</span><br /><span class="token punctuation">)</span><br /><br /><span class="token operator">>></span><span class="token operator">></span> df<span class="token punctuation">.</span><span class="token builtin">apply</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x <span class="token operator">+</span> <span class="token string">" doing stuff"</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span><br /> col1 col2<br /><span class="token number">0</span> element1 doing stuff element1 doing stuff<br /><span class="token number">1</span> element2 doing stuff element2 doing stuff</code></pre>
<p>You will generally see lambdas used with higher order functions like <code>map()</code>, <code>reduce()</code> and <code>filter()</code>. E.g.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Map a anonymous function against all elements of df['col1']</span><br /><span class="token operator">>></span><span class="token operator">></span> <span class="token builtin">map</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x <span class="token operator">+</span> <span class="token string">" doing stuff"</span><span class="token punctuation">,</span> df<span class="token punctuation">[</span><span class="token string">'col1'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /><span class="token punctuation">[</span><span class="token string">'element1 doing stuff'</span><span class="token punctuation">,</span> <span class="token string">'element2 doing stuff'</span><span class="token punctuation">]</span></code></pre>
<p>You may also see a lambda used to define a function. e.g.</p>
<pre class="language-python"><code class="language-python">functionName <span class="token operator">=</span> <span class="token keyword">lambda</span> x<span class="token punctuation">:</span> x <span class="token operator">+</span> <span class="token string">" doing stuff"</span></code></pre>
<p>This is poor practice when a standard function definition would have been sufficient. The Python community has a bit of a split over the use of anonymous functions (lambdas) vs. <a href="https://www.digitalocean.com/community/tutorials/understanding-list-comprehensions-in-python-3">list comprehensions</a>. The <a href="https://en.wikipedia.org/wiki/Benevolent_dictator_for_life">benevolent dictator</a> of the Python community <a href="https://en.wikipedia.org/wiki/Guido_van_Rossum">Guido van Rossum</a> has <a href="https://www.artima.com/weblogs/viewpost.jsp?thread=98196">argued for lambda's complete removal from python 3</a>.</p>
<h2 id="in-conclusion">In Conclusion <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#in-conclusion">#</a></h2>
<p>Anonymous functions can make your code base harder to read. It can also make debugging harder. However, they can save you time from having to define yet another function in your code base. When should you use anonymous functions? According to <a href="http://hadley.nz/">Hadley</a> <em>"when itâs not worth the effort to give it a name"</em>. A good example is when you won't use the function anywhere else in your code. Or when you want to <code>apply()</code> a couple of pre-defined functions in one call e.g. <code>lapply(df, function(x) secondFunction(firstFunction(x)))</code>.</p>
<h2 id="resources">Resources <a class="direct-link" href="https://lukesingham.com/anonymous-functions-in-r-python/#resources">#</a></h2>
<ul>
<li>
<p><a href="http://www.curiousefficiency.org/posts/bloggercom1999blog-9320223post-110439602874040740.html">Anonymous functions in Python</a></p>
</li>
<li>
<p><a href="http://adv-r.had.co.nz/Functional-programming.html#anonymous-functions">Hadley on Anonymous Functions</a></p>
</li>
<li>
<p><a href="https://dbader.org/blog/python-lambda-functions">Lambda Functions in Python: What Are They Good For?</a></p>
</li>
</ul>
Shiny (R) Web App Performance - Profiling2017-08-06T00:00:00Zhttps://lukesingham.com/shiny-r-performance-profiling/<p>Introduced at the 2016 R conference, the <code>profvis</code> package offers a visual way of inspecting the call stack and highlights the most memory and computationally intensive parts of your code.</p>
<h2 id="run-profvis">Run Profvis <a class="direct-link" href="https://lukesingham.com/shiny-r-performance-profiling/#run-profvis">#</a></h2>
<pre class="language-r"><code class="language-r"><span class="token comment"># Load library</span><br />library<span class="token punctuation">(</span>profvis<span class="token punctuation">)</span><br /><br /><span class="token comment"># Run profiler on shiny app with optional arg to save output</span><br />profvis<span class="token punctuation">(</span><span class="token punctuation">{</span> runApp<span class="token punctuation">(</span><span class="token string">'Projects/path_of_app'</span><span class="token punctuation">)</span> <span class="token punctuation">}</span><br /> <span class="token punctuation">,</span> prof_output <span class="token operator">=</span> <span class="token string">'/path_to_save_output'</span><span class="token punctuation">)</span></code></pre>
<p>At this point your app will launch, to get the shiny app code to execute, interact with the parts of the application that you are interested in. Once you have completed the interactions, close the page and press the 'stop' button at the top of the console in RStudio. Profvis will recognise you have stopped running the shiny app, display the profvis interface and <em>optionally</em> save the file. The file name is randomly generated and should look something like this <code>file108f93bff877b.Rprof</code>.</p>
<blockquote>
<p>Each block in the flame graph represents a call to a function, or possibly multiple calls to the same function. The width of the block is proportional to the amount of time spent in that function. When a function calls another function, another block is added on top of it in the flame graph.</p>
</blockquote>
<p>Source: <a href="https://rstudio.github.io/profvis/">Profvis Overview</a></p>
<p>To reload the saved profvis profile:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Load saved profvis</span><br />profvis<span class="token punctuation">(</span>prof_input <span class="token operator">=</span> <span class="token string">'/path_to_save_output/file108f93bff877b.Rprof'</span><span class="token punctuation">)</span></code></pre>
<h2 id="reload-a-saved-profvis">Reload a Saved Profvis <a class="direct-link" href="https://lukesingham.com/shiny-r-performance-profiling/#reload-a-saved-profvis">#</a></h2>
<p>You can also save as a webpage using the following code:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Assign to variable</span><br />p <span class="token operator"><-</span> profvis<span class="token punctuation">(</span>prof_input <span class="token operator">=</span> <span class="token string">'/path_to_save_output/file108f93bff877b.Rprof'</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Save as a webpage</span><br />htmlwidgets<span class="token operator">::</span>saveWidget<span class="token punctuation">(</span>p<span class="token punctuation">,</span> <span class="token string">"/path_to_save_output/profile.html"</span><span class="token punctuation">)</span></code></pre>
<h2 id="resources">Resources <a class="direct-link" href="https://lukesingham.com/shiny-r-performance-profiling/#resources">#</a></h2>
<ul>
<li>
<p><a href="https://www.rstudio.com/resources/videos/profiling-and-performance/">Video of Winstan Chang presenting ProfViz</a></p>
</li>
<li>
<p><a href="https://blog.rstudio.com/2016/05/23/profiling-with-rstudio-and-profvis/">Profiling code integrated into RStudio IDE, great for non-shiny code</a></p>
</li>
</ul>
Binary Classification in Python - Who's Going to Leave Next?2017-07-12T00:00:00Zhttps://lukesingham.com/whos-going-to-leave-next/<p>This post goes through a binary classification problem with Python's machine learning library <a href="http://scikit-learn.org/">scikit-learn</a>.</p>
<h2 id="aim">Aim <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#aim">#</a></h2>
<blockquote>
<p>Create a model that predicts who is going to leave the organisation next. Commonly known as churn modelling.</p>
</blockquote>
<p>To follow along, I breakdown each piece of the coding journey in this post. Alternatively, you can find a <a href="https://github.com/ucg8j/kaggle_HR">complete copy of the code on github</a>. You'll need the following packages loaded:</p>
<pre class="language-python"><code class="language-python"><span class="token keyword">import</span> os<br /><span class="token keyword">import</span> pandas <span class="token keyword">as</span> pd<br /><span class="token keyword">import</span> numpy <span class="token keyword">as</span> np<br /><span class="token keyword">import</span> matplotlib<span class="token punctuation">.</span>pyplot <span class="token keyword">as</span> plt<br /><span class="token keyword">import</span> seaborn <span class="token keyword">as</span> sns<br /><span class="token keyword">from</span> random <span class="token keyword">import</span> sample<br /><span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>linear_model <span class="token keyword">import</span> LogisticRegression<br /><span class="token keyword">from</span> sklearn <span class="token keyword">import</span> tree<br /><span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>ensemble <span class="token keyword">import</span> RandomForestClassifier<br /><span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>svm <span class="token keyword">import</span> SVC<br /><span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>neighbors <span class="token keyword">import</span> KNeighborsClassifier<br /><span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>naive_bayes <span class="token keyword">import</span> GaussianNB<br /><span class="token keyword">from</span> sklearn<span class="token punctuation">.</span>cross_validation <span class="token keyword">import</span> cross_val_score<br /><span class="token keyword">from</span> sklearn <span class="token keyword">import</span> metrics<br /><span class="token keyword">from</span> IPython<span class="token punctuation">.</span>display <span class="token keyword">import</span> Image <br /><span class="token keyword">from</span> pydotplus <span class="token keyword">import</span> graph_from_dot_data</code></pre>
<p>The dataset for this exercise was originally found on <a href="https://www.kaggle.com/">kaggle</a>. It's no longer available via Kaggle, so you can download a copy from my GitHub repo by running:</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">curl</span> https://raw.githubusercontent.com/ucg8j/kaggle_HR/master/HR_comma_sep.csv <span class="token operator">>|</span> HR_comma_sep.csv</code></pre>
<p>Then to read in the data:</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Set working directory</span><br />path <span class="token operator">=</span> os<span class="token punctuation">.</span>path<span class="token punctuation">.</span>expanduser<span class="token punctuation">(</span><span class="token string">'~/Projects/kaggle_HR/'</span><span class="token punctuation">)</span><br />os<span class="token punctuation">.</span>chdir<span class="token punctuation">(</span>path<span class="token punctuation">)</span><br /><span class="token comment"># Read in the data</span><br />df <span class="token operator">=</span> pd<span class="token punctuation">.</span>read_csv<span class="token punctuation">(</span><span class="token string">'HR_comma_sep.csv'</span><span class="token punctuation">)</span></code></pre>
<p>It contains data of 14,999 employees who are either in the organisation or have left, and 10 columns.</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> df<span class="token punctuation">.</span>shape<br /><span class="token punctuation">(</span><span class="token number">14999</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">)</span><br /><span class="token operator">>></span><span class="token operator">></span> df<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>tolist<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token punctuation">[</span><span class="token string">'satisfaction_level'</span><span class="token punctuation">,</span><br /> <span class="token string">'last_evaluation'</span><span class="token punctuation">,</span><br /> <span class="token string">'number_project'</span><span class="token punctuation">,</span><br /> <span class="token string">'average_montly_hours'</span><span class="token punctuation">,</span><br /> <span class="token string">'time_spend_company'</span><span class="token punctuation">,</span><br /> <span class="token string">'Work_accident'</span><span class="token punctuation">,</span><br /> <span class="token string">'left'</span><span class="token punctuation">,</span><br /> <span class="token string">'promotion_last_5years'</span><span class="token punctuation">,</span><br /> <span class="token string">'sales'</span><span class="token punctuation">,</span><br /> <span class="token string">'salary'</span><span class="token punctuation">]</span></code></pre>
<p>We need to get some sense of how balanced our dataset is...</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Some basic stats on the target variable</span><br /><span class="token keyword">print</span> <span class="token string">'# people left = {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>df<span class="token punctuation">[</span>df<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token comment"># people left = 3571</span><br /><br /><span class="token keyword">print</span> <span class="token string">'# people stayed = {}'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>df<span class="token punctuation">[</span>df<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token comment"># people stayed = 11428</span><br /><br /><span class="token keyword">print</span> <span class="token string">'% people left = {}%'</span><span class="token punctuation">.</span><span class="token builtin">format</span><span class="token punctuation">(</span><span class="token builtin">round</span><span class="token punctuation">(</span><span class="token builtin">float</span><span class="token punctuation">(</span><span class="token builtin">len</span><span class="token punctuation">(</span>df<span class="token punctuation">[</span>df<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span> <span class="token operator">==</span> <span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span> <span class="token operator">/</span> <span class="token builtin">len</span><span class="token punctuation">(</span>df<span class="token punctuation">)</span> <span class="token operator">*</span> <span class="token number">100</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token number">3</span><span class="token punctuation">)</span><br /><span class="token comment"># % people left = 24.0%</span></code></pre>
<p>Knowing that 76% of the workforce will stay helps us not be excited by machine learning diagnostics that aren't much better than 76%. If I predicted that <strong>all</strong> staff would remain in the organisation, I would be 76% accurate!</p>
<p>I next check for missing values.</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> df<span class="token punctuation">.</span><span class="token builtin">apply</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> <span class="token builtin">sum</span><span class="token punctuation">(</span>x<span class="token punctuation">.</span>isnull<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">0</span><span class="token punctuation">)</span><br />satisfaction_level <span class="token number">0</span><br />last_evaluation <span class="token number">0</span><br />number_project <span class="token number">0</span><br />average_montly_hours <span class="token number">0</span><br />time_spend_company <span class="token number">0</span><br />Work_accident <span class="token number">0</span><br />left <span class="token number">0</span><br />promotion_last_5years <span class="token number">0</span><br />sales <span class="token number">0</span><br />salary <span class="token number">0</span></code></pre>
<p>No missing values, that's great. Next I look for correlations in the data.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Correlation heatmap</span><br />correlation <span class="token operator">=</span> df<span class="token punctuation">.</span>corr<span class="token punctuation">(</span><span class="token punctuation">)</span><br />plt<span class="token punctuation">.</span>figure<span class="token punctuation">(</span>figsize<span class="token operator">=</span><span class="token punctuation">(</span><span class="token number">10</span><span class="token punctuation">,</span> <span class="token number">10</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br />sns<span class="token punctuation">.</span>heatmap<span class="token punctuation">(</span>correlation<span class="token punctuation">,</span> vmax<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">,</span> square<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> annot<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> cmap<span class="token operator">=</span><span class="token string">'cubehelix'</span><span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/07/heatmap_correl.png" alt="" /></p>
<p>Unsurprisingly, <code>satisfaction_level</code> has the largest correlation with the decision to stay or leave. Next up, pairwise plots provide a lot information for one diagram.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># take a 5% sample as this is computationally expensive</span><br />df_sample <span class="token operator">=</span> df<span class="token punctuation">.</span>sample<span class="token punctuation">(</span>frac<span class="token operator">=</span><span class="token number">0.05</span><span class="token punctuation">)</span><br /><span class="token comment"># Pairwise plots</span><br />pplot <span class="token operator">=</span> sns<span class="token punctuation">.</span>pairplot<span class="token punctuation">(</span>df_sample<span class="token punctuation">,</span> hue<span class="token operator">=</span><span class="token string">"left"</span><span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/07/pairwise.png" alt="" /></p>
<p>On the diagonal we can see the distribution of each variable, including the target variable of who has left (this visualises the balance). There are some interesting groupings of those who leave (Green dots) when looking at the plots of <code>satisfaction_level</code> by <code>last_evaluation</code> and by <code>average_montly_hours</code> - they look to be the same sub-populations. E.g. Low satisfaction but high evaluation and monthly hours, there's also a grouping of a high number of projects completed with low satisfaction levels by those who leave. Perhaps these are <em>high performers</em> that easily leave when they are not satisfied by their job. There are other interesting groupings so it's worth spending some looking at this output examining these relationships. It's great to have these groupings, as our classifier should be able to differentiate between those who leave and those that remain. If there were no relationships revealed in this plot, we would face a very difficult problem and perhaps have to go back to examine what data we could collect to help predict the outcome.</p>
<p>Two variables were left out of the above plots, <code>Salary</code> and <code>Sales</code>. I need to deal with these separately prior to modelling, as our machine learning classifier will expect numbers to crunch rather than text.</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> df<span class="token punctuation">.</span>salary<span class="token punctuation">.</span>unique<span class="token punctuation">(</span><span class="token punctuation">)</span><br />array<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'low'</span><span class="token punctuation">,</span> <span class="token string">'medium'</span><span class="token punctuation">,</span> <span class="token string">'high'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> dtype<span class="token operator">=</span><span class="token builtin">object</span><span class="token punctuation">)</span><br /><span class="token operator">>></span><span class="token operator">></span> df<span class="token punctuation">.</span>sales<span class="token punctuation">.</span>unique<span class="token punctuation">(</span><span class="token punctuation">)</span><br />array<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'sales'</span><span class="token punctuation">,</span> <span class="token string">'accounting'</span><span class="token punctuation">,</span> <span class="token string">'hr'</span><span class="token punctuation">,</span> <span class="token string">'technical'</span><span class="token punctuation">,</span> <span class="token string">'support'</span><span class="token punctuation">,</span> <span class="token string">'management'</span><span class="token punctuation">,</span> <span class="token string">'IT'</span><span class="token punctuation">,</span> <span class="token string">'product_mng'</span><span class="token punctuation">,</span> <span class="token string">'marketing'</span><span class="token punctuation">,</span> <span class="token string">'RandD'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> dtype<span class="token operator">=</span><span class="token builtin">object</span><span class="token punctuation">)</span></code></pre>
<p><code>Salary</code> has 3 values <em>low</em>, <em>medium</em> and <em>high</em>. These factor variables have an order. I'm going to go use dummy encoding as a simple and common method to handle factor variables. The downsides to this is I am not handling the <em>ordering</em> and will introduce extra dimensions to the data (see <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">Curse of dimensionality</a> for why this is bad). Helmert, Forward/Backward differencing or Orthogonal encoding would <em>probably</em> be better.</p>
<hr />
<h3 id="as-an-aside...">As an aside... <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#as-an-aside...">#</a></h3>
<p><code>R</code> makes factors really easy to handle. For example, let's say we have data with an unordered factor variable <code>df$unordered.variable</code>:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Order a factor variable in R</span><br />df<span class="token operator">$</span>ordered.var <span class="token operator"><-</span> factor<span class="token punctuation">(</span>df<span class="token operator">$</span>unordered.var<span class="token punctuation">,</span> levels <span class="token operator">=</span> c<span class="token punctuation">(</span><span class="token string">'low'</span><span class="token punctuation">,</span> <span class="token string">'medium'</span><span class="token punctuation">,</span> <span class="token string">'high'</span><span class="token punctuation">)</span></code></pre>
<hr />
<h2 id="dummy-encoding">Dummy Encoding <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#dummy-encoding">#</a></h2>
<p>Dummy encoding, or <em>one hot encoding</em>, transforms categorical variables into a series of binary columns. Before I one hot encode the <code>sales</code> and <code>salary</code> I prepend the column names to the categories, that way I know later which column each new column came from. This helps particularly in cases where the columns use the same category names e.g. 'high' could apply to <code>sales</code> and <code>salary</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Prepend string prior to encoding</span><br />df<span class="token punctuation">[</span><span class="token string">'salary'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">'$_'</span> <span class="token operator">+</span> df<span class="token punctuation">[</span><span class="token string">'salary'</span><span class="token punctuation">]</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">)</span><br />df<span class="token punctuation">[</span><span class="token string">'sales'</span><span class="token punctuation">]</span> <span class="token operator">=</span> <span class="token string">'sales_'</span> <span class="token operator">+</span> df<span class="token punctuation">[</span><span class="token string">'sales'</span><span class="token punctuation">]</span><span class="token punctuation">.</span>astype<span class="token punctuation">(</span><span class="token builtin">str</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Create 'salary' dummies and join</span><br />one_hot_salary <span class="token operator">=</span> pd<span class="token punctuation">.</span>get_dummies<span class="token punctuation">(</span>df<span class="token punctuation">[</span><span class="token string">'salary'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />df <span class="token operator">=</span> df<span class="token punctuation">.</span>join<span class="token punctuation">(</span>one_hot_salary<span class="token punctuation">)</span><br /><br /><span class="token comment"># Create 'sales' dummies and join</span><br />one_hot_sales <span class="token operator">=</span> pd<span class="token punctuation">.</span>get_dummies<span class="token punctuation">(</span>df<span class="token punctuation">[</span><span class="token string">'sales'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />df <span class="token operator">=</span> df<span class="token punctuation">.</span>join<span class="token punctuation">(</span>one_hot_sales<span class="token punctuation">)</span></code></pre>
<p>If we proceeded to modelling without dropping columns, we would run into the <em>dummy variable trap</em>. To illustrate what the dummy variable trap is, let's say that our data has a column for <code>Gender</code> in the form <code>['male','female','male','female'...]</code>. If we use <code>pd.get_dummies()</code></p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Generate 'Gender' variable</span><br />Gender <span class="token operator">=</span> np<span class="token punctuation">.</span>random<span class="token punctuation">.</span>choice<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'male'</span><span class="token punctuation">,</span><span class="token string">'female'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><br /><span class="token comment"># Create DataFrame</span><br />df1 <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>Gender<span class="token punctuation">,</span> columns<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'Gender'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /><span class="token comment"># One hot encode Gender</span><br />dummies <span class="token operator">=</span> pd<span class="token punctuation">.</span>get_dummies<span class="token punctuation">(</span>df1<span class="token punctuation">[</span><span class="token string">'Gender'</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /><span class="token comment"># Join dummies to 'df1'</span><br />df1 <span class="token operator">=</span> df1<span class="token punctuation">.</span>join<span class="token punctuation">(</span>dummies<span class="token punctuation">)</span><br /><span class="token keyword">print</span> df1<br /> Gender female male<br /><span class="token number">0</span> male <span class="token number">0</span> <span class="token number">1</span><br /><span class="token number">1</span> female <span class="token number">1</span> <span class="token number">0</span><br /><span class="token number">2</span> male <span class="token number">0</span> <span class="token number">1</span><br /><span class="token number">3</span> female <span class="token number">1</span> <span class="token number">0</span></code></pre>
<p>We can see that the dummy columns of <code>female</code> and <code>male</code> reflect the <code>Gender</code> column, a row with column <code>Gender='male'</code> has <code>female=0</code> and <code>male=1</code>. In modelling, we are looking for <em>independent</em> variables to predict a <em>dependent</em> outcome. One of our dummy variables is completely <em>dependent</em> on the other (<a href="https://en.wikipedia.org/wiki/Multicollinearity">see Multicollinearity</a>). We can naturally drop one of the dummy variables to avoid this trap whilst not losing any valuable information.</p>
<p>To avoid the dummy variable trap in the Kaggle HR dataset, I'm dropping the columns, <code>sales_IT</code> and low salary <code>$_low</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Drop unnecessary columns to avoid the dummy variable trap</span><br />df <span class="token operator">=</span> df<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'salary'</span><span class="token punctuation">,</span> <span class="token string">'sales'</span><span class="token punctuation">,</span> <span class="token string">'$_low'</span><span class="token punctuation">,</span> <span class="token string">'sales_IT'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span></code></pre>
<h2 id="split-into-test-and-training-sets">Split Into Test and Training Sets <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#split-into-test-and-training-sets">#</a></h2>
<p>Now to split the dataset up into our training, testing and <em>optionally</em> our validation sets. This is best done randomly to avoid having one subgroup of the data overrepresented in either our training or testing datasets, hence <code>df.sample()</code>.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Randomly, split the data into test/training/validation sets</span><br />train<span class="token punctuation">,</span> test<span class="token punctuation">,</span> validate <span class="token operator">=</span> np<span class="token punctuation">.</span>split<span class="token punctuation">(</span>df<span class="token punctuation">.</span>sample<span class="token punctuation">(</span>frac<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token builtin">int</span><span class="token punctuation">(</span><span class="token number">.6</span><span class="token operator">*</span><span class="token builtin">len</span><span class="token punctuation">(</span>df<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">,</span> <span class="token builtin">int</span><span class="token punctuation">(</span><span class="token number">.8</span><span class="token operator">*</span><span class="token builtin">len</span><span class="token punctuation">(</span>df<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br /><span class="token keyword">print</span> train<span class="token punctuation">.</span>shape<span class="token punctuation">,</span> test<span class="token punctuation">.</span>shape<span class="token punctuation">,</span> validate<span class="token punctuation">.</span>shape<br /><span class="token comment"># (8999, 20) (3000, 20) (3000, 20)</span><br /><br /><span class="token comment"># Separate target and predictors</span><br />y_train <span class="token operator">=</span> train<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><br />x_train <span class="token operator">=</span> train<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />y_test <span class="token operator">=</span> test<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><br />x_test <span class="token operator">=</span> test<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />y_validate <span class="token operator">=</span> validate<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><br />x_validate <span class="token operator">=</span> validate<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Check the balance of the splits on y_</span><br /><span class="token operator">>></span><span class="token operator">></span> y_test<span class="token punctuation">.</span>mean<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token number">0.23933333333333334</span><br /><span class="token operator">>></span><span class="token operator">></span> y_train<span class="token punctuation">.</span>mean<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token number">0.24091565729525502</span></code></pre>
<p>Feature importance can be ascertained from a random forest, alternatively <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">Principle Components Analysis (PCA)</a> can be used.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Variable importance</span><br />rf <span class="token operator">=</span> RandomForestClassifier<span class="token punctuation">(</span><span class="token punctuation">)</span><br />rf<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token keyword">print</span> <span class="token string">"Features sorted by their score:"</span><br /><span class="token keyword">print</span> <span class="token builtin">sorted</span><span class="token punctuation">(</span><span class="token builtin">zip</span><span class="token punctuation">(</span><span class="token builtin">map</span><span class="token punctuation">(</span><span class="token keyword">lambda</span> x<span class="token punctuation">:</span> <span class="token builtin">round</span><span class="token punctuation">(</span>x<span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">,</span> rf<span class="token punctuation">.</span>feature_importances_<span class="token punctuation">)</span><span class="token punctuation">,</span> x_train<span class="token punctuation">)</span><span class="token punctuation">,</span> reverse<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span><br /><span class="token comment">#[(0.3523, 'satisfaction_level'), (0.1738, 'time_spend_company'), (0.1705, 'number_project'), (0.158, 'average_montly_hours'), (0.1043, 'last_evaluation'), (0.0077, '$_high'), (0.0065, 'Work_accident'), (0.0059, '$_medium'), (0.0042, 'sales_technical'), (0.0034, 'sales_sales'), (0.0028, 'sales_support'), (0.0024, 'sales_hr'), (0.0023, 'sales_accounting'), (0.0019, 'sales_management'), (0.0014, 'sales_RandD'), (0.001, 'sales_product_mng'), (0.001, 'sales_marketing'), (0.0006, 'promotion_last_5years')]</span></code></pre>
<p>A very rough plot of <code>importances</code> reveals that past the first 5 columns, we really don't get that much more variance explained by the additional columns.<br />
<img src="https://lukesingham.com/content/images/2017/07/rf_variable_importance-1.png" alt="" /></p>
<p>From hereon in I'll only use the first 5 variables.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Create variable lists and drop</span><br />all_vars <span class="token operator">=</span> x_train<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>tolist<span class="token punctuation">(</span><span class="token punctuation">)</span><br />top_5_vars <span class="token operator">=</span> <span class="token punctuation">[</span><span class="token string">'satisfaction_level'</span><span class="token punctuation">,</span> <span class="token string">'number_project'</span><span class="token punctuation">,</span> <span class="token string">'time_spend_company'</span><span class="token punctuation">,</span> <span class="token string">'average_montly_hours'</span><span class="token punctuation">,</span> <span class="token string">'last_evaluation'</span><span class="token punctuation">]</span><br />bottom_vars <span class="token operator">=</span> <span class="token punctuation">[</span>cols <span class="token keyword">for</span> cols <span class="token keyword">in</span> all_vars <span class="token keyword">if</span> cols <span class="token keyword">not</span> <span class="token keyword">in</span> top_5_vars<span class="token punctuation">]</span><br /><br /><span class="token comment"># Drop less important variables leaving the top_5</span><br />x_train <span class="token operator">=</span> x_train<span class="token punctuation">.</span>drop<span class="token punctuation">(</span>bottom_vars<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />x_test <span class="token operator">=</span> x_test<span class="token punctuation">.</span>drop<span class="token punctuation">(</span>bottom_vars<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />x_validate <span class="token operator">=</span> x_validate<span class="token punctuation">.</span>drop<span class="token punctuation">(</span>bottom_vars<span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span></code></pre>
<h2 id="model-evaluation-basics">Model Evaluation Basics <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#model-evaluation-basics">#</a></h2>
<p>The formula for <strong>accuracy</strong> is:<br />
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><mtext>Accuracy</mtext><mo>=</mo><mfrac><mrow><mi>T</mi><mi>P</mi><mo>+</mo><mi>T</mi><mi>N</mi></mrow><mrow><mi>T</mi><mi>P</mi><mo>+</mo><mi>T</mi><mi>N</mi><mo>+</mo><mi>F</mi><mi>P</mi><mo>+</mo><mi>F</mi><mi>N</mi></mrow></mfrac></math><br />
</p>
<ul>
<li>
<p><strong>TP</strong> is the number of true positives</p>
<p><em>Predicted as leaving and they do</em></p>
</li>
<li>
<p><strong>TN</strong> is the number of true negatives</p>
<p><em>Predicted as remaining and they do</em></p>
</li>
<li>
<p><strong>FP</strong> is the number of false positives</p>
<p><em>Predicted as leaving, but they aren't - whoops!</em></p>
</li>
<li>
<p><strong>FN</strong> is the number of false negatives</p>
<p><em>Predicted as remaining and they don't - eek!</em></p>
</li>
</ul>
<p>As mentioned before, if we were aiming to predict those who remain (76%) then if we predicted 100%, we would have <code>(0 + 11428) / 14999</code> or 76% accurracy. As the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy Paradox wiki</a> states precision and recall are probably more appropriate evaluation metrics. Here's a really quick definition of sensitivity (AKA recall), specificity and precision:</p>
<ul>
<li><strong>Sensitivity/Recall</strong> - <code>TP / (TP + FN)</code> how well the model <em>recalls</em>/identifies those that will leave. AKA the true positive rate.</li>
<li><strong>Specificity</strong> - <code>TN / (TN + FP)</code> how well the model identifies those that will stay.</li>
<li><strong>Precision</strong> <code>TP / (TP + FP)</code> how believable is the model? A low precision model will alarm you to those who are leaving that are actually staying.</li>
<li><strong>F1 score</strong> <code>2 * (precision * recall)/(precision + recall)</code>is the harmonic mean betwen precision and recall or the <em>balance</em>.</li>
</ul>
<p>For this problem, we are perhaps most interested in knowing who is going to leave next. That way an organisation can respond with workforce planning and recruitment activities. Therefore, precision and recall will be the metrics we focus on.</p>
<blockquote>
<p>Intuitively, precision is the ability of the classifier not to label as positive a sample that is negative, and recall is the ability of the classifier to find all the positive samples.</p>
</blockquote>
<p>Source: <a href="http://scikit-learn.org/stable/modules/model_evaluation.html#precision-recall-and-f-measures">scikit-learn</a></p>
<h2 id="modelling">Modelling <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#modelling">#</a></h2>
<h3 id="1.-logistic-regression">1. Logistic Regression <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#1.-logistic-regression">#</a></h3>
<p>Starting off with a very simple model, logistic regression.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Instantiate</span><br />logit_model <span class="token operator">=</span> LogisticRegression<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token comment"># Fit</span><br />logit_model <span class="token operator">=</span> logit_model<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment"># How accurate?</span><br />logit_model<span class="token punctuation">.</span>score<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment">#0.7874</span><br /><br /><span class="token comment"># How does it perform on the test dataset?</span><br /><br /><span class="token comment"># Predictions on the test dataset</span><br />predicted <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>logit_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token comment"># Probabilities on the test dataset</span><br />probs <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>logit_model<span class="token punctuation">.</span>predict_proba<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token keyword">print</span> metrics<span class="token punctuation">.</span>accuracy_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br /><span class="token number">0.785</span></code></pre>
<p>Now let's look at the confusion matrix and the metrics we care about.</p>
<pre class="language-python"><code class="language-python"><span class="token operator">>></span><span class="token operator">></span> <span class="token keyword">print</span> metrics<span class="token punctuation">.</span>confusion_matrix<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br /><span class="token punctuation">[</span><span class="token punctuation">[</span><span class="token number">2100</span> <span class="token number">182</span><span class="token punctuation">]</span><br /> <span class="token punctuation">[</span> <span class="token number">463</span> <span class="token number">255</span><span class="token punctuation">]</span><span class="token punctuation">]</span><br /><span class="token operator">>></span><span class="token operator">></span> <span class="token keyword">print</span> metrics<span class="token punctuation">.</span>classification_report<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br /> precision recall f1<span class="token operator">-</span>score support<br /><br /> <span class="token number">0</span> <span class="token number">0.82</span> <span class="token number">0.92</span> <span class="token number">0.87</span> <span class="token number">2282</span><br /> <span class="token number">1</span> <span class="token number">0.58</span> <span class="token number">0.36</span> <span class="token number">0.44</span> <span class="token number">718</span><br /><br />avg <span class="token operator">/</span> total <span class="token number">0.76</span> <span class="token number">0.79</span> <span class="token number">0.77</span> <span class="token number">3000</span></code></pre>
<p>Let's double check the <code>classification_report</code>. Focusing on those who will leave.</p>
<blockquote>
<p>Precision = TP/(TP+FP) = 255/(255+182) = 0.58<br />
Recall = TP/(TP+FN) = 255/(255+463) = 0.36</p>
</blockquote>
<p>Perhaps we could optimise this model, however there are a range of models to test that may perform better out-of-the-box. I'll quickly run through a range of models...</p>
<h3 id="2.-decision-tree">2. Decision Tree <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#2.-decision-tree">#</a></h3>
<p>Decision trees are highly interpretable and tend to perform well on classification problems.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Instantiate with a max depth of 3</span><br />tree_model <span class="token operator">=</span> tree<span class="token punctuation">.</span>DecisionTreeClassifier<span class="token punctuation">(</span>max_depth<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">)</span><br /><span class="token comment"># Fit a decision tree</span><br />tree_model <span class="token operator">=</span> tree_model<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment"># Training accuracy</span><br />tree_model<span class="token punctuation">.</span>score<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><br /><span class="token comment"># Predictions/probs on the test dataset</span><br />predicted <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>tree_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br />probs <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>tree_model<span class="token punctuation">.</span>predict_proba<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Store metrics</span><br />tree_accuracy <span class="token operator">=</span> metrics<span class="token punctuation">.</span>accuracy_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />tree_roc_auc <span class="token operator">=</span> metrics<span class="token punctuation">.</span>roc_auc_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> probs<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />tree_confus_matrix <span class="token operator">=</span> metrics<span class="token punctuation">.</span>confusion_matrix<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />tree_classification_report <span class="token operator">=</span> metrics<span class="token punctuation">.</span>classification_report<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />tree_precision <span class="token operator">=</span> metrics<span class="token punctuation">.</span>precision_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />tree_recall <span class="token operator">=</span> metrics<span class="token punctuation">.</span>recall_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />tree_f1 <span class="token operator">=</span> metrics<span class="token punctuation">.</span>f1_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># evaluate the model using 10-fold cross-validation</span><br />tree_cv_scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>tree<span class="token punctuation">.</span>DecisionTreeClassifier<span class="token punctuation">(</span>max_depth<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">,</span> x_test<span class="token punctuation">,</span> y_test<span class="token punctuation">,</span> scoring<span class="token operator">=</span><span class="token string">'precision'</span><span class="token punctuation">,</span> cv<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># output decision plot</span><br />dot_data <span class="token operator">=</span> tree<span class="token punctuation">.</span>export_graphviz<span class="token punctuation">(</span>tree_model<span class="token punctuation">,</span> out_file<span class="token operator">=</span><span class="token boolean">None</span><span class="token punctuation">,</span><br /> feature_names<span class="token operator">=</span>x_test<span class="token punctuation">.</span>columns<span class="token punctuation">.</span>tolist<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span><br /> class_names<span class="token operator">=</span><span class="token punctuation">[</span><span class="token string">'remain'</span><span class="token punctuation">,</span> <span class="token string">'left'</span><span class="token punctuation">]</span><span class="token punctuation">,</span><br /> filled<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> rounded<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">,</span> <br /> special_characters<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span> <br />graph <span class="token operator">=</span> graph_from_dot_data<span class="token punctuation">(</span>dot_data<span class="token punctuation">)</span><br />graph<span class="token punctuation">.</span>write_png<span class="token punctuation">(</span><span class="token string">"images/decision_tree.png"</span><span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/07/decision_tree.png" alt="" /></p>
<h3 id="3.-random-forest">3. Random Forest <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#3.-random-forest">#</a></h3>
<pre class="language-python"><code class="language-python"><span class="token comment"># Instantiate</span><br />rf <span class="token operator">=</span> RandomForestClassifier<span class="token punctuation">(</span><span class="token punctuation">)</span> <br /><span class="token comment"># Fit</span><br />rf_model <span class="token operator">=</span> rf<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment"># training accuracy 99.74%</span><br />rf_model<span class="token punctuation">.</span>score<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><br /><span class="token comment"># Predictions/probs on the test dataset</span><br />predicted <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>rf_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br />probs <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>rf_model<span class="token punctuation">.</span>predict_proba<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Store metrics</span><br />rf_accuracy <span class="token operator">=</span> metrics<span class="token punctuation">.</span>accuracy_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />rf_roc_auc <span class="token operator">=</span> metrics<span class="token punctuation">.</span>roc_auc_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> probs<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />rf_confus_matrix <span class="token operator">=</span> metrics<span class="token punctuation">.</span>confusion_matrix<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />rf_classification_report <span class="token operator">=</span> metrics<span class="token punctuation">.</span>classification_report<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />rf_precision <span class="token operator">=</span> metrics<span class="token punctuation">.</span>precision_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />rf_recall <span class="token operator">=</span> metrics<span class="token punctuation">.</span>recall_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />rf_f1 <span class="token operator">=</span> metrics<span class="token punctuation">.</span>f1_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Evaluate the model using 10-fold cross-validation</span><br />rf_cv_scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>RandomForestClassifier<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">,</span> x_test<span class="token punctuation">,</span> y_test<span class="token punctuation">,</span> scoring<span class="token operator">=</span><span class="token string">'precision'</span><span class="token punctuation">,</span> cv<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span><br />rf_cv_mean <span class="token operator">=</span> np<span class="token punctuation">.</span>mean<span class="token punctuation">(</span>rf_cv_scores<span class="token punctuation">)</span></code></pre>
<p>This has performed possibly a little too well!</p>
<h3 id="4.-support-vector-machine">4. SUPPORT VECTOR MACHINE <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#4.-support-vector-machine">#</a></h3>
<pre class="language-python"><code class="language-python"><span class="token comment"># Instantiate</span><br />svm_model <span class="token operator">=</span> SVC<span class="token punctuation">(</span>probability<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span><br /><span class="token comment"># Fit</span><br />svm_model <span class="token operator">=</span> svm_model<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment"># Accuracy</span><br />svm_model<span class="token punctuation">.</span>score<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><br /><span class="token comment"># Predictions/probs on the test dataset</span><br />predicted <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>svm_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br />probs <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>svm_model<span class="token punctuation">.</span>predict_proba<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Store metrics</span><br />svm_accuracy <span class="token operator">=</span> metrics<span class="token punctuation">.</span>accuracy_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />svm_roc_auc <span class="token operator">=</span> metrics<span class="token punctuation">.</span>roc_auc_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> probs<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />svm_confus_matrix <span class="token operator">=</span> metrics<span class="token punctuation">.</span>confusion_matrix<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />svm_classification_report <span class="token operator">=</span> metrics<span class="token punctuation">.</span>classification_report<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />svm_precision <span class="token operator">=</span> metrics<span class="token punctuation">.</span>precision_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />svm_recall <span class="token operator">=</span> metrics<span class="token punctuation">.</span>recall_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />svm_f1 <span class="token operator">=</span> metrics<span class="token punctuation">.</span>f1_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Evaluate the model using 10-fold cross-validation</span><br />svm_cv_scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>SVC<span class="token punctuation">(</span>probability<span class="token operator">=</span><span class="token boolean">True</span><span class="token punctuation">)</span><span class="token punctuation">,</span> x_test<span class="token punctuation">,</span> y_test<span class="token punctuation">,</span> scoring<span class="token operator">=</span><span class="token string">'precision'</span><span class="token punctuation">,</span> cv<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span><br />svm_cv_mean <span class="token operator">=</span> np<span class="token punctuation">.</span>mean<span class="token punctuation">(</span>svm_cv_scores<span class="token punctuation">)</span></code></pre>
<h3 id="5.-knn">5. KNN <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#5.-knn">#</a></h3>
<pre class="language-python"><code class="language-python"><span class="token comment"># instantiate learning model (k = 3)</span><br />knn_model <span class="token operator">=</span> KNeighborsClassifier<span class="token punctuation">(</span>n_neighbors<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">)</span><br /><span class="token comment"># fit the model</span><br />knn_model<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment"># Accuracy</span><br />knn_model<span class="token punctuation">.</span>score<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><br /><span class="token comment"># Predictions/probs on the test dataset</span><br />predicted <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>knn_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br />probs <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>knn_model<span class="token punctuation">.</span>predict_proba<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Store metrics</span><br />knn_accuracy <span class="token operator">=</span> metrics<span class="token punctuation">.</span>accuracy_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />knn_roc_auc <span class="token operator">=</span> metrics<span class="token punctuation">.</span>roc_auc_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> probs<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />knn_confus_matrix <span class="token operator">=</span> metrics<span class="token punctuation">.</span>confusion_matrix<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />knn_classification_report <span class="token operator">=</span> metrics<span class="token punctuation">.</span>classification_report<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />knn_precision <span class="token operator">=</span> metrics<span class="token punctuation">.</span>precision_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />knn_recall <span class="token operator">=</span> metrics<span class="token punctuation">.</span>recall_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />knn_f1 <span class="token operator">=</span> metrics<span class="token punctuation">.</span>f1_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Evaluate the model using 10-fold cross-validation</span><br />knn_cv_scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>KNeighborsClassifier<span class="token punctuation">(</span>n_neighbors<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">,</span> x_test<span class="token punctuation">,</span> y_test<span class="token punctuation">,</span> scoring<span class="token operator">=</span><span class="token string">'precision'</span><span class="token punctuation">,</span> cv<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span><br />knn_cv_mean <span class="token operator">=</span> np<span class="token punctuation">.</span>mean<span class="token punctuation">(</span>knn_cv_scores<span class="token punctuation">)</span></code></pre>
<h3 id="6.-two-class-bayes">6. TWO CLASS BAYES <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#6.-two-class-bayes">#</a></h3>
<pre class="language-python"><code class="language-python"><span class="token comment"># Instantiate</span><br />bayes_model <span class="token operator">=</span> GaussianNB<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token comment"># Fit the model</span><br />bayes_model<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><span class="token comment"># Accuracy</span><br />bayes_model<span class="token punctuation">.</span>score<span class="token punctuation">(</span>x_train<span class="token punctuation">,</span> y_train<span class="token punctuation">)</span><br /><br /><span class="token comment"># Predictions/probs on the test dataset</span><br />predicted <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>bayes_model<span class="token punctuation">.</span>predict<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br />probs <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span>bayes_model<span class="token punctuation">.</span>predict_proba<span class="token punctuation">(</span>x_test<span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Store metrics</span><br />bayes_accuracy <span class="token operator">=</span> metrics<span class="token punctuation">.</span>accuracy_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />bayes_roc_auc <span class="token operator">=</span> metrics<span class="token punctuation">.</span>roc_auc_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> probs<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><br />bayes_confus_matrix <span class="token operator">=</span> metrics<span class="token punctuation">.</span>confusion_matrix<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />bayes_classification_report <span class="token operator">=</span> metrics<span class="token punctuation">.</span>classification_report<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">)</span><br />bayes_precision <span class="token operator">=</span> metrics<span class="token punctuation">.</span>precision_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />bayes_recall <span class="token operator">=</span> metrics<span class="token punctuation">.</span>recall_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br />bayes_f1 <span class="token operator">=</span> metrics<span class="token punctuation">.</span>f1_score<span class="token punctuation">(</span>y_test<span class="token punctuation">,</span> predicted<span class="token punctuation">,</span> pos_label<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Evaluate the model using 10-fold cross-validation</span><br />bayes_cv_scores <span class="token operator">=</span> cross_val_score<span class="token punctuation">(</span>KNeighborsClassifier<span class="token punctuation">(</span>n_neighbors<span class="token operator">=</span><span class="token number">3</span><span class="token punctuation">)</span><span class="token punctuation">,</span> x_test<span class="token punctuation">,</span> y_test<span class="token punctuation">,</span> scoring<span class="token operator">=</span><span class="token string">'precision'</span><span class="token punctuation">,</span> cv<span class="token operator">=</span><span class="token number">10</span><span class="token punctuation">)</span><br />bayes_cv_mean <span class="token operator">=</span> np<span class="token punctuation">.</span>mean<span class="token punctuation">(</span>bayes_cv_scores<span class="token punctuation">)</span></code></pre>
<h2 id="results">Results <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#results">#</a></h2>
<pre class="language-python"><code class="language-python"><span class="token comment"># Model comparison</span><br />models <span class="token operator">=</span> pd<span class="token punctuation">.</span>DataFrame<span class="token punctuation">(</span><span class="token punctuation">{</span><br /> <span class="token string">'Model'</span><span class="token punctuation">:</span> <span class="token punctuation">[</span><span class="token string">'Logistic'</span><span class="token punctuation">,</span> <span class="token string">'d.Tree'</span><span class="token punctuation">,</span> <span class="token string">'r.f.'</span><span class="token punctuation">,</span> <span class="token string">'SVM'</span><span class="token punctuation">,</span> <span class="token string">'kNN'</span><span class="token punctuation">,</span> <span class="token string">'Bayes'</span><span class="token punctuation">]</span><span class="token punctuation">,</span><br /> <span class="token string">'Accuracy'</span> <span class="token punctuation">:</span> <span class="token punctuation">[</span>logit_accuracy<span class="token punctuation">,</span> tree_accuracy<span class="token punctuation">,</span> rf_accuracy<span class="token punctuation">,</span> svm_accuracy<span class="token punctuation">,</span> knn_accuracy<span class="token punctuation">,</span> bayes_accuracy<span class="token punctuation">]</span><span class="token punctuation">,</span><br /> <span class="token string">'Precision'</span><span class="token punctuation">:</span> <span class="token punctuation">[</span>logit_precision<span class="token punctuation">,</span> tree_precision<span class="token punctuation">,</span> rf_precision<span class="token punctuation">,</span> svm_precision<span class="token punctuation">,</span> knn_precision<span class="token punctuation">,</span> bayes_precision<span class="token punctuation">]</span><span class="token punctuation">,</span><br /> <span class="token string">'recall'</span> <span class="token punctuation">:</span> <span class="token punctuation">[</span>logit_recall<span class="token punctuation">,</span> tree_recall<span class="token punctuation">,</span> rf_recall<span class="token punctuation">,</span> svm_recall<span class="token punctuation">,</span> knn_recall<span class="token punctuation">,</span> bayes_recall<span class="token punctuation">]</span><span class="token punctuation">,</span><br /> <span class="token string">'F1'</span> <span class="token punctuation">:</span> <span class="token punctuation">[</span>logit_f1<span class="token punctuation">,</span> tree_f1<span class="token punctuation">,</span> rf_f1<span class="token punctuation">,</span> svm_f1<span class="token punctuation">,</span> knn_f1<span class="token punctuation">,</span> bayes_f1<span class="token punctuation">]</span><span class="token punctuation">,</span><br /> <span class="token string">'cv_precision'</span> <span class="token punctuation">:</span> <span class="token punctuation">[</span>logit_cv_mean<span class="token punctuation">,</span> tree_cv_mean<span class="token punctuation">,</span> rf_cv_mean<span class="token punctuation">,</span> svm_cv_mean<span class="token punctuation">,</span> knn_cv_mean<span class="token punctuation">,</span> bayes_cv_mean<span class="token punctuation">]</span><br /><span class="token punctuation">}</span><span class="token punctuation">)</span><br /><span class="token comment"># Print table and sort by test precision</span><br />models<span class="token punctuation">.</span>sort_values<span class="token punctuation">(</span>by<span class="token operator">=</span><span class="token string">'Precision'</span><span class="token punctuation">,</span> ascending<span class="token operator">=</span><span class="token boolean">False</span><span class="token punctuation">)</span><br /><br /> Model F1 Accuracy Precision cv_precision recall<br /> r<span class="token punctuation">.</span>f<span class="token punctuation">.</span> <span class="token number">0.976812</span> <span class="token number">0.989333</span> <span class="token number">0.994100</span> <span class="token number">0.988294</span> <span class="token number">0.960114</span><br /> d<span class="token punctuation">.</span>Tree <span class="token number">0.915088</span> <span class="token number">0.959667</span> <span class="token number">0.901798</span> <span class="token number">0.899632</span> <span class="token number">0.928775</span><br /> SVM <span class="token number">0.902098</span> <span class="token number">0.953333</span> <span class="token number">0.885989</span> <span class="token number">0.863300</span> <span class="token number">0.918803</span><br /> kNN <span class="token number">0.902959</span> <span class="token number">0.953000</span> <span class="token number">0.873502</span> <span class="token number">0.831590</span> <span class="token number">0.934473</span><br /> Bayes <span class="token number">0.530179</span> <span class="token number">0.808000</span> <span class="token number">0.620229</span> <span class="token number">0.831590</span> <span class="token number">0.462963</span><br />Logistic <span class="token number">0.327684</span> <span class="token number">0.762000</span> <span class="token number">0.483333</span> <span class="token number">0.526624</span> <span class="token number">0.247863</span></code></pre>
<p>From the above results table, it is clear the Random Forest is the best model based on Accuracy, Precision and Recall metrics. The performance is so good, that I would be concerned about overfitting and a lack of generalisation to future periods of data. However, the classifier was evaluated with K-folds cross-validation (K=10) and also evaluated on the test dataset.</p>
<p>To use the trained model later, it's first best practice to re-train on the entire dataset to get the best trained model. Then save the pickle i.e. serialised model.</p>
<pre class="language-python"><code class="language-python"><span class="token comment"># Create x and y from all data</span><br />y <span class="token operator">=</span> df<span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><br />x <span class="token operator">=</span> df<span class="token punctuation">.</span>drop<span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token string">'left'</span><span class="token punctuation">]</span><span class="token punctuation">,</span> axis<span class="token operator">=</span><span class="token number">1</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Re-train model on all data</span><br />rf_model <span class="token operator">=</span> rf<span class="token punctuation">.</span>fit<span class="token punctuation">(</span>x<span class="token punctuation">,</span> y<span class="token punctuation">)</span><br /><br /><span class="token comment"># Save model</span><br /><span class="token keyword">import</span> cPickle<br /><span class="token keyword">with</span> <span class="token builtin">open</span><span class="token punctuation">(</span><span class="token string">'churn_classifier.pkl'</span><span class="token punctuation">,</span> <span class="token string">'wb'</span><span class="token punctuation">)</span> <span class="token keyword">as</span> fid<span class="token punctuation">:</span><br /> cPickle<span class="token punctuation">.</span>dump<span class="token punctuation">(</span>rf_model<span class="token punctuation">,</span> fid<span class="token punctuation">)</span></code></pre>
<p>The unabridged version of the above code can be found <a href="https://github.com/ucg8j/kaggle_HR">here on github</a>.</p>
<h2 id="good-resources">Good Resources <a class="direct-link" href="https://lukesingham.com/whos-going-to-leave-next/#good-resources">#</a></h2>
<ul>
<li>
<p><a href="http://www.statsmodels.org/dev/contrasts.html">Contrasts/Encoding in Python</a></p>
</li>
<li>
<p><a href="https://stats.idre.ucla.edu/sas/webbooks/reg/chapter5/regression-with-saschapter-5-additional-coding-systems-for-categorical-variables-in-regressionanalysis/">UCLA's Contrasts/Encoding in SAS</a></p>
</li>
<li>
<p><a href="http://rstudio-pubs-static.s3.amazonaws.com/65059_586f394d8eb84f84b1baaf56ffb6b47f.html">Contrasts/Encoding in R</a></p>
</li>
<li>
<p><a href="http://scikit-learn.org/stable/modules/preprocessing.html">Sci-kit Learn's "Preprocessing" Doc</a></p>
</li>
<li>
<p><a href="http://www.algosome.com/articles/dummy-variable-trap-regression.html">Simple Explanation of the Dummy Variable Trap</a></p>
</li>
<li>
<p><a href="https://stats.stackexchange.com/questions/199549/how-can-machine-learning-models-gbm-nn-etc-be-used-for-survival-analysis/263946#263946">Cross Validated: How can machine learning models be used for survival analysis?</a></p>
</li>
<li>
<p><a href="https://ragulpr.github.io/2016/12/22/WTTE-RNN-Hackless-churn-modeling/">Healthy criticism of the approach I took</a></p>
</li>
<li>
<p><a href="https://tryolabs.com/blog/2013/03/25/why-accuracy-alone-bad-measure-classification-tasks-and-what-we-can-do-about-it/">A very well written (human speak) blog on accuracy vs. recall & precision</a></p>
</li>
<li>
<p><a href="http://www.damienfrancois.be/blog/files/modelperfcheatsheet.pdf">Model Performance Cheat Sheet</a></p>
</li>
<li>
<p><a href="https://classeval.wordpress.com/introduction/">Very simple explanation of ML evaluation metrics with nice visuals</a></p>
</li>
<li>
<p><a href="https://kevinzakka.github.io/2016/07/13/k-nearest-neighbor/">A good blog on KNN using Python</a></p>
</li>
<li>
<p><a href="http://www.datasciencecentral.com/profiles/blogs/customer-churn-logistic-regression-with-r">Good process schema</a></p>
</li>
<li>
<p><a href="https://www.capgemini.com/wp-content/uploads/2017/07/Predictive_Modeling_Using_Transactional_Data.pdf">High level overview of using transactional data for churn modelling</a></p>
</li>
</ul>
Rooting a Oneplus 3T Phone2017-07-05T00:00:00Zhttps://lukesingham.com/rooting-a-oneplus-3t-phone/<p>I recently rooted my Oneplus 3T phone on Android 7.1.1. Finding the right information was not as easy as I thought it might be, so here's my summary. I used my macbook, you will need your USB cord to connect the phone. <strong>Warning</strong> - you will end up wiping your phone during this process, I'd recommend performing a backup prior. I didn't as I figured most app data are synced to the cloud, the only things I noticed missing was SMSs and things like numbers and Whatsapp history where a sync to the cloud occurred on a weekly basis.</p>
<h2 id="1.-unlock-the-bootloader">1. Unlock the <a href="https://www.androidcentral.com/what-is-android-bootloader">Bootloader</a> <a class="direct-link" href="https://lukesingham.com/rooting-a-oneplus-3t-phone/#1.-unlock-the-bootloader">#</a></h2>
<p>a. If you haven't already done so, you will need to enable <code>Developer Options</code> on your phone by tapping 7x on the <code>Build Number</code> found in your phone under <code>Settings > About Phone > Build Number</code>. Once enabled, navigate to <code>Settings > Developer Options > USB debugging</code> and ensure this is toggled to ON.</p>
<p>b. You will now need the to obtain the <a href="https://dl.google.com/android/android-sdk_r24.4.1-macosx.zip">Android Software Development Kit to run commands on your phone from your computer terminal</a>. This provides several tools you will use including <a href="https://developer.android.com/studio/command-line/adb.html">Android Development Bridge (ADB)</a> and <a href="https://www.androidcentral.com/android-z-what-fastboot">Fastboot</a>. Once downloaded, unzip and place the files in a place where you can find them. Then navigate to the folder and launch the Android SDK manager by double clicking on <code>android-sdk-macosx/tools/android</code>. This will provide a GUI to install the tools needed as per the following screenshot.</p>
<p><img src="https://lukesingham.com/content/images/2017/07/Screen-Shot-2017-07-05-at-14-16-57.png" alt="" /></p>
<p>Now you are ready to run some commands:</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Navigate to the tools</span><br />$ <span class="token builtin class-name">cd</span> /android-sdk-macosx/platform-tools<br /><span class="token comment"># Check that your phone is connected properly</span><br />$ adb devices<br />List of devices attached<br />fc031cd8 device<br /><br /><span class="token comment"># Reboot phone into bootloader</span><br />$ ./adb <span class="token function">reboot</span> bootloader<br /><span class="token comment"># Unlock the phone</span><br />$ ./fastboot oem unlock</code></pre>
<p>The last command will wipe your phone. Once complete you will then boot up the phone and need to repeat the process in step 1.a.</p>
<h2 id="2.-install-twrp-custom-recovery">2. Install TWRP Custom Recovery <a class="direct-link" href="https://lukesingham.com/rooting-a-oneplus-3t-phone/#2.-install-twrp-custom-recovery">#</a></h2>
<p>I chose the open source <a href="https://en.wikipedia.org/wiki/TWRP">TWRP recovery software</a>. You can get a copy of TWRP <a href="http://techerrata.com/browse/twrp2/bacon">here</a>. Without the installation of TWRP, Step 3 will not be possible (I tried but the OnePlus recovery won't allow you to install SuperSu).</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Check that your phone is connected properly</span><br />$ adb devices<br />List of devices attached<br />fc031cd8 device<br /><br />$ ./fastboot flash recovery /YOUR/<span class="token environment constant">PATH</span>/TO/TWRP.img<br />$ ./fastboot <span class="token function">reboot</span></code></pre>
<h2 id="3.-root-via-flashing-supersu">3. Root Via Flashing SuperSU <a class="direct-link" href="https://lukesingham.com/rooting-a-oneplus-3t-phone/#3.-root-via-flashing-supersu">#</a></h2>
<p>If you haven't already <a href="https://www.android.com/filetransfer/">download Android File Transfer</a>. Transfer the <a href="https://download.chainfire.eu/696/SuperSU/UPDATE-SuperSU-v2.46.zip">SuperSU zip</a> into the root of your phone. Ensure <code>Settings > Developer Options > Advanced Reboot</code> is toggled to on. Then press the power off button and select <code>Reboot > Recovery</code>. From here you will have the new TWRP menu to select <code>Install</code> and choose the SuperSu zip file. You will need to swipe right to confirm the Flash.</p>
<h3 id="one-small-hiccup">One Small Hiccup <a class="direct-link" href="https://lukesingham.com/rooting-a-oneplus-3t-phone/#one-small-hiccup">#</a></h3>
<p>I encountered one issue in this process. SuperSu was not intalling from the TWRP menu. I stopped the above process in between Step 2 and Step 3 as I had to go to bed for work the next day. On booting my phone for the day, I let my android install apps associated with my google account. Normally a great feature when signing into a new phone or one that has been formatted. However, the phone must be freshly wiped to be able to install SuperSu. So I had to separately wipe the phone again using the following commands found on this <a href="https://stackoverflow.com/a/39357553/3691003">SO answer</a>:</p>
<pre class="language-bash"><code class="language-bash">$ ./adb <span class="token function">reboot</span> bootloader<br /><span class="token comment"># Wait a few seconds...</span><br /><br /><span class="token comment"># Check the phone is in bootloader</span><br />$ ./fastboot devices<br /><span class="token comment"># Wipe user data</span><br />$ ./fastboot <span class="token parameter variable">-w</span></code></pre>
<h2 id="useful-resources">Useful Resources <a class="direct-link" href="https://lukesingham.com/rooting-a-oneplus-3t-phone/#useful-resources">#</a></h2>
<ul>
<li><a href="https://www.androidexplained.com/oneplus-3t-root/">Android Explained</a></li>
<li><a href="https://forums.oneplus.net/threads/guide-for-mac-how-to-unlock-bootloader-install-custom-recovery-root-macbook-pro.279125/">OnePlus Forums</a></li>
</ul>
Benchmarking the Performance of R Code2017-06-18T00:00:00Zhttps://lukesingham.com/benchmarking-the-performance-of-r-code/<p>To assess the performance of R code there's a great little package called <a href="https://cran.r-project.org/web/packages/microbenchmark/index.html">microbenchmark</a>.</p>
<pre class="language-r"><code class="language-r">install.packages<span class="token punctuation">(</span><span class="token string">'microbenchmark'</span><span class="token punctuation">)</span><br />library<span class="token punctuation">(</span>microbenchmark<span class="token punctuation">)</span></code></pre>
<p>I was particularly interested in the performance increase of a <a href="https://shiny.rstudio.com/">shiny application</a> that reads in a bunch of <code>.csv</code> data. To test the performance increase I ran the following code:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># For all *.csv files in "data" directory</span><br />csv.files <span class="token operator"><-</span> list.files<span class="token punctuation">(</span>path <span class="token operator">=</span> <span class="token string">"data"</span><span class="token punctuation">,</span> pattern <span class="token operator">=</span> <span class="token string">"*.csv"</span><span class="token punctuation">,</span> full.names <span class="token operator">=</span> <span class="token boolean">TRUE</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># How many csvs are there?</span><br />length<span class="token punctuation">(</span>csv.files<span class="token punctuation">)</span><br /><span class="token comment">#[1] 13</span><br /><br /><span class="token comment"># Benchmark</span><br />results <span class="token operator"><-</span> microbenchmark<span class="token punctuation">(</span><br /> read.csv2.test <span class="token operator">=</span> lapply<span class="token punctuation">(</span>csv.files<span class="token punctuation">,</span> read.csv2<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> read_csv.test <span class="token operator">=</span> lapply<span class="token punctuation">(</span>csv.files<span class="token punctuation">,</span> read_csv<span class="token punctuation">)</span><span class="token punctuation">,</span><br /> times <span class="token operator">=</span> <span class="token number">50</span><span class="token punctuation">)</span></code></pre>
<p>So that's 50 iterations of reading 13 files, or 650 reads for each function. Now to have a look at the results:</p>
<pre class="language-r"><code class="language-r"><span class="token operator">></span> results<br />Unit<span class="token operator">:</span> milliseconds<br /> expr min lq mean median uq max neval<br /> read.csv2.test <span class="token number">111.18612</span> <span class="token number">118.28930</span> <span class="token number">124.71949</span> <span class="token number">123.38192</span> <span class="token number">131.20016</span> <span class="token number">144.3691</span> <span class="token number">50</span><br /> read_csv.test <span class="token number">67.56371</span> <span class="token number">71.28666</span> <span class="token number">76.22763</span> <span class="token number">74.19215</span> <span class="token number">78.06555</span> <span class="token number">105.8433</span> <span class="token number">50</span></code></pre>
<p>Not the most impressive gain from the <code>readr</code> package, where the <a href="https://cran.r-project.org/web/packages/readr/README.html">documentation purports up to 10x faster reads</a>, but still substantial and worth implementing. For a simple guide on UX and the implications of lag check out this <a href="https://ux.stackexchange.com/a/3836/78865">stackexchange answer</a>. And if you want even faster read/write operations have a look at <a href="https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html">data.table</a>'s <code>fread()</code> function.</p>
<h2 id="examine-the-distribution-of-evaluations">Examine the Distribution of Evaluations <a class="direct-link" href="https://lukesingham.com/benchmarking-the-performance-of-r-code/#examine-the-distribution-of-evaluations">#</a></h2>
<p>The <a href="https://cran.r-project.org/web/packages/microbenchmark/index.html">microbenchmark</a> package includes a <code>boxplot()</code> function which is a really good idea to look at prior to implementing a particular change in your code base. Here's an example of a comparison between a custom function and the same function compiled using the <code>compiler::cmpfun()</code> function.</p>
<p>Here's the a print out of the results:</p>
<pre class="language-r"><code class="language-r">print<span class="token punctuation">(</span>results<span class="token punctuation">)</span><br /><span class="token comment">## Unit: microseconds</span><br /><span class="token comment">## expr min lq mean median uq max neval</span><br /><span class="token comment">## plain 87.550 90.5355 227.71949 91.8995 94.5685 13308.578 100</span><br /><span class="token comment">## compiled 87.254 90.9845 95.71651 92.4605 96.1630 194.724 100</span></code></pre>
<p>If I only compared the mean of the two functions, then I would expect to get a considerable 58% speed increase.</p>
<p><img src="https://lukesingham.com/content/images/2017/08/microbenchmark_compiled_function.png" alt="Microbenchmark Boxplot" /></p>
<p>However, the boxplot of the results <code>boxplot(results)</code> quickly reveals an outlier in the 100 evaluations performed that heavily impacts the mean. In this case, the median actually shows a negligible difference between the function and the byte compiled function.</p>
<p>If you prefer a fancier looking plot, microbenchmark has included a <a href="http://ggplot2.org/">ggplot2</a> based <code>autoplot()</code> function.</p>
<pre class="language-r"><code class="language-r">library<span class="token punctuation">(</span>ggplot2<span class="token punctuation">)</span><br />autoplot<span class="token punctuation">(</span>results<span class="token punctuation">)</span></code></pre>
<p><img src="https://lukesingham.com/content/images/2017/08/microbenchmark_autoplot.png" alt="Microbenchmark Autoplot" /></p>
<h1 id="when-should-you-use-microbenchmarking%3F">When Should You Use Microbenchmarking? <a class="direct-link" href="https://lukesingham.com/benchmarking-the-performance-of-r-code/#when-should-you-use-microbenchmarking%3F">#</a></h1>
<p>I use the microbenchmark package whenever I am concerned about speeding up an application. Old hands know the obligatory computer science mantra and the Knuth quote that must go along with this microbenchmarking blog post.</p>
<blockquote>
<p>Make it work, make it right, make it fast.</p>
</blockquote>
<p>-- Kent Beck</p>
<blockquote>
<p>... premature optimization is the root of all evil...</p>
</blockquote>
<p>-- Donald Knuth</p>
<p>Knuth goes on to say in <a href="http://web.archive.org/web/20130731202547/http://pplab.snu.ac.kr/courses/adv_pl05/papers/p261-knuth.pdf">his paper</a>,<em>'It is often a mistake to make a priori judgments about what parts of a program are really critical'</em>. Which is why you should first profile your code to identify where your efforts are best spent.</p>
<p><strong>Postscript</strong> - <a href="https://m.signalvnoise.com/cant-crack-that-programming-problem-go-to-sleep-or-take-a-walk-930c767e1119">After a few nights of sleep I realised</a> why the compiled vs. non-compiled functions had the distributions they did. <a href="https://stat.ethz.ch/pipermail/r-announce/2017/000612.html">Version 3.4 of R</a> has <a href="https://en.wikipedia.org/wiki/Just-in-time_compilation">Just-In-Time (JIT) compiling</a> at level 3 by default. The function I was comparing contains a <code>for</code> loop. Therefore, the plots show the difference between the new <code>JIT(3)</code> speed of R vs pre-compiling your functions that contain for loops. The first run of the non-compiled function is not byte-compiled, however all subsequent runs are byte-compiled. Hence one outlier but otherwise very similar distributions.</p>
How to Stop R Projects and Scripts Breaking2016-11-07T00:00:00Zhttps://lukesingham.com/using-package-management-in-r/<p><em>A how-to in getting started with Package Management in R</em></p>
<p><strong>The problem:</strong> Let's say you coded a project in <code>R</code> in 2014 using the package <a href="http://dplyr.tidyverse.org/"><code>dplyr</code></a> and you called the project <code>2014project</code>. Over time packages get updated, sometimes changing syntax and function names, let's say some major syntax changes occurred in 2015 to <code>dplyr</code> and you used the latest version of <a href="http://dplyr.tidyverse.org/"><code>dplyr</code></a> in other projects in 2015. Now you come to 2016 and you want to re-run the code in the <code>2014project</code>, but you find a bunch of errors because in 2015 dplyr had those syntax changes, such that the syntax you used in 2014 no longer computes. Now for the <strong>solution</strong>...</p>
<p>R scripts breaking over time is annoying for anyone, but it's worth highlighting that organisations using proper package management will significantly decrease their costs in the form of maintenance time spent by developers, data scientists and data analysts fixing projects broken due to changes in package dependencies. In contrast to the Python community who seem very familiar with their <a href="https://virtualenv.pypa.io/en/stable/"><code>virtualenv</code> package management</a>, the R community doesn't seem to have widely adopted the practice. In the data science world, I attribute this difference between the R and Python communities to Python coming out of the computer science discipline and R coming from the statistics discipline. If you are an organisation heavily using R, you have a lot of productivity to gain from the adoption of package management by your R developers.</p>
<h2 id="packrat">Packrat <a class="direct-link" href="https://lukesingham.com/using-package-management-in-r/#packrat">#</a></h2>
<pre class="language-r"><code class="language-r"><span class="token operator">></span> readLines<span class="token punctuation">(</span>system.file<span class="token punctuation">(</span><span class="token string">"DESCRIPTION"</span><span class="token punctuation">,</span> package <span class="token operator">=</span> <span class="token string">"packrat"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">[</span>c<span class="token punctuation">(</span><span class="token number">3</span><span class="token punctuation">,</span><span class="token number">9</span><span class="token punctuation">,</span><span class="token number">10</span><span class="token punctuation">)</span><span class="token punctuation">]</span><br /><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token string">"Title: A Dependency Management System for Projects and their R Package"</span><br /><span class="token punctuation">[</span><span class="token number">2</span><span class="token punctuation">]</span> <span class="token string">"Description: Manage the R packages your project depends on in an"</span><br /><span class="token punctuation">[</span><span class="token number">3</span><span class="token punctuation">]</span> <span class="token string">"isolated, portable, and reproducible way."</span> </code></pre>
<h2 id="getting-started">Getting Started <a class="direct-link" href="https://lukesingham.com/using-package-management-in-r/#getting-started">#</a></h2>
<p>To get the RStudio niceties/integrations it seems neccessary to treat your project folder as an R Project. I've created a new project <code>/testpackrat2</code> and with the latest RStudio you should be able to select both <code>git</code> and <code>packrat</code> to be initialised. This runs the <code>packrat::init()</code> function to enter packrat mode, takes a snapshot of the package dependencies (of which there are currently none) and places the binaries in the project folder under <code>testpackrat2/packrat/src/</code>; finally <code>init()</code> runs <code>restore()</code> to apply the latest snapshot to the project folder. <code>R</code> will restart once this process is finished.</p>
<p><img src="https://lukesingham.com/content/images/2016/11/Screen-Shot-2016-11-08-at-09-19-09.png" alt="New Packrat Project RStudio" /><br />
To double check packrat has been initialised properly run:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Check I am in the right directory</span><br /><span class="token operator">></span> getwd<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span> <span class="token string">"/Users/lukesingham/Projects/testpackrat2"</span><br /><br /><span class="token comment"># See if packrat is working</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>status<span class="token punctuation">(</span><span class="token punctuation">)</span><br />Up to date.</code></pre>
<p>You should also notice the RStudio integration in the packages area, where it will show your project version against that which Packrat has the binary for and the source of that binary:</p>
<p><img src="https://lukesingham.com/content/images/2016/11/Screen-Shot-2016-11-07-at-10-03-25.png" alt="packages window RStudio " /></p>
<p>For test purposes I created <code>script.R</code> with only one package.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Create script</span><br /><span class="token operator">></span> system<span class="token punctuation">(</span><span class="token string">"echo 'library(stringr)' >| script.R"</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Confirm project contents</span><br /><span class="token operator">></span> system<span class="token punctuation">(</span><span class="token string">"ls"</span><span class="token punctuation">)</span><br />packrat<br />script.R<br />testpackrat.Rproj</code></pre>
<p>If I open <code>script.R</code> in RStudio and try and load the library this is the expected behaviour:</p>
<pre class="language-r"><code class="language-r"><span class="token operator">></span> library<span class="token punctuation">(</span>stringr<span class="token punctuation">)</span><br />Error <span class="token keyword">in</span> library<span class="token punctuation">(</span>stringr<span class="token punctuation">)</span> <span class="token operator">:</span> there is no package called âstringrâ</code></pre>
<p>I have <code>stringr</code> installed on my system, but now I am in an isolated packrat project which is only reading from the project folders list of packages. Now to run the packrat commands to <code>snapshot()</code> the addition of <code>stringr</code> to the project and install it ready for use in the project.</p>
<pre class="language-r"><code class="language-r"><span class="token operator">></span> packrat<span class="token operator">::</span>status<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><br />The following packages are referenced <span class="token keyword">in</span> your code<span class="token punctuation">,</span> but are not present<br /><span class="token keyword">in</span> your library nor <span class="token keyword">in</span> packrat<span class="token operator">:</span><br /><br /> stringr<br /><br />You will need to install these packages manually<span class="token punctuation">,</span> then use<br />packrat<span class="token operator">::</span>snapshot<span class="token punctuation">(</span><span class="token punctuation">)</span> to record these packages <span class="token keyword">in</span> packrat.<br /><br /><span class="token comment"># Well I do have stringr in my library, anyway...</span><br /><span class="token comment"># now to take a snapshot</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>snapshot<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><br />Adding these packages to packrat<span class="token operator">:</span><br /> _<br /> magrittr <span class="token number">1.5</span> <br /> stringi <span class="token number">1.1</span><span class="token number">.2</span><br /> stringr <span class="token number">1.1</span><span class="token number">.0</span><br /><br />Fetching sources <span class="token keyword">for</span> magrittr <span class="token punctuation">(</span><span class="token number">1.5</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span> OK <span class="token punctuation">(</span>CRAN current<span class="token punctuation">)</span><br />Fetching sources <span class="token keyword">for</span> stringi <span class="token punctuation">(</span><span class="token number">1.1</span><span class="token number">.2</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span> OK <span class="token punctuation">(</span>CRAN current<span class="token punctuation">)</span><br />Fetching sources <span class="token keyword">for</span> stringr <span class="token punctuation">(</span><span class="token number">1.1</span><span class="token number">.0</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span> OK <span class="token punctuation">(</span>CRAN current<span class="token punctuation">)</span><br />Snapshot written to <span class="token string">'/Users/lukesingham/Projects/testpackrat2/packrat/packrat.lock'</span><br /><br /><span class="token comment"># Now to install/'restore' those sources to the snapshot just taken</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>restore<span class="token punctuation">(</span><span class="token punctuation">)</span><br />Installing magrittr <span class="token punctuation">(</span><span class="token number">1.5</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span><br /> OK <span class="token punctuation">(</span>built source<span class="token punctuation">)</span><br />Installing stringi <span class="token punctuation">(</span><span class="token number">1.1</span><span class="token number">.2</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span><br /> OK <span class="token punctuation">(</span>built source<span class="token punctuation">)</span><br />Installing stringr <span class="token punctuation">(</span><span class="token number">1.1</span><span class="token number">.0</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span><br /> OK <span class="token punctuation">(</span>built source<span class="token punctuation">)</span></code></pre>
<p>It's probably worth pointing out what packrat is doing to your library path, particularly when you go to install new packages in your project.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Turn packrat off</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>packrat_mode<span class="token punctuation">(</span><span class="token punctuation">)</span><br />Packrat mode off. Resetting library paths to<span class="token operator">:</span><br /><span class="token operator">-</span> <span class="token string">"/usr/local/lib/R/3.2/site-library"</span><br /><span class="token operator">-</span> <span class="token string">"/usr/local/Cellar/r/3.2.3/R.framework/Versions/3.2/Resources/library"</span><br /><br /><span class="token comment"># Turn packrat back on</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>packrat_mode<span class="token punctuation">(</span><span class="token punctuation">)</span><br />Packrat mode on. Using library <span class="token keyword">in</span> directory<span class="token operator">:</span><br /><span class="token operator">-</span> <span class="token string">"~/Projects/testpackrat2/packrat/lib"</span></code></pre>
<p>So as long as you are in <code>packrat_mode</code>, when you run <code>install.packages()</code> or remove <code>remove.packages()</code> it will only be modifying the project <code>testpackrat2/</code> folder.</p>
<h2 id="existing-project-with-out-of-date-packages">Existing Project with Out-Of-Date Packages <a class="direct-link" href="https://lukesingham.com/using-package-management-in-r/#existing-project-with-out-of-date-packages">#</a></h2>
<p>I want to test the scenario where I have existing projects using old versions of packages. To do this I ran:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Find old packages</span><br />old.packages<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><span class="token operator">></span> Installed Built ReposVer<br />lubridate <span class="token string">"1.5.6"</span> <span class="token string">"3.2.3"</span> <span class="token string">"1.6.0"</span><br />htmlwidgets <span class="token string">"0.6"</span> <span class="token string">"3.2.3"</span> <span class="token string">"0.7"</span></code></pre>
<p>I then created a folder with these packages, to simulate an old project that I want <code>packrat</code> to manage.</p>
<pre class="language-bash"><code class="language-bash"><span class="token function">mkdir</span> testpackratOutofDatePackages<br /><span class="token builtin class-name">echo</span> <span class="token string">"library(lubridate); library(htmlwidgets)"</span> <span class="token operator">>|</span> testpackratOutofDatePackages/oldScript.R</code></pre>
<p>Now go to RStudio and open the existing project <code>testpackratOutofDatePackages</code>. Click on <code>File > New Project > Existing Directory</code></p>
<p><img src="https://lukesingham.com/content/images/2016/11/Screen-Shot-2016-11-07-at-19-58-24.png" alt="New project Rstudio" /></p>
<p>Unfortunately there is not an option to <code>init</code> packrat as part the process. So...</p>
<pre class="language-r"><code class="language-r"><span class="token operator">></span> packrat<span class="token operator">::</span>init<span class="token punctuation">(</span><span class="token punctuation">)</span><br />Initializing packrat project <span class="token keyword">in</span> directory<span class="token operator">:</span><br /><span class="token operator">-</span> <span class="token string">"~/Projects/testpackratOutofDatePackages"</span><br /><span class="token ellipsis">...</span> <span class="token punctuation">(</span>edited down the consolte printout<span class="token punctuation">)</span><br />Fetching sources <span class="token keyword">for</span> htmlwidgets <span class="token punctuation">(</span><span class="token number">0.6</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span> OK <span class="token punctuation">(</span>CRAN archived<span class="token punctuation">)</span><br />Fetching sources <span class="token keyword">for</span> lubridate <span class="token punctuation">(</span><span class="token number">1.5</span><span class="token number">.6</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span> OK <span class="token punctuation">(</span>CRAN archived<span class="token punctuation">)</span></code></pre>
<p>Ok great, it fetches the old binaries. Now let's test upgrading these packages and then reverting back to the initial project state.</p>
<pre class="language-r"><code class="language-r"><span class="token operator">></span> install.packages<span class="token punctuation">(</span><span class="token string">"htmlwidgets"</span><span class="token punctuation">)</span> <span class="token comment"># v0.7</span><br />Installing package into â<span class="token operator">/</span>Users<span class="token operator">/</span>lukesingham<span class="token operator">/</span>Projects<span class="token operator">/</span>testpackratOutofDatePackages<span class="token operator">/</span>packrat<span class="token operator">/</span>lib<span class="token operator">/</span>x86_64<span class="token operator">-</span>apple<span class="token operator">-</span>darwin15.<span class="token number">2.0</span><span class="token operator">/</span><span class="token number">3.2</span><span class="token number">.3</span>â <span class="token ellipsis">...</span><br /><span class="token operator">*</span> DONE <span class="token punctuation">(</span>htmlwidgets<span class="token punctuation">)</span></code></pre>
<p>So it seems packrat auto snapshots the project, such that I couldn't run <code>restore()</code> and return to htmlwidgets v0.6. Let's try that again without autosnapshot and with my out-of-date <code>lubridate</code>:</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Turn off auto.snapshot</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>set_opts<span class="token punctuation">(</span>auto.snapshot<span class="token operator">=</span><span class="token boolean">FALSE</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Check status</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>status<span class="token punctuation">(</span><span class="token punctuation">)</span><br />Up to date.<br /><br /><span class="token comment"># Now try updating lubridate</span><br /><span class="token operator">></span> install.packages<span class="token punctuation">(</span><span class="token string">"lubridate"</span><span class="token punctuation">)</span><br />DONE<br /><br /><span class="token operator">></span> packrat<span class="token operator">::</span>status<span class="token punctuation">(</span><span class="token punctuation">)</span><br /><br />The following packages are out of sync between packrat and your current library<span class="token operator">:</span><br /> packrat library<br /> lubridate <span class="token number">1.5</span><span class="token number">.6</span> <span class="token number">1.6</span><span class="token number">.0</span><br /><br />Use packrat<span class="token operator">::</span>snapshot<span class="token punctuation">(</span><span class="token punctuation">)</span> to set packrat to use the current library<span class="token punctuation">,</span> or use<br />packrat<span class="token operator">::</span>restore<span class="token punctuation">(</span><span class="token punctuation">)</span> to reset the library to the last snapshot.<br /><br /><span class="token comment"># Return to old version of lubridate</span><br /><span class="token operator">></span> packrat<span class="token operator">::</span>restore<span class="token punctuation">(</span>overwrite.dirty<span class="token operator">=</span>T<span class="token punctuation">)</span><br /><br />Downgrading these packages <span class="token keyword">in</span> your library<span class="token operator">:</span><br /> from to<br /> lubridate <span class="token number">1.6</span><span class="token number">.0</span> <span class="token number">1.5</span><span class="token number">.6</span><br /><br />Do you want to continue? <span class="token punctuation">[</span>Y<span class="token operator">/</span>n<span class="token punctuation">]</span><span class="token operator">:</span> Y<br />Replacing lubridate <span class="token punctuation">(</span>downgrade <span class="token number">1.6</span><span class="token number">.0</span> to <span class="token number">1.5</span><span class="token number">.6</span><span class="token punctuation">)</span> <span class="token ellipsis">...</span> OK <span class="token punctuation">(</span>built source<span class="token punctuation">)</span></code></pre>
<h2 id="conclusion">Conclusion <a class="direct-link" href="https://lukesingham.com/using-package-management-in-r/#conclusion">#</a></h2>
<p>By default packrat <code>auto.snapshot</code> my project when I updated a package. Crucially, this meant I could not use <code>packrat::restore()</code> because I was <code>already up to date</code>. Of course, if I had been using <code>git</code> and making commits before and after, then there wouldn't have been a problem. However, without <code>git</code>, running <code>packrat::restore()</code> to downgrade a package requires the <code>overwrite.dirty=TRUE</code> option and only seems possible if you switched off <code>auto.snapshot</code>. I'd highly recommend turning it off anyway, as the manual update will allow for a better understanding and control over packrat.</p>
<h2 id="other-resources">Other Resources <a class="direct-link" href="https://lukesingham.com/using-package-management-in-r/#other-resources">#</a></h2>
<ul>
<li><a href="https://www.rstudio.com/resources/webinars/managing-package-dependencies-in-r-with-packrat/">RStudio Video presentation of Packrat</a></li>
<li><a href="https://cran.r-project.org/web/packages/packrat/packrat.pdf">Packrat Package Documentation</a></li>
<li><a href="https://rstudio.github.io/packrat/">Packrat website</a></li>
<li><a href="https://mran.microsoft.com/rro/#repos">An alternative to packrat, Microsoft's MRAN and Enhanced R</a></li>
<li><a href="https://groups.google.com/forum/#!forum/packrat-discuss">Packrat discussion board</a></li>
</ul>
Map of Australia Using OpenStreetMaps, PSMA, R and Leaflet.js2016-08-10T00:00:00Zhttps://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/<p><strong>Aim:</strong> To use GNAF (Geocoded National Address File) data to display the adminstrative boundaries (e.g. States) on top of Open Street Maps. Additionally, I want the user when hovering over a state to see summary statistics. I want an Australian version of <a href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#leaflet.js">this</a>.</p>
<h2 id="data">Data <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#data">#</a></h2>
<p>G-NAF, is data containing all Australian addresses with geocoding, shape files for boundaries of states, electorates, municipals. It is the geocoding of a nation. And finally was released on <a href="http://data.gov.au/">data.gov.au</a> this year. <a href="https://docs.google.com/viewer?url=http%3A%2F%2Fminus34.com%2Fopendata%2Fintro-to-gnaf.pptx&embedded=true&chrome=false&dov=1">Here's a nice GNAF data overview.</a> N.B. One set of data that is missing from the release is the <a href="https://www.psma.com.au/products/postcode-boundaries">postcode boundaries</a>.</p>
<h3 id="state-boundaries">State Boundaries <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#state-boundaries">#</a></h3>
<p>My initial geojson to get started was sourced from this <a href="https://raw.githubusercontent.com/edwinsteele/d3-projects/master/data/au-states.geojson">github repo</a>.</p>
<h3 id="postcode-boundaries">Postcode Boundaries <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#postcode-boundaries">#</a></h3>
<p>I obtained <a href="http://www.abs.gov.au/AUSSTATS/abs@.nsf/DetailsPage/1270.0.55.003July%202011?OpenDocument">the shape file for Australian postcodes</a> from the Australian Bureau of Statistics. I then converted the shapefile to geojson (helpful tutorial <a href="http://ben.balter.com/2013/06/26/how-to-convert-shapefiles-to-geojson-for-use-on-github/">here</a>):</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Convert shp file to geojson</span><br />$ ogr2ogr <span class="token parameter variable">-f</span> GeoJSON <span class="token parameter variable">-t_srs</span> crs:84 au-postcodes.geojson POA_2011_AUST.shp</code></pre>
<p>The resulting geojson is a whooping 201.1mb. There are two more processes to consider. Firstly, simplification of the polygons to reduce the load on the browser. The most popular algorithm to use for this is the <a href="https://en.wikipedia.org/wiki/Ramer%E2%80%93Douglas%E2%80%93Peucker_algorithm">RamerâDouglasâPeucker algorithm</a>. In R have a look at <a href="http://www.inside-r.org/packages/cran/rgeos/docs/gSimplify"><code>gSimplify()</code></a> however I am going to use the <a href="https://github.com/mbloch/mapshaper">mapshaper command line tool</a> as I prefer command line tools and mapshaper has a couple of implementations of Visvalingam's polygon simplification algorithm.</p>
<pre class="language-bash"><code class="language-bash"><span class="token comment"># Install package</span><br />$ <span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">-g</span> mapshaper<br /><br /><span class="token comment"># Run Visvalingam simplification retaining 10% of polygons</span><br />$ mapshaper <span class="token parameter variable">-i</span> au-postcodes.geojson <span class="token parameter variable">-simplify</span> <span class="token number">10</span>% <span class="token parameter variable">-o</span> au-postcodes-Visvalingam-0.1.geojson</code></pre>
<p>The size of the <code>au-postcodes-Visvalingam-0.1.geojson</code> file is about 10% of the size of <code>au-postcodes.geojson</code> file.</p>
<p><img src="https://lukesingham.com/content/images/2017/08/polygon-simplification.gif" alt="Polygon Simplification" /></p>
<p>Secondly, we may want to reduce the file size by 'minifying' or 'packing' the data by removing all spaces and new lines. This can be done using <a href="https://github.com/igorti/geojson-minifier">geojson-minify</a>. Packing the <code>au-postcodes.geojson</code> file from above with the command below reduced the file size to 19.1mb.</p>
<pre class="language-bash"><code class="language-bash">$ <span class="token function">npm</span> <span class="token function">install</span> <span class="token parameter variable">-g</span> geojson-minifier<br />$ geojson-minifier <span class="token parameter variable">-o</span> pack <span class="token parameter variable">-f</span> au-postcodes.geojson</code></pre>
<h2 id="maps">Maps <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#maps">#</a></h2>
<p>So <code>library(rgdal)</code> seems pretty essential but if you're using a mac like I am, here you run into some pain. If you try running <code>install.packages('rgdal')</code> it will exit with a status of 0. Installing the underlying <code>gdal</code> dependency fixes the problem.</p>
<pre class="language-bash"><code class="language-bash">$ brew <span class="token function">install</span> gdal</code></pre>
<p>After installing gdal, <code>install.packages('rgdal')</code> ran just fine.</p>
<h2 id="adding-data-to-the-shapefile">Adding Data to the Shapefile <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#adding-data-to-the-shapefile">#</a></h2>
<p>I used R for this process, <code>rgdal</code> allows you to do joins on shape files so you can attribute data to the geographical areas.</p>
<pre class="language-r"><code class="language-r">library<span class="token punctuation">(</span>rgdal<span class="token punctuation">)</span> <span class="token comment"># Spatial data processing</span><br />library<span class="token punctuation">(</span>jsonlite<span class="token punctuation">)</span> <span class="token comment"># Read json files</span><br />library<span class="token punctuation">(</span>readr<span class="token punctuation">)</span> <span class="token comment"># Fast I/O</span><br /><br /><span class="token comment"># Postcode shape json</span><br />gdal.postcodes <span class="token operator"><-</span> readOGR<span class="token punctuation">(</span><span class="token string">"au-postcodes-Visvalingam-0.1-density.geojson"</span><span class="token punctuation">,</span> <span class="token string">"OGRGeoJSON"</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Read data you want to join to the shape file</span><br />df <span class="token operator"><-</span> read_csv<span class="token punctuation">(</span><span class="token string">"your_data.csv"</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Change to character for leading 0s in NT postcodes</span><br />df<span class="token operator">$</span>cleaned_postcode <span class="token operator"><-</span> sprintf<span class="token punctuation">(</span><span class="token string">"%04d"</span><span class="token punctuation">,</span> df<span class="token operator">$</span>postcode<span class="token punctuation">)</span><br /><br /><span class="token comment"># Join agg values onto postcode (returns)</span><br />gdal.postcodes<span class="token operator">@</span>data <span class="token operator"><-</span> sp<span class="token operator">::</span>merge<span class="token punctuation">(</span>gdal.postcodes<span class="token operator">@</span>data<span class="token punctuation">,</span> df <span class="token punctuation">,</span> by.x <span class="token operator">=</span> <span class="token string">"POA_NAME"</span><span class="token punctuation">,</span> by.y <span class="token operator">=</span> <span class="token string">"cleaned_postcode"</span><span class="token punctuation">,</span> all.x <span class="token operator">=</span> <span class="token boolean">TRUE</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Write to a new file</span><br />writeOGR<span class="token punctuation">(</span>gdal.postcodes<span class="token punctuation">,</span> <span class="token string">'au-postcodes-Visvalingam-0.1-density.geojson'</span><span class="token punctuation">,</span><span class="token string">'spDf'</span><span class="token punctuation">,</span> driver <span class="token operator">=</span> <span class="token string">'GeoJSON'</span><span class="token punctuation">,</span> check_exists <span class="token operator">=</span> <span class="token boolean">FALSE</span><span class="token punctuation">)</span></code></pre>
<h2 id="leaflet.js">Leaflet.js <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#leaflet.js">#</a></h2>
<p>To display the information in a standalone single page app I started with this <a href="http://leafletjs.com/examples/choropleth-example.html">leaflet choropleth example</a>.</p>
<iframe frameBorder="0" width="820" height="520" src="https://leafletjs.com/examples/choropleth/example.html#map">
</iframe>
<p>There's a few things to customise to an Australian version:</p>
<h3 id="1.-getting-the-data-into-the-webpage">1. Getting the Data into the Webpage <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#1.-getting-the-data-into-the-webpage">#</a></h3>
<p>I modified the page to read <code>au-postcodes-Visvalingam-0.1-density.js</code>. You can't just read a <code>.json</code> file in the head of your page, for options on how to get data to load on your page see this <a href="http://stackoverflow.com/a/13576588/3691003">helpful SO answer</a>. I went with option two, I opened the <code>au-postcodes-Visvalingam-0.1-density.geosjson</code> and inserted <code>var postcodesData =</code> at the beginning of the data array and put a <code>;</code> to end the variable declaration.</p>
<pre class="language-js"><code class="language-js"><span class="token keyword">var</span> postcodesData <span class="token operator">=</span> <span class="token punctuation">{</span><br /><span class="token operator">...</span><br /><span class="token punctuation">}</span><span class="token punctuation">;</span></code></pre>
<p>Then save as <code>.js</code>. Now you can read the <code>au-postcodes-Visvalingam-0.1-density.js</code> file in the <code>head</code> of your html.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>head</span><span class="token punctuation">></span></span><br /> <span class="token comment"><!-- load geojson data as "postcodeData" --></span><br /> <span class="token tag"><span class="token tag"><span class="token punctuation"><</span>script</span> <span class="token attr-name">src</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>au-postcodes-Visvalingam-0.1-density.js<span class="token punctuation">"</span></span><span class="token punctuation">></span></span><span class="token script"></span><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>script</span><span class="token punctuation">></span></span><br />...<br /><span class="token tag"><span class="token tag"><span class="token punctuation"></</span>head</span><span class="token punctuation">></span></span></code></pre>
<p>At the bottom of your page you will need to ensure the <code>postcodesData</code> is referenced in the declaration of the <code>geojson</code> variable. This is also when the leaflet functions defining style and hover events are pointed to the postcode shapes.</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// add all options to the polygons</span><br />geojson <span class="token operator">=</span> <span class="token constant">L</span><span class="token punctuation">.</span><span class="token function">geoJson</span><span class="token punctuation">(</span>postcodesData<span class="token punctuation">,</span> <span class="token punctuation">{</span><br /> <span class="token literal-property property">style</span><span class="token operator">:</span> style<span class="token punctuation">,</span><br /> <span class="token literal-property property">onEachFeature</span><span class="token operator">:</span> onEachFeature<br /><span class="token punctuation">}</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">addTo</span><span class="token punctuation">(</span>map<span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h3 id="2.-setting-initial-location-and-zoom-on-page-load">2. Setting Initial Location and Zoom on Page Load <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#2.-setting-initial-location-and-zoom-on-page-load">#</a></h3>
<p>To open on Australia instead of the US the key piece of code you need to set is the latitude and longitude coordinates along with the zoom level. This required a little bit of playing around to get the desired look. N.b I set the zoom level to 4 in the code below, don't try finely tuning with decimal places as tiles are only available for integers. I learnt the hard way!</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// set up the map</span><br /><span class="token keyword">var</span> map <span class="token operator">=</span> <span class="token constant">L</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token string">'map'</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">setView</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">27.833</span><span class="token punctuation">,</span> <span class="token number">133.583</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h3 id="3.-dynamically-fit-the-map-to-fill-the-device-screen-size">3. Dynamically Fit the Map to Fill the Device Screen Size <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#3.-dynamically-fit-the-map-to-fill-the-device-screen-size">#</a></h3>
<p>There is one issue with setting the zoom, since the zoom of the map is fixed and the device size is variable depending on the device being used. For example, if you set the zoom using your desktop screen, then loaded the page on a mobile device with a fraction of the screen real estate you will only see a fraction of Australia.</p>
<h4 id="a.-set-the-viewport">a. Set the Viewport <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#a.-set-the-viewport">#</a></h4>
<p><a href="http://www.w3schools.com/css/css_rwd_viewport.asp">Check out W3's intro to the viewport</a>. Included the following in the head of your html.</p>
<pre class="language-html"><code class="language-html"><span class="token tag"><span class="token tag"><span class="token punctuation"><</span>meta</span> <span class="token attr-name">name</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>viewport<span class="token punctuation">"</span></span> <span class="token attr-name">content</span><span class="token attr-value"><span class="token punctuation attr-equals">=</span><span class="token punctuation">"</span>width=device-width, height=device-height, initial-scale=1.0<span class="token punctuation">"</span></span><span class="token punctuation">/></span></span></code></pre>
<h4 id="b.-change-the-%23map-styling">b. Change the #map Styling <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#b.-change-the-%23map-styling">#</a></h4>
<p>The leaflet choropleth example has the following styling applied to the <code>div</code> map:</p>
<pre class="language-css"><code class="language-css"><span class="token selector">#map</span> <span class="token punctuation">{</span> <span class="token property">width</span><span class="token punctuation">:</span> 800px<span class="token punctuation">;</span> <span class="token property">height</span><span class="token punctuation">:</span> 500px<span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre>
<p>This is a fixed size, we want to change this to be relative. The following will fit to any screensize</p>
<pre class="language-css"><code class="language-css"><span class="token selector">#map</span> <span class="token punctuation">{</span> <span class="token property">position</span><span class="token punctuation">:</span> absolute<span class="token punctuation">;</span> <span class="token property">top</span><span class="token punctuation">:</span> 0<span class="token punctuation">;</span> <span class="token property">right</span><span class="token punctuation">:</span> 0<span class="token punctuation">;</span> <span class="token property">bottom</span><span class="token punctuation">:</span> 0<span class="token punctuation">;</span> <span class="token property">left</span><span class="token punctuation">:</span> 0<span class="token punctuation">;</span> <span class="token punctuation">}</span></code></pre>
<h4 id="c.-set-relative-zoom">c. Set Relative Zoom <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#c.-set-relative-zoom">#</a></h4>
<p>The steps before fix the filling of the screen. So now we have variable screen fill and fixed zoom. What we want is variable screen fill and variable zoom. That way, leaflet will automatically zoom to the maximum it can whilst keeping all of Australia visible on the device. Thankfully, leaflet provides a function for this, <code>fitbounds</code>. We need to change this:</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// set up the map</span><br /><span class="token keyword">var</span> map <span class="token operator">=</span> <span class="token constant">L</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token string">'map'</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">setView</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">27.833</span><span class="token punctuation">,</span> <span class="token number">133.583</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<p>If you imagine a box containing Australia, then draw a diagonal through that box, the latitude and longitude of the diagnoal line's endpoints are what the <code>fitBounds()</code> constructs as a container from. I picked a point at the south easter tip of Tasmania and the other to the north west of Western Australia. Here's the code:</p>
<pre class="language-js"><code class="language-js"><span class="token comment">// set up the map</span><br /><span class="token keyword">var</span> map <span class="token operator">=</span> <span class="token constant">L</span><span class="token punctuation">.</span><span class="token function">map</span><span class="token punctuation">(</span><span class="token string">'map'</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">setView</span><span class="token punctuation">(</span><span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">27.833</span><span class="token punctuation">,</span> <span class="token number">133.583</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">.</span><span class="token function">fitBounds</span><span class="token punctuation">(</span><br /><span class="token punctuation">[</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">43.21</span><span class="token punctuation">,</span> <span class="token number">147.8</span><span class="token punctuation">]</span><span class="token punctuation">,</span> <span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">12.5</span><span class="token punctuation">,</span> <span class="token number">117.2</span><span class="token punctuation">]</span> <span class="token punctuation">]</span><br /><span class="token punctuation">)</span><span class="token punctuation">;</span></code></pre>
<h3 id="other-useful-resources-encountered">Other Useful Resources Encountered <a class="direct-link" href="https://lukesingham.com/map-of-australia-using-osm-psma-and-shiny/#other-useful-resources-encountered">#</a></h3>
<ul>
<li>
<p><a href="http://gadm.org/country">Shapefiles for all countries</a></p>
</li>
<li>
<p><a href="https://github.com/Robinlovelace/Creating-maps-in-R/blob/master/vignettes/geoJSON.Rmd">Creating maps in R</a></p>
</li>
<li>
<p><a href="https://github.com/rstudio/leaflet/issues/55">Useful leaflet Github comments</a></p>
</li>
<li>
<p><a href="http://rpubs.com/jcheng/us-states-2">Joe Cheng, as always a pioneeR</a></p>
</li>
<li>
<p><a href="https://github.com/mbostock/topojson/wiki/Command-Line-Reference">Convert shape file formats</a></p>
</li>
</ul>
A Quick Look into Stamp Duties & Land Taxes2015-07-22T00:00:00Zhttps://lukesingham.com/a-look-into-land-taxes/<h3 id="stamp-duties">Stamp Duties <a class="direct-link" href="https://lukesingham.com/a-look-into-land-taxes/#stamp-duties">#</a></h3>
<p>Stamp duties are a transaction tax. Prior to the digital era, the transfer of legal ownership presented a convenient time for governments to levy a tax. Upon the transfer of ownership a stamp duty was paid and the relevant transfer of ownership document was either stamped...<br />
<img src="https://www.ozrevenues.com/Revenue-Railway-Local-Perfin-Catalogue/nsw-revs/202.jpg" alt="stamp-duty-stamped" /></p>
<p>Or a stamp was affixed...<br />
<img src="https://lukesingham.com/content/images/2017/08/stamp-duty-stamp-affixed.jpeg" alt="stamp-duty-stamp" /></p>
<p>Nowadays, most Australians recognise stamp duties as the tax paid upon the purchase of a property. For example, in NSW the rate of stamp duty ranges from 1.25% to 7%.</p>
<p><img src="https://lukesingham.com/content/images/2015/07/Screen-Shot-2015-07-17-at-10-59-47-am.png" alt="nsw-stamp-duty-rates" />Source: <a href="http://www.osr.nsw.gov.au/taxes/transfer-land/about">osr.nsw.gov.au</a></p>
<h3 id="land-value-tax">Land Value Tax <a class="direct-link" href="https://lukesingham.com/a-look-into-land-taxes/#land-value-tax">#</a></h3>
<p>A popular alternative, or at least with economists, is that of a Land Value Tax (LVT). Despite the idea having been argued for since <a href="https://en.wikipedia.org/wiki/Adam_Smith">Adam Smith</a>, the idea is most strongly associated with <a href="https://en.wikipedia.org/wiki/Henry_George">Henry George</a> who famously argued in his best selling book <a href="https://en.wikipedia.org/wiki/Progress_and_Poverty"><em>Progress and Poverty</em></a> for an LVT to replace all other inefficient taxes.</p>
<p>The crux of the argument is that since land is fixed (as opposed to say the supply of labour or level of consumption) disincentives or <em>distortionary</em> effects of land taxes are not applicable. On the other hand, a stamp duty causes a distortion to the property market by disincentivising transactions. Logically, stamp duties decrease the propensity of someone to move, thus reducing labour mobility. Those individuals who have to relocate often pay more in stamp duties, which unless there was some reason to discourage people from moving, is inequitable. The <a href="http://taxreview.treasury.gov.au/content/finalreport.aspx?doc=html/publications/papers/final_report_part_1/chapter_6.htm">Treasury's graphs</a> below show the shape of the effective tax rate of a stamp duty over years of occupancy. The panel B graph shows the impacts of several moves resulting in a much higher effective tax rate. A flat rate shows the effect of a land tax (flat rate).</p>
<p><img src="https://lukesingham.com/content/images/2015/07/Screen-Shot-2015-07-19-at-11-56-38-am.png" alt="effective-tax-rate-stamp-duty" /></p>
<p>By replacing stamp duties with an annually payable LVT land hoarding behaviour would be heavily discouraged. Governments would stand to benefit as stamp duty revenue fluctuates with the property market, whereas a land value tax would provide a steady annual revenue changing only when land values are revised.</p>
<p>The <a href="http://taxreview.treasury.gov.au/content/finalreport.aspx?doc=html/publications/papers/final_report_part_1/chapter_6.htm">Henry Review</a> makes several important points in considering a land tax:</p>
<ul>
<li>
<p>A land tax could encompass all property types such that the use of land is more likely to go that which is most productive, whether that be commercial, residential or industrial.</p>
</li>
<li>
<p>Tax-free thresholds could apply to exempt low value land. Particularly of benefit to agricultural land holders.</p>
</li>
<li>
<p>Different rates of land tax could be constructed in a progressive manner. Much like the increasing rates of income tax or stamp duty seen above.</p>
</li>
<li>
<p>Replacing stamp duty with land tax would remove the supply impediments caused by the tax system. However, the tax system itself is not the major source of supply constraints.</p>
</li>
<li>
<p>Currently land tax policy exempts owner-occupied housing, removing 60% of land by value from the potential revenue base.</p>
</li>
<li>
<p>To avoid double taxation of those who have already paid stamp duty, the tax could be introduced by only applying to new transfers.</p>
</li>
</ul>
<p><a href="https://grattan.edu.au/wp-content/uploads/2014/04/Game_Changers_Web.pdf">The Grattan Institute estimated in 2012</a> that replacing stamp duty with a land tax would increase GDP by $25 billion. Other <a href="http://www.ahuri.edu.au/downloads/publications/EvRevReports/AHURI_Final_Report_No182_The_spatial_and_distributional_impacts_of_the_Henry_Review_recommendations_on_stamp_duty_and_land_tax.pdf">research based on Melbourne house prices</a> suggested that the introduction of a land tax would decrease average house prices by 5% and would have a greater reduction on inner CDB house prices with up to 12% reductions.</p>
<p>One potential issue with the introduction of the policy is the impact it would have on the asset rich but income poor i.e. pensioners. However, as <a href="http://www.abc.net.au/news/2013-02-01/janda-stamping-out-inefficient-duties/4496356">Michael Janda points out:</a></p>
<blockquote>
<p>... it would be very simple for state governments to allow such people to defer their land tax payments, which would be indexed at a suitable rate and only fall due when the property next changed hands. Thus the land tax bill could be automatically deducted from the sale proceeds of the property when it was eventually sold.</p>
</blockquote>
<p>Another administrative consideration is the difficulty in valuing land. However as <a href="http://www.economist.com/blogs/economist-explains/2014/11/economist-explains-0">the Economist points out</a>, "<em>the efficiency of the tax does not depend on accurate valuations."</em></p>
<p><strong>The biggest issue is political...</strong></p>
<p>People don't like paying taxes. The issue with a land tax is that it is hard to avoid and an annually payable (or annually painful). It would be very easy to politically tar such a economically sound policy, as the <em>great big new tax grab</em>. This hasn't stopped up to <a href="http://www.elibrary.imf.org/view/IMF001/20531-9781484369050/20531-9781484369050/20531-9781484369050.xml">twenty countries having implemented or are considering</a> implementing the policy at some level.</p>
<p>In Australia, the ACT introduced a land tax three years ago to replace stamp duties. This year, the South Australian government has removed stamp duties on business transfers and <a href="http://www.afr.com/news/sa-budget-2015-stamp-duty-on-commercial-property-to-go-20150618-ghqt4d">will remove stamp duties on commercial property transfers in 2016</a>. However, they have backed down on their <a href="http://www.abc.net.au/news/2015-02-11/sa-government-releases-discussion-paper-on-tax-system/6085352">suggestion earlier this year</a> of replacing stamp duties with a land tax on residential properties for political reasons.</p>
<p>There are of course bodies such as the <a href="http://www.udia.com.au/">Urban Development Institute of Australia (UDIA)</a> and the <a href="http://www.propertyoz.com.au/">Property Council</a> who would stand to gain from a tax change that would increase property market activity. Active engagement of such bodies along with public education would be necessary for a government wanting to minimise political fallout.</p>
<p>For now it remains the economist's consensus and the politician's discord.</p>
Post War but not Post Confict2013-08-21T00:00:00Zhttps://lukesingham.com/post-war/<p>"<em>What happened in Sri Lanka was a major Rwanda-like atrocity, in a different scale, where the West didn't care. There was plenty of early warning. This [conflict] has been going on for years and decades. Plenty of things could have been done [to prevent it]. But there was not enough interest</em>." -N. Chomsky</p>
<p>A bloody civil war that gripped the island nation of Sri Lanka for over a quarter of a century came to a horrific end in 2009. The Liberation Tigers of Tamil Eelam (LTTE) were overwhelmingly defeated by the state led Sinhalese majority government forces. The atrocities committed by the state to reach such a conclusion have been vastly underreported. Estimates of civilian deaths in the final five months of the war range between 9,000 and 75,000 (<a href="http://www.bbc.co.uk/news/world-asia-21873551">BBC, 2013</a>). One United Nations report puts the number at 40,000; most recently they have reported that up to 70,000 civilians could have died in the final 5 months (<a href="http://www.un.org/News/dh/infocus/Sri_Lanka/POE_Report_Full.pdf">UN, 2011</a>, & <a href="http://www.un.org/News/dh/infocus/Sri_Lanka/The_Internal_Review_Panel_report_on_Sri_Lanka.pdf">UN, 2012</a>). If this is true, as Frances Harrison of the Guardian states, the magnitude of the estimated civilian deaths is on âthe scale of Syria but condensed in speed into 5-6 months whereas Syria has been going on for 18 months [to] two years.â Former Norwegian diplomat Erik Solheim said âThere were massive war crimes in the last phase which were probably the biggest bloodshed of the twenty first century.â</p>
<p>A UN panel of experts reported that the government of Sri Lanka has engaged in:</p>
<ul>
<li>Killing of civilians through widespread shelling</li>
<li>Shelling of hospitals and humanitarian objects</li>
<li>Denial of humanitarian assistance</li>
<li>Human rights violations suffered by victims and survivors of the conflict, including both Internally Displaced People (IDP) and suspected LTTE cadres</li>
<li>Human rights violations outside the conflict zone, including against the media and other critics of the Government</li>
</ul>
<p>An insidious and well-planned strategy by the government has meant very little independent reporting, foreign aid bodies being prevented from going into the warzone and to date, the resistance of any post-war independent investigation.</p>
<p>The host of CHOGM (Commonwealth Heads of Government Meeting) is the same government, the same people who are ultimately responsible. The question of accountability will not go away, particularly for the Tamil people, but also for those courageous journalists now in exile who dared to question the government.</p>
<p>Sadly, the end of the war has not meant the end of human rights abuses perpetrated by the Sri Lankan government. The once Tamil controlled northern areas are now under military occupation. Many displaced Tamils are unable to return home since the government have seized their property for âmilitary purposesâ or for âagricultural purposesâ. Ongoing disappearances, rape, sexual abuse, land grabbing and beatings are stories that are still coming out of Sri Lanka leading some critics to call the post-war, but not post-conflict scenario a âstructural genocideâ. The Sri Lankan government is unsurprisingly against transparency and judicial independence, as demonstrated by the unconstitutional impeachment and removal of the Chief Justice earlier this year, further showing the abuse of power wielded by the Executive.</p>
<p>For many progressives, the recent Labor asylum policy announcement borne out of political pragmatism rather than compassion and morality has come as an utter disappointment. However, it raises the challenge of changing the hearts and minds of the electorate. This will occur through learning about the plight that refugees flee. Each countryâs refugee has its unique and often complex story. This motion aims to shed light on the Sri Lankan Tamilsâ story while also advocating for diplomatic pressure to be placed on the Sri Lankan Government.</p>
<p><strong>What can we do about it?</strong></p>
<p>International pressure has been shown to be an effective tool in pressuring governments to do the right thing. The Sri Lankan government may not care about the Tamils but they do care about the Commonwealth. A boycott of CHOGM would make a good start. Apartheid is a good example of an international sporting boycott that raised awareness and pressured a nation towards corrective action. A boycott of Sri Lankan cricket, I believe, would mount enough pressure both internationally and domestically on the government to accept the calls for an independent investigation into the final stages of the war.</p>
<p>If you would like to find out there are few links below:</p>
<p><strong><a href="https://www.change.org/en-AU/petitions/prime-minister-reconsider-chogm-2013-in-sri-lanka">Boycott CHOGM Petition</a></strong></p>
<p><strong>Documentaries</strong></p>
<ol>
<li>
<p><a href="https://www.youtube.com/watch?v=r3yPzyM0KMU&bpctr=1582841718">Killing Fields of Sri Lanka</a></p>
</li>
<li>
<p><a href="https://www.youtube.com/watch?v=EWzlQeVKcUg">Killing Fields of Sri Lanka follow-up documentary</a></p>
</li>
</ol>
How to Make a Wordcloud Using R2013-06-19T00:00:00Zhttps://lukesingham.com/how-to-make-a-word-cloud-using-r/<p>Recently I have been using R for some basic data visualisations, outputs like word clouds and heat maps. I don't have a programming background so upon first look the R command line based environment can seem a little daunting. However, the ease at which I have been able to create some pretty amazing outputs with very little code has surprised me. In this blog I will attempt to share the steps in a simple process as well as the small amount of code that is needed.</p>
<h4 id="1.-rstudio-%2B-packages">1. RStudio + Packages <a class="direct-link" href="https://lukesingham.com/how-to-make-a-word-cloud-using-r/#1.-rstudio-%2B-packages">#</a></h4>
<p>First of all, you will need to install <a href="http://cran.rstudio.com/">RStudio</a>. The program gives the user a nice interface to operate within. The code can be typed in the window to the top left of the program, useful particularly if you want to save your code as a script. The code can be sent to the command line from there, or you can simply start typing the code into the Console.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Install required packages</span><br />install.packages<span class="token punctuation">(</span>c<span class="token punctuation">(</span><span class="token string">"tm"</span><span class="token punctuation">,</span> <span class="token string">"wordcloud"</span><span class="token punctuation">,</span><span class="token string">"SnowballC"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Load libraries</span><br />library<span class="token punctuation">(</span>tm<span class="token punctuation">)</span><br />library<span class="token punctuation">(</span>wordcloud<span class="token punctuation">)</span><br />library<span class="token punctuation">(</span>SnowballC<span class="token punctuation">)</span></code></pre>
<h4 id="2.-load-the-text">2. Load the Text <a class="direct-link" href="https://lukesingham.com/how-to-make-a-word-cloud-using-r/#2.-load-the-text">#</a></h4>
<p>This is the point where you load the text with which you would like to create your word cloud with. For this example I am using <a href="http://en.wikipedia.org/wiki/We_choose_to_go_to_the_Moon">JFK's 'We choose to go to the Moon'</a> speech.</p>
<p>Create a new folder e.g. <code>~/Desktop/test/</code> containing a <code>speech.txt</code> file.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Create a corpus variable</span><br />mooncloud <span class="token operator"><-</span> Corpus<span class="token punctuation">(</span>DirSource<span class="token punctuation">(</span><span class="token string">"~/Desktop/test/"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><br /><span class="token comment"># Make sure it has loaded properly - have a look!</span><br />inspect<span class="token punctuation">(</span>mooncloud<span class="token punctuation">)</span></code></pre>
<h4 id="3.-format-and-clean-the-text">3. Format and Clean the Text <a class="direct-link" href="https://lukesingham.com/how-to-make-a-word-cloud-using-r/#3.-format-and-clean-the-text">#</a></h4>
<p>These commands will remove various things like punctuations and english words you aren't particularly interested in for the cloud like conjunctions. Additionally it will format the case of the text, I am going to go with lowercase, however you can run various combinations of these arguments including arguments not listed here.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Strip unnecessary whitespace</span><br />mooncloud <span class="token operator"><-</span> tm_map<span class="token punctuation">(</span>mooncloud<span class="token punctuation">,</span> stripWhitespace<span class="token punctuation">)</span><br /><span class="token comment"># Convert to lowercase</span><br />mooncloud <span class="token operator"><-</span> tm_map<span class="token punctuation">(</span>mooncloud<span class="token punctuation">,</span> tolower<span class="token punctuation">)</span><br /><span class="token comment"># Remove conjunctions etc.</span><br />mooncloud <span class="token operator"><-</span> tm_map<span class="token punctuation">(</span>mooncloud<span class="token punctuation">,</span> removeWords<span class="token punctuation">,</span> stopwords<span class="token punctuation">(</span><span class="token string">"english"</span><span class="token punctuation">)</span><span class="token punctuation">)</span><br /><span class="token comment"># Remove suffixes to the common 'stem'</span><br />mooncloud <span class="token operator"><-</span> tm_map<span class="token punctuation">(</span>mooncloud<span class="token punctuation">,</span> stemDocument<span class="token punctuation">)</span><br /><span class="token comment"># Remove commas etc.</span><br />mooncloud <span class="token operator"><-</span> tm_map<span class="token punctuation">(</span>mooncloud<span class="token punctuation">,</span> removePunctuation<span class="token punctuation">)</span><br /><br /><span class="token comment">#(optional) arguments of 'tm' are converting the document to something other than text, to avoid, run this line</span><br />mooncloud <span class="token operator"><-</span> tm_map<span class="token punctuation">(</span>mooncloud<span class="token punctuation">,</span> PlainTextDocument<span class="token punctuation">)</span></code></pre>
<h4 id="4.-word-cloud-time!">4. Word Cloud Time! <a class="direct-link" href="https://lukesingham.com/how-to-make-a-word-cloud-using-r/#4.-word-cloud-time!">#</a></h4>
<p>Time to produce a word cloud, run the following command and watch RStudio populate the 'Plots' window to the right of the console.</p>
<pre class="language-r"><code class="language-r"><span class="token comment"># Time to generate a wordcloud!</span><br />wordcloud<span class="token punctuation">(</span>mooncloud<br /> <span class="token punctuation">,</span> scale<span class="token operator">=</span>c<span class="token punctuation">(</span><span class="token number">5</span><span class="token punctuation">,</span><span class="token number">0.5</span><span class="token punctuation">)</span> <span class="token comment"># Set min and max scale</span><br /> <span class="token punctuation">,</span> max.words<span class="token operator">=</span><span class="token number">100</span> <span class="token comment"># Set top n words</span><br /> <span class="token punctuation">,</span> random.order<span class="token operator">=</span><span class="token boolean">FALSE</span> <span class="token comment"># Words in decreasing freq</span><br /> <span class="token punctuation">,</span> rot.per<span class="token operator">=</span><span class="token number">0.35</span> <span class="token comment"># % of vertical words</span><br /> <span class="token punctuation">,</span> use.r.layout<span class="token operator">=</span><span class="token boolean">FALSE</span> <span class="token comment"># Use C++ collision detection</span><br /> <span class="token punctuation">,</span> colors<span class="token operator">=</span>brewer.pal<span class="token punctuation">(</span><span class="token number">8</span><span class="token punctuation">,</span> <span class="token string">"Dark2"</span><span class="token punctuation">)</span><span class="token punctuation">)</span></code></pre>
<h4 id="5.-the-result">5. The Result <a class="direct-link" href="https://lukesingham.com/how-to-make-a-word-cloud-using-r/#5.-the-result">#</a></h4>
<p><img src="https://lukesingham.com/content/images/2015/04/mooncloud.png" alt="moon-speech-wordcloud" /><br />
VoilĂ ! Where there is a 'will' there is a way. It's hard to imagine a current leader of a western country announcing such a bold and expensive policy now - despite the world increasing its wealth substantially in the last half century of rapid development and therefore being even more capable than it was in the 60s.</p>
<p>Back to technical aspects, for further information and optional arguments you can use to customise the word cloud please see the following:</p>
<ul>
<li><a href="http://cran.r-project.org/web/packages/tm/vignettes/tm.pdf">tm</a> - the text mining package</li>
<li><a href="http://cran.r-project.org/web/packages/wordcloud/wordcloud.pdf">wordcloud</a> - the cloud generator, also does commonality clouds e.g. compare two political speeches for common themes</li>
<li><a href="http://cran.r-project.org/web/packages/SnowballC/SnowballC.pdf">SnowballC</a> - multi-language stemming algorithm package</li>
</ul>
<p>I learnt to do this from a fair amount of Googling. The most helpful blog I came across was <a href="https://georeferenced.wordpress.com/2013/01/15/rwordcloud/">Georeferenced</a> - so credit where credit is due. I learnt this method due to <a href="http://www.wordle.net/">Wordle</a> being blocked from my workplace, so of course use the website if you want to keep your hands clean!</p>