The Epic Journey to the new Image Science Website

I wrote this history largely for me - I've been working on this project for 3 years alongside the day to day running of Image Science - the business I run with my wonderful wife Amy - and it's been this great big thing in the background of my life for a long time.  It's certainly the biggest project I've ever worked on - in a largely solo way - and so this article stands as a document of the paths travelled and why it worked out the way it did.

But along the way many people have been involved or had an interest in the project and almost everything I talk about here is something I've been asked about by someone along the way.  So I thought I would write it all down so I'd have something to point those folks to if they asked again.

Apologies it's so long!  Brevity, counter-intuitively - takes time, and I've spent more than enough on this already!

Before 2009 all versions of the Image Science website that I'd put together were quite simple - first just directly hand coded html pages, then DreamWeaver templates with Mal's eCommerce for a simple shop system bolted on. But of course we outgrew this sort of approach, and so in 2008 we researched extensively, and then spent about 6 months developing a new website.  In early 2009 we launched this new website based on what we thought would be a solid platform for the future.

After much research I'd settled on an Australian developed shop and knowledgebase platform from a company then called Interspire.  As part of this research I had extensive back and forth discussions with them, commissioned some custom development from there as test runs for the future to get a sense of their working model, and was many times promised a solid base on which to work - and all sorts of exciting plans for the future.  A key part of my choice was that it was a self hosted system, with the opportunity for customisation for our specific and unique business needs.

However within 6 months of the launch, the company (then Interspire, now Big Commerce) - had suddenly and without warning (even their own support staff seemed unaware of the forthcoming drastic change) - abandoned the products I'd built with, and shifted to an online only, no customisation possible platform. And moved all operations to Austin Texas (where their cowboy style of doing business is probably right at home!).

They burned a lot of their early customers - providing no information and no path forward for customers they'd made promises to only months earlier - and ran away to Texas without a look back.   (Since then, Big Commerce have become darlings of the venture capital world - and I'm always suspicious of technology companies built on that sort of speculation).

Reading those early emails back now almost makes me laugh at my own naivety - I bought on promises, and I suppose with a distinct lack of caveat emptor.  And of course many companies move to hosted 'Software As A Service' models around this time - I'm sure it made excellent sense for them, but it certainly left us in the lurch and they were telling me - right up until their sudden announcement - the opposite.

Despite these troubles, the product I'd chosen, and with some significant custom development I then added on top, suited our needs very well and the website was actually pretty great....for 2009.  We got endless compliments on it for years.  (Indeed, we were still getting compliments on it right up until the launch of our current site, in 2016!).  

But increasingly, of course, it was showing it's age and in particular the services ordering side of the site felt very bolted on and was not intuitive.  It was also entirely unsupported code, and eventually some breaking change with an external system like shipping quotes or payment gateways would come along and then we'd be stuck - so I obviously had to make a move to something more modern.

So from around 2012, I started to seriously research alternatives again.  And of course the world of the web was changing rapidly around me.

Image Science - The Business

We have a lot of content - our old site had around 1200 solid pages, including many products, a whole lot of service information, 300 articles, many long form, and an Ebook of about 80 A4 pages worth of text.

Our customers are ... potentially almost everyone.  That is to say, we work with both complete beginners who are just starting out with their new camera or have just begun illustrating - right through to household names - some of Australia's finest artists and photographers.  

So we need something that manages to be accessible and not overwhelming, AND has the meaty detail that more developed artists are looking for (& that we're famous for providing).  I refer to this as an iceberg website - where you have 1/10ths of the information on show for the people who want to get 'the skinny' very quickly, but in a very discoverable way still have the 9/10ths of detail available under the surface.

Other Systems

Of course our website doesn't operate in a vacuum. 

Increasingly, we need to integrate successfully and seamlessly with other web-based systems - like handling many large file uploads/downloads, send things to our accounting software, and helpdesk/ticket software for dealing with the many incoming enquiries we process - as a high service business it's critical we make this as easy as possible for our customers when they come looking for support or advice.

Furthermore, we wanted to choose a sensible platform to build our future on - a system where we could easily and quickly make extensions & behaviour changes, using standardised APIs and modern tools.

Content & Commerce

The classic problem for anyone dealing with both serious content and serious e-commerce concerns is this: 

There are some great content systems available, but they typically have pretty terrible shopping cart systems that are bolted on the side.  

Alternatively you have some very sophisticated e-commerce platforms that have poor content management systems added as an after-thought.  For years I have search for the best solution to this problem.

Shopping Carts

I identified early on that I wanted a system that was great at both managing content and handling sophisticated e-commerce.  

This meant evaluating systems in quite some detail - not just looking at feature lists - as no system will ever have, 'out-of-the-box', everything I want/need.  Rather it's looking at both what the system currently offers, and how extensible a particular system would be to achieve what I wanted, and for the longer term.  This means a very time consuming rinse and repeat process: initial research, then if they show promise, installing them, adding substantial and varied test data, and running through various exercises from making orders through to getting right in to the code and attempting the process of making extensions or modifying behaviour.

Obviously many systems I could dismiss for a variety of reasons quite quickly - for example WordPress (now amazingly some 25% of the web runs on this!) - is a well known security nightmare, and pig's breakfast of a system to work with from a code point of view - and we'd already had some substantial experience of that (the blog section of our old site was an embedded WordPress system).

I tried many cart packages (Prestashop, OpenCart, etc etc) extensively but none of them have good enough content management - and in fact most of them are actually (and surprisingly) pretty feeble at the more complex commerce stuff (like for example they mostly do have variant product systems - which is essential when you're selling multiple versions of essentially the same products - but those variants then never support custom sizes and weights per variant - obviously essential if it costs $10 to ship one variant and $100 to ship another...).  Typically they're all built with, and heavily promoted for, flashy features, but when you actually come to invest time in using those features in practice, they simply don't work properly for business needs beyond the basic case.  Much like Dell monitors, it's endless ticking of feature boxes, not well considered and tested solutions to problems.

Magento is very flexible in theory and often pointed to for complex e-commerce sites, but has a notoriously heavy template system and requires a very large server to run effectively - AND you have no support without a minimum cost of entry of ~ US$15000 (I think that's even per year!) - other than community support.  For the complexity of our build, we saw this as a big problem.  No man is an island, and the company that backs the product is probably as important as the product itself (this is where we went wrong with Interspire - a good product for it's time, for sure, but with a company behind it that was simply and demonstrably willing to behave unethically at the time).

While various hosted cart services have appeal & tend to be very pretty (Shopify etc.), none of them are close to flexible enough for our needs, and lack decent content management systems beyond general support for basic 'static pages' - so we dismissed most of those straight away.  And of course I even evaluated the current offering from Interspire, Big Commerce.  Nice for what it is, but again positively feeble at content management, and I don't think anything could bring me to hitch my wagon to that company again!

Completely Custom? Carts as Infrastructure?

One thing I didn't want to do was go completely custom - i.e. build the entire thing from the ground up, in custom code.  I have the necessary skills I think, but it's simply too much work to do!  E-commerce, with all the requirements with tax, shipping, checkout & gateway requirements etc., quickly becomes complex and developing all that on your own is a huge time burden, let alone testing it properly etc.  That's without even thinking about security, performance, UI/UX design etc - it's a *massive* task, realistically beyond one person and certainly beyond one person simultaneously running a full-time, busy business.

Good content management is a problem that entire businesses with very intelligent people have taken years to solve - indeed I'd argue it's really only been the last few years that really excellent systems have come along.  

To build both CMS & e-commerce in full, in house, would take resources we simply don't have and put us forever more in the position of always needing to do things in house as custom development.  We would not be standing on the shoulders of giants and would substantially be re-inventing the wheel.

Carts as infrastructure, or carts attached to frameworks like Laravel, certainly have some appeal, but many are still quite under-developed and lack real traction at this point, so these felt risky from a future platform point of view.  And you'd still be left with a LOT more work to do with these.

No - for us we wanted a pre-existing system with some support, reasonable cost, but ideally with a fair bit of already built, excellent functionality for both content and e-commerce.  And this was actually really difficult to find in 2012.

In 2012, after much testing and research, we chose ExpressionEngine by Ellislab, and CartThrob as the the systems on which to build.  

ExpressionEngine (AKA EE) - was (and presumably is!) a major and proven player in the content management world with some very big names clients like Sony, Warner Bros and Adobe.  A current example would be Donald Trump's 2016 campaign site which is clearly handling massive traffic.  There was an active and high quality community of developers, and support forums and paid support options.  At that time, EE was without doubt a highly regarded product.

CartThrob, by a company called Mighty Big Robot, was the premier cart system for ExpressionEngine and was specifically designed to take advantage of all the content management features of EE.  It was super flexible and there were a number of complex stores with a lot of content out there (Warner Bros. Australia amongst them), that led us to believe this would be up for the job (after an initial install and some weeks of testing).  We  did have some concerns about support here - they had forums etc, and commercial support - but at that time there were some examples of compaints of slow responses etc.  However - it really did seem this was the only tool with the right level of flexibility for us.

Why Build It Myself?

Why not simply pay someone else to solve this problem for us, and leverage their expertise?

This came down to a few factors.

I started by listing out a bunch of requirements and essentially scoping the project.  I then talked in depth with a few companies to get a sense of what this would cost us, and I spoke to some similar business about their experiences and costs in making their sites (although finding anything really comparable was (and is) difficult to be honest).

1. Cost

I soon established a budget of around $15000 is needed for even a basic ecommerce site and we wanted something WAY WAY beyond basic.

I identified a few business with experience in ExpressionEngine and in some cases had long meetings with these folks, and discussed pricing.  The minimum quotes were around $45000 and some were talking more like $100000 for everything we wanted.  And the more they understood the scope of what we were doing, the more the cost increased of course.   

(These quotes would prove to be very reasonable once we did the work ourselves - indeed they were probably substantial under-estimates of the real work involved in the end!).

But an inexact sort of estimate of north of $50000 seemed to be the best quote we would really work with - and that's a BIG cost for a small business.  While not impossible to contemplate - the website is our major business asset, and we expect to get at least 5 years out of this new one - so the per year cost is not so crazy....and ideally we'll be using this platform (with updates and improvements as we go of course) - for many more than 5 years.

2. Control

In building a platform for the future, a primary goal was understanding the platform so I could make further substantial changes myself - I didn't want to have to talk to an agency every time I wanted to make changes - even fairly fundamental ones.  

I very much wanted a nimble platform that would allow us/me be more experimental with how we present (and sell) our products and services.  

I wanted to avoid too much 'technology debt' - that is using a mish mash of add-ons from a large variety of developers and then being beholden to them for updates and fixes as the platform evolves.  Some technology debt is almost unavoidable of course - it's finding the right balance between re-inventing the wheel and using, solid, support components when it's appropriate.  

3. Quality

If you want something done right, do it yourself.  

That's not always true - but if you have the skills, it's unlikely anyone else will ever pay the same attention to detail and be willing to work and re-work things until they are 'just right' in the same way as someone with a strong vested interest in the result.  And no one is more 'vested' than me!

And of course no one can understand our business better than we do - we work in it all day everyday and we know exactly how our customers approach us and what they need from us - and indeed we've surveyed them and discussed precisely this with many, many of them over the years.  

4. Shifting Sands

The reality is that in developing something this big, with this massive scope - there never really is a firm target!  Thoughts change and the scope evolves as you make it.  Often dramatically.

It's all very well wireframing out a site as you see it, but when you come to actually use it once implemented - on phone, tablet and desktop - it often simply looks and feels at odds with your expectations and the initial designs.  Furthermore, you think of new, better ways of achieving things, so you go back to the drawing board.

The reality is, it's often only the second or third time you build something this big that you begin to get it really right.

5. Challenge

This was a more personal thing - I felt like I needed a real challenge again after several years working in and on the establishment of Image Science.  A really big task to sink my teeth in to.  I love what I do, but I can't spend all my life helping people choose a monitor or calibrate their printers...that's too easy.  I needed something much harder to help me stay motivated.

Slow Progress

2012 was not a great year for my family.  Without going in to too much detail, most of 2012 was spent dealing with some health issues that had a dramatic impact on our lives for some time....so after making our choice of platform, everything stalled.  As our old website was still ticking over, it was an obvious side task we could abandon to make our lives simpler, and so I simply ignored things for about a year whilst we got properly back on our feet.

In mid 2013 I was able to again find time and motivation to work on things, and in mid 2013 I finally bought licenses for EE & Cartthrob and begin serious work in earnest - although always juggling running the actual business along the way of course - meaning I very often had only a few hours in a week to work.

I made, at this point, a couple of mistakes.  

My first mistake was to simply run with what we'd chosen without taking stock of the choice - I wanted to get in to the problem and start solving it, not spend more time on research and testing.  It's easy to see this mistake with hindsight, but also hard to see whether there really was a better choice to make at that time.  

My second mistake was that I began to speak about the project - even publicly.  This was way too premature, but ... well, I was excited to be getting in to it.  Oops.

In any case, I began.  I did all the things developers do at the start of big, new projects - I set up environments in which to work, configured all my tools just the way I like them, and experimented with different major structural approaches until something felt 'right'.  Basically I procrastinated while I circled around the edges of how I would start to tackle this major project.  

And then, I really began.  As months passed, a few things became clear:

• I'd underestimated, and fairly spectacularly, how much work was involved and/or how much available time I would have to do it
• I'd over-estimated how much EE templating would be like real programming (in which I am quite experienced) - it isn't.  It's got a LOT of fundamental issues with parsing (I mean seriously, a regular expression based parser??!) - and data availability that make complex things vastly harder than they need to be.  Coming from a classic programming background, this was a rude shock!
• EE needed an absurd amount of add-ons to achieve (what should be) simple things
• EE was rapidly losing favour with developers due to a perceived lack of progress with the system
• Modern frameworks had appeared in the web world with significant advantages that EE could not have without major re-engineering
• Mighty Big Robot had all but disappeared.  However there were occasional re-assurances of things coming and getting things back on track with support etc.
• There was a new kid on the block getting a lot of good press in the content world - Craft CMS.  

So at some point around mid 2014, I was working in earnest on the system and finally making good progress - and spending noticeably more time on it than I had predicted!  

But I was not actually wonderfully happy with the tools I'd chosen.  However there comes a point where (to coin a poker term) you're 'pot committed' and backing out is more expensive and risky than staying in.  And while this perhaps reads like things were awful, I was in fact building a nice, flexible, and very complex system with it and making quite good progress.

Whilst Cartthrob was a bit of a pig to work with and, much like EE, made some things that should be simple very difficult, it was proving to be very very flexible and we were well on our way to a finished system.  (I do think writing a (very complex) custom shipping calculator for CT was the most diabolical and unhappy piece of programming I've ever had to do, though!).  

There was no question that a system that was very very good at both content and e-commerce was indeed coming along. But - given my time investment particularly - it was disturbing to sense the developer community around EE and particularly CT cooling noticeably on the products.  I kept looking on the bright side and telling myself I'd made the right choice, and that EE3 and CT3 would come along in due course, and help restore everyone's faith in the systems.  Titbits of news seemed to suggest good things were coming  although looking back now a lot of that was perhaps based on my wishful reading of things more than actual fact.

As I worked through 2014 and in to 2015, I was having major 'the grass is greener on the other side' dissonance with my decision as more and more of the best folks from the EE community jumped ship to Craft and appeared to be loving it.  But - and the major problem - there was simply no serious e-commerce solution for Craft. This meant, of course, it simply wasn't an option - or we'd have had to bolt on a full Cart as Infrastructure system to it (which I seriously investigated but did not run with for the reasons mentioned above).  

I even installed Craft more than once, played with it, and it seemed nice - the UI was obviously light years ahead of EE (responsive, fast, clean!) - but to be honest it wasn't immediately obvious from these quick tests that it offered major benefits in content modelling etc.  (Or perhaps I just didn't want to see it?).

In any case, progress with EE and CT was steady but slow.  We had to wait more time than was ideal for some work from our designer along the way (the only bit we weren't doing in house was the graphic design side), and various major service jobs took up a lot of time, and of course just the general day to day stuff of running a business, and things like staff changeovers, took up a lot of time as well.  But slowly, steadily, a lot of work was being done, and I was getting close to the finish line.

And again, I made the mistake of telling people we were close to finishing.  

Pride.

And then, the Fall.

So, around April 2015 we were about 85% finished with our EE site - indeed it was largely just some content entry and visuals that needed to be done, the underlying systems were all very much in place after an epic amount of work.  We planned to finish and finally launch within a couple of months.  

Then, two things happened at roughly the same time (co-incidence?  I'm not sure...)

1. MightyBigRobot finally publicly admitted they were stuffed, and shut up shop - CartThrob was abandoned and we hadn't even launched!  Waking up to this news - which I had in fact now expected for some time - despite them only a month or so before announcing progress with CartThrob3 - was quite simply awful.  All this work, started in fact because we had previously built on an abandoned platform....only to have the new platform go defunct before launch....  I was not happy, to put it mildly.  But really the writing had been on the wall for some time (and I might well have shifted earlier had I seen a realistic alternative).

On this - the closing of MightyBigRobot was in fact years after they should have done this, and I think an appalling example of a way to run a company.  I have no idea of the underlying whys of it, and I suspect they had the best intentions early on, but the visible behaviour of the company from around 2013 on was just grossly irresponsible and they let a lot of folks dangle in the wind.  

To say they could have handled it better is an under-statement of grand proportion.  Again, I have not walked in their shoes and I am sure there were underlying causes and what not - but the fact remains if you publicly offer a product you have, I think, a certain responsibility to your customers to do the basics of keeping them honestly informed, if nothing else.  

No one could argue that going from tweeting:

    We’re still here and working furiously on CT3, some highlights of CT3: EE Native Grid support, Inclusive tax Fieldtype, Tax Classes. #eecms

...to (and this tweet seems to have been removed so I am paraphrasing!) -
    
    'We're closed for business, don't let the door hit you on the way out'

.... just a month later....- well that's nothing less than an insult to the people who bought, worked with, and invested in their product.

2. Pixel and Tonic, the makers of CraftCMS, announced in April 2015 they had acquired a fledgling e-commerce addon for their system, then called Market.  Market was being developed by an Australian based in Perth, Luke Holder.  They were renaming it to Commerce and the list of features announced seemed solid and enticing.

This was HUGE news - at last, there seemed to be a best-in-class content management system with a forthcoming IN HOUSE flexible commerce system - the holy grail! Better yet - the primary developer was local so I could talk directly to him and get a proper sense of things.  BUT - it wasn't out yet, with a 'beta later this year' being all we really knew.  It certainly sounded like what we were after - powerful, flexible, and completely integrated with the content management side of things. But it was new, completely untested and really impossible to say whether it would really be capable of handling our needs.

So there I was - depressed at the demise of CT, excited at the arrival, finally, of the dreamed-for integrated content and e-commerce system I had literally been thinking about for years.

And so, of course, I wanted this shiny new toy.  Oh the lights, how they dazzled.  

But - could I actually switch?  Could I really abandon what was effectively two years of (on/off) work at this point - work that was nearly finished?  Was I really contemplating doing this?  And would my wife Amy divorce and/or kill me if I did?

Not A Slam Dunk

Ok, so we had major issues - a now defunct commerce platform (Cartthrob), based on an ageing very-good-but-also-frustrating content management system (ExpressionEngine)...but despite these issues  - we DID have an almost finished site, and an impressive and sophisticated one at that - perhaps only weeks away from launch.  

But launch on a dead platform?  Argh!

By this time, it had become quite clear that Craft had won the 'content wars' - as it has matured, people just continue to love it more (something I would say is fairly unusual in my 20 odd years of IT experience).  Sure, there are some other good systems that have appeared as well, but Craft just seems to get things right more often than any of them.

Craft, of course, actually grew from EE - Pixel and Tonic made the best add-ons for EE and, themselves frustrated with the underlying platform - they 'simply' went out and built a (much) better one that addresses all of EEs many frustrations. But at it's heart there's a very similar, very effective approach to content modelling shared by the two systems.

Much has been written about this but it's the quality of the people behind Craft - even more than the product itself - that makes it so appealing.  Their approach is one of quality through and through - and they're open and communicative about things.  They say no just as often as they say yes - very important in a software business where the temptation is to continually add endless features and integrations for requested edge cases - which then pollute your code base and become maintenance costs for forever more.

So I was completely confident Craft itself was up to the job and backed by a great company....but Commerce was as yet an unknown, a risk.

So I contemplated it - with much fretting and back and forthing with Amy - to even contemplate such a late switch seemed madness - but I felt I had to at least ask the questions and find out more.

So I contacted Luke over in Perth and we set up a Skype. That first chat was confusing, but encouraging.  Honestly I didn't know enough about Craft to ask a lot of sensible questions at that point, but what I did know very well were our commerce needs, and I quizzed poor Luke quite mercilessly on those.  I liked his honest approach - he was happy to tell me what things were done, or planned, and (just as importantly) what things were definitely not planned.  He seemed a good fit for P & T and very quickly I could see why they'd hired him and acquired Commerce.

After this I spent a few sleepless nights actually contemplating switching - and my ever wonderful and long suffering wife Amy was completely supportive, despite the fact this overwhelming website task had (has) been such a time-draining part of our lives, and for years now.  She knew how unhappy the dead platform issue was making me, and she was more quickly convinced than me that drawing a line under EE and CT and jumping ship was the way to go.

And so, I jumped.

So, May of 2015 I jumped to Craft and Commerce.  Still not yet 100% sure, though, I began with just getting to know the system and setting up for developing on it.  After two years in the EE world and with EE thinking in my head, it took some getting used to Craft's really clean, modern approach.  I devoured all the resources on it I could find, looking for best practices and impressed with the quality of the community and the work it puts out.

By June last year it seems I bought a Craft license - clearly I had decided in that short time it was the system for us.  And a year and a significant amount of use later, I'm more sure about that than ever.  Craft is one of those rare things - especially in computing - that simply gets better and more enjoyable the more time you spend with it.

So often you'll say to yourself 'wouldn't it be great if I could do this' - and you'll write it, and it will just work.  It's structured and methodical and logical, all the way through.  It's built on excellent, solid foundations like Yii and Twig, and adheres to standards where possible.  The support systems behind it are easy to access, quick, and generous.  Whether it's P & T staff answering questions promptly and with bang on responses (and they all seem to!), or whether it's the excellent and amazingly generous Craft community on Slack and Twitter etc. - you never feel alone, stuck or unsupported.

It can do complex things, and makes those complex things more simple than any other CMS I've seen.  It's got a great experience for all who use it - both developers and content editors find it a joy to work with.

It is perfect?  No.  Documentation could be better, and like any big complex thing, there's some things better than others (Craft's globals aren't a patch on EE's Low variables for instance) - and a few oddities that seem out of place (user photos aren't assets?!).   But it's as close to perfect as I've found (and I really have looked far and wide).  Craft 3 - to be released later this year most likely - will undoubtedly have all sorts of new coolness.  I don't think those P & T folks can help themselves - they do clever cool stuff as a matter of course, it's just who they are.

I really, really like Craft.

Since that first conversation with Luke, I've barely left the poor fellow alone.  We've had longs chats about many things.  For a while three of us, all co-incidentally Australian, had a private beta Slack room before the public beta expanded the reach.  Ah, glory days.  And I think, to be honest, I've even had a (very) small part in helping Commerce become the very solid product it already is - and it's just a baby!  

Only just past V1, it's already a remarkably powerful e-commerce system - it certainly filled all our requirements admirably with the best handling of content and variations I've seen - by a long way.  Much like it's parent system it's got great, clear thinking behind it and it's generally a joy to work with.

Is it perfect?  No.  Documentation could be better (there's a theme here huh?) - and so far it's got a few notable shortcomings (e.g. addresses aren't elements, and promotional stuff like sales/discounts could do with more work) - and the back end order processing stuff is still towards the basic end.  

But Commerce absolutely shines at the content side of e-commerce - and all the basics of cart handling and checkout are in, solid and working well.  It's already a *much* better system than Cartthrob (which, despite its flaws, is a very clever system indeed so that's by no means faint praise - and I say this having used them both as deeply as anyone alive right now most likely - I have very thoroughly kicked the tires of both).

More Delays...

Craft Commerce was not even out yet - by 'this year' was about as precise as it got.  So we had to wait for it to be more fully developed - in the end it came out in about November 2015 If I recall correctly.

So, by the end of 2015 we had mostly caught up to where we had been with EE, but based now on Craft/Commerce.  It's testament to Craft and Commerce that we could re-develop so much, so quickly - and with much simpler, neater, DRY code (DRY = Don't Repeat Yourself - a coding principle I'm a little addicted to!).

Around this time our designer - an absolutely lovely guy we've worked with for two years at that point - had his own serious health issues.  We decided to wait and see how his treatment would go to see if we could carry on working with him, so there was a few months delay at the beginning of 2016 while we waited on the outcome of that (but of course we were doing lots of content entry and polishing work while waiting!).

It became clear we unfortunately needed to find someone new, so around April we asked another developer to help us finish things off.  But of course in this gap we also had some scope creep and thus added some new things to the site, so we ended up needing another 4 months to get to the finish line!

With much help along the way of course - I built....this.  This monster of a website.  And learned an enormous amount along the way, of course. And effectively, I built this twice.  Of course the second build, in Craft, benefited greatly from the first build in EE, and the lessons learned.

But - by golly - did I underestimate the work.  Spectacularly and utterly, I got it wrong on that side of things.  Websites are *hard* and they're a LOT of work.  I imagine it's much like building a house yourself, or writing a novel - at some point you're half way along and you can't face the prospect of finishing because you now really know just how much you have left to do.  But of course you can't go back, either, or it will have been for nothing.  And with big tasks like these - for years nobody even really even sees what you're doing, so to the public eye it probably seems like you're not actually doing anything!

Motivation is a constant struggle - because although the task might be interesting and enjoyable - it's still work, and there's still lots of pressure attached - self imposed or not.  So it's really quite hard to keep your spirits up along the way as the task list in front of you quite literally runs to in to the thousands of little tasks in front of you.  At times I'd simply be unable to  motivate myself to carry on with the site - as it was over and above all my normal work, sometimes I just couldn't bring myself to get over a particular hurdle, or take on yet another mammoth part of the task.

List of Things Built

There's a website here.  

It's got a lot of content.  

There's around 900 products. Although many are now done as variants WITH their individual sizes and weights.

There's around 300 articles.

Maybe a 100 blog posts.

There's an embedded 85 A4 page eBook (that was a solid months work on it's own - editing & re-writing it, and creating a PanDoc based system to use one source text to generate the website, PDF, ePub and Mobi versions).

There's over 2000 lines of code in the shipping calculator alone - with grouping, boxing, live rates and table based rates, as well as support for free shipping, drop shipping, pick ups, etc etc.  

Tens of thousands of other lines of custom Twig and PHP code.

There's an integrated helpdesk/ticketing system (essentially embedded Fresdesk).

There's connections to Mailchimp and Mailgun for email & marketing stuff.

There's connections to eParcel for consignees, TNT for live rates.

There's a whole bunch of stuff I have forgotten to list simply because it's all too big to keep in my head at one time!

There's an agile deployment system built on Ansible that can take a bare VPS and be a full production ready image science server in less than an hour.

Tools

These are the major tools I ended up using:

• Craft
• Commerce
• Addons - Imager, SEOmatic, shortlist, SuperTable, LJ Dynamic Fields etc.  A few too many really, I'm hoping to reduce this in time.
• PHP7
• nodejs/gulp
• less
• pandoc

DevOps

(For the server nerds...)

• Custom Ansible deployment system to:
• ServerPilot managed Ubuntu 16.04 VPS - on Vultr
• Cloudflare to handle DNS & various performance aspects
• S3 for image and file serving
• HTML5 Boilerplate
• Also LiquidFiles for a dedicated file handling server (on a second Vultr VPS).

It's mostly done - or - it's done to a point.  It certainly does vastly more than our old site, and does it vastly better.  

But nothing this big is ever really finished.

Some parts are more successful than others, of course, and the idea here - now that all this previously hidden work is finally out in the open and in real world use - is to see how it works for the next few months and work out which bits really need a proper 'Version 2' go over.  Without doubt the monitor and paper 'selectors' are just basic things at this point - I have plans for big improvements there.  And with search, too.

I certainly feel, now, finally like I have summited the mountain.  Sure, it's a false summit and the real peak is always just around the corner - but all in all I am quite happy with it, the tools used to make it, and I'm confident it will serve our business very well.

What's Next?

Ha - I've learned my lesson!  I'm keeping that to myself while I work on it!!