Nick Craig-Wood's home page

Xmas 2025

nick@craig-wood.com (Nick Craig-Wood) — Thu, 25 Dec 2025 00:00:00 +0000

2025 has been a year of courses, A-levels and a building project.

Highlights include: a cookery course for Loveday and Nick at River Cottage; skiing in Kitzbühel; hosting Britney from Nanjing; a (baffling to some) visit to the Minecraft experience for Dougal’s 18th birthday; a party to celebrate the end of school; a fantastic family holiday in Sardinia with Loveday’s siblings scattered around Porto Rafael in seaside villas; a holiday with uni friends in Cornwall; the start of the renovations at The Mill and a crochet holiday in Morocco for Loveday.

Amy is doing well at her work at The Land App when she can spare time between holidays and van adventures with her boyfriend Sam and Oggy (her dog). She bravely drove her van to Sardinia stopping off for climbing in Fontainebleau with Doug.

Ed is still working with Nick. He and Phoebe (who is making an appearance in the card for the first time) have just celebrated 5 years together. They moved out of Guildford in the summer into their own flat in London, even taking most of their stuff with them.

Issy finished her degree – hooray! As Ed moved out of the flat above the garage, Issy moved in. She is now taking some time off to think about what she is doing next.

Dougal did a lot of studying sitting in the corner in Nick’s office. He got an excellent set of A-levels and has applied to study Computer Science at university next year. He had a great trip to the US with his cousin Olly and is now also working for Nick writing software. He still finds plenty of time to shout at his computer while gaming with his friends.

Loveday and Nick have been enjoying swimming together a couple of times a week and having more time to walk the coast path in Cornwall. As we did last year, we travelled to Sardinia on the train enjoying time in Paris and Lake Como which was very special. Loveday vetoed all the pictures of her looking like a beetroot after we climbed to Brunate so you’ll just have to imagine it.

Loveday has been extremely busy this year with Quakers, various courses (gardening, Alpha, agroforestry), and the start of work at The Mill. Nick has given up trying to remember where she is.

Nick has had a good year having a lot of fun working with the boys. The business has exceeded expectations. He has been having fun with recreational maths (arXiv 2508.08307) discovering an great new formula for Pi!

In October Loveday demolished The Mill. It was nice knowing it. What remains of it is floating in a sea of mud with a few builders mixed in. Or that is what Nick understands since he refuses to come and visit due to the fear of getting stuck in the mire. Hopefully it will get put back together some time next year.

Unicode Block-Style Morse Encoder

nick@craig-wood.com (Nick Craig-Wood) — Tue, 12 Aug 2025 00:00:00 +0000

Text

How: Type text and press Enter or Convert. Each conversion appends to the output below. Unsupported characters are skipped.

Output uses special block characters (U+1FB03, U+1FB0B) and spaces (U+2001) so it stays properly spaced in both mono and proportional fonts.

Output

Xmas 2024

nick@craig-wood.com (Nick Craig-Wood) — Wed, 25 Dec 2024 00:00:00 +0000

We've enjoyed time at home in 2024 with plenty of lunches with the boys as Ed is working with Nick and Doug usually drops in for his lunch hour. Highlights of the year have included: a ski trip to Andorra with Loveday's brother and Doug's cousins; quality time for Loveday getting muddy at The Mill and planting 300 trees; hosting two Chinese students from Nanjing, Albert and Alice; an Easter trip to Cornwall with friends and family; celebrating our 30th wedding anniversary with a stay at The Pig on the Beach in perfect weather; seeing Loveday Hambly's (Loveday's famous Quaker namesake) blue plaque being unveiled at Tregongeeves in Cornwall; and attending an Emmanuel College re-union.

Our big adventure of the year was a summer holiday in Sardinia. The family wanted to do watersports in the sun – but Loveday refused to fly. How to square that circle? Loveday and Nick took the train, stopping off in Paris, Milan and Florence, spent 2 weeks by the sea with the family, and then returned via Pisa, Genova, Ventimiglia and Nice. The train journeys bookending the trip made it very memorable holiday.

Amy started the year travelling while the grand-puppy (Oggy) stayed at Fairholme. In May she joined The Land App as a digital designer after a longer-than-expected gap. She works a nine day fortnight leaving time for adventures in her converted van accompanied by Oggy and her boyfriend, Jake.

Ed has been helping Dad with the business side of Rclone Services when he can fit it in between learning and filming skateboard tricks and sleeping. He had a great interrailling trip with Phoebe visiting eastern Europe and skateboarding with his mates in Prague and Poland. Dad is getting him an alarm clock for Christmas.

Issy is in her third year at uni studying liberal arts, for some definition of studying at least. She hasn't been in the best of health this year but she is happier in her new flat. She enjoyed visiting her friend Emily in New York and won a poetry competition with her poem Love Can Give You This Cool Shark (Journal Entry).

Dougal's deep in Year 13 now, grinding hard on his skateboard, CompSci and Electronics projects. Man's fully locked in for his A-levels. He'd say I'm glazing him, but trust me - bro's gonna get the W.

Nick has been busy this year with his new business. All good fun though – writing open source software and supporting paying customers is a nice change of pace and much less stressful than previous enterprises. In his spare moments he has been accumulating half-completed maths, radio, software and electronics projects.

Loveday at last decided on a design for The Mill and we got planning permission! Whether it will get built anytime soon is a different matter. She and Issy volunteered on the children's program at Quaker BYM. She has also been very busy with local Quakers, Zero Carbon Guildford and her pesky family always wanting things!

Xmas 2023

nick@craig-wood.com (Nick Craig-Wood) — Mon, 25 Dec 2023 00:00:00 +0000

2023 has seen lots of foreign travel for everyone except Loveday. It started with Amy leaving for Dubai, South Africa and Senegal. Nick, Ed, Phoebe (Ed’s girlfriend) and Dougal went on the ski train to France and enjoyed a week of late season skiing - with no broken bones!

Nick, Issy and Dougal had a spectacular trip to Japan in the summer for wonderful food, toilets, sights, cleanliness, trains, Anime and cheap amateur radio equipment. It was very hot though! We finished the trip with 3 days in Korea and ate lots of delicious barbecued meats, fried food and desserts.

Trips to The Mill in Cornwall: lots, number of redesigns considered: →∞, number of designs finalized: 0, number of things decided: NaN.

The children remain alive, but one year older ignoring relativistic time dilation effects due to the travelling.

Amy has had a busy year converting her van, travelling, surfing, climbing and working at Olio. Sadly Olio seem to be running out of runway so Amy was made redundant in November. She immediately booked her dog into the dog sitters (us!) and a flight to Sri Lanka for 3 weeks of surfing in January!

Ed started the year as an advisor for Reassured but decided to come and work with Dad for a few months (while making skateboard videos on the side) before setting off travelling with Phoebe for 3 months round SE Asia, Australia and NZ.

Issy is in her second year at uni studying liberal arts (a bit like literary arts with a helping of TV studies). When not working she has been helping out with the children at Quaker meeting. She has taken up a troglodyte existence in a basement in Archway and would love a visit if you don’t mind the fungus!

Dougal got an excellent set of GCSEs and has started at a local school for 6th form. At last, after 23 years, Loveday is free of school runs! Dougal is doing Maths, Further Maths, More Maths and Even More Maths (disguised as Computer Science and Electronics). He is also doing a lot of climbing (rocks rather than social).

Nick had a paper published on his Pi obsession (Euler found the first binary digit extraction formula for π in 1779) so can concentrate on other stuff finally! He also set up a little company Rclone Services Ltd to run support contracts for his open source software. Ed has been very useful helping with sales and marketing.

Loveday has monetised Nick this year giving him the push to set up his new company. With her new found free time she’s been knitting at 44.7 ms⁻¹. She has also has become the treasurer of Guildford Quaker Meeting, been volunteering at Zero Carbon Guildford and demonstrating for the climate and peace.

Infinity mirror with Raspberry Pi Pico, Neopixels and Micropython

nick@craig-wood.com (Nick Craig-Wood) — Sat, 11 Feb 2023 00:00:00 +0000

Infinity mirror with Raspberry Pi Pico, Neopixels and Micropython

Here is an account of how I renovated our old infinity mirror to this:

From this:

This isn't a recipe on how to do this yourself (though you could certainly use it like that), its an account of exactly what I did, including all the bits where I wasn't sure where I was going in the hope it will help less experienced constructors with the thought process.

The mirror

A long time ago my wife and I were looking in a lighting shop and we came across the infinity mirror. I was immediately captivated by it and we had it installed in our bathroom when we had it renovated.

Fast forward more than a decade and the lights in the infinity mirror had stopped working. I had never taken it apart to see how to fix it which I really should have done but you know how you get used to things not working...

Just before Christmas 2022 I decided I would like to have a project to work on over the holiday period, so after a very brief bit of research I ordered:

40 plated through hole neopixel LEDs
2 raspberry pi pico W microcontrollers
5V 2A PSU
40 x 330uF capacitors
40 x 330Ohm resistors
Some more wire since I had run out

I chose the plated through LED style neopixels because I knew they'd be visible in the mirror and I wanted them to look nicer than the neopixel strips. I knew I needed 34 LEDs but I ordered some extra for testing. I ended up breaking a couple of them so just as well.

I chose the pi pico because I'd done a project with that before with C++ but I thought it would be fun to try micropython (which I hadn't tried before).

The resistors and caps I chose after a bit of looking at the datasheet. Spoiler - I didn't need these in the end and I did need some decoupling caps instead - see later.

I made that order in the full knowledge that it wouldn't be everything I needed and that there may be stuff in there that I didn't need. When getting going on a project its much more important for me to make a start even if it is in the wrong direction than to wait until I've planned everything out to the last degree.

Later I ended up ordering:

40 x 10nF ceramic capacitors
50 x bezel mounts for the 5mm LEDs
5 x 100k pots with nice knobs
5 x switches

Neopixels

Once the stuff arrived I took some photos to see how the neopixels worked inside.

You can see the 3 LED chips and also the controller chip all connected together with very fine wire in the LED encapsulation.

Neopixels have 4 connections, Volts, Ground, Data In and Data Out. You chain them up Data In to Data Out and you connect the first Data In to your microcontroller (in this case a Raspberry Pi Pico). The microcontroller generates a serial data stream with 24 bit color for each neopixel. The first neopixel uses the first block of pixel data to color itself and passes the rest of the stream on. This means you can chain hundreds of them together with only one pin on your microcontroller and address each one individually.

Neopixels and micropython

The first thing I wanted to do was to make a neopixel work from micropython to give me some confidence that the project would work.

I made a plan like this:

Install micropython on the pi pico and make it work
Breadboard a single neopixel to see if that works
Breadboard a chain of neopixels to see if they work

I fully expected these steps to be very hard - bringing up new hardware is always troublesome.

Installing micropython

Installing micropython turned out to be very easy. I installed the micropython firmware from here

https://www.raspberrypi.com/documentation/microcontrollers/micropython.html

Using Pico W version rp2-pico-w-20230116-unstable-v1.19.1-803-g1583c1f67.uf2

I then needed to work out how to communicate with micropython. I tried thonny but I don't really like graphical clicky GUIs and eventually I found that Ubuntu packaged rshell which worked and let me use a REPL on the board and transfer files into the flash.

This was so easy compared to microcontroller development I've done in the past - well done Raspberry Pi and Micropython.

Having a REPL on the board meant I could do little experiments to see if bits of hardware were working just by typing python snippets.

First light

I soldered on 7 pins to the pico (just enough for power, IO and to stop it wobbling) then I plugged it into a breadboard with a single neopixel.

It took me a little while to discover that there is a built in neopixel library in the pico micropython. I found an external library first but decided to use the built in library after evaluating it.

I could then use the REPL from rshell to write code and try to make neopixel work.

This should set the LED to a pink color:

from neopixel import NeoPixel
import machine
pin = machine.Pin(0)
pixels = NeoPixel(pin, 1)
pixels.fill((255, 5, 20))
pixels.write()

However it didn't set it to pink at all! Eventually I worked out that the ordering of RGB seems to be wrong. I looked in the source for the neopixel python library and saw I could fix it like this:

NeoPixel.ORDER = (0, 1, 2, 3)

So a test of the LED became:

from neopixel import NeoPixel
NeoPixel.ORDER = (0, 1, 2, 3)
import machine
pin = machine.Pin(0)
pixels = NeoPixel(pin, 1)
pixels.fill((255, 5, 20))
pixels.write()

It was a relief to get over that hurdle - it seemed then that the project might work after all.

Fifth light

I then extended the breadboard with 4 more neopixels for 5 in total and tried to make that work.

I then had fun noodling around with python to try to make some interesting patterns.

Note that the neopixels are extremely bright when on full power so I did most of this testing with a bit of white paper over the neopixels. When the neopixels are in the mirror they will shine sideways not towards the viewer to make interesting internal reflections. This worked out very well.

Knobs

At this point I realised it would be nice if the user had some control over the display. A brightness knob at minimum would be nice!

The Pico has 3 ADC pins so I bought some 100k pots with knobs off eBay and wired them up and soldered them to the pico.

Later I discovered that 100k was too big a value as they are a bit noisy and can't quite pull up and down to the rails due to the internal resistance of the pico ADC. I chose the 100k to save a bit of power but it was a mistake - 10k pots would have been better. I left them at 100k though and did a bit of work in the software to make sure they could hit completely off and completely on.

I also at this point decided to add two buttons for mode up and mode down but I didn't make them until later.

Mechanical work

Now that I had proved the concept of the pico and neopixels it was time for me to disassemble the mirror. I really should have done this before I started, but I guess I enjoy the feeling of not knowing what I'm doing ;-)

You can see the mirror has what is essentially a string of Christmas lights inside. These gave a beautiful creamy light so I hope I don't destroy that with the LEDs.

Each Christmas light came out through a hole about 10mm in diameter and I had to work out how to fit the LEDs which are 5mm in diameter into those holes neatly. This is all very visible so it had to look nice.

This was probably the hardest part of the project for me. If I had a 3D printer I guess I could have printed something up.

After a lot of messing about, I eventually bought some panel mounts for the LEDs. This made them up to 8 mm in diameter. They were not quite big enough and it took a lot of force to get the LEDs far enough into the panel mounts. I broke a couple of LEDs doing it!

Then I cut about 1cm off the end of all the Christmas lights (with a saw!) so I could re-use the little bit of green plastic tube you see below and put the panel mount into those which luckily fit perfectly.

At last the LEDs were ready to fit into the 10mm holes in the frame.

I then soldered the wires to connect up the neopixels. I also soldered a 10nF decoupling capacitor across the power lines for each one. The datasheet recommended feeding each one through a resistor with a reservoir capacitor, but in my tests the 5V PSU I had bought was plenty capable of dealing with the power surges so I just went for a decoupling capacitor to minimise noise.

This was a lot of soldering - it took about 4 hours to make and solder all the wires. There are 3 wires for each LED that needed making and 4 solder joints, so 204 wire ends to strip and tin and 102 joints to make with 3 or 4 things being soldered together!

With the benefit of 20/20 hindsight and a bit more searching eBay and Amazon I could have bought a string of Christmas light neopixels which probably would have done the job with much less soldering.

I then connected the breadboard and made sure the lights worked.

Software

Now the hardware unknowns were beaten for the moment it was time to get back to the software.

You can download the software from github.

One thing I did do which I recommend you always do when working with embedded stuff is write a simulator.

I used pygame for this. This meant I could test and debug the display modes on my desktop which was a much quicker cycle than testing on the raspberry pi pico. It's also a nice thing to have - you can download the code from the github repo and try it yourself even if you don't have any hardware.

I went through a few iterations of how exactly to split the code between the simulator and micropython but in the end passing a mirror object to each mode turned out the easiest.

I experimented with different frame rates and I found that 50 Hz was too fast - some of the modes were dropping frames so I settled on 25 Hz.

I also experimented with running the refresh off a timer interrupt in micropython. This worked for a while and then locked up and I conjecture I was doing too much on the interrupt. I switched it over to a timed sleep and it became 100% reliable after that.

I experimented with lots of different modes and had the idea of using the board's temperature sensor for a temperature display mode and using the on board WiFi to pick up the current time for a clock mode. The clock mode actually works really well - with a few minutes' practice you can read it just like any analogue clock.

Final assembly

Now that the software was looking good I went back to the hardware side for the final assembly.

I needed to get the pico off the bread board and mounted into the mirror somehow.

In the end I decided to make a piece of wood which I would velcro underneath the mirror (there is a small gap so it won't be visible) and put the pico and knobs there.

This taxed my meagre carpentry skills. I made a mistake with the knobs - the bit of wood I chose was too thick to put the nuts on the knob shafts to hold them in which was a bit of a disaster. However I decided to epoxy them and the buttons in place which made a nice result provided I don't ever want to take them off again.

I fed the power (5V, GND) in to the top of the mirror and out of the bottom of the mirror comes 3 wires, 5V, GND and the input to the string of neopixels. The 5V line goes via a diode to VSYS on the pico so that the pico can be powered via the USB or by the external 5V PSU. This arrangement is outlined in the datasheet for the pico.

VBUS is the 5V input from the micro-USB port, which is fed through a Schottky diode to generate VSYS. The VBUS to VSYS diode (D1) adds flexibility by allowing power ORing of different supplies into VSYS.

and later

The simplest way to safely add a second power source to Pico is to feed it into VSYS via another Schottky diode (see Figure 15). This will 'OR' the two voltages, allowing the higher of either the external voltage or VBUS to power VSYS, with the diodes preventing either supply from back-powering the other.

We will run the LEDs directly from an external 5V power supply. Since the VSYS input range is 1.8V to 5.5V we don't need to worry about a diode drop, so any silicon diode should be fine.

So via a diode connect external 5V PSU to VSYS
Leave VBUS unconnected (since it connects directly to the USB power)

This will allow us to connect a computer and not blow up the computer with the external 5V PSU.

The neopixel drive is connected to GP0 via a 330Ohm resistor. Why the resistor? Just in case the pin gets shorted to 5V - that will save the pico in that case.

I made connectors for the 3 wires to the pico board so I could take it off and things are very nearly done.

Conclusion

That was a lot of fun. The mirror is somewhat mesmerising in the color cycling modes. It also made the wife happy to have the infinity mirror working again, so win-win-win :-)

Thanks

Dougal for helping me with the wire stripping
Dave Lawrence for writing the Matrix and ChaseRGB modes

Xmas 2022

nick@craig-wood.com (Nick Craig-Wood) — Sun, 25 Dec 2022 00:00:00 +0000

We’d love to blame the lateness of this card on the postal strikes; in reality it is because we have been even less organised than usual! We hope you had a lovely Christmas and this reaches you in time for New Year.

2022 has been a relatively quiet year for us. Nick booked a ski trip in March but unfortunately covid hit just before the trip so Ed, Phoebe (Ed’s girlfriend) and Dougal went without him. Amazingly, they came back in the same number of pieces that they went!

In August, Nick, Loveday and Dougal took the ferry to northern Spain where Issy and her friend Emily joined us after interrailling through France and Spain. We stayed near A Coruña and enjoyed surfing and exploring cities such as Santiago and Lugo, and the beautiful coastline. The roads were excellent but Loveday was disappointed by the endless eucalyptus plantations (for paper) which have replaced much of the native flora.

We’ve had several trips to The Mill in Cornwall, including Easter with friends, where we went for walks along the cliff paths, worked on the garden and made plans for future renovations.

Amy (26) left Just Eat to join Olio as a UX Designer in a fully remote job. She bought a new van and spent many happy hours converting it into a camper and getting Dad to do the electrics before taking off to Ireland for a fortnight. She’s currently a digital nomad on her way to South Africa for 2 months, leaving Oggy behind.

Ed (25) spent part of the year working for an outfit selling whiskey barrels by the pallet and is about to start a job in Basingstoke as an insurance protection adviser. He had a lovely holiday with Phoebe in Canada.

Issy (21) completed her Early Years Education course in June and had a summer of fun, including Shambala Festival with Amy. She has survived first term of university at King’s College London, studying Liberal Arts.

Dougal (15) has been enjoying skateboarding with Ed and his school friends. He’s passed the 6th form entrance exam for his new school in Guildford so will be able to walk to school – hooray. GCSEs first though!

Nick (M5NCW) built his first portable transceiver and used it to make Morse code contacts in Europe while sitting in the garden in Cornwall. Having been a fan of Formula 1 for many years it was a real treat to attend the British Grand Prix for the first time to see an especially eventful race. Nick continues to do consultancy and work on his rclone project.

Loveday started the year very busy with her postgraduate studies in Ecology and Sustainability completing her diploma but deciding not to do the master’s dissertation. She has been heavily involved in Quaker activities including organising a weekend retreat. She loves meeting up with her book group friends and volunteering at Zero Carbon Guildford, where she helps run the community fridge.

Xmas 2021

nick@craig-wood.com (Nick Craig-Wood) — Sat, 25 Dec 2021 00:00:00 +0000

We had a very exciting end to 2020 as we completed the purchase of The Mill in Port Isaac just before Christmas. Our new Cornish stronghold is an old miller’s cottage and converted watermill. Our plan is to move there in a few years but we are open for holidays now if you’d like to visit.

In March 2021 Pepper had a lovely multicoloured litter of 5 puppies. Pepper and her puppies accompanied us on our first family visit to The Mill as soon as lockdown lifted, the children having been unable to visit before. Pepper was an excellent mother and the puppies have all gone to friends and family, with Amy adopting Oggy (our first grandpuppy!).

In June we went on Holiday to Plockton in Scotland. The weather was amazing for exploring the beautiful scenery. While there we met up with old friends and Tristram and family. Dougal and his cousins had lots of fun jumping into cold lochs.

We decamped to Cornwall for most of the summer to explore our new demesne. Lots of friends and family came to see us there – having new visitors every week was great fun!

Amy bought her first house in Bristol and moved in with Oggy and Ziggy her red van. She is a UX designer for Just Eat and hangs out with the ‘Waves and Paves’ girls. Amy, boyfriend Sam, Oggy and Ziggy enjoyed the NC 500 this summer.

In March Ed moved out. He and his girlfriend Phoebe are renting a flat together in Woking though he returns home regularly to raid the fridge. Ed worked for Money Expert selling energy and broadband for most of the year but now has a new challenge learning how to be a financial adviser with Neilson Place.

Issy has been doing the second year of her Early Years Education course at Guildford College and has loved doing work experience in our local nursery, pre-school and primary school. She has been having fun with her new college friends and is looking forward to studying Liberal Arts at university next year.

Dougal has started his GCSE courses at school. Lockdown school went much better this year, helped by a dedicated laptop (no games!) and the school running live online lessons. He did his first overnight adventure with the school for his Bronze DofE. When not at school he enjoys gaming, surfing and skateboarding, both with and without his big brother.

In June Nick passed his Radio Amateur’s exam and now has the callsign 2E0WPM. At the Gophercon UK conference in October he gave a talk about deadlocks which went very well. He’s been working on rclone and doing software consultancy while taking time out to learn Morse Code, do radio experiments and help Loveday make lots of Uncle Rick’s spicy pickle!

Loveday is continuing to do her Masters very slowly. She has been getting The Mill ready to let, doing lots of Quaker related activities including marching with her banner during COP26 and has been volunteering with Zero Carbon Guildford.

Deadlocks: the dark side of concurrency

nick@craig-wood.com (Nick Craig-Wood) — Mon, 25 Oct 2021 00:00:00 +0000

This is a transcript of a talk I gave at Gophercon UK on 2021-10-25. You can watch it on YouTube (47 minutes) or see the slides and read the words here.

Go makes it so easy to write concurrent programs that sooner or later you will run into a deadlock. A deadlock is where your program (or part of it) just stops working for mysterious reasons.

This talk will explain what deadlocks are and why the Go deadlock detector isn't very useful in real world programs.

Deadlocks can be very difficult to find and fix and this talk will give lots of tips on how to figure out why your program deadlocked and how to fix it and avoid deadlocks in the future.

You will need an understanding of go routines, locking (sync.Mutex) and channels to get the most out of this talk. No previous deadlock experience required!

Introduction

Hello and welcome everyone.

I’m Nick Craig-Wood and I’m going to be talking to you about deadlocks.

This is an intermediate level talk, so I’m assuming you know what a go routine is and have some idea what a channel and a mutex is.

I’ve put a link to the code samples up there – you don’t need them as I’ll be putting them on the screen, but you can have them for reference during the talk or later.

In this talk we are going to talk about what a deadlock is, how to debug them and how to avoid them in future.

It will have lots of real world examples culled from dozens of painful examples in the rclone codebase!

I learned about deadlocks the hard way by having to debug lots of them and I’m aiming of for this to be the talk I would have like to listened to when I knew nothing.

Hopefully everyone, no matter how experienced will pick up a few tips.

But first a little bit about me. At the moment I’m mostly doing Open Source coding on rclone with a bit of consultancy.

I’ve been using Go for 9 years since before version 1.0. I got into it when I was a busy CTO and found I was doing nearly 100% people management and strategy with no time for coding. So I decided I’d learn Go in my spare time because it sounded interesting and I needed a programming activity to keep me sane.

After writing a few small things in Go, I started started the rclone project in 2012 and its been growing ever since, now with 30 thousand stars on GitHub and millions of downloads a month.

That’s been quite a ride but I’ll save that story for a different talk!

Since all the real world examples in this talk come from rclone I’ll tell you a little bit about it.

Rclone is a highly concurrent command line tool to sync your data to and from any cloud storage system. It supports over 40 providers including s3, dropbox, google drive, azure and lots more.

It tries to do everything it can to maximise your network and disk bandwidth and minimise the chance of data corruption.

It can also mount your cloud storage system as a network drive using its FUSE filing system.

To make it run fast rclone is riddled with Go routines, and the inevitable channels and locks which go with them and in turn the inevitable deadlocks.

Before I started on the rclone journey I knew nothing about deadlocks!

What is a deadlock?

So what is a deadlock?

Here is Wikipedia’s rather abstract definition of a deadlock – something to do with one process holding something another process needs.

I prefer to think of them as a catch 22 situation.

Rather like a newly graduated developer: you need experience to get a job, but you can’t get experience until you get a job.

In a deadlock the other process has something you want, and you have something the other process wants and you are both prepared to sit there and block forever until the other process gives you what you need.

Let’s go through a deadlock in diagram form.

Here we have two go routines (the blue circles labelled Process 1 and Process 2) and two locks (the pink squares).

Each lock could be a sync.Mutex or a Read Write Mutex or maybe even a write to a channel.

The first step is Process 1 takes Lock 1. All good so far.

Next step is process 2 takes lock 2. Again all good, nothing to see here.

Now process 1 attempts to takes lock 2. This causes Process 1 to block and wait for lock 2 to be released as lock 2 is already taken by process 2.

We haven’t reached deadlock yet as everything could be resolved as soon as Process 2 drops Lock 2.

However it doesn’t do that...

Process 2 now tries to take Lock 1.

This is unsuccessful because Lock 1 is taken by Process 1 so Process 2 blocks waiting for it to be released.

Now Process 1 and Process 2 are blocked. They will never proceed as each is waiting for the other to release a lock.

These two processes are now deadlocked.

This means that some part of your program has stopped working and will never work again.

Let’s see what that might look like in Go with a simple bit of code.

We have process 1 and process 2 just like in the previous example and they are going to sit there in a loop, taking and releasing lock1 and lock2.

The only difference between process1 and process2 is that they take and release the locks in a different order.

This code looks like it should work, but it will deadlock pretty soon.

This is the output once you run the code.

First thing to notice is that the Go runtime has detected the deadlock giving the nice friendly message “all go routines are asleep – deadlock”.

The runtime gives a backtrace of all the Go routines that are running which is very useful for working out what went wrong.

The disadvantage of this is that they can be very long – even this tiny example needed 2 slides to fit it on.

Go routine 1 is waiting on the select with no cases that we put in to stop the program ending too soon. We can ignore that.

Goroutine 17 is in state semacquire – its deadlocked calling the sync Mutex implementation.

Here is the next bit of the output.

In goroutine 19 you can see a different bit of our code in also calling the sync Mutex implementation. Again it is deadlocked in the semacquire state.

I’ve put arrows at the bits of code that the backtrace indicates the go routines are executing.

You can see process1 has taken lock1 and is blocked taking lock2, whereas process 2 has taken lock2 and is blocked taking lock1.

This is a clear deadlock. Whenever you see a trace where a single go routine takes more than one lock, you should be thinking about deadlocks.

This is exactly what we expected from our initial diagram, process 1 is blocked taking lock 2 and process 2 is blocked taking lock 1.

It is nice to see theory and practice agreeing!.

You’ll have noticed that Go can detect deadlocks.

This is very useful, but unfortunately it very rarely detects deadlocks in actual code.

This is because all the Go routines in your entire program have to be asleep for it to trigger and in my experience that almost never happens except in toy examples like these.

There is always something else going on, like log collection or statistics calculations or something like that which means the race detector does not fire.

This time if we add one extra go routine as indicated the deadlock detector does not kick in, and the program deadlocks silently.

Or rather part of the program - process 1 and process 2 - is deadlocked but the rest of it is working fine.

Unfortunately this is the usual state of affairs when debugging deadlocks – some part of your code just stops working with no other symptoms.

Enough of your code may be working to give you the idea its it isn’t broken but some important part isn’t progressing.

We’ll see in the next section of the talk what to do at this point.

A quick diversion, as I expect this question has popped into your minds already.

Can we use channels instead of those nasty Mutexes to avoid deadlocks?

Alas, no, you can cause a deadlock with channels just as easily as with Mutexes and we’ll see an example of that later.

In my my experience with rclone only about 10% of the deadlocks are with channels so they are less likely certainly.

So why do we get deadlocks?

There are two ingredients.

The first is concurrency. Without concurrency, that is things happening at the same time, we can’t have deadlocks.

This means that single threaded code can’t have deadlocks which is a point we’ll come back to later.

The next magic ingredient for deadlocks is locking.

When I say Locking here I’m including channels. Reading and writing to channels can block your program just as easily as Mutexes.

And if I wanted to be really precise I’d say shared resource acquisition instead of Locking because you can get deadlocks using other things other than Mutexes and channels, for example using file system locks.

So where did these locks come from?

My excuse is the race detector made me do it :-)

So what do we do to pacify the race detector? We put locking in, and sooner or later once you’ve put enough locking in you will cause a deadlock.

I’m joking here of course, the go race detector is wonderful.

It tells us about data races which would have caused silent corruption in our programs.

Let’s just sing the virtues of the race detector for a moment. It is a technological marvel and very close to magic.

The race detector will tell you when your program runs a risk of memory corruption due to unsafe accesses.

If you are writing concurrent programs you should run your tests with the race detector, and if you can, some of your production binaries too.

Any problems it finds are always real problems – I’ve never known it make a mistake.

However when fixing the things the race detector points out, do think about what deadlocks you are about to create.

In particular think about what locks might be already held and what locks you might want to take in this go routine.

How to debug a deadlock

Here is an example I’m going to use quite a few times.

It is a simplified example from rclone’s FUSE based filing system for mounting remote cloud storage as if it were a local disk.

On the left you can see the definition of the File struct. It has a name, size and a parent directory. It has a Path method which returns the full path of the File and it needs to ask its parent directory for its Path in order to do that. It has a Size method to return its size and a Truncate method showing how the size of the file might get changed.

On the right we see the definition of the Directory struct. This has a name, a parent directory, a slice of files it contains and a slice of directories it contains. It has a Path method for returning its full path and it needs to find out the Path of its parent here. It also has a Size method which adds up the sizes of all the files in the directory.

There is no locking in the code above, yet, but a few runs with the race detector reveals that locking is needed.

To prevent those nasty data races we add a sync.Mutex called mu into the File structure. The name, lower case mu is traditional in Go code.

Then in each of the methods we lock the mutex with mu.Lock() and defer the unlock of the mutex with the defer mu.Unlock().

This means that none of the methods will ever run concurrently and it means our instance variables are safe from data race corruption – hooray!

This is a standard pattern used in Go concurrent programming and hopefully you’ve seen it before.

Take the lock at the start of each method and defer its release to the end – what could possibly go wrong?

We do the same thing in the Directory structure, add a sync.Mutex called mu to the struct and then call mu.Lock() in each method and defer the mu.Unlock() until the method returns.

However we’ve unwittingly caused a potential deadlock by doing this.

I say potential deadlock here because when you run this code it will quite likely work just fine until just the right conditions are met.

Most deadlocks are like this – they don’t stop your code the first time you create them, they come back to bite you later when you are least expecting them.

Usually just after you’ve put your code into production.

Here is a bit of code to exercise the Directory and File abstractions and which will provoke the deadlock.

Process1 just calls File.Path in a loop continuously. Not exactly normal usage, but we’re trying to provoke the deadlock here.

Process2 just calls Directory.Size in a loop continuously.

In our main function we create a root directory and put a file into it then start go routines to call process 1 and process 2.

We also start a race detector defeating go routine which just sleeps for a minute.

And finally the empty select statement to sleep the main go routine forever. Remember if we exit the main go routine then our program will stop, and we don’t want that.

So we run the program and it is now sitting there doing a whole heap of nothing while deadlocked.

How can we find out what it is doing?

If you are running on a Unix based platform (for example Linux or macOS) then you can send the process a SIGQUIT using the kill command or press Control Backslash. This will produce a nice backtrace.

Windows platform users will have to work a little harder and install the HTTP debug handler from net/http/pprof and I’ll show how to do that a bit later.

This is what happened when I sent the File and Directory example program the QUIT signal.

I got this nice backtrace. It’s quite long but there is a lot of stuff we can ignore.

The output is divided into stanzas and at the start of each one is a little description, Here it says goroutine 0, idle.

Following that is a reference to a line of source code, plus the code itself, once for each level of the call stack.

In fact this is a system go routine doing something in the runtime – we can ignore it because it doesn’t have any of our code in the backtrace.

Next we see go routine 1, select, no cases. This is the wait forever select code which we put in to stop the main function existing. We can ignore this too.

Next we see go routine 5 semacquire. This one is important, semacquire means it is blocked when trying to acquire a lock. For most deadlocks you are searching for these semacquire lines first.

Below the semacquire you can see a couple of lines of code in the runtime – these are in the sync.Mutex code, and finally a bit of our code in Dir.Path.

We can see that Dir.Path was called by File.Path so by the time the go routine tries to acquire the Directory lock it has already acquired the File lock.

Next we see goroutine 6 – also blocked on semacquire.

It looks quite similar to the last one, two lines in sync.Mutex code and then it tring to acquire the mutex in File.Size.

Looking up the backtrace you can see that File.Size was called from Dir.Size which means that this go routine acquired the Dir lock first and is now blocked on trying to acquire the File lock.

If you have an editor you can click on those file names to jump to the code in question it is really helpful at this point so you can look up and down the call stack to see what other locks might be held.

Under that we see go routine 7 sleep which is our deadlock defeating go routine which we can ignore.

And finally the register dump which you can ignore for this purpose.

I’m sure the register dump is useful for low level debugging, maybe if you are writing assembler, but I’ve never needed it to debug deadlocks.

Here I’ve annotated the code from the backtrace showing where each go routine is blocked.

The fact that both parts of the code are blocked on acquiring a Mutex are very suggestive that there is a deadlock.

Let’s look at Dir.Path on the left hand side of the screen. We saw from examining the backtraces that this go routine already holds the file lock and is now blocked trying to acquire the directory lock.

On the right File.Size is already holding the directory lock but is blocked trying to acquire the file lock.

This means that these two functions are deadlocked and will block forever because the locks they are waiting for will never be released.

This deadlock occurs whenever the Size method on Dir is called concurrently with the Path method on File.

When thinking about deadlocks it is helpful to write out a timeline of the two processes involved. The numbers on the left of the code show the ordering of the calls to each process.

The arrows point from the blocked locks to the place the lock is held, and them crossing is very suggestive of a deadlock.

You might think that there is very little opportunity for this to happen given the short windows of opportunity – each of these instructions will be over in a few nano seconds won’t it? However, if the deadlock is possible it will happen sooner or later.

One thing to bear in mind is that locking instructions are particularly slow – they use slow memory locked instructions or schedule go routines or processes - which often turns something you thought would never happen into something which always happens.

I’ll just quickly loop back to how to install the HTTP debug handler which you can use to collect backtraces when running Windows.

Look up the docs for net/http/pprof – everything you need is in there.

You can collect the output with curl or just look at in your browser.

You can use this on any platform of course, and you might consider leaving it in your production code for when your production code has deadlocked and you want to find out why.

The output from the HTTP debug handler is similar to the output from SIGQUIT with one stanza for each go routine and a trace showing position in file and the actual line of code being executed.

It doesn’t call out the state of each go routine in the same way as the SIGQUIT output, but it is easy to see that the two go routines here are each blocked on SemaqcuireMutex which means blocked on acquiring a mutex as you might expect.

One important difference is that this output collapses goroutines with identical stacks which can very useful if you’ve got 1000s of goroutines.

When you are tracking down a deadlock in a real world program with lots of concurrency, you’ll find that the backtraces get very long, very quickly.

Deadlocks have a nasty tendency to spread through a good fraction of your running go routines until they are all waiting on a lock kind of like the way something crystallizes out of a solution.

Here is an example of an backtrace a user sent me while debugging an rclone deadlock.

It has nearly 2000 lines covering 260 go routines and is par for the course when dealing with lots of concurrency. There are 14 go routines blocked on acquiring a mutex, 123 blocked in a select statement, 121 blocked on sending or receving to a channel and 1 in a system call and 1 idle.

If that sounds like looking for a needle in a haystack, it is, we need some help here!

Luckily there is a great tool called panic parse which helps enormously here.

It condenses down that 2000 line log into a nicely coloured 142 line log with one stanza for each different type of go routine.

It shows that there are only 14 different types of go routine here which makes working out what is going on so much easier.

It also has a nice web output which I haven’t tried yet, but I’ve put up the screenshot from the github page so you can see what it is like.

This tool has saved me so much time when debugging deadlocks – thank you Marc-Antoine Ruel!

The text output of panic parse shows a stanza for each go routine. It collapses go routines with similar stacks together like the HTTP debug handler, but better.

Here is what a stanza looks like. On the left you can see a number – in this case 7 – this is how many go routines like this there were, and really usefully the time it has been paused. Looking for the first goroutine to pause can often be a big clue.

It is then reasonably easy to see that there are only a few go routines that could possible be the problem, namely the ones that are waiting on locks that have taken locks already so we can immediately ignore nearly all the go routines.

I’m not going to lie, it’s still tricky working out exactly what the cause of the deadlock from here is, but it is so much easier with panic parse.

How to avoid deadlocks

We’ve talked so far about what a deadlock is and how to find them and debug them, but a better strategy would be to avoid them completely.

I’m going to talk through a number of different techniques for avoiding deadlocks. Unfortunately there isn’t just one thing I can say – do this and you’ll never get a deadlock again, alas.

We’re going to start with some easy things to fix then move on to some harder problems.

I said there was going to be an example of deadlocks with channels a bit later and here it is!

Let’s start with a Job. You can see its struct at the top.

It has a constructor NewJob which doesn’t do a lot in this example.

The Process method for each job runs the Job. However in the process of running a Job, more Jobs might be created. In this example it picks a random number as to whether to create more jobs or not.

Those jobs are then returned in a slice for further processing.

This is simplified from an rclone example where each Job listed a directory and returned jobs to list sub directories.

Here the ProcessJobs function takes a WaitGroup and a channel of jobs. When running it takes a job out of the channel, processes it and adds any created jobs back onto the channel.

It is written like this with a channel so we can run a few copies of ProcessJobs concurrently to increase the throughput and that is what happens in the main routine - We start 3 copies of the ProcessJobs function to increase the throughput.

Its worth explaining what the sync.WaitGroup does here. We are using it to keep track of the number of active jobs – whenever a job is created it is increased with the Add(1) method and whenever a job is finished it is decreased with the Done method.

When the wait group goes to zero, we know all the jobs have finished and we wait for that moment with the Wait method.

When we run it, it works fine for a while, running jobs, creating new ones, adding them to the channel, until...

… you guessed it, we get a deadlock.

The first go routine shown, go routine 1, is our sync WaitGroup waiting for the others to finish.

Goroutine 6, 7 and 8 are all in state chan send. That is they are waiting for a channel to be emptied before they can send some data down it. You can see that they are all waiting in ProcessJobs.

The problem is that somehow all three of our go routines are blocked writing to the channel and none of them are reading from it.

Because those 3 go routines are the only place that we read from the channel, it means that the channel will never get emptied and those 3 go routines will never get unblocked, creating a deadlock.

This is the code with the deadlock positions marked on it.

The three ProcessJobs functions are all trying to write a new Job into the jobs channel. However none of them is reading from the channel, so none of those writes will ever succeed A classic deadlock, this time with channels.

We can also see the sync.Waitgroup Wait method being called to wait for the Job processing to be complete. This is now part of the deadlock too as it will never proceed.

The Waitgroup Wait isn’t the cause of the deadlock, but it is collateral damage and this is very common when looking at deadlocks. Often the collateral damage overwhelms the cause of the deadlock and that makes it very hard to work out what is going on.

So what is the solution?

We need the ProcessJobs function to be either reading from the channel or processing a Job. We don’t ever want it to be blocked on writing to the channel, because if one go routine can get blocked then all of them can.

The easiest way of achieving this is by sending into the channel in a new go routine. The original Go routine is released to receive jobs from the channel while the new go routine sends the new jobs to the channel.

This has the consequence that there might be an indefinite number of go routines with a few jobs to add to the jobs channel hanging around. If that is a problem then a different design is needed.

Note we also moved the activeJobs.Done call into the new go routine – we can’t signal a job is finished until all its Job children are safely queued. If we do then the main function will end too early.

The moral of this channel deadlock is simple. Don’t send and receive to the same channel in the same go routine – it is a recipe for deadlocks.

As I said earlier this example came straight from rclone. This dead lock took some time to appear and it took me a few iterations to a find a solution which didn’t deadlock.

Concurrency is hard!

Here is an example with read write locks.

We have two main functions RO and RW. RO takes a read lock then calls MoreRO which in turn takes the Read lock again. This is allowed – read locks can be taken multiple times.

The RW function takes the lock for Write.

This look harmless and it works most of the time until suddenly it doesn’t and you made a deadlock as we’ll see in a moment.

This example is simplified from a real world example which was caused by a bit of careless refactoring.

The next step is the Read Write lock being taken in go routine 2.

If you read the docs you’ll see a Read Write lock prevents any new Read locks happening so that the lock eventually becomes available.

There is also a note about recursive locking not being allowed if you read carefully which is exactly what this example is about.

The next step is the Read Write lock being taken in go routine 2.

If you read the docs you’ll see a Read Write lock prevents any new Read locks happening so that the lock eventually becomes available.

There is also a note about recursive locking not being allowed if you read carefully which is exactly what this example is about.

The deadlock happens when go routine 1 attempts to take the read lock again. This is called recursive locking and none of the standard library locking routines support recursive locking.

This lock is blocked because the Read Write lock in go routine 2 is in the process of being taken.

This means that MoreRO function will never return and the RO function will never release its lock, so in its turn the RW function will never acquire the mutex.

Thus, a deadlock!

And the moral of this story is Don’t take a read lock twice in the same go routine.

It might sound completely obvious but it is really easy to do by accident.

It does warn explicitly against doing this in the docs too, but of course I didn’t notice that until after I caused this deadlock!

I could show a commit message for every one of the examples in this talk, but since we haven’t got all day I’ll just put this one up so you can feel my pain.

I always find debugging deadlocks hard even with all these nice tools, so when you have to debug one yourself, just remember that!

Objects which contains locks that call each others methods are a deadlock waiting to happen.

This happens naturally in more complicated systems, for example file systems, and great care needs to be taken not to cause deadlocks.

Going back to our Directory and File example, the problem is caused by the fact that Directory and File both call each other’s methods.

The general solution to this is called Total Lock Ordering but we’re going to look at some easier mitigations first.

Here is one example fix for the deadlock. I call this technique shortening the locking Window.

What we’ve done here is released the mutex in the File Path method as soon as we can and in doing that the lock and unlock of the File and Directory mutexes no longer overlap.

At the top you can see the original function.

Below that you can see we’ve modified it to make a temporary copy of name and parent from the File then unlocked the mutex.

This is enough to fix the deadlock as shown before. It does introduce the idea that the Path function might return an out of date Path if the name or parent changes during the call of Path. Some thought is required here as to whether this is OK or not.

In order to minimize the chance of deadlocks, release locks as soon as possible to shorten the locking window.

Specifically release them before calling any other functions or methods which might take locks.

I’ve even resorted to using the defer statement to defer calls to get them outside the locking of the current method.

Another easy mitigation is to notice which members of your object are read only. These should be assigned to once in the initialization of the object and never written again.

These don’t need any kind of locking at all to access. Concurrent reads are always safe.

So, for example, here we say that the name of files and their parent directories never change.

I’ve moved them above the mutex definition to give the idea that they aren’t and don’t need to be protected by the mutex.

In this rename free world, we don’t need any locking in the Path method and we fixed the deadlock by not having any locking in the File.Path method at all.

If you can identify variables which are read only, you don’t need locking for them and they can be accessed outside the lock.

No locking means no possibility of deadlocks.

Here is a sketch of some other things you can do which I didn’t have room to make examples for.

Once way to reduce deadlock possibilities is to split the lock into multiple locks, so use more fine grained locking. This makes it less likely the locks will overlap.

Or maybe you can use atomics for some accesses (perhaps you are incrementing a counter). Atomics never block your process so can’t cause a deadlock. They come with their own caveats so use with care.

Using a Read write mutex is a great help for reducing lock contention provided you’ve got methods which only read things. Allowing many read only methods to run at once stops them blocking and minimizes the chance of deadlock.

And if you are using channels, buffered channels help avoid deadlocks because they don’t synchronize two go routines. Deadlocks can still occur when the channels get full though so it wont cure everything.

Now its time to bring out the big guns.

The ultimate solution to this kind of locking problem is to impose total lock ordering. This means that you are sure that if the locks overlap then you always take one lock first.

So if you have two locks A and B, you always take lock A before taking lock B in the same go routine. This makes sure we never have a deadlock.

This, however, is much easier said than done in a structured program with locks inside objects.

In practice with object based locks and methods, this means making one object super-ordinate and one subordinate.

For example this might mean that Directory can call File methods with the Directory lock held but File methods mustn’t call Directory methods with the File lock held.

In a perfect world we’d be able to specify what we want to a linter. I don’t think such a linter exists yet – if it does do tell! In the mean time comments like this one from rclone are what we have to live with.

It explains to the reader that the Cache and Item objects are tightly linked and that if you don’t want to cause a deadlock then don’t call Cache methods from Item with the Item lock held.

This means that when you change code you need to be very careful to uphold those invariants. If you don’t, deadlocks will ensue.

So make write of comments and do lots of code review when writing this sort of code.

Debugging deadlocks can get very hard, very quickly and I used to think I was pretty good at this computer science stuff, but debugging deadlocks frequently makes me feel very stupid.

So if the locking gets too complicated then you can bail out and use a single threaded engine to remove all the locking.

Single threaded likely means lower performance but maybe having no deadlocks is worth it.

You’d communicate with the engine over channels in a request / response style. Typically you’d send the channel for the response over the request channel.

This makes things much simpler and is very much in the classic Go style but making everything single threaded will drop your performance which may be unacceptable.

You’ll notice there’s been a lot of struggle with deadlocks in the previous slides and it would be nice if there were tools to help us find the deadlocks before they happen.

Well there are a couple of notable tools for finding deadlocks in go code.

Dingo hunter is based of a theoretical approach for finding channel based deadlocks. You may have caught the talk about it here in 2016. It only works for unbuffered channels, alas, which limits its scope.

Go deadlock is a practical tool for finding violations of total lock ordering and has been used to find deadlocks in Cockroach DB for example. I’ve used it on the rclone code base a few times.

It comes up with error messages that look rather like those from the race detector, but unlike the race detector it detects potential deadlocks before they happen so you can run this on your code which hasn’t deadlocked yet.

Go deadlock only works for mutex deadlocks, not channel deadlocks. As I said before this is probably 90% of the deadlocks I see so it is an excellent start.

Go deadlock is very easy to use. Here’s the example from its GitHub page.

You just replace your sync imports with a go deadlock import and when you run your code it will give total lock ordering errors.

This is what our File and Directory example looks like when run with go deadlock.

As you can see it nicely points out the potential deadlock. The top half is the first go routine and it’s saying the Directory lock then the File lock was taken but in the other go routine (the bottom half) the File lock was taken first, then the Directory lock. It isn’t saying a deadlock happened, it is just saying that because of this ordering of locks a deadlock could happen.

I’ve used go-deadlock on more complicated examples and I have to say I’ve found the output quite difficult to interpret. This is because deadlocks are hard to understand rather than a shortcoming of the tool. So if you do plug go deadlock into your massive concurrent program, expect a bit of head scratching while you work out what the error messages mean!

When I want to use go deadlock with rclone I run this little script.

It searches and replaces all the sync Mutex and sync RWMutex with their deadlock detector versions.

Unfortunately I find you can’t leave Go deadlock in production code as it has too much of a performance impact.

I’d like to have a neater way of enabling it for testing, but I haven’t figured that out yet.

Conclusion

This has been quite an intense talk and we’ve gone over a lot of concurrency related things, but I hope you all now have some idea of what a deadlock is, how do debug deadlocks and how to avoid them in the future.

Deadlocks are the curse of concurrent programming, however there good tools for finding and debugging them and there are some promising tools for helping to prevent them happening in any code base.

So if, or rather when, you get to debug your first deadlock, just remember, Don’t Panic, or rather...

...Do panic and collect a backtrace :-)

Thank you all very much for listening.

Xmas 2020

nick@craig-wood.com (Nick Craig-Wood) — Fri, 25 Dec 2020 00:00:00 +0000

2020 has been the best of times and the worst of times. As a family of dedicated introverts (with one exuberant exception) we’ve enjoyed spending more time at home together, meals in the garden and long walks in the local area with Pepper.

The year opened with Nick finalising the sale of Memset after more than a year of intensely stressful negotiations, made harder by Nick’s sister and business partner Kate’s deteriorating mental health. Luckily Ed’s position teaching snowboarding in Austria gave the perfect excuse for a February half term holiday. Unluckily - and perhaps predictably, if you’ve been reading these messages long enough - Ed broke his arm just before we were due to visit Ellmau meaning we had to go skiing without him. Thankfully the sale of Memset completed in March. Unfortunately this was just before the first lockdown which meant Nick was unable to say a proper goodbye to everyone who worked there.

Lockdown meant Issy finished Upper Sixth very suddenly, without taking her A-levels or experiencing the usual rites of passage that come with leaving school. Dougal loved spending more time at his computer but home schooling and remote Common Entrance exams turned out to be less fun than Roblox. To entertain ourselves in lockdown we incubated and hatched some eggs - we now have two fine hens and their noisy brothers have moved to the countryside.

Amy, who works for Just Eat, spent the first part of the year in Bristol. In May she returned home in her new red van (which we keep mistaking for the postman!). She used her time here to build in a tiny kitchen and a very ingenious folding bed. Nick helped her with the electricals and then she set off to explore the UK with friends over the summer.

Ed’s broken arm took a while to heal which made getting a job in lockdown challenging. He briefly had a job in Guildford but since September has been working in Godalming, helping people save money by switching their broadband and electricity suppliers. With his first month’s pay he bought a new car (and the new wing mirror he needed after taking it out for the first time). Issy hopes he will have enough money to move out soon so that she can have his bedroom.

Through the year Issy helped organise Quaker Senior Conference. Normally this is a residential event but Issy and the team rose to the challenge of taking it online this August. We weren’t allowed to join in, but Issy’s timetable on the door let us know what she was up to. Since September Issy has been studying Early Years Education at Guildford College.

Sadly, Kate died in September of what we think was a drug overdose. This was quite a blow as she’d just come back from the Priory and we had hoped she was getting better. Having a lockdown funeral was strange with only 17 people allowed to come, but we sent her off in style with a pink glittery coffin and a motorcycle hearse.

Loveday is continuing her part time MSc in between Zooming with the Quakers, knitting, gardening and looking after the family. Nick is gradually de-stressing and has been putting more energy into his open source software rclone. The 3 week Italian adventure we were looking forward to was cancelled, but we did manage holidays in Cornwall and took a wet and windy trip to Northumberland in August. Hopefully we will have some Cornwall related news to share in 2021.

Xmas 2019

nick@craig-wood.com (Nick Craig-Wood) — Wed, 25 Dec 2019 00:00:00 +0000

We were obviously very naughty in 2018 because Santa brought us norovirus for Christmas which spread through the family and laid us low for a week. Hopefully this year we’ve behaved better!

Nick and Dougal had 2 ski trips this year. The first in February with Amy and cousins Olly & Felix to Sauze d’Oulx, Italy. Things went well except when Amy’s blood sugar dropped too low and the time Dougal lost his snowboard down a red run… The second time was in April; this time with Ed, who, in true Ed style, managed to send his passport to the Russian embassy for a visa just before the trip, leading to a mad panic on the day of the flight and Ed missing half the holiday!

In April it was our 25^th wedding anniversary. We had a party at home and it was lovely to see so many of you there. Revisiting the wedding video provided plenty of entertainment and lots of “who’s that” moments.

In July we went to the west coast of Scotland for a “summer” holiday. When the weather was good it was lovely and when it wasn’t we got very wet. We played our traditional game of cooking sausages wherever we went. The highest scoring round was cooking on a rock in the middle of a river. We went to the Isle of Canna so Loveday could revisit her teenage years and were lucky enough to see a large pod of dolphins on the way.

Amy (23) and boyfriend Jack converted an old transit van into a camper (see pic). They spent the summer travelling round Europe, returning briefly to have Amy’s braces removed (hooray!) and again to fix a bit of engine bother (“seaweed in the tank”). In August, Amy left home (sad face) and has ~~run away to the circus~~ gone to live in Bristol, joining the graduate program for “Just Eat” in their design department where she works on the user experience (UX) of their app and website.

Ed (22) left Vodafone in March to travel through Northern Europe to Moscow for 6 weeks. He then went again ending up in Scandinavia this time. After getting underfoot at home for a few weeks we packed him off on a snowboard instructor course in Austria. He passed his test (in German!) and has now just been let loose teaching in Ellmau for the season.

Isabel (18) completed the job of chopping off her hair (for charity). She has enjoyed more Quaker events this year. A-levels are looming but she has decided to take a year out after them to delay having to fill in the dreaded UCAS form.

Dougal (12) has spent almost every moment of his spare time playing computer games. His voice has been breaking and he has been growing, soon Amy will be the smallest in the family! School and rugby have been going well this year. Pepper won the “loveliest lady” prize in the school dog show; thank heavens she wasn’t wearing her “Foxy Lady” scent!

Memset has been keeping Nick busy, celebrating 10 years being carbon neutral (you can see Nick with the award in the pic). Loveday enjoyed crochet and cookery courses and a trip to Iceland in August for her sister’s birthday where much lounging around in bathrobes was done. She then decided she was bored so signed up for a part time MSc in “Sustainability & Ecology” from the Centre for Alternative Technology in Wales. New Year’s resolution: learn to type!

Xmas 2018

nick@craig-wood.com (Nick Craig-Wood) — Tue, 25 Dec 2018 00:00:00 +0000

Firstly apologies that the photomontage didn’t make it to press in 2017!

Alas our beloved Daisy left us in November 2017. We remained dog-less until January when we went to see Pepper, a black and tan cocker spaniel. Pepper came to live with us in March and has been eating everything ever since. High scores so far: 2 pairs of glasses, 1 passport and Nick’s favourite Rubik’s cube! Fortunately she has the sweetest temperament so it is impossible to be cross with her for long.

In March, Amy and the boys went to Sauze d’Oulx in Italy for skiing and pizza. In July 4 of us (+dog) had a fun week in a caravan in Scotland on the Ardnamurchan peninsula. We took inflatable kayaks with us and had great fun paddling round the beautiful scenery in between the showers and driving to distant lay-bys to get 3G signal for Issy. August saw us in Cornwall with a wet visit to Boardmasters festival. Ed competed in the Guildford skate jam unfortunately breaking something in his foot (we couldn’t have a message without a broken something from Ed!).

In October we reluctantly had to say goodbye to another old friend: Treharrock Manor in Cornwall. All the Hambly family came to help with the mammoth clear out. Fairholme now looks like an antique dealers storeroom. Nick hopes to be able to get his car back in the garage sometime next year!

Amy (22) graduated with a degree in digital design from Brunel and moved back home bringing her boyfriend Jack. In October she finally had the jaw operation she has been expecting for 7 years. At last she can bite things things properly! She has been working for Dad making Memset’s new website which she has done a splendid job of.

Ed (21) has celebrated his 21st birthday, passed his driving test and completed a successful year in sales at Vodafone. He is enjoying working all day then going out every night (ah to be young!).

Isabel (17) survived her GCSEs in the summer and was happy with the results. She has now moved to a new school for the 6th form, and in an epic act of rebellion against her scientific parents is studying English, History and Religion & Philosophy. She continues to enjoy young Quaker activities and has chopped off most of her hair.

Dougal (11) has moved to a new school. It is a positive and encouraging environment and he is more relaxed at school than before. He has been playing football for a local club and doing a lot of computer gaming in his spare time. He has made his old man proud by learning to solve the Rubik’s cube.

Nick has taken on the CEO role at Memset which has brought new challenges but he has the support of a great team. His spare time hobby has been his Open Source project rclone. Nick even gave a talk about rclone which you can find on YouTube. Loveday’s year has been busy managing and preparing to sell Treharrock. This has been both very sad and a great relief. She is looking forward to a slightly quieter life next year with time to read all the books from her book clubs.

Rclone and Go

nick@craig-wood.com (Nick Craig-Wood) — Wed, 21 Nov 2018 00:00:00 +0000

This is a transcript of a talk I gave at the Go London User Group on 2018-11-21. You can watch it on YouTube (26 minutes) or see the slides and read the words here.

It's about rclone and how rclone uses the Go programming language.

Introduction

Good evening.

I’m Nick Craig-Wood and I’m here to talk about rclone. Rclone is an Open Source command line program to transfer files to and from cloud storage started by me as a hobby in 2012.

First a bit about me. I’ve loved programming ever since I was 13 with my ZX80 and have been doing it ever since.

Now-a-days I’m CTO of Memset Ltd who do cloud hosting and storage. I don’t get the chance to do a lot of coding at work any more so I now do Open Source coding as a hobby.

I’ve always had a keen interest in storing data, keeping it safe and transferring it about – hence rclone which is an embodiment of those things!

I used to be a data hoarder, but the house filled up with servers and disks and my wife complained about the electricity bill so I’m mostly reformed now.

Thank heavens for cloud storage!

My talk this evening is going to start with a quick summary of rclone and its history.

I’ll dive into how it works, and show a bit of code.

I’ll then discuss how you test a project like rclone and round off with a look at some of my favourite libraries.

But before I get going I don’t want you to get the impression rclone is all my work. I’d like to thank my excellent core developers, the super helpful group of people on the forum and anyone who has sent a PR or raised an issue – rclone wouldn’t be what it is today without you.. Oh and thank you for everyone who has sent a PR correcting my spelling :-)

What is rclone?

In summary rclone is a command line program to sync your data to and from cloud providers.

Rclone takes lots of care with data integrity, checking hashes at every step and preserves timestamps on your files.

It can transfer data directly between cloud providers so you can use it to back up your s3 objects to google cloud storage.

It has some optional features like encryption, but by and large rclone maps objects 1:1 from source to destination.

There is lots more info on the website.

Many of you will be familiar with rsync. Which is (see slide)

Rclone syncing works very much like rsync. I’ve tried to keep the flags the same where possible to make it familiar to rsync users.

There is one thing rclone does not do though and that is implement the extremely clever delta encoding that rsync uses for transferring objects.

That unfortunately isn’t possible when trying to copy objects 1:1 to cloud providers as they just don’t provide the API to do it.

It would be possible to implement it but you’d lose that 1:1 mapping of objects.

Rclone supports most major cloud providers. Well I think so, they keep making new ones!

Certainly all the big names are here, both for business use like s3, google cloud storage, azure blob storage and for personal like google drive and dropbox.

There are also the standard protocols like FTP, SFTP, HTTP.

I particularly like the HTTP backend – you point it file listing webpage (for example the ones apache generates) and it can pull the files off there recursively, or you can mount it as a disk.

All rclone operations are supported by all providers.

Thanks to Go’s amazing cross compiling abilities, rclone runs on nearly every platform you can think of.

Rclone is licenced under a very liberal MIT licence which means that it gets bundled into Synology NAS and other products like that.

Nearly all the platforms are cross compiled on a linux host as part of the Travis build. All except Windows and macOS which I had to build natively to fix various problems.

As I began to do less programming at work, I realised I needed to do some programming at home in order to retain my sanity.

In 2012 I decided to learn Go as it had just reached the exciting milestone of a 1.0 release.

I made a few test programs then I wrote my first major go library, an interface for the openstack swift object storage system.

I need a program to exercise that “swiftsync” which became “rclone”.

There were some things that attracted me immediately to Go.

We’d been using Python a lot at work so the single binary for a deployment was just breath of fresh air.

Go excels in concurrency which is really important for rclone too. Everything works concurrently in rclone such as directory listings and transfers of objects.

I started using Go as a new challenge and it has really brought back the joy of programming for me personally.

Lastly Go is a really easy language to pick up, especially if you’ve done any programming in a language similar to C such as Java.

From its humble beginnings rclone has grown into quite a large project.

As of today we’ve got about 11,000 stars on github, 200 contributors who’ve made 500 pull requests. Rclone get’s downloaded quarter of a million times a month and it has been packaged by the major Linux, Windows and Mac distributions.

Exactly how rclone got all those users is a mystery, but being on the front page of Hacker news added about 1,000 github stars overnight. Unfortunately at that time I was in the Outer Hebrides with no Internet so I had no idea this was happening!

Sorry, your browser doesn't support embedded videos.

Here is a visualisation of rclone’s source code made with Gource.

Rclone was created in 2012 and it got off to a slow start with just me – that is me whizzing about.

Rclone reached v1.00 in 2014 and supported 3 backends: s3, drive and swift.

Rclone gained its first external contribution in 2015. That was when it implemented Amazon Drive and it started to get popular.

In 2016 it gained some traditional protocols, SFTP and FTP and by 2017 I’d finally figured out I needed a VFS layer. Which brings us to the frenzy of 2018!

In 2017 I was surprised to find rclone in the middle of a minor media storm.

This is how...In 2015 Amazon launched their Unlimited Drive Storage product.

This, for a certain kind of person that you can find on the data hoarders subreddit was a personal challenge as you can imagine.

The screenshot on the left was sent to me from that reddit and all the data (1 Peta byte of it) was uploaded with rclone.

Rclone was banned from Amazon Cloud Drive, probably as a consequence of this, though ostensibly it was the use of encrypted keys within the binary. Amazon also shut down the unlimited Drive product in the US

I was surprised to have an interview with a journalist from The Register.

Rclone is still banned from Amazon Cloud Drive unless you have your own keys.

How does it work?

I’ll now move on from the history to how it works.

There are a number different functionalities of rclone. Each of these is implemented as a sub command otherwise known as a verb of the main program.

You write the verbs after the command so “rclone config” or “rclone sync”. More parameters can come after the verb.

For data transfer the main 3 verbs are sync, copy and move.

The first command you run after installing rclone is “rclone config”.

This starts up the old school config wizard.

I think I first saw this style of config wizard in perl when configuring CPAN.

It isn’t the prettiest in the world but it gets the job done!

When it needs to do oauth it opens up your browser for you and redirects back to a local webserver to receive the results.

The next thing you are likely to want to do is copy some files.

Here is a quick demo of some files being copied using “rclone copy” and “rclone ls” to list the files.

Always start with “rclone copy” - don’t use “rclone sync” as that can delete files in the destination.

This shows the use of rclone sync.

Rclone sync differs from copy in that it makes the source and destination identical.

This means that it copies files to the destination and deletes any files in the destination which should be there.

Always try with –dry-run first!

Here is rclone copy in diagram format.

On the left are the source and destination before the command and on the right after the command.

You can see that it can copy a file, ignore an unchanged file, overwrite a stale file and that it doesn’t touch files which exist in the destination already but aren’t in the source.

Whereas rclone sync’s job is to get the source and destination identical

You can see that it can copy a file, ignore an unchanged file, overwrite a stale file and that it deletes files which exist in the destination already but aren’t in the source.

Rclone sync, copy and move are all implemented by the same routine as most of the code is identical between them.

Rclone mount is one of my favourite features. Mount any backend supported by rclone as a FUSE mount.

This works on linux, mac and Windows.

Rclone supports seeking on the files too so you can play video quite well from your cloud storage system and I know that lots of rclone users do exactly this.

However if you want to open a file for both read and write or seek while writing then rclone will need to download the file and work on a cached copy. This is part of the VFS layer in rclone which adapts rclone objects to work more like an actual file system.

Here is another of my favourite features.

It is a blatant rip-off or homage as I prefer to call it of the ncdu tool. If you’ve never used it I recommend you install it right away

It is the best tool for answering the question “Where is all my disk space?”. It lets you explore your local disk using a really easy to use curses interface.

I had the idea of implementing that in rclone and I needed something to do on a plane journey so hence rclone ncdu was born.

It uses the excellent termbox library to implement the text based user interface.

Dive into code

Now let’s dive into a bit of code.

Rclone implements its backends using interfaces which won’t be a suprise. This shows the interface which all rclone backends must fulfil.

Rob Pike tells us: “The bigger the interface, the weaker the abstraction” - this is a big interface :-(

It is broken down into 3 sub parts in the rclone source but I’ve presented the whole interface here as it all as to be implemented for a backend.

You can see the kind of methods you might expect, List, NewObject, Put and Mkdir.

There are also some missing from that list which I’ll cover in a moment, like Copy.

This is the interface a cloud storage object must fulfil.

Again pretty straight forward. Note the Hash method which you call with one of the types the Hashes method in the Fs object returns.

Open reads the object and Update sets its contents.

You can also read the modification time and set the modification time – something that is very important for rclone syncing.

All the backends support the interfaces in the previous two slides. However some backends can do more, and rclone can use that.

For example the Purge interface deletes a directory and everything under it. Some backends can do that and some can’t.

For example, say you were doing a server side move of an object. Some backends can do that immediately with the Move method (for example Google Drive). Some support Copy but not Move (for example S3). Rclone will Copy the object then Delete it. And some don’t support either (eg B2) in which case rclone will download the object and upload it again.

Here is how you might use those optional interfaces – with an type assertion. That is straight forward go code.

However, rclone backends can wrap other backends, for example the encryption backend. The outer backend wants to expose what the inner backend can do so must implement all optional methods. The inner backend may not implement all optional methods so we’ve got a problem.

The traditional solution to this is a sentinel error which we check. Here we call Purge and check the err value against ErrCantPurge.

This works very well, provided you don’t want to know which methods are used in advance. Unfortunately rclone does need to know….

The solution to that is a good old fashioned struct full of function pointers. This is filled in with a bit of reflection so doesn’t take any manual work to keep up to date.

Backends which wrap other backends logical AND their own features with the features of the wrapped backend to make the final Features struct.

To use Purge, say, you just check to see if the pointer is nil. If it isn’t nil that means the method is supported so you can call it.

Those of you who’ve used C++ will realise this is a re-implementation of the vtable or virtual table. Go made the implementation straight forward though.

Testing

Lets go into how rclone is tested.

As you saw earlier rclone supports lots of providers, it has lots of commands, architectures OSes. That all makes it really hard to test.

Anyone think writing mocks for 27 different cloud providers would be much fun for a hobby project? No?

So rclone does lots of integration tests. These give real confidence that rclone works properly and keeps on working. After all cloud providers often change stuff.

Rclone has a full continuous integration pipeline.

I think we are amazingly lucky as open source programmers to be able to use Github, Travis and Appveyor for free – they are fantastic tools.

The CI tests run all the non integration tests and builds binaries for all the platforms, uploading them for beta release automatically for every branch pushed to the main repository.

You’ll find those at beta.rclone.org

The integration tests run on a separate server.

These run the tests, which are just normal go unit tests with some extra flags against all the backends.

Running these tests costs money and takes a long time so I don’t run them against every pull request, only against the master branch and only once per day.

The integration tests produce a report for the developers to look at.

This was a good day with only one failure out of 105 tests.

Often there are lots of failures, usually because of some cloud provider glitch, but sometimes because of a change in the rclone code.

Making a pretty report (well that is pretty by my standards) wasn’t high on my priority list. However I completely messed up one of the releases by misreading the old report, so I made a report which couldn’t be misunderstood.

The integration tests rclone uses are standard go tests. It is easy to add flags to the go tests and rclone adds a “-remote” flag to tell it to run the tests against an actual backend.

Unlike unit tests, integration tests often fail. The internet is a messy place – things go wrong. The test_all framework re-runs the tests that fails to try to get them to pass.

I should probably take the time to open source this framework as it could be useful elsewhere!

In go1.6 the concept of nested tests was introduced.

This made rclone’s life much easier. As you can see it is only a few lines of code to run the integration test suite for a backend. Some backends run the suite more than once with different parameters.

When making a new backend, you can use the integration tests as a guide for what you need to work on next and it makes the process quite enjoyable.

Pre go1.6 code generation was used to make the integration tests which was rather painful so thank you to the Go team for making nested tests.

This is something that not many people know, but can make your life much easier.

You can add flags to tests. You don’t need to do anything special, just define flags with the standard library flag package.

There are some great examples of this in the standard library, for instance using a “-golden” flag to make the tests files.

Rclone uses flags to indicate that tests should run with a real backend, and to help debug those tests.

Rclone is a reasonably large program now, approaching 100 thousand lines of Go code.

However it is built on a fantastic set of libraries some of which I’ll talk about in a moment.

And all of that in turn is built on the go standard library.

Rclone uses the vendor directory to keep that all straight, and we recently switched to “go mod” to manage that.

I don’t think we can drop the vendor directory just yet as rclone needs to support old go versions for the Linux distros that package rclone. Maybe in a year or two.

Tools

I’ll just run through a few of my favourite libraries and tools that rclone uses.

There are loads of other libraries I could have mentioned here so apologies to any authors I’ve missed out.

My first choice is goimports. Install it in your editor and never type an import statement again.

You can see me editing a small file in emacs here and the import statements being automatically added while the code is formatted for you. It works in vim too so I’m told ;-)

Cobra is excellent for organising command line programs with lots of verbs.

I’m sure many of you have used kubectl to control kubernetes or used docker – that is cobra in action.

Cobra comes with a flag library pflag which allows you to create POSIX flags. Flags made by the go flag library are somewhat old school in comparison.

Cobra also creates your bash completion script for all those people like me who use bash, but hate writing bash scripts!

Cobra helps with the documentation. When you write a command cobra makes command help for it and then has a really easy way of turning that into markdown for the website.

A lot of the help on the rclone website was generated this way and it really helps keeping everything consistent.

I then use pandoc to turn the markdown into a man page and one big HTML document for the download.

If you’ve been keeping up with go news you’ll have read the errors proposal for go 2.

The pkg/errors package was one of the motivations for the proposal, so I imagine at some time I’ll be retiring it when go 2 comes along.

You use it to annotate your errors as they bubble up through your code.

This means that rather than receiving a log message with “unexpected EOF” you receive a helpful message which gives you some idea as to what really went wrong.

Conclusion

I’m just going to talk briefly about what to do if your open source package takes off.

It is easy to get daunted by 100s of issues most of which are support requests so the first thing to do is open a forum and get the users helping themselves. I use Discourse – it is free and easy to setup and maintain.

Don’t be shy of asking users who make issues for help. You might need to give them a bit of a helping hand, but that is a positive process for both parties.

When you’ve had a few good pull requests from someone, recruit them!

Oh yes, write lots of docs – maybe that should have gone first.

Finally get octobox.io – rather than managing 100s of emails from Github manage them in a nice web interface where you can filter etc.

Thank you all for listening :-)

Xmas 2016

nick@craig-wood.com (Nick Craig-Wood) — Sun, 25 Dec 2016 00:00:00 +0000

In March all 6 of us piled in the trusty people carrier and drove to a friend’s self catering ski chalet in Les Arcs. We had fun skiing and finding the many uses of WiFi on holiday. Nick and Dougal went skiing together every day logging their top speeds on the resort app. Amy found 3 friends in the resort. Ed slept a lot, snowboarded a bit and broke nothing! Issy did a small amount of skiing but mostly watched Netflix. Loveday learned to crochet thanks to YouTube (see if you can find it in the picture).

The summer saw Nick taking July off work which we mostly spent in Cornwall on a DIY marathon. At the end of August we did our next great drive. We started out by going to Reading Festival to see “twenty one pilots” leaving there late to drive to Stafford, slept for 4 hours then carried on up to Ullapool to catch the ferry to Stornaway on Lewis and thence to Valtos. The next day was glorious and the two scenes at the top of the picture are from that day. The weather went downhill, but we enjoyed: seeing standing stones; lots of kite flying; a happy day tie dying in the garden; a boat trip and seeing seals, black houses and iron age dwellings. This holiday we enjoyed not having WiFi or phone signal unless standing in the sea where you can see the unprecedented sight of Issy talking to her friend on her phone.

Amy (20) is in her 3rd year at uni doing a year in industry. She managed to land herself a great job just before the deadline as a front-end developer for Dusted, a digital agency in London. She enjoys photography and reluctantly lives at home.

Ed (19) now has two jobs, one at the café at the sports centre, the other at Topman. He is still trying to work out what he is doing next but his parents are pleased he is gainfully employed.

Isabel (15) has had an unusually active year, getting a surf board for her birthday (“Doris”), and doing her Duke of Edinburgh award which involved actually walking for really quite a long time. She is trying to watch every TV show in existence in her spare time and has almost succeeded. She has also done some homework.

Dougal (9) along with learning to ski, has also learnt to unicycle (wobbling in father’s wheel track). He has been enjoying football, rugby and cricket at school. In Cornwall he has really got into surfing, taking over Ed’s old board (see picture).

Nick has been changing role at work trying to do less management and more technical strategy which has been a welcome transformation. He has also continued art classes and writing open source software (eg rclone).

Loveday has been busy. The management of Treharrock has required lots of trips to Cornwall and many hours sorting out the waste contract. She has started an upholstery course which is a very expensive way of getting second hand furniture! She has also finished knitting a jumper for Issy and even better it still just about fits.

Snake Puzzle Solver

nick@craig-wood.com (Nick Craig-Wood) — Sun, 25 Sep 2016 00:00:00 +0000

My family know I like puzzles so they gave me this one recently:

Fork me on Github

The code for this along with a Jupyter notebook for it is on github.

When you take it out the box it looks like this:

And very soon after it looked like this (which explains why I've christened the puzzle "the snake puzzle"):

The way it works is that there is a piece of elastic running through each block. On the majority of the blocks the elastic runs straight through, but on some of the it goes through a 90 degree bend. The puzzle is trying to make it back into a cube.

After playing with it a while, I realised that it really is quite hard so I decided to write a program to solve it.

The first thing to do is find a representation for the puzzle. Here is the one I chose:

# definition - number of straight bits, before 90 degree bend
snake = [3,2,2,2,1,1,1,2,2,1,1,2,1,2,1,1,2]
assert sum(snake) == 27

If you look at the picture of it above where it is flattened you can see where the numbers came from. Start from the right hand side.

That also gives us a way of calculating how many combinations there are. At each 90 degree joint, there are 4 possible rotations (ignoring the rotations of the 180 degree blocks) so there are:

>>> 4**len(snake)
17179869184

17 billion combinations. That will include some rotations and reflections, but either way it is a big number.

However it is very easy to know when you've gone wrong with this kind of puzzle - as soon as you place a piece outside of the boundary of the 3x3x3 block you know it is wrong and should try something different.

So how to represent the solution? The way I've chosen is to represent it as a 5x5x5 cube. This is larger than it needs to be but if we fill in the edges then we don't need to do any complicated comparisons to see if a piece is out of bounds. This is a simple trick but it saves a lot of code.

I've also chosen to represent the 3d structure not as a 3d array but as a 1D array (or list in python speak) of length 5 x 5 x 5 = 125.

To move in the x direction you add 1, to move in the y direction you add 5 and to move in the z direction you move 25. This simplifies the logic of the solver considerably - we don't need to deal with vectors.

The basic definitions of the cube look like this:

N = 5
xstride=1    # number of pieces to move in the x direction
ystride=N    # number of pieces to move in the y direction
zstride=N*N  # number of pieces to move in the z direction

In our list we will represent empty space with 0 and space which can't be used with -1:

empty = 0

Now define the empty cube with the boundary round the edges:

# Define cube as 5 x 5 x 5 with filled in edges but empty middle for
# easy edge detection
top = [-1]*N*N
middle = [-1]*5 + [-1,0,0,0,-1]*3 + [-1]*5
cube = top + middle*3 + top

We're going to want a function to turn x, y, z co-ordinates into an index in the cube list:

def pos(x, y, z):
    """Convert x,y,z into position in cube list"""
    return x+y*ystride+z*zstride

So let's see what that cube looks like:

def print_cube(cube, margin=1):
    """Print the cube"""
    for z in range(margin,N-margin):
        for y in range(margin,N-margin):
            for x in range(margin,N-margin):
                v = cube[pos(x,y,z)]
                if v == 0:
                    s = " . "
                else:
                    s = "%02d " % v
                print(s, sep="", end="")
            print()
        print()

>>> print_cube(cube, margin = 0)
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 

-1 -1 -1 -1 -1 
-1  .  .  . -1 
-1  .  .  . -1 
-1  .  .  . -1 
-1 -1 -1 -1 -1 

-1 -1 -1 -1 -1 
-1  .  .  . -1 
-1  .  .  . -1 
-1  .  .  . -1 
-1 -1 -1 -1 -1 

-1 -1 -1 -1 -1 
-1  .  .  . -1 
-1  .  .  . -1 
-1  .  .  . -1 
-1 -1 -1 -1 -1 

-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1 
-1 -1 -1 -1 -1

Normally we'll print it without the margin.

Now let's work out how to place a segment.

Assuming that the last piece was placed at position we want to place a segment of length in direction. Note the assert to check we aren't placing stuff on top of previous things, or out of the edges:

def place(cube, position, direction, length, piece_number):
    """Place a segment in the cube"""
    for _ in range(length):
        position += direction
        assert cube[position] == empty
        cube[position] = piece_number
        piece_number += 1
    return position

Let's just try placing some segments and see what happens:

>>> cube2 = cube[:] # copy the cube
>>> place(cube2, pos(0,1,1), xstride, 3, 1)
>>> print_cube(cube2)
01 02 03 
 .  .  . 
 .  .  . 

 .  .  . 
 .  .  . 
 .  .  . 

 .  .  . 
 .  .  . 
 .  .  .

>>> place(cube2, pos(3,1,1), ystride, 2, 4)
>>> print_cube(cube2)
01 02 03 
 .  . 04 
 .  . 05 

 .  .  . 
 .  .  . 
 .  .  . 

 .  .  . 
 .  .  . 
 .  .  .

>>> place(cube2, pos(3,3,1), zstride, 2, 6)
>>> print_cube(cube2)
01 02 03 
 .  . 04 
 .  . 05 

 .  .  . 
 .  .  . 
 .  . 06 

 .  .  . 
 .  .  . 
 .  . 07

The next thing we'll need is to undo a place. You'll see why in a moment.

def unplace(cube, position, direction, length):
    """Remove a segment from the cube"""
    for _ in range(length):
        position += direction
        cube[position] = empty

>>> unplace(cube2, pos(3,3,1), zstride, 2)
>>> print_cube(cube2)
01 02 03 
 .  . 04 
 .  . 05 

 .  .  . 
 .  .  . 
 .  .  . 

 .  .  . 
 .  .  . 
 .  .  .

Now let's write a function which returns whether a move is valid given a current position and a direction and a length of the segment we are trying to place.

def is_valid(cube, position, direction, length):
    """Returns True if a move is valid"""
    for _ in range(length):
        position += direction
        if cube[position] != empty:
            return False
    return True

>>> is_valid(cube2, pos(3,3,1), zstride, 2)
True

>>> is_valid(cube2, pos(3,3,1), zstride, 3)
False

Given is_valid it is now straight forward to work out what moves are possible at a given time, given a cube with a position, a direction and a length we are trying to place.

# directions next piece could go in
directions = [xstride, -xstride, ystride, -ystride, zstride, -zstride]

def moves(cube, position, direction, length):
    """Returns the valid moves for the current position"""
    valid_moves = []
    for new_direction in directions:
        # Can't carry on in same direction, or the reverse of the same direction
        if new_direction == direction or new_direction == -direction:
            continue
        if is_valid(cube, position, new_direction, length):
            valid_moves.append(new_direction)
    return valid_moves

>>> moves(cube2, pos(3,3,1), ystride, 2)
[-1, 25]

So that is telling us that you can insert a segment of length 2 using a direction of -xstride or zstride. If you look at previous print_cube() output you'll see those are the only possible moves.

Now we have all the bits to build a recursive solver.

def solve(cube, position, direction, snake, piece_number):
    """Recursive cube solver"""
    if len(snake) == 0:
        print("Solution")
        print_cube(cube)
        return
    length, snake = snake[0], snake[1:]
    valid_moves = moves(cube, position, direction, length)
    for new_direction in valid_moves:
        new_position = place(cube, position, new_direction, length, piece_number)
        solve(cube, new_position, new_direction, snake, piece_number+length)
        unplace(cube, position, new_direction, length)

This works by being passed in the snake of moves left. If there are no moves left then it must be solved, so we print the solution. Otherwise it takes the head off the snake with length, snake = snake[0], snake[1:] and makes the list of valid moves of that length.

Then we place each move, and try to solve that cube using a recursive call to solve. We unplace the move so we can try again.

This very quickly runs through all the possible solutions:

# Start just off the side
position = pos(0,1,1)
direction = xstride
length = snake[0]
# Place the first segment along one edge - that is the only possible place it can go
position = place(cube, position, direction, length, 1)
# Now solve!
solve(cube, position, direction, snake[1:], length+1)

Wow! It came up with 2 solutions! However they are the same solution just rotated and reflected.

But how do you use the solution? Starting from the correct end of the snake, place each piece into its corresponding number. Take the first layer of the solution as being the bottom (or top - whatever is easiest), the next layer is the middle and the one after the top.

After a bit of fiddling around you'll get...

I hope you enjoyed that introduction to puzzle solving with computer.

If you want to try one yourselves, use the same technique to solve solitaire.

Download the entire code snake_puzzle_solver.py or the iPython notebook.

Xmas 2015

nick@craig-wood.com (Nick Craig-Wood) — Fri, 25 Dec 2015 00:00:00 +0000

We brought in this year by spending Christmas and New Year at Polzeath beach in Cornwall where we had a lovely time with family and friends. It was our last Christmas with Loveday's mother Elizabeth who died of pancreatic cancer in February after a 4 month illness. She was cared for at home by her children until the end which was an intense but rewarding experience.

After that we needed a holiday so all 6 of us spent Easter in Barcelona where we had good weather and Nick got to see the Dali museum which has been a lifelong ambition. Not all the children were terribly impressed by the culture! Soon after we started rebuilding our garage, attaching it to the house with extra rooms above, so imagine the rest of the year to the background of builders. The builders have done a great job and finished just in time for Christmas.

In July 5 of us spent 2 weeks in Turkey near Marmaris mostly relaxing with a bit of watersport and a few visits to local highlights such as the mud baths. Dougal learnt to sail, windsurf and waterskii all in two weeks which was amazing. Later in the summer we had a really enjoyable week at Treharrock with university friends and their families.

Amy (19) is in her second year at Brunel University. She is enjoying leaving projects to the last minute and, having swapped 4 legs for 4 wheels, driving between home, uni and boyfriend. Her passion for horses is being kept alive at university where she is captain of the riding team. This seems to mostly involve lots of meetings and sorting out arguments. She is currently working at PC World selling cameras and declining unwanted offers of marriage.

Ed (18) and school had a mutually agreeable parting of the ways after his AS results and he is now studying at the university of life. He spend a few weeks as a builder on our building site and gave the garage a thorough christening by inviting 100 people for his 18th party. He is now working full time at Côte restaurant as a commis waiter where he enjoys visits from his family. He still loves skateboarding despite having broken his collarbone while going off piste down a grassy bank in the spring.

Isabel (14) became the second member of the family to help organise a Quaker event (YPP). She loves these events and has been to several this year. She continues her iPad addiction and hasn’t stopped complaining about the consequent WiFi curfew! She enjoyed bringing her friends to Cornwall and has been making good progress at school.

Dougal (8) is really getting stuck in at his new school proving himself a determined football and rugby player. At home his new hobby is weaponising stationery and using Isabel for target practice.

Loveday and Nick have both been extremely busy in 2015. Nick has being doing more management at Memset but has preserved his sanity with art classes and a writing open source software. Loveday has overseen the building work, had many trips to Cornwall and excitingly taken on the management of Treharrock with a bit of Yoga to de-stress.