PICO-8 keeps on throwing errors at me saying that I can't change my directory!
I've uninstalled and re-installed, but still no dice.

Whoops, I posted too soon.
I somehow managed to fix it by deleting the pico-8 folder in C:\Users\yourname\AppData\Roaming\ and running pico-8 again.

I had some similar problems once ... but it was after running a certain cartridge and I think this is among the fixes for 0.1.4 but I might be wrong.