Because 7 ate 9.

No, seriously, why is color #6 so close to color #7 ?

I mean, if you look at the decimal values for the three colors of 5, 6, and 7, you have:

N .. Red . Grn . Blu
05 . 095   087   079
06 . 194   195   199
07 . 255   255   255

You can calculate what color #6 should be based on taking each of the color guns and subtracting the difference from 7 and 5 adding the division by 2.

You would get:

N .. Red . Grn . Blu
05 . 095   087   079
06 . 175   171   167
07 . 255   255   255

So why the disparity ? And why are "gray" not matched all across the board ? If you median them out, (adding the 3 together and dividing by 3), you would have:

N .. Red . Grn . Blu
05 . 087   087   087
06 . 171   171   171
07 . 255   255   255

So why the current color disparity between grays ?

Cause a good palette is not just a linear ramp of colors (certainly not rgb for that matter)

Well, how do you make a gray ramp then, where it doesn't jump like that ?

dither or use the palette differently. Color theory helps, and you wont find that in programmer manuals.

Came across this which looks interesting:

https://coolors.co/browser/latest/1

I'm afraid there's no magic 16-colour palette "to rule them all". We have got only a few fixed colours to work with and I'm sure they have been chosen very carefully and thoroughly by Zep to fit most games/cartridges. It's explained in the video Zep talk: PICO-8 and the Search for Cozy Design Spaces. Of course, "we could have" a built-in function to choose each palette entry at runtime from a full RGB24/RGB16/... palette, but I'm afraid Pico8 palette is one of the disctintive harsh limitations the following phrase refers to: "The harsh limitations of PICO-8 are carefully chosen to be fun to work with, encourage small but expressive designs and hopefully to give PICO-8 cartridges their own particular look and feel."