Tuesday, 22 December 2015

ZFS, like a work of art

A subtle thing with ZFS is you'll notice how the drive L.E.D.s flash quite differently to typical storage arrays, when you understand more under the hood you'll know why that is. So just looking in a DC you'd be able to observe this across which servers for example. You can see this type of effect here to illustrate - https://www.youtube.com/watch?v=LS3cfl-7n-4

ofc thats ZFS on linux.. which is implemented as a FUSE so less efficent than that of a FS in kernel space as elaborted across various posts, some examples: https://lkml.org/lkml/2007/4/16/133 , https://lkml.org/lkml/2007/4/16/83

example pool using raidz2 with hot spares, which will autoreplace in the event a drive or 2 fail. Creating with brackets like this is always easier - c4t{0..1}d0. Also have to get the order of commands to be correct or you may be second guessing...

# zpool create data c0t50004CF210AD1C22d0 c0t50004CF210BE51F1d0 c0t50004CF210BE51F3d0 c0t50004CF210BE5214d0 c4t{0..1}d0 raidz2
Unable to build pool from specified devices: invalid vdev specification: raidz2 requires at least 3 devices

# zpool create -o atime=off -o compress=lz4 data raidz2 c0t50004CF210AD1C22d0 c0t50004CF210BE51F1d0 c0t50004CF210BE51F3d0 c0t50004CF210BE5214d0 c4t{0..1}d0
# zpool add data spare c4t3d0 c5t3d0
# zpool status
  pool: data
 state: ONLINE
  scan: none requested

        NAME                       STATE     READ WRITE CKSUM
        data                       ONLINE       0     0     0
          raidz2-0                 ONLINE       0     0     0
            c0t50004CF210AD1C22d0  ONLINE       0     0     0
            c0t50004CF210BE51F1d0  ONLINE       0     0     0
            c0t50004CF210BE51F3d0  ONLINE       0     0     0
            c0t50004CF210BE5214d0  ONLINE       0     0     0
            c4t0d0                 ONLINE       0     0     0
            c4t1d0                 ONLINE       0     0     0
          c4t3d0                   AVAIL  
          c5t3d0                   AVAIL

Then as always test the assumption and it works as expected. I've got hot swap capabilities so pulled a drive out to simulate then try write some data and looks to have worked.

# zpool status -xv
  pool: data
 state: DEGRADED
status: One or more devices are unavailable in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or 'fmadm repaired', or replace the device
    with 'zpool replace'.
  scan: resilvered 136K in 1s with 0 errors on Wed Dec 23 05:53:44 2015


    NAME                         STATE     READ WRITE CKSUM
    data                         DEGRADED     0     0     0
      raidz2-0                   DEGRADED     0     0     0
        c0t50004CF210AD1C22d0    ONLINE       0     0     0
        c0t50004CF210BE51F1d0    ONLINE       0     0     0
        spare-2                  DEGRADED     0     0     0
          c0t50004CF210BE51F3d0  UNAVAIL      0    24     0
          c4t3d0                 ONLINE       0     0     0
        c0t50004CF210BE5214d0    ONLINE       0     0     0
        c4t0d0                   ONLINE       0     0     0
        c4t1d0                   ONLINE       0     0     0
      c4t3d0                     INUSE  
      c5t3d0                     AVAIL  

device details:

    c0t50004CF210BE51F3d0      UNAVAIL       too many errors
    status: FMA has faulted this device.
    action: Run 'fmadm faulty' for more information. Clear the errors
        using 'fmadm repaired'.
       see: http://support.oracle.com/msg/ZFS-8000-FD for recovery

Saturday, 12 December 2015

Night Shifts....

For several years now I've had to do night shifts, it isn't something I ever wanted to do as I understand it is simply not good for your health. (neither is sat in a chair for almost 12 hours a day or night either) but in this industry it is common and in my case it is part of the job so overall it seems to be the best choice. Can't always get everything so just picking and choosing what's important.

If you have to do night shifts here is how you can be better prepared and avoid some issues that I've faced in past experience.

When I first started I had trouble even remembering things, to the point that I'd forget stuff within minutes. I had problems remaining awake even with adequate sleep to the point that I had my head on the desk struggling and the desire to sleep was overpowering. Those things can and have lead to silly, unnecessary mistakes.

I was told early on "drink lots of coffee :)" and of course sleep enough in the day. Firstly I tried this several times (some tea and or coffee) then I was so tired one shift I decided it may be a good idea to eat raw instant coffee, that just tasted bad and this has limited effectiveness. One colleague told me over a beer that scientific studies comparing tea, coffee and sugar intake for tests to remain awake for long periods overall proved sugar is the most effective way to keep yourself awake. Tea/Coffee although having a half life of 6 hours (subject to types,volume etc) is only initially suppressing sleep as it is binding to receptors within the brain which ordinarily are done by the chemical adenosine which would be binding to neurons reducing activity making you feel drowsy. After the caffeine wears off those chemicals are still present  and with this increasing the firing rate of neurons activity may over time prove to be less effective. Long story short having sugar seems the most effective way to remain awake is this is your brain and bodies main fuel source, glycogen used as energy. If you have sugar in various forms it only stimulates you to help remain active. If you keep putting in fuel the engine still runs. I found it hard to believe at first as it meant I was wrong for a few years and its commonly showed even on TV, we need to stay awake so drink coffee... one episode from Star gate SG-1 springs to mind... When I've tried sugar instead I've kept awake easier and much longer with even less sleep prior. (was still a wreck though)

The sleep part is the vast majority of people have sleep cycles which are 90 mins but can range from 70-120 mins which is less common. Waking on an aligned sleep cycle is essential for the entire day. Being woken up mid cycle your just zombified and it doesn't go away sometimes no matter what you try. So most cases you want 4-6 of these cycles and wake right at the waking stages to feel much more awake that will last the entire day.

A key useful thing here that helps either in day or night if you are able is drink a tea or coffee then have a nap which ranges from a minimum 5 minutes up to a maximum of 20 mins. so 5,10,15,20 as a target. This way while you digest and process the effects of some caffeine you have reduced the amount of adenosine residing in your brain just after then the caffeine can kick in. Also a good time  to do this is in the middle of your waking hours (so midday or midnightish). That may look contradictory to the previous statement about sugar and tea/coffee but the point is to make some tests personally and find what works, so the thing here is to have and rely on less amount of caffeine and primarily increase sugar intake if you need to. The more coffee/tea you drink on a daily basis the lesser the effects are which is why you want to have it when it is required.

Get an alarm such as the natural Phillips wake-up light, I saw this first on gadget show and it uses natural simulated light to wake you up that gradually increases in luminosity which is much more preferable to loud beeping, repetitive alarms. and always force yourself to wake on the alarm and not snooze it! That conditions your brain to not react to this and you go back into a sleeping state making it harder to subsequently awake. I always have the primary wake up light alarm, and then a backup alarm in case for any reason it does not go off. Best alarm I've ever used. If you time this right you should just wake up properly, if it is hard to get out of bed this implies the sleep cycles you set haven't aliened up right.

This other point was useful brought to my attention too: buy a blackout blind. That helped as during the day this completely stops any sunlight getting through and are relatively cheap and easy to install. For me it helped a bit, other people who invested in it said it made a great difference. We as people having seen sunlight naturally causes us to awake by us having chemical reactions take place in the brain, likewise when it gets dark and we see this we again react to make ourselves feel more tired/sleepy. Even if your in bed eyes shut some light shining inside will have effects which is why you want to limit it or cut it off outright as you can still detect it to some degree.

Key pointed summary.

  • Have some caffeine if needed, but primarily you want more sugar.
  • Buy and use a blackout blind to cut off sunlight when sleeping in the day
  • Get a naturally sun simulated wake up alarm, wake on first alarming!
  • Drink a tea/coffee then nap of 5-20 mins as temp. boost. 
  • Target and align for 4-6 of 90 min sleep cycles and wake on lightest sleep state.
  • Experiment to find what works and make minor adjustments, gradually.