Sunday, July 27, 2014

Runners Manifesto -- draft

It seems like the running community needs something along the lines of what the Hacker Manifesto was meant to be, except for running. There are many articles describing running to runners but perhaps this can be a document to introduce non-runners to runners... or even a document for runners to cherish. 


Hi, I am a runner. I may not run faster than you, but I might run a lot further. Or maybe I can run more consistently every morning or evening. Or maybe I just enjoy lacing up a pair of shoes, or running barefoot.

First, let's be clear. I don’t run away from things, I run towards them. I make goals, have dreams and connect with other runners because we have a fundamental understanding of each other. We like to run.

Why do I run? Well, why do you breath? Running is kinda like breathing. Yeah, I could stop for a while but I would start getting light headed and dizzy… kinda like before a marathon. You really don’t want to bother me then.

I, like so many others, used to not understand runners. In high school, I played soccer and I could run. Im college, I biked around quite a bit. It wasn’t until life forced me to discover new things that I finally found my inner runner. Once I did, I realized that running fast was fun. Running far was fun. Trying to do both was dangerous… and also fun. Ultimately, neither was fundamental to what I did on a daily basis but somehow running becomes a way of life.

I was lucky when I started; I met a group of runners who had run a long time. There were wise old owls to give sage advice and young yet dedicated runners to keep the energy up. Just about everyone could whup my butt except I found that it didn’t matter. We were all friends and supported each other in our goals. Were we running the same race? Cool! Let’s get together and keep each other motivated to do better. Running similar races? Awesome … let’s get together and help keep each other motivated. Completely different races? Let’s get together and keep each other motivated. Essentially, I walked into what was a hidden community that hides in plain sight and supports itself.

What defines a runner? For me, its someone who gets out there and pushes themselves to achieve their goals. It may start with running and seeing a 5k to accomplish, and then trying to beat their 5k time. Maybe it takes you twice as long to do a 5k as me, that means you are working for twice as long to do that and I respect that. Maybe you respect that I can do it twice as fast. Either way, you rock for doing it.

If you stop running, are you still a runner? Yeah, I think so. Running is a way of life, not an activity you do. I’ve met people who used to run and I can tell they are runners. They say they used to be runners but I know better. They still run, just not on their feet.

If you are a non-runner, that’s cool too. Please don’t ask why I run and I won’t ask why you don’t. Also, if you want to challenge me to a race, that’s awesome. Come and train with me and we’ll run whatever I’ve signed up for next.

See you out there!

Wednesday, March 19, 2014

Host configuration -- use the hostname to configure the network

As I imagine many people to be, when bringing up hosts, I'm still stuck in the days of:

Step 1: (node) configure networking
Step 2: (node) configure the hostname
Step 3: (service1) configure DNS to match
Step 4: (service2) plug this into some form of configuration management
Step 5: profit

The problem is that you often have a complicated back and forth between configuration of the node and configuration of the node. If you decide to semi-automate this you might try and add dhcp into the mix:

Step 1: (service1) Allocate IP address for a node
Step 2: (node) Get MAC address from node
Step 3: (service2) Plug MAC+IP into DHCP configuration
Step 4: (service1+service2) Push out changes to dhcp/dns
Step 5: (node) initialize networking using DHCP
Step 6: profit

Adding in IPv6 and SLAAC, things get worse since you have to grab the SLAAC address after networking is brought up and that means two different changes to DNS. Pretty soon, you are spending several minutes just moving basic data between services on your network.

ENTER IPv6 and a slight amount of thought:

I recently had some other push factors (related to our private cloud) to try and minimize the effort spent here. I now use SLAAC and an IPv6 DNS address as a temporary configuration point. Then I query DNS for the information I need to configure the node. This basically just comes down to IPv4 and IPv6 address at the moment:

Step 1: (service1) Allocate IPv4 and IPv6 address
Step 2: (node) bring up node with temporary (SLAAC) config
Step 3: (node) query DNS for addresses and plug the values into a static configuration
Step 4: profit

Now all I need is a script on the new node to make this much faster and viola! My configuration is down to two steps:

Step 1: (service1) Allocate addresses
Step 2: (node) run: init_host new-hostname
Step 3: profit

Of course, init_host runs dig and plugs values into the key places. Finally, it runs some form of configuration management (puppet in our case) to get the rest of the host configured. Since many hosts are virtual instances under OpenStack, we can simply leave an init_host script in the base image for convenience. Time taken to bring up a new node has gone from minutes of error-prone copy/paste to seconds of error-prone typing. I'm much more likely to be careful over a period of seconds with a few steps than a period of minutes with many steps. Hopefully our infrastructure will benefit ... and hopefully yours will too!

Monday, March 3, 2014

Napa Valley Marathon - 2014

I live in a beautiful city that is practically ideal for runners in the winter. There are long flat areas to run as well as hilly and mountainous regions to train in depending on your goals. There is also an abundance of sunshine and ever so slight lack of oxygen at 2500' above sea level. To clarify, it's an ideal place for winter training for an early spring marathon since the summer gets pretty hot.

This training season has been non-ideal in many ways (stress fracture which meant one month with no running) and very little speed/strength training. I made my focus entirely on my long runs which I was just barely able to schedule before the marathon with two 20 mile runs on two successive weekends. The last one was on flat-as-a-pancake pavement with two weeks (one weekend) worth of tapering before the marathon. Given my somewhat ad-hoc training plan, I didn't have a solid goal pace in mind and was planning on winging it somewhat.

On race weekend, my diet varied from very decent hotel breakfasts to dark chocolate peanut butter cups. Suffice to say, I should have been a little more regimented with my food. Pre-race dinner wasn't bad but I probably could have done better by adding a potato somehow for potassium. I had chicken, rice and veggies. I did manage to avoid wine and drank plenty of water throughout the day so that I would be able to spend less time worrying about hydration during the run. In hindsight, I may have had a little too much water and not enough salt/potassium to stay balanced.

Ok, ready, set, go! I woke up on race morning with very little sleep. My neighbors in the hotel decided to come back at 2 am in the morning the previous night and have an hour long conversation. This night wasn't as bad since they started at midnight instead of 2 am. Of course, I had to wake up at 4:00 so I could leave by 4:30 to walk to the busses that left the finish area at 5:15 for the race that started at 7. To be fair to the race directors, I think this was just enough time for all logistics to be worked out for the crowd but I did spend 30 minutes on the bus after we arrived waiting to be kicked off. I think I will try to stay in Calistoga (close to the start) instead of Napa for those precious extra few hours of sleep if I run this again.

Chatter on the bus while driving up was the usual range of topics between runners. "This is my first marathon" ... "doesn't matter if it's your first, your 50th or you are 20 or 60, the feeling at the beginning is always the same" ... "does anyone know what the weather is supposed to be?" Ok, the last one may not be as typical on race day but the weather report had varied wildly in the past week. The forecast went from "no rain" on race day to "it's gonna rain" and everywhere in between. The night before the race I checked and there was a 0% chance. I took that and planned for only slightly worse. Thankfully I did plan for slightly worse.

At the start, I saw some friends in passing and made a few new ones. I saw a guy standing on his own and decided to make simple conversation. Jim  would be happy for a sub-4 and hadn't run a marathon in a while. He asked what I was planning on and I let him know that I was hoping for 3:20 - 3:40 but was going to just see what happened. He seemed impressed and we continued chatting for a little bit. I saw another friend a little bit ahead, went to wish her good luck and then made my way back through the pack so I didn't get caught up in the rush of the start. I tend to go out to fast if I am too far in the front and so I compensate by starting a little further back and taking it easy for a mile.

After some poetry, the national anthem and the usual positive heckling of the announcer, we were off! I started further back than usual because people were walking all the way up to the starting line (which was chip timed.) I wasn't trying to PR so I did my best to just go with it and then gently pass people until I reached a slightly faster group. I did my usual identify people by race shirts they had on and say, "go Ragnar!" for people with Ragnar shirts. This worked well for a while until about mile 2 where I was able to hit my target range.

I caught up with some friends who were running the course for the heck of it. They originally signed up for the race but had injuries during training and thus planned on not doing it. The night before, there was wine and "good" decision making so the whole group decided to start the race and see what happened. I caught up to a few people in the group and passed them without realizing it was them. Once I had gone a little ways ahead they called me on it and caught up. We ran together for a short period and they definitely had their own conversation going. I decided to let the ladies talk while I keep focusing on my run which was starting to feel increasingly difficult.

I was breathing much harder than I would have at the same pace which seemed odd given that I was 2000' of elevation lower than usual. Eventually they pulled past and I started throttling my pace back. My body wasn't hurting or feeling tired but I was starting to feel a lot of mental resistance; Far more than should happen around mile 3 or 4. I couldn't really lock on what was problematic and so I equated the stress to negative thoughts and tried pulling myself out of them.

I passed Jim and he surprised me, "Hey Tim, way to go!" I had to look back a little to recognize him and cheered him on too. There was still a lot positive energy in the air and some surrounding runners chuckled at the exchange. I think he finished with, "I'll see you at the end!"

The spectators were great along the sides of the road. Nice posters and positive (but not over the top) comments for the passing runners. One group was even offering an entire line of high-fives. I think that was the fastest part of the course for me. I am a total sucker for that sort of thing and love to play into it. My gps log shows a single peak at 2 minutes faster than my target pace. I'm glad only one crowd was that organized with the high-fives.

I kept the pace, or at least I thought I was for a while. The resistance was getting unusually difficult to wade through. I tried everything to get positive thoughts flowing. I looked around at the beautiful grape fields and hills, enjoyed the cool temperatures, appreciated the runners around me, and came back to thinking of wine at the end. Just pushing through wasn't working either. Nothing helped. It's a downhill race with an uphill battle. Finally, I dropped my pace by two minutes for a half mile to try and reset a little bit. There was no way I could sustain this mentally even though my body was feeling mostly ok. Finally, I decided to walk. That's a tough decision for me, especially before mile 13.

The resistance I felt just would not give up. I started employing tactics such as 1 mile on and then a short walk/jog rest to try and break out of the head space. Once I figured out that I could walk, I started walking more and my overall pace was starting to really suffer. Jim caught up to me and asked what was wrong. I just kinda shook my head and said, "it's all mental." He knew exactly what I was saying and gave me an encouraging pat on the back. After talking for a little bit he continued on, "I'll wait for you at the finish!" At that point, a course motor cycle monitor looked back and asked for a thumbs up/thumbs down. I gave him a thumbs up but I noticed he took a moment to believe me and he didn't ask someone ahead of me that was also walking.

Finally, I suspected that this may be food related. I didn't plan on taking anything but salt pills and aid-station water which had worked fine for long runs in the past. There was an emergency GU in my pocket but I did not want to resort to that if I didn't have too. I started accepting gatorade and eventually bananas and even went back for seconds a few times. By mile 19, I was starting to feel a little less mental fatigue but all the walk breaks and cold had told my body, "ok, we are basically done." I then walked for half a mile trying to just focus on gradually increasing steam. I was hoping I could build up a little momentum and start into a jog that would eventually become a run again. I started an easy pace and decided that I had to hold it for a mile to see what happened.

I walked again for a brief period, started up and then it happened; My right calf started knotting up when I would land/push off. No longer was it just a question in my mind about running vs not, my body was enforcing that I could not run. By this point I had gotten past the, "maybe I should drop" thoughts and graduated to, "if I have to walk 6 miles then so be it." My mental state was getting much better but my body worse which reinforced the fuel-related diagnosis.

What was nice about passing mile 20 is that others were also hitting a wall. I wasn't alone anymore! There were people passing and walking ahead. Some people walked around me and someone even passed and said, "go Ragnar!" since I had my oldest Ragnar shirt on. That managed to bring a smile to my face.

I walked for two miles and tried again. Nope! Not happening. Potassium doesn't come from mid-air but rather from aid-station volunteers. Finally, an aid station! Lots of gatorade, water to help dilute it, a salt pill and as many bananas as I could hold with a cup of water. I gave myself 10 minutes to let the salt be absorbed into my system and let the food ease its way into a safe place. "Ok, let's try running again," I thought to myself. Nope. Nope nope nope! One more time? Nope. Fine, little more walking, but damnit, I'm making it!

In the the final stretch, once I got over all the people passing me, I found new motivation. If I couldn't run, I was going to walk as fast as I could. Constant forward motion was my goal. I tried running a few times and after analyzing the knots trying to form in my calf I finally figured out that if I used my heel a little more and my forefoot a little less, I could jog. Normally, this is not how I optimize my run. I wear minimal footwear and heel strikes are considered dangerous. The shoes I was wearing had several millimeters of padding, however, so I figured I could get away with it.

I thought to myself, "I'm going to make it, no matter what I need to do." The light at the end of the tunnel was starting to twinkle through slowly and the only thing keeping me from shedding a few tears of happiness were well placed crowds of people cheering on runners. They probably wouldn't have noticed given the light but constant rain that was falling by now. Runners themselves were saying motivating things as they pass. Sometimes it was for themselves via a mantra to keep going, other times it was definitely aimed at me. Either way, it was inspiring.

Ok, new goal with new information: I'm going to run through the finish. Oh, someone has homemade sorbet! I stopped and asked briefly about flavors. I finally chose the lightly colored one and it was SO GOOD. If the person who made that and stood out in the cold to serve sorbet is reading this, "Thank you! Something about it made me feel instantly better." I jogged out of sight and the cramps came back. So close! Walk for a moment... then start up again. Nope. Ok, let's walk and conserve a little energy. I'm running through this finish no matter what.

One spectator had measured 0.2 miles beyond the 25 mile post. She was trying to inspire people to keep going since there was exactly one mile. Again, being a sucker for spectators, I played into it, got a fist bump and started running. Oops, slowly, right ... cramps. Ok, I can do a mile! I can always do a mile!

With the finish area in sight, both of my calves were starting to cramp up even with my modified run. I finally decided I didn't care and just did whatever I had to in order to keep my body moving forward through the finish. I vaguely saw the finish clock in so much as to recognize a "4" as the first number... which I was well aware of anyway. I crossed all three finish matts and was greeted by someone immediately after I stopped running. She was looking me very intently in the eyes and I was trying really hard to recognize this person. Did I know her? Was she part of the group I came with? She had a lot of stuff on to stay warm/dry but I finally figured out she was trying to assess my state. I mentioned some cramping and she mentioned some medical tents. Someone placed a medal around my neck while another placed a heat blanket around my shoulders. The medal is cool but, hands down, it was the heat blanket that won in my mind.

This was definitely not the marathon I planned to run but there is no way you could get me to change a thing about it for the world. My time was an hour slower than anticipated, my body still hurts and I'm pretty sure I'll have a few nagging injuries for a while but the memories will last a lifetime.

Tuesday, February 25, 2014

OpenStack and Glance Image Types vs Size

I have looked around for good information on images and support for sparse storage and compression. After experimenting a bit I decided to compile a few key results. There are two attributes I care about here: storage and network transfer.

In order for efficient storage of an image, the image must be as small as possible. If there is a run of no data on an image (free space that has been filled with zeros) then that should not reserve bits on the filesystem. The key term here is "sparse image." Once you transfer a sparse image to another host, the "sparse bits" actually get transmitted as long strings of zeros... thus you transfer 10 GB of data for a 10 GB sparsely allocated image with (say) 6 GB of actual filesystem data.

In order to be efficient for a network transfer, the image has to actually be small which ends up meaning compression. (eg: gzip/bzip2/...)

I used an LVM backed instance for my source data and in order to fill zeros into the free space of the block storage I ran this inside the instance:

dd if=/dev/zero of=/tmp/zeros bs=4M; rm /tmp/zeros; halt

This basically just wrote zeros into a file under /tmp, removed it and then halted the instance. You may need to use a different location on your host because /tmp is sometimes mounted as a different filesystem (eg: tmpfs.)

Now, my lv is ready to convert to different image types. I used qemu-img in order to read from the block device to create images of different varieties:

qemu-img convert -c -O qcow2 /dev/vg/kvm-woof.snap host-woof.compressed.qcow2
qemu-img convert -O qcow2 /dev/vg/kvm-woof.snap host-woof.qcow2
qemu-img convert -O vdi /dev/vg/kvm-woof.snap host-woof.vdi
qemu-img convert -O vmdk /dev/vg/kvm-woof.snap host-woof.vmdk

The differences between sizes somewhat surprised me:

root@os-ph1:/staging# ls -lh host-woof.* 
-rw-r--r-- 1 root root 2.1G Feb 25 17:01 host-woof.compressed.qcow2
-rw-r--r-- 1 root root 6.3G Feb 25 16:52 host-woof.qcow2
-rw-r--r-- 1 root root  10G Feb 25 16:42 host-woof.raw
-rw-r--r-- 1 root root 6.9G Feb 25 16:49 host-woof.vdi
-rw-r--r-- 1 root root 6.3G Feb 25 16:49 host-woof.vmdk

A raw image is pretty much the worst thing you can do. It turns out that only the qcow2 format supports compression within the image itself. There are some downsides to qcow2 with performance but certainly if you are creating/destroying VMs a lot then you can save yourself some network bandwidth by transferring smaller images. Creating the compressed qcow2 image took roughly twice as long (no hard numbers here) as the uncompressed version. YMMV as not all data is created equal (or equally compressible).

It's also interesting to note that a compressed qcow2 image may slowly expand through use. New writes to the image may not be compressed.

-- EDIT --

I use LVM to back instances on hosts in OpenStack. It turns out that regardless of the format chosen, the whole (raw) image still needs to be written out to the logical volume. Unless you have really fast disks, this is by far the slowest part of the process. At least in my environment (with spinning rust) all I gain is some space on my image store and less overall used network bandwidth. Also, images are cached in a raw format on each node. The image also acts something like a sparse image:

root@os-ph12:/var/lib/nova/instances/_base# ls -lh abcca*
-rw-r--r-- 1 nova nova 10G Feb 25 21:05 abcca5bbe40c5b147f8a110bf81dab8bbb65db25
root@os-ph12:/var/lib/nova/instances/_base# du -h abcca*
6.2G    abcca5bbe40c5b147f8a110bf81dab8bbb65db25

Tuesday, January 7, 2014

"Unavailable console type spice" -- Another OpenStack Error

Tonight, I found myself trying to implement the spice proxy out of curiosity in havana. I found a nice post that talked about how to do this after banging my head on the wall for a while:

Anyway, this lead to getting "console is currently unavailable. Please try again later. Reload" when I tried to load a console of an already running host. I dove into the stack trace I received in the nova-compute logs:

2014-01-07 22:41:51.402 5706 ERROR nova.openstack.common.rpc.amqp [req-4ad10aaf-60c0-4f88-8964-cb3f6dd06814 6c978326923a4fa997a6a83b3fdbd11e 47eedd8414b84466a731289a5d6dee35] Exception during message handling
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/", line 461, in _process_data
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     **args)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/", line 172, in dispatch
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/", line 439, in inner
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     return catch_client_exception(exceptions, func, *args, **kwargs)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/", line 420, in catch_client_exception
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     return func(*args, **kwargs)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/", line 90, in wrapped
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     payload)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/", line 73, in wrapped
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/", line 271, in decorated_function
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/", line 258, in decorated_function
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/", line 3579, in get_spice_console
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     connect_info = self.driver.get_spice_console(instance)
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 2199, in get_spice_console
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     ports = get_spice_ports_for_instance(instance['name'])
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/", line 2197, in get_spice_ports_for_instance
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp     raise exception.ConsoleTypeUnavailable(console_type='spice')
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp ConsoleTypeUnavailable: Unavailable console type spice.
2014-01-07 22:41:51.402 5706 TRACE nova.openstack.common.rpc.amqp 
2014-01-07 22:41:51.404 5706 ERROR nova.openstack.common.rpc.common [req-4ad10aaf-60c0-4f88-8964-cb3f6dd06814 6c978326923a4fa997a6a83b3fdbd11e 47eedd8414b84466a731289a5d6dee35] Returning exception Unavailable console type spice. to caller

and the final hint was a comment in the source:

# head -2197 /usr/lib/python2.7/dist-packages/nova/virt/libvirt/ | tail -3
            # NOTE(rmk): We had Spice consoles enabled but the instance in
            # question is not actually listening for connections.
            raise exception.ConsoleTypeUnavailable(console_type='spice')

Turns out that running VMs have VNC or SPICE nailed into their configuration:

root@os-ph12:/# EDITOR=cat virsh edit instance-000000a1  | egrep 'graphics.*(spice|vnc)' 
root@os-ph12:/# EDITOR=cat virsh edit instance-000000a0  | egrep 'graphics.*(spice|vnc)' 

thus the VMs needed to be rebuilt on the host with spice.