Wells Blog – Page 19

Disney and V&A’s

This past weekend I went to Disney World with some friends of mine. Now, I hold an annual pass so I get in there for free, as do they. These friends of mine like to go there more to stay that the resorts and get away from things back home and at work, whereas I prefer to go to the parks in order to get away from everything.

We drove down on Friday evening, and about half way there I realized that I had left my wallet in my car back at their house, so I was completely without any funds, ID, or anything else. This… sucked. I had my pass with me, so I would be able to get into the parks, but if I wanted to do anything else I was completely reliant upon them for funding, etc.

I also kept feeling my ass because my wallet was missing, but that is a different matter.

So, as we were approaching Disney we were debating where to eat. Now, I don’t have any real preference on where we eat, as I can eat just about anything anywhere, so I let the others in the truck argue over where to go. There was some debate on going to Jiko which is in Animal Kingdom Lodge, and we even managed to get reservations there. Then they decided that it might be more fun to go to Citrico’s in the Grand Floridian instead. We have a waiter there who is a good friend, so we managed to get reservations there instead without any issue.

We got to Disney and checked into our hotel (Buena Vista Palace Hotel & Spa), then drove over to Citrico’s for dinner. Nice dinner, as always.

On Saturday, my friends were a bit… slow… in getting going. We went to breakfast at the Concourse Steakhouse in the Contemporary Resort. Decent little place to get breakfast, though I am not a big breakfast person. From there we went to Hollywood Studios (formerly known as MGM Studios) and went to see exactly one thing… Walt Disney: One Man’s Dream. After that we left the park. Yeah… we went and saw one thing. As I said… they go to enjoy the resorts, I go to enjoy the parks. Ah well. We then went back to the hotel and rested until late afternoon, when we got all gussied up to go to Victoria & Albert’s in the Grand Floridian.

Now, Victoria & Albert’s (aka V&A’s) is not some restaurant where you can just check in at anytime. This is a restaurant where you have to be in coat and tie (if you remove your coat at the table, they ask you to put it back on) or a dress… basically you have to be dressed to the nines just to get in. This is a full-service restaurant where you are escorted to the bloody bathroom if you have to go (no, they don’t follow you inside, but the do escort you to the door.) The prices are fixed (>$100 a seat, guaranteed) but you get a full seven course meal that is very very good.

The maitre-de and waiters know their stuff as well, and if you want a specific wine they probably have it, or can recommend a number of substitutes for what you want.

Anyway, a really good meal was had, and I even found a new wine that I enjoy, which is bloody impressive seeing as I only like sweet wines that don’t have that bitter dryness to them.

Afterwards we got back to the hotel, and I was still feeling antsy because I just had not moved around enough that day, so I walked across the street and the whole way through Downtown Disney and back before going to bed.

The next day we went and had breakfast at Rainforest Cafe just outside of Animal Kingdom, then went in to see the Finding Nemo show and then go on Expedition Everest. From there we went home.

Again, I would have done a lot more in the parks during my stay, but my friends are just not ones to do so, which is unfortunate.

Infiniband Network Mapping

So, recently I had to figure out a way to produce a map of the infiniband fabric. What this means, and how to display it, was a different matter. I had a number of tools available to do so, but the most important one I found was the ibnetdiscover tool that is packaged with the OFED (OpenFabrics Enterprise Distribution) software stack.

This tool allowed me to find the infiniband connections between the different switches in our fabric and the connections to individual nodes. Of course, all it would tell me is the IB GUID associated with each switch and node, although the nodes would let me know that they were connected to via “HCA-1”.

To fix the switches, I had to go to each switch and get their GUIDs manually and enter them into a file that ibnetdiscover supports as a lookup file. Not so bad for the 24-port edge switches that we have from Cisco, but the Silverstorm switches are unmanaged, and we cannot login to them. Fortunately I was smart when I installed the Silverstorm switches and grabbed the GUIDs off of them when I installed them.

The larger Cisco 9600 series (also known as TopSpin 270) switch was a much larger problem, because none of the GUIDs detailed by the software matched anything that the 9600 series switch shows in queries sent to it. Mapping that out was a pain, and the way that things are apparently connected in a 9600 series switch is interesting. There are three different kinds of blades in the 9600 series:

Port blades where you physically plug infiniband cables into the card
Management blades which plug into the back of the switch and manage the port blades
Super management blades which manage the management blades.

These are all interconnected: 2 port blades connect to a single management blade, which in turn are connected to each of the super management blades. So, when a message comes in across one of the cables, it enters through a port blade, goes to a management blade, which in turn decides whether it should go out another port on that same port blade, through a port on the other port blade that is attached to the management blade, or up to one of the super management blades. All of this is of course figured out via the subnet manager engine which runs on the 9600 series switch (in our case). That subnet manager can also be run in software on any host node, but because of the size of the infiniband network that we have, it was better to offload that to an actual hardware based subnet manager. I was able to figure out these interconnects on the 9600 switches by using the ibtracrt command against LIDs.

So, I had the different GUIDs allocated to different switches, but what about the individual nodes connected? All I was getting was “HCA-1” out of them, and trying to override via the lookup file wasn’t working. The problem turned out to be that when infiniband is firing up on the nodes, it was firing up so early that getting the hostname was not occurring, so all it had to go on was that it was connecting via port 1 on the infiniband HCA. By setting up a command to cat the hostname into the sys value on each node, this was fixed and the proper names were being displayed.

Now that I could get an output to come out that was right, it was a simple matter to write a perl script that would take this output and convert it to an output that would work well as an input for the program “dot”, which took that input and produced a very nice GIF of the infiniband network mappings.

📂 Linux