David Nesting

Project: Wireless solar-powered garden moisture sensor

2011-11-06T06:44:00.000-08:00

We recently built a box garden in our back yard, and we integrated it into the yard's existing irrigation system. Unfortunately tuning the amount of water it should receive was somewhat clumsy, and it occurred to me that a sensor monitoring the moisture of the soil would be helpful in ensuring our garden gets the right amount of water.

Features

Wireless, solar-powered
Measures soil moisture (dielectric constant method)
Measures soil temperature
Measures solar panel voltage

Hardware cost: about $125 (+$25-30 for the hardware to program the XBee)

Note that this post does not discuss the "base station" side of the wireless system. I'll follow up with a separate post about that.

Hardware

Parts
I chose XBee because I have an established XBee network, and I'm familiar with the devices and they seem to be power-efficient. The XBee also has four analog-to-digital converters built in, so I also decided to try building this without a microcontroller at all, to further save power. This meant all of my sensors had to be analog.

For power, if I have the XBee sleep for 30 seconds at a time, it should be able to run at less than 3-5mA on average (1.6mA for the device while it's sleeping, and less than 40mA while it's awake). For the battery, we should have enough energy to last several days without sunlight, and when it's sunny we should be able to fully recharge in a day or so. I ended up with a 1300mAh LiPo battery (10 days at 5mA) and a 2W solar panel (333mA at 6V).

For the soil moisture sensor, I wanted something analog, maintenance-free and something I could bury. There are many sensors (especially DIY sensors) of the "stick two nails into the soil" variety. These are unreliable and corrode over time. Vegetronix makes a VG400 and VH400 analog sensor that measures the dielectric constant of the soil. I went with the VH400 sensor because I figured I wanted better precision at the expense of a little bit of power.

Parts list (proto-board friendly approach):

Vegetronix VH400 soil moisture sensor ($34.95), analog 0-3V output (also consider the VG400)
Insulated thermistor (Adafruit $4.00), 10k ohms
XBee 2mW Wire Antenna - Series 1 or 2 (ZB) (SparkFun $25.95)
XBee break-out, to mate the XBee with a breadboard or proto board

Breakout Board for XBee Module (SparkFun $2.95)
2x 2mm 10pin XBee Socket (SparkFun $2.00)

XBee computer interface for programming, either XBee Explorer USB (SparkFun $24.95), or XBee FTDI break-out (Adafruit kit $10.00) and FTDI cable (Adafruit $20.00)
USB LiPoly Charger - Single Cell (SparkFun $14.95, Adafruit $20.00)

Note: The SparkFun charger comes set to 5v. The selection jumper (a solder blob) must be removed and jumpered to 3.3v for the Xbee.
Note: The two chargers linked above are different. The Adafruit charger would probably require the "XBee Explorer Regulated" (see links in the substitutions paragraph below).

LiPo battery (SparkFun 2000mAh $16.95, Adafruit 1300mAh $12.00)
Solar Panel, 6V 2W, weather-resistant (Adafruit $25.00)

This could probably be lower power, but this was the only one I could find that was also weather and UV resistant.
Pay attention to the connector on the charger above. You may also need to grab a mini USB cable, for instance, and solder it to the solar panel.

Break Away Headers - Straight (SparkFun $2.50) -- I always keep some of these around because I use them a lot when mating break-out boards to breadboards.
Breadboard or proto board

I started with: Breadboard Clear Self-Adhesive (SparkFun $5.95)
And then transferred everything to: ProtoBoard - Rectangle 3" (SparkFun $4.50)

Consider also getting the female complement to the Break Away Headers above to avoid making some of the connections permanent.

A weather-proof case, e.g.:

Small waterproof OtterBox (Adafruit $10.00)
SparkFun Project Case - Black (SparkFun $9.95)

Resistors for three voltage dividers (I used a mix of what I had handy). The XBee only measures up to 1.2v.

3.0v -> 1.2v for measuring the moisture sensor (I used a 6.8k and two 5.6k in series)
3.3v + 10k ohm thermistor -> 1.2v (I used 3.3k)
6v -> 1.2v for measuring solar panel voltage (I used a 3.3k and two 6.8k in series)

Substitutions are possible. The battery charger just needs to be able to accept the voltage provided by the solar panel, and must be compatible with whatever battery you're using. The charger should permit simultaneous powering of your circuit while charging (or not), and should provide a regulated 3.3v output. If you can't get a regulated output, substitute something like the "XBee Explorer Regulated (SparkFun $9.95)" for the Xbee break-out board and be aware that the pins on the XBee Regulated are different from SparkFun's break-out board.

Assembly
Here's a basic breadboard arrangement of the above:

The LiPo charger is the red component at the top, and has the "SYS" or "Project" lines connected to the breadboard as shown (inside rail is positive, outside is ground). The third wire on the left side of the charger connects to the solar panel as well, though you could conceivably connect it to the battery's positive terminal if you'd rather see battery voltage instead. The solar panel then connects to the LiPo charger (not depicted).

The two-port screw terminal connects to the thermistor. Orientation doesn't matter. The four-port screw terminal connects to the VH400 sensor. As depicted here, port 1 is the signal (black wire of the VH400), port 3 is the supply voltage (red) and port 4 is ground (bare).

I strongly suggest modifying this approach to power the VH400 from the XBee's ON pin, so that it's only powered when the XBee is awake. The XBee is not guaranteed to supply a full 3.3v from this pin, so you may also need to use a transistor to power the VH400 from Vcc. The layout above is what I ended up doing, though.

Software/Configuration

Since the node itself has no microcontroller, I only needed to be concerned about (a) configuration of the XBee, and (b) the software on the server side to interpret and record the data.

XBee Configuration
I'm using a Series 2 (ZB) XBee module, so the instructions below might vary if you're using something different. I'm not taking advantage of any of the Zigbee features of the Series 2, so the Series 1 XBees should work fine (and they're cheaper). Use the XBee Explorer USB or FTDI adapter to connect the XBee to your computer, and use XCTU to flash the 'router' firmware onto the XBee. Search the web for tutorials on obtaining and using XCTU with XBee. The 'router' firmware will give you a flexible set of features. The 'end device' firmware will automatically put the XBee in a cyclic sleep mode, which is a pain to work with. With the right firmware in place, you can use XCTU or a serial terminal program to configure the XBee. You may want to start with "ATFR" or "ATRE" to reset the XBee's configuration to factory defaults. After each set of commands, you'll probably also want to do an "ATWR" to write your settings to non-volatile memory.

First, let's identify the module:

ATNI "GARDEN"
ATDH
ATDL

ATDH and ATDL will give you the upper and lower 32-bits of the node's 64-bit address. Record these for later. Now let's configure the ADIO pins to be analog inputs (2).

ATD0 2
ATD1 2
ATD2 2

And let's have the XBee periodically report them (say, every 5s (0x1388 ms)). We leave the destination address defaulted to 0, which will cause the samples to be sent to the XBee coordinator (our base station).

ATIR 1388

Since my network is encrypted, I also need to set an encryption key and enable encryption before this will work. Skip setting KY and EE unless you need to.

ATKY 0011223344556677
ATEE 1

A few seconds later, the XBee should be joined to the network and transmitting samples to your coordinator. Verify that you're receiving these before continuing. Once the XBee seems to be reliably sending its analog samples, there's a few more changes we can make to reduce its power consumption.

First, we can reduce the transceiver power. By default, the PL setting is at full transmit power (4), and the PM receive boost setting is enabled (1). Try reducing these as much as you can, while still allowing you to receive messages. You may need to do this once your device is installed in its final home.

ATPL 0
ATPM 0

Finally, we can put it to sleep between samples. I have mine set to wake every 30s, but 1 to 5 minutes might be acceptable too, and will extend the battery life. The time spent asleep is the product of the SP value (in 10ms increments up to 2800ms (0xaf0)) and SN (a simple multiplier). We'll also tell it to stay awake for only 100ms (0x64).

ATSN 1
ATSP AF0
ATST 64
ATSO 4
ATSM 4

After sending the ATSM 4 command, the XBee will start sleeping as instructed. Your ability to make further configuration changes will be hampered. If you're using XCTU, you'll get an error trying to send anything while the XBee is sleeping. While XCTU is giving you this error, you can wake the XBee up by grounding the RST pin, at which point XCTU should discover it again. Alternatively, you can make your changes remotely via another XBee (using the "remote AT" API), which will buffer commands until the XBee wakes up and checks in with its parent node. You may also have success using ATSM 5, and using the wake pin feature to keep it awake while you're messing with it. I haven't tried that.

You'll want to use the same (or greater) SN and SP values for the parent XBee node (either the coordinator, or an upstream router). Otherwise, the parent node will forget about the child since it will be asleep so much, causing you to lose commands that you might want to send to it, and requiring the child to do a more expensive join to the network.

Base Station Software
On the receiving end, you'll need another XBee and a computer interface (similar to what you needed to program the XBee above) acting as a controller. With the XBee accessible by serial port, and in API mode, you can adapt this Python script to receive samples.

Results

The graphs below should have live data, constructed with RRDtool. (Essentially, I just used the sample Python script above to execute an 'rrdtool update' command that adds all three measurements to an RRD file that I can then graph.) If there are gaps in the data (red lines), note that I'm still tuning the solar panel placement.

Update 2012-05-07: After about a year in operation, I've stopped getting readings. I suspect the battery has died. Until I get it working the graphs below will be blank or just give you outside temperatures (measured separately).

Learnings

The VH400 was an unnecessary power expense. I think I could have gotten by with the VG400 and saved a lot of power in the process (~7mA versus 0.6mA).
The VH400 could have been driven from the XBee's ON pin, so that it was only consuming current when the XBee was awake. This is undoubtedly the biggest power drain.
Cold solder joints will haunt you when deploying something like this someplace with lots of temperature variation.
Solar panel placement is important. Despite getting a lot of California sun during the day, I ended up getting about half a day of charge out of my solar panel each day. I'm still tuning this, given that my garden is growing up all around and finding a place without shadows is difficult.
Some way to monitor battery voltage would have been nice. There's one more ADIO port on the XBee that could be used for this, but the LiPo charger I used didn't have an easy way to access the battery terminal. The XBee is capable of reporting its own voltage, but this would have been regulated at 3.3v by the LiPo charger.

Converting Windows XP to a VMware VM in Windows 7

2009-11-13T08:24:00.000-08:00

I recently bought a new Windows 7 computer. I ended up with some hardware that wasn't compatible with Windows 7, and since my new machine had an obscene amount of RAM, I wanted to play with VMware, so I set out to virtualize my old Windows XP system within my new Windows 7 system. Since Windows 7 does not allow raw disk access to user mode processes, you can't just boot off of the XP hard drive within VMware. I also wasn't keen on installing any fast-and-loose tools or drivers to enable raw disk access. In the event my experience is useful, here's a step-by-step of how I ended up doing this:

Install the XP hard drive into the Windows 7 computer.
A method of building a .vmdk disk image from the raw disk is needed. I'm familiar with how to do this in Linux, and I wanted a Linux VM anyway, so that's how I approached this. There's almost certainly a way to do this without "leaving" Windows. You need enough free space on your target disk to hold the entirety of the XP drive.
1. Download an ISO for a Linux distribution (e.g. Ubuntu).
2. In VMware, create a new VM from this ISO. It doesn't need much space (I gave it 10G, and Ubuntu Desktop is only using ~3G).
3. In the hardware configuration for the VM, add a new Hard Drive, choose Physical Disk, and select the Windows XP disk (the whole disk, not just a partition). You'll only be reading from this disk.
4. In the settings for this VM, under Options, enable Shared Folders and map a folder on your host on the drive where you might want to store the resulting .vmdk. Once you have a working system, you can reformat the XP drive and move this file over there.
5. Once you have the Linux VM running, install qemu-img. In Ubuntu, this is how you do it: apt-get install qemu-img.
6. Identify the Windows XP disk. If you're familiar with Linux and how partitioned devices appear, you can probably just pick it out of the list at /dev/sd*. (You can see which drive is mounted as the root filesystem by typing mount. With only two drives, one is the XP filesystem.) Otherwise, you can try examining /dev/disk/by-id or by-path or by-uuid to try and figure it out.
7. Identify the location of your host's shared folders. This will probably be a directory under /mnt/hgfs: ls /mnt/hgfs
8. Use qemu-img to convert the raw disk to a VMDK file: qemu-img convert -O vmdk /dev/sda /mnt/hgfs/vm-images/winxp.vmdk
Depending on the size, the conversion process may take several minutes. After it's done, go back to the host, configure a new Windows XP VM under VMware, and select this VMDK file for your drive.
If it boots up, and seems usable, then you're probably done. Install VMware Tools, remove obsolete software/drivers, and possibly re-activate Windows (this was painless for me; there must be a certain number of hardware upgrades you can do without requiring you to call them up and justify your change).
Consider shrinking your VMDK to try and recover some of the "free" space.
If you want to move the VMDK back over to the original drive, you'll probably need to shrink the size of the VMDK first, so that it will fit within a filesystem on the old drive (it should be at least as big as the original drive was, and since the overhead of a filesystem reduces the available disk space, you can't just store it as a file).

Review: Indiana Jones 4 Sucked

2008-06-02T08:18:00.000-07:00

As far as I'm concerned, there are only three Indiana Jones movies. The fourth simply does not count. Lucas took a great thing and tried to stretch it just a little too far and ended up with something horrifying. There are spoilers below, so if you haven't seen the movie, stop reading now.

The first are the big changes. The film takes place the 1950s. No more Nazis. Our villains this time are the commie Russians. Our main antagonist, played by Cate Blanchett, feels like a James Bond villain rather than the villains we've known from earlier films. Way overdone. And then there were the aliens. But more on that later.

Ford was great. The same Indy we've grown to love. But his character went through a number of events that were completely implausible. I could write off some of the fantastic events of previous movies by getting into the story. Maybe there was a Grail. If the Ark was real, maybe it would have powers. But I couldn't do that with this movie.

First there was the "magnetism". Jones threw up some gunpowder to see where it would be drawn through the air. First, neither gunpowder nor lead shot is magnetic, so whatever it was that was attracting this stuff, it wasn't magnetism. Second, magnetic fields follow the inverse square law. Double the distance, quarter the strength of the field. If throwing this stuff up in the air caused it to fly toward the source, he should have felt it tugging on the stuff while it was in his hand, and given the speed it was moving, gravity should have yanked it back down harder than the "magnetic" field. But fine, this was something "alien" so maybe it wasn't really a magnetic field. But it was the first sign that something wasn't quite right.

Next was the atomic bomb. Jones found himself in a mock town about to be blown up by a nuclear test. He hides in a lead-lined refrigerator to protect himself from the radiation, which seems like it could be effective (no idea if the lead would be enough), but when the shock wave hits, destroying the town, he's thrown (seemingly) miles away, landing in the desert, a safe distance from the explosion. No human would survive the forces of the initial blast nor cratering in the desert floor inside a freakin' refrigerator. It's getting worse.

Jones and crew then survived three falls over giant waterfalls, inside a vehicle. My physics may be a bit rusty, but once the front of the vehicle hits bottom, your body is still in a free-fall, and the vehicle is now stopped just as surely as it had been if it were waiting at the bottom of the falls.

And then there was the flying saucer. The "crystal skulls" were in fact alien skulls, and you put them back together and they become a real alien, who then starts up his giant flying saucer and disappears, with what appears to be a cubic mile worth of dirt and rock swirling around him.

How can you have a flying saucer in an Indiana Jones movie?

But beyond all of the above, the movie simply didn't have an engaging story. I wasn't motivated to solve the puzzle with Indy. This was a sad day.

Risk: Who should pay for it?

2007-12-19T12:25:00.000-08:00

America is widely regarded as an increasingly litigious society. We're sue-happy, and lots of people are discovering that the system can be (ab)used to win jackpots. While some consider this trend an unredeemable negative, I view this as a reflection on the desires of the community to shift risk around. This may be stupid and short-sighted, but the market normally can correct for it. I divide lawsuits into three categories:

A justified suit where the plaintiff believes he has been wronged and seeks equitable compensation
A frivolous suit where the plaintiff does not believe he has been wronged, but seeks to win a jackpot by settling the case
A justified suit where the plaintiff believes he has been wronged and seeks to get as much money as possible, above and beyond equitable compensation

No matter what reforms are proposed, we must not prevent those falling into the first category from getting what they deserve.

The second category is clear-cut as well. When insurance companies are paying for the cost of a defense, they look at the economics first. If it looks like it will cost you $100,000 to defend a lawsuit, and the plaintiff would be willing to walk away if you gave them $50,000 cash, you'll save $50,000 by settling, even if you're certain you would win the case. There is frequently no economic/legal incentive in the US to avoid these situations (such as a "loser pays" system common in other countries), so there will always be people willing to try.

The third category (the basis for most of the rest of my post) arises because people are greedy, and juries frequently base their decisions on emotional factors rather than strictly rational ones. They may award millions of dollars for claims of pain and suffering, or punitive damages to "send a message". There is an incentive, then, for plaintiffs to lie, exaggerate or mislead so as to maximize their potential profit.

But more importantly, there is an economic incentive here for people to avoid personal responsibility in the first place. You don't win as much money if you admit that you could have exercised some common sense to avoid the harm in the first place.

Juries are supposed to be representative of the community. If a jury claims that someone should have anticipated a certain risk and done more to prevent the harm it caused, that's effectively setting policy for that community. It is impossible to be completely free from risk. All you can do is draw a line somewhere and devote some percentage of your resources to mitigating it. Where that line gets drawn is entirely a cultural thing, and there's no correct place to draw it.

Guiding philosophies

On one side, you have the conservative buzzword of "personal responsibility". Pay attention to what you're doing, and when you make a mistake, learn from your mistake and move on. The costs of mistakes should be borne by those that make them, giving them an incentive to be more responsible, lowering risk-related costs for society and punishing those that fail to do so.

On the other side, you have the more liberal stance that people should be protected from their own stupidity and incompetence. People should feel safe and secure at all times and the people making my life riskier should be responsible for mitigating that. These costs should be borne by those creating the risk, giving them an incentive to make the lives of everyone safe and secure.

In the "personal responsibility" camp, the costs of errors are not shared uniformly. Those that are less educated or less capable will make more mistakes. A consequence of being less educated and less capable is that these people probably make less money and are less able to pay for the costs of those mistakes. This exaggerates income disparity and decreases the quality of life for those on the lower end of the socioeconomic spectrum.

In the more liberal camp, the costs of errors are shared more uniformly among the community. A jury may think they're doing the right thing by inflating an award for damages to send a message, but what they've in fact done is tell everyone that their community places more of the responsibility for risk on the part of the businesses, and less on the part of the individual. Insurance companies will react immediately to these developments by increasing their liability/malpractice premiums in that community, and businesses will have to raise their prices to compensate (or go out of business). There's no such thing as a free lunch, and there will always be economic costs associated with risks.

Though I identify more as a liberal than a conservative, I tend to side more with the conservative viewpoint here. We do not grow as individuals by pointing fingers at someone else when we fail to use our common sense. This gives stupid people license to be stupid, and potentially smart people license to remain children for the rest of their lives. If the costs of my stupidity are going to be borne by society, what incentive do I have to not do stupid things? If I have good common sense and know how to avoid unnecessary risk, why should I have to pay for those that don't?

On the other hand, I'll acknowledge that this is more of a cultural issue than one that can be decided by the numbers. Is it necessarily bad that a community wants to spread out the cost of risks? Help the little guy at the expense of everyone else? That's not all bad, is it, if that's how the community (via their juries) wants to spend its resources?

Medical malpractice in particular

The goal (ostensibly) of the civil court system is to compensate people for harm caused by a doctor's errors. All doctors will make errors, and given the nature of their work, the harm caused by those errors, on average, will be staggering. Doctors (or their insurance companies) should be paying out settlements or judgments all the time. That's not a bad thing; it's a cost of doing business. Medicine is inherently risky.

Generally speaking, the market should always compensate for cultural shifts involving responsibility for risk. If juries start deciding to require that their businesses assume liability for more of these risks, the costs of doing business in that community should go up, and prices should go up correspondingly. On the other hand, if juries decide that individuals should assume more of these risks, the consumers end up having to pay more when those risks are realized. Note that the consumer normally pays either way, just from completely different directions, with costs distributed differently.

Unfortunately, with some industries, the market is not allowed to correct for these shifts. Most seekers of medical care are doing so with health insurance. Health insurance companies (including the federal government) negotiate fixed reimbursement rates for health care services. In communities with plaintiff-friendly juries, the health care industry is squeezed from both sides. They have communities pushing doctors to assume more risk-associated costs, with health insurance providers preventing doctors from increasing prices to compensate.

The result? The doctors simply stop practicing, switch to a low-risk specialty, or move someplace else. Your community can no longer sustain the same level of health care that your neighbors can. If you find your community is shifting risk around (by way of civil judgments), you have to pay attention to the effects and do something about them.

AT&T Home Entertainment sucks

2007-09-08T15:04:00.000-07:00

(Note: Updates below.)

So after I'd committed to my brand new apartment, it was time to start setting up my utilities. The conversation with the leasing office went something like this:

"Your cable TV will be provided by AT&T Home Entertainment."

"That's it? No Comcast? DirecTV?"

"The apartment is pre-wired for AT&T, and you won't be able to use Comcast. But you might be able to do DirecTV or DISH on your balcony if you have a view to the satellite. Let's look at the map. Hmm, yeah, I don't think you have the right view to see their satellites. Sorry."

So I was a bit nervous about this. Having worked for AT&T, I know exactly how much money and attention the services that they simply rebrand and resell get. For those that don't know, ATTHE basically wires up commercial structures for DISH or DirecTV service, resells the service, and pads the bill. So I was a little optimistic since DISH isn't really that bad, and DISH supports HD, has an HD+DVR receiver, and does local channels in high definition. So I figured I was going to be set.

Of course, this story would be boring if that were actually the case. This is the actual conversation (basically) I had with one of the sales reps at ATTHE:

"Do you have an HD receiver with a DVR?"

"No, we only have the HD receiver, and the standard-definition DVR receiver. We don't have an HD+DVR receiver."

"But you're just re-selling DISH, right? DISH has an HD+DVR receiver."

"Yes, but we don't use DISH receivers."

"Umm, OK. Well since it's just DISH service, couldn't I just buy my own DISH HD+DVR receiver and use it with your service?"

At this suggestion, the sales lady became nearly hostile. "No, sir, it wouldn't work. We use different satellites than DISH." WTF? "Our satellites can't handle the signal that the DVR boxes need."

(Ignoring the ridiculous satellite comment) "But you have a standard-definition DVR?"

"Yes, but the satellites can't do both high definition and DVR at the same time."

I am not making this up. I told her this was a deal-breaker and went to look at my other options. I knew I had none at this point but I didn't want to give her my sale. A few days later, I figured I had no choice, so I called back and decided to order.

"So, just to confirm, I'll be able to get all of the local channels in high definition, right?"

"Yes sir, absolutely."

Nope. When the installer came, hooked me all up, and I started browsing my channels, I asked him why I wasn't seeing locals in HD. "Oh, all you'll get is those 5 HD channels. We don't support locals in HD here."

!!!!!!!!!!!

I let him go, and I called DirecTV. If there's any way at all they can work with the southeasterly view that I have, I want them to try. I am not going to give AT&T a penny more than I have to. They deliver a shitty, stripped-down version of DISH, and the version that they put in my apartment complex is even shitter and even more stripped down than that. AVOID IF POSSIBLE.

Update 2008-05-15

So, eight months later, here's the scoop:

I eventually got my locals in HD by telling it to use off-the-air channels,vwhich were fortunately being provided with my cable signal. I had to figure this out myself. Twice they attempted to send someone out to fix my cable box, because they believed it was a problem with the box. The first time, no one showed up. The second time, the person showed up 10 days late. Seriously. I got a hang tag on my door and an annoyed voice mail saying I wasn't there at the appointed time to let them in. They were supposed to be there a week and a half prior.

It also turned out that the number of channels I did have available was a fraction of the channels DISH provided to their own customers. DISH customers got something like 30 or 50 HD channels, and I only got five. I called a total of three times to try to understand what was happening here, and got wildly
different explanations. After researching this on my own, I learned that the boxes they use are out of date by years. They did not support MPEG2, which is what a large number of DISH channels are encoded with.

A month or so ago, one of my receivers stopped working intermittently (maybe ~50% of the time, it was reporting no signal). I explained this over the phone, and I was told that I could set up a service appointment, but when they came out, if it was working, I would be billed $45. It was suggested that I
wait until it's failing all of the time before having someone come out, to avoid this charge. Alternatively, I could schedule a visit anyway, and if it started working before he arrived, I could quickly call and cancel the visit so as to avoid the $45 charge. This discussion was surreal.

A week later I decided to risk it, and when the tech came out, he told me that they'd just upgraded my complex, and they were now offering MPEG2-capable receivers, which included an HD+DVR receiver! Unfortunately, they would have to charge me to schedule an independent visit to have someone come out and hook it up. They would not just send me the replacement receivers. And the new service was going to cost me more.

So while most of my programming concerns are now satisfied, I am still their customer not because I choose to be, but because I have no other option if I want cable TV. Their customer service reps have proven, time after time, to be incredibly incompetent ("We use different satellites."), to the point of infuriating. It's clear that AT&T does not value this business unit or its customers and I strongly urge you to avoid them unless you can't.

Oh, and they recently added "online ordering"! Wow, great, now I can make programming changes online without having to deal with a real person. How does it work? It's just a text box on a web form that says, "Tell us what you want to change. We'll follow up with you after you submit." State of the art!

Update 2009-03-19

So, three weeks ago, my local HD channels stopped working. Both of my DISH boxes were stuck in a crash loop. I called up, and was told that there's a "problem" in the Bay Area and that HD locals would be out until sometime in May. I've since determined that other DISH customers in the Bay Area are fine, so the problem is specific to AT&T HE. Fortunately, I recently moved to a new unit that has a southerly view, so my hands are no longer tied! I've dropped AT&T HE. Fuck them. I've never been so dissatisfied with a utility provider in my entire life. I am disgusted and appalled by the level of service I've received, the technical abilities of the staff, and the failure to plan. As I said before, this company does not value its customers (they have Uverse now, after all), and gets its money entirely through exclusive agreements with apartment complexes. Its customers are hostages. They have no incentive to provide good customer service or to provide anything other than the bare minimum they can get by with. FOR THE LOVE OF GOD, AVOID AT ALL COSTS.

If you're considering moving into an apartment complex that has one of these exclusive agreements, bitch and moan about it, and DO NOT get a unit that does not have a southerly view.

Medical malpractice

2007-07-24T12:08:00.001-07:00

A popular complaint among physicians is that America's medical malpractice system is this crushing force, arbitrarily assigning jackpot awards to people whose only harm was not getting better after they saw a doctor. With average awards running so high, malpractice insurance premiums have to be high as well. The novelty of hearing about a physician flaunt his lack of insurance, making him an unprofitable target for malpractice litigation, usually stirs up debate.

Both sides of the issue have merit, but, as usual, the disagreements are more a matter of perspective than something that you can act upon.

As a patient, it's completely unreasonable to expect that your physician will be perfect. There are people that walk into hospitals and never walk out. The physician could have done everything conceivable, anticipated every possible complication, and done everything perfectly, and the patient still might have died. These are poor outcomes that have nothing to do with the abilities of the physician. People die. We need to move on.

Further, doctors make errors. They're human beings like the rest of us. There will be patients that do not wake up from their operations because the surgeon goofed.

From the doctor's perspective, it's unreasonable to expect that you won't make an error, or that you should be exempt from having to pay for the consequences of those errors. As a driver, I know there's a chance I may make a mistake. I've done it before, and caused a (small) accident. Who paid to make everything right? Me, of course, because it was my error. As an employee of a company, if I'm given a large amount of responsibility, and I make a mistake and cause a tremendous loss for the business, the business pays for that, either directly or indirectly. I almost certainly pay personally here too, either by losing my job, or at least my standing. But either way, the harm that my mistake caused was borne either by me, or by the person I was working for.

The problem is that medicine deals with large risks. Your own life is the ultimate risk. A single error in a doctor's career could cause the loss of a person's life. Is the doctor's life more valuable than the patient's? "An eye for an eye" would suggest that this mistake be held up against the very life of the doctor that made it.

Of course, you can't do that. The physician is just doing his or her job, and just because it's a high-risk job doesn't mean the physician should have to pay for that risk. That's why we have a malpractice system that uses insurance companies. This system works on the assumption that doctors usually don't cause errors, but when they do, the costs are usually staggering. Premiums should necessarily be high, and those costs should be passed on to the patients. Doctors that make more mistakes pay high premiums, and they're forced out of the business because they can't afford it.

So, in theory, the system isn't fundamentally broken, in my opinion. However, there are still two problems that need to be addressed:

The first problem is that a jury of your peers hasn't a fucking clue what constitutes malpractice, as opposed to just a bad outcome. Consequently, the skills of the lawyer combined with the believability of each side's "expert witnesses" decide cases. I'm an "objectivist", so any system where the skills of the people arguing decide fairness is, in my eyes, an unfair system. I still think a jury system could work here, but letting both sides hire witnesses still seems wrong to me. There is an inherent conflict of interest when someone pays you to take the stand to testify in their favor.

Some solutions to this problem include specialized "health courts", or an impartial panel of experts that review and certify malpractice cases, rather than relying on the abilities of a lawyer to convince or confuse a possibly uneducated jury.

The second problem is that the justice system is really there to fix injustices. It disgusts me that there are literally industries that operate around litigation. Litigation should be a last resort, not a first. What we're really talking about here is the inevitable error that a physician makes. What's so hard about (a) agreeing that an error was made, and (b) identifying fair compensation for that error?

Answer: the promise of jackpots. The introduction of non-economic damages in civil suits was a horrible mistake, in my opinion. Anyone can claim "pain and suffering" damages in the millions, and if their story is sad enough, juries will agree. It's impossible to quantify these things. It's just a subjective judgment call, which makes it inappropriate for a court to force people to agree to. In my opinion, a government and its justice system ought to be serving the public interest, in everything it does. By permitting people to cry and whine to a jury in order to be awarded millions of dollars of "compensation", we are encouraging the population to be a bunch of fucking whiny cry-babies. Get a spine! Grow up!

But it's not just non-economic jackpots that I take issue with. It's actually my understanding that, generally speaking, awards tend to be consistent with the injury suffered. The problem is that a potential plaintiffs (and their lawyers) have an economic incentive to call a bad outcome a case of malpractice. All they have to do is convince or confuse a jury into believing them over the doctor. (Remember that civil cases do not require them to believe either side "beyond a reasonable doubt.") So if the costs of bringing the suit aren't that significant compared to the potential rewards, lawyers will gladly take the case, because even if they lose most of them, the few that pay off will pay off big. So yes, the awards might even be adequate compensation for the plaintiff's poor outcome, but was it really fair to make the doctor (or his insurance company) pay for it in the first place?

So instead of a physician's mistakes being considered an unfortunate cost of practicing medicine, handled expediently and fairly for both sides, it becomes an ugly, disgusting example of everything that's wrong with America's justice system. Physicians are vilified, called "baby killers", with the most hurtful, shameful language you can imagine appearing in an official court document. To the lawyer, it's just another day at the office. To the doctor, it's devastating and heart-wrenching. This isn't a simple cost of doing the job, it's a personal attack. It's evil.

Solutions to this problem? Surprisingly, the same as the first: stop letting medical malpractice trials become a circus of confusion of claims that are dubious to begin with. People that have legitimate claims should be compensated. Those that don't have legitimate claims should have a harder time. With less willingness for lawyers to pursue dubious claims, it should be possible to better predict how you will fare at trial, which allows both sides to see cases a little more objectively, and to have a better idea up front what sort of compensation is appropriate. These cases can then be fairly settled out of court and everyone can move on with their lives. Better for injured patients, better for doctors, not so great for lawyers.

Renewing License Plates in Missouri

2007-05-01T06:42:00.000-07:00

I can't stand it when things are harder than they need to be. Many times, things are hard because someone stupid is making the decisions, or someone lazy is doing the implementation.

My plates are up for renewal. I always wait until the last minute to do these types of things.

Fortunately, Missouri offers a neat little online system for accomplishing this. No waiting in line! Unfortunately, it would appear that an early design meeting included a bit of dialog like this:

"Do we want to let people use the system if their plates have expired?"
"Hell no! Screw them!"
"Yeah! Down with the citizens! Make it harder for them to renew!"

So if you're late getting your plates renewed, not only do you have to pay a penalty, but they rob you of the convenience of getting them renewed online (arguably much more of a penalty than the late fee). Why???

Of course, some would say it's my own fault for not renewing them on time. It's true that I didn't renew them on time. But what exactly is the point of this policy? Is it indeed some form of additional punishment? Why punish people by making it harder for them to get into compliance? Doesn't that just discourage people from doing the right thing? "Well I was all ready to renew online, but now I have to drive to an office, so I'm just going to put it off until I get a ticket..."

My little rant doesn't end here, though. Here's the list of documents that you're required to have to renew your plates:

Renewal application
Paid personal property tax receipts for the previous two (2) years
Safety inspection
Emissions inspection
Proof of insurance

Each of these items is verified by the system as you enter them in.

Think about that for a second. The system already knows about most (if not all) of these documents. It would have to in order to verify that they're valid. So why do I have to go through the house and scrounge up all of the paper copies to enter them into the system? Why can't I just enter my license plate number, or my name and address, and have the system just pull up and verify that I'm all in compliance and ready to go? There's no reason why renewing plates can't be a simple matter of sending them the fee (electronically! even if it's past due!). Why make people do extra work?

Frequently, if this were a deliberate choice, the reason would boil down to costs. "Adding this feature might save everyone 5 minutes of time, but it costs $50,000! It's not cost-effective." There is an error in this logic. I see this a lot at work (AT&T), but frequently the time/money saved is a little less obvious. Let's do some math. If this feature were used by even, say, 100,000 people, we're talking about 500,000 minutes of time, or over 8000 man-hours. In a business, it might be standard practice to multiply this by $50 or $100 per hour of time. But since this is a municipality, let's value our citizens' time at $10 per hour instead. This hypothetical feature would thus save the public over $80,000 a year. It makes sense, then, to implement it if it costs less than this. That $50,000 is cost-effective after all!

Tamm Ave. Bridge Demolition

2007-04-20T22:07:00.000-07:00

They blew up the Tamm Ave. bridge tonight, the first bridge of many that will be demolished to make way for ~~a new hyperspace bypass~~ the upcoming I-64/40 construction. It was actually pretty neat. People were getting into it. A drunk woman next to me kept trying to chat me up. Fortunately she wouldn't shut up long enough for me to respond to anything she said.

I managed to shoot this crappy, out-of-focus video:

Firefox date handling in Page Info

2007-02-13T17:31:00.000-08:00

Today I was researching an HTTP caching problem and I discovered what I thought was a bug. When viewing the Page Info for an HTTP resource in Firefox, there is a "Modified" and "Expires" date that should reflect the modification date of the resource and when the resource should expire from the browser's cache. Here is what I was seeing:

Modified:  Tuesday, February 13, 2007 6:38:58 PM
Expires:   Tuesday, February 13, 2007 6:38:03 PM

For the life of me, I could not understand why the modification date was after the expiration date. A helpful user wrote in with his own numbers that gave the expiration date as being a minute-and-a-half in the future of the modification date. It was supposed to expire exactly 20 seconds after it was modified. What was going on here?

Take a look at the response headers. This request was made when the local clock on my PC was exactly 00:38:00 GMT:

Date:          Wed, 14 Feb 2007 00:39:14 GMT
Last-Modified: Wed, 14 Feb 2007 00:38:58 GMT
Expires:       Wed, 14 Feb 2007 00:39:18 GMT
Cache-Control: max-age=3

See what's going on yet?

The clocks on servers and client systems can't be guaranteed to be synchronized. So when a server sends a Last-Modified or Expires date, the client can't really be sure when those events occur. HTTP solves this by including a Date header, which should match the server's clock at the time the response was sent. Browsers can then compare this clock to the client PC's clock to figure out when the server really intended the resource to expire.

Firefox is really doing the only reasonable thing it can do here. Cache expiration is something only Firefox can do. Since Firefox needs to know when, relative to its local clock, a resource should expire, it needs to compare the server's intended expiration date with its own clock. This means it has to adjust the resource's expiration date to compensate for the difference in clocks. The modification date, on the other hand, is something that's really only meaningful to the server. When Firefox decides that it needs to make a request to the server, it has to provide the Last-Modified date exactly as the server did, so that the server knows when it's appropriate to send back an updated resource, or just tell the browser to use what it already has cached.

The result? The Modified and Expires dates on the Page Info tab are showing times from the perspective of two different clocks. They cannot be compared to one another!

More reasons people don't want to deal with HTTP caching!

Evolution vs. Creationism: Not the debate you think it is

2007-02-05T15:50:00.000-08:00

The "evolution vs. creationism" debate bugs me. I normally love to discuss things like this and debate the merits of science versus philosophy, but the problem with this debate is that people don't actually understand what it is that's being debated, which makes any educated discussion somewhat difficult.

Creationism is an explanation for the creation of life, one that that presumes a creator. "Strict creationism" would go so far as to say that all of this happened just a few thousand years ago, while a more allegorical interpretation would allow for life to be "created", perhaps millions of years ago, and subsequently evolve according to a creator's design.

Evolution is not an explanation for the creation of life. It is the observation that life changes over time. There is zero scientific evidence that evolution does not occur. In fact, when you take a step back and look at what we do know about life, it's clear that evolution is inevitable and can't not occur.

Evolution of life over time

If you don't believe in things like "DNA" or "chemicals", you might as well stop reading now and go put your head back in the sand outside. We have an excellent understanding of how DNA functions. We may not quite understand how all of the instructions put within DNA molecules get the effects that they do, but this is all an issue of overwhelming complexity, not a lack of understanding of the processes that are involved. When you take a strand of DNA and let it "run", you get the same proteins and the same form of life. When that DNA is mutated by the environment, or there is a failure to accurately copy it during the process of cell division or reproduction, changes are passed on to offspring.

Now normally when you read about evolution, this is the part where you read that nature "selects" the more capable organism and the less capable ones die off and disappear. This glosses over the whole process and you miss out on why this is inevitable. DNA is like an assembly line at a factory. When you change things around, either with the line or with the raw materials, you get a slightly different product. Usually it isn't something you want, but occasionally you come up with something really novel that lets you outcompete someone else. (For practical reasons, this doesn't actually occur in real factories, but when you're spread out over the surface of the earth and have a few hundred million years to lounge around, shit happens.)

So long as you always have changes occurring in the mechanics of life (just like we always have solar and cosmic radiation, or errors in cell division), it is inevitable that these changes will result in offspring that are different from their parents. When you always have environmental changes (like with climate changes, deforestation, natural disasters, changes in the availability of food, or the presence of a new species of predator that wasn't there before), you always open the door for some of those mutations to provide a benefit to organisms that have it. Maybe the mutation will allow them to digest a certain kind of food they weren't able to digest previously, and a food shortage means they need to try. Maybe it made the organism hairier, just when the temperatures were starting to drop. Maybe it changed muscle density somewhere and allowed the organism to run slower, but over longer distances, which turned out to be helpful when food was starting to thin out.

If environmental change is inevitable, and mutations are inevitable, then adaptation is inevitable. Evolution is inevitable. It's not a "theory", it's obvious, and it's staring you right in the face.

Whenever you hear about an antibiotic-resistant strain of bacterial infection, that is a strain of bacteria that has evolved in the last few decades. When you invent an imperfect but effective poison that kills off bacteria, the inevitable mutations that occur with DNA, combined with the extremely rapid rate of reproduction of bacteria, means that at some point in the near future, a mutated variant of that bacteria will have properties that allow it to survive the poison. There is no disagreement in the scientific community that evolution occurs, and the non-scientists telling you differently are flat out lying, or simply do not understand what evolution is. Doctors and all types of biologist are way beyond accepting evolution. It's now part of every-day practical application for many of these professions. When someone says, "There's no such thing as evolution," it's like telling a construction worker that nails do not hold pieces of wood together.

Creation of life

Most people arguing about evolution are actually trying to argue about the creation of life. Evolution, per se, is not incompatible with creationism, even if you subscribe to the strict 6000-year-old-universe religious view. We see evolution all the time, right around us.

Where there is disagreement is on how life got started in the first place. We can clearly see life evolving right in front of our eyes, but what did it originally evolve from? Evolution does not attempt to answer this question, so it is completely inappropriate to attack evolution just because you believe in creationism.

Strict creationists, though, might still have a beef, because the evidence for evolution stretches back not just through the recent past, but hundreds of millions of years. Even if they could accept that life evolves today, they can't accept that the gradual appearance of proto-human species documented by the fossil record ended up with us. "Maybe they existed, or maybe God just put the fossils there to test our faith. Even if they did exist, they must have all died off or something before God put modern humans here."

A more liberal, allegorical interpretation of creationism usually ignores the 6000-year-old rule, and even might accept the scientific viewpoint that humans evolved from earlier forms of life, only instead of this happening due to selective pressure and environmental adaptation alone, our evolution was guided by a god. This helps to reconcile the two theories, and lets some creationists sleep at night, but it can't survive scientific scrutiny.

The reason scientists look to explain the creation of life without creationism is because creationism is not a scientific theory. This is another point of contention among proponents of both sides. "Creationism is just another theory!" In layman's terms, it's a "theory". In scientific terms, it is not. A scientific theory must be something that is falsifiable, meaning that we can devise an experiment that demonstrates that the theory does not accurately predict reality. Any "theory" that isn't falsifiable can not advance science, and science must necessarily disregard it. Feel free to discuss it as philosophy, but it is not science. There are many people that view philosophy and science as two different branches that attempt to discover "truth". What non-scientists frequently fail to realize is that while philosophy must necessarily incorporate some amount of science to further its pursuit of truth, the reverse is not true. Science is all about the scientific method. It cannot incorporate philosophical viewpoints, biases or methodologies. If you cannot phrase an assertion as something that can be experimentally tested, it is not scientific and must be excluded.

Consequently, when scientists study the possible origin of life, they work based on their observations and the currently accepted views of how all of this works. Since evolution guides the development of life on an ongoing basis, it's reasonable to assume that evolution guided the development of early life. It makes sense, then, when trying to see how life came to be in the beginning, to focus on how evolution might have worked with early forms of life, and work our way back to see what initial components could have spontaneously come together to produce the first rudimentary life-like processes.

And for a discussion of that, I must point you to Wired magazine's community-provided take, which (presently) gives an excellent discussion of the current scientific thinking about all of this.

If you wish to debate evolution, please come prepared with an understanding of what it is that you're debating.

Game of Tag

2006-10-21T17:57:00.000-07:00

It took me eight months, Will, but here are my responses:

Four jobs I've had

Sonic cook
Internet provider, helpdesk, web stuff
Electronics manufacturer, providing network/PC support
AT&T!

Four movies I can watch over and over

Fifth Element
Alien*
Star Wars*
Fight Club

Four TV shows I love(d) to watch

Star Trek: TNG
Battlestar Galactica
Stargate SG-1/Atlantis
Attack of the Show (mostly for Olivia)

Four places I've been on vacation

My first thought is to put Alaska, but I lived there for several years, so I don't think it should count.

Padre Island, Texas
Saranac, New York
Las Vegas, Nevada
Misc. lesser trips

Maybe I should have listed Alaska after all.

Four favorite dishes

Chicken in a basil cream sauce
Ice cream
Cheese & Macaroni
Pizza, naturally

Four websites I visit daily

That last one is probably cheating, since it's a feed reader, but it's true. Every day.

Four places I'd rather be

Exploring the universe
Exploring the world
Exploring a restaurant
Exploring my fridge

OK, I'm back.

Four I'm tagging

I respect my friends too much to do that. :)

(Photo by sergeymk.)

Net Neutrality: You pay either way!

2006-10-21T16:45:00.000-07:00

So I've been seeing this commercial off and on a bit, and it really annoys me, because it's extremely misleading. The voiceover goes like this:

Are you google-eyed with confusion over 'net neutrality? No wonder. It's all just clever mumbo-jumbo.

'Net neutrality is nothing more than a scheme by the multi-billion-dollar Silicon Valley tech companies, to get you, the consumer, to pay more for their services.

Forget all the mumbo-jumbo. 'Net neutrality simply means: you pay.

Those in support of 'Net neutrality are guilty of similarly misleading comments and I don't like it. To understand 'Net neutrality, you need to understand a few things:

Internet providers today are already compensated

Internet providers are simply companies that provide people with Internet connections. Those companies must then contract (peer) with other Internet providers, to ensure their customers can send and receive traffic to everyone on the global Internet. The Internet itself isn't really a single thing. All of these providers connect up with each other and relay traffic to/from each other, and they call that interconnected mesh "the Internet".

Every physical Internet connection costs money. When you subscribe to the Internet through your ISP, you get the benefit of connectivity. Your ISP doesn't really benefit from the connection itself. Consequently, the flow of cash goes one way: from you to them.

Your favorite web site, for example, is in the same situation. They must obtain a much larger Internet connection from an ISP. The ISP doesn't really benefit from having that connection per se, so the money flows from your favorite web site to their chosen ISP.

Your ISP and their ISP are in a different situation, though. If they aren't connected to each other, their customers aren't going to like it and will find another ISP who is. If your ISP wasn't connected up to anyone else, your ability to pull up any of your favorite web sites would be hampered, right? So if these two ISPs have roughly the same amount of traffic they want to pass to the other, they might agree to set up a connection between themselves, where traffic from one side balances out traffic from the other side, so both sides end up paying mostly just for the maintenance of the connection.

Opponents of 'Net neutrality like to say, "They're the popular web site sucking up all of this traffic. They should have to pay!" Except they do pay. Their connection with their ISP ain't free, and they're probably charged based on the amount of data they send/receive. If this traffic between their ISP and your ISP is no longer in your ISP's favor, that's their fault for not writing a better peering agreement. If your use of your "unlimited" flat DSL connection is going up and costing your Internet provider more, that's something you and your Internet provider have to work out between themselves. The reason flat rate Internet connections work is because the under-utilizers subsidize the costs of the over-utilizers, and the flat rate ends up being a rough average. If the average utilization goes up, it's a no-brainer that the flat rate has to go up to compensate. You can't try to keep your flat pricing low by pretending the increase is caused by 3rd-parties that you have no business relationship with whatsoever.

Quality of Service

The utilization "problem" is a red herring, though. The real issue is one of Quality of Service (QoS), and who pays for it.

When you make a request against your favorite web site, it might pass through a handful of different ISPs. These ISPs may have no business relationship at all to the content provider, or to you, the consumer. Consequently, these providers aren't going to care when you tell them that your live video stream shouldn't be interrupted and should take precedence over random pieces of e-mail that could be delayed a few seconds without incident. Neither you nor your favorite web site have any ability to guarantee the quality of the service they provide you.

Within a single ISPs network, though, that's not the case. If you're an AT&T customer, and you have a nice big IP connection from them, and they want to sell you live video on top of it, AT&T controls the entire network at that point, so they have the ability to guarantee the quality of service for that live video (TV) stream, and can say that TV takes precedence over file downloads and the like. This raises red flags for competitors, that want to sell you TV over the public Internet. Since they cannot guarantee that their own video stream will have a guaranteed quality of service, their live video will be treated just like other random Internet traffic. If you start downloading some software, your live video stream from them will start breaking up and be unwatchable, while AT&T's live video service will still look great.

This is a non-neutral Internet. The ISP is able to provide services that others are unable to provide. The ISP's solution (AT&T's position) is simply to sell guaranteed connectivity to these content providers, at a premium. If you want to guarantee the quality of service of your Internet traffic to their customers, you need a dedicated connection where everything between the provider and the consumer is QoS-aware, and special efforts to configure the AT&T network to guarantee that level of service all the way to your home. These specialized networks necessarily cost more.

The content providers, understandably, think it's ridiculous to pay extra for this type of thing, and if they have to pay more to deliver their services to you, they will pass that charge on to you. There's no such thing as a free lunch.

Net Neutrality

What these content providers advocate for instead, is a "neutral" Internet provider, one that does not give preference to their own IP packets, or allows 3rd-party providers to give themselves the same degree of preference without charging them extra for that ability.

The problem is that the latter capability is still going to cost something, costs they will pass on to you.

Either way, you pay! The battle between these two sides has nothing to do with whether or not you pay. It's all about who you pay (i.e. who the "bad guy" is that has to raise rates).

And, of course, this all makes the assumption that we even want guaranteed quality of service for the video we watch on the Internet. For stuff like Google Video and YouTube, we seem to be doing OK today without QoS. So who's upset about the situation? Those that want to provide TV service over the Internet. When you dial up Skinemax with your remote, you're not going to want to deal with 10 seconds of "Buffering..." every few minutes, or an episode of Lost that cuts out when your kids hop on Second Life. It has to work without disruption, and it has to work right now. Somebody has to pay for that no matter how "neutral" the Internet is, and ultimately, it's going to be you either way.