wiki/ news/ 2005-09-01 - General Meeting: Post-mortem analysis of LV2's wreckage

A bunch of us gathered to do a post mortem on LV2 this evening. We spread tarps and white sheets down on the FAB 155 table and began trying to do a "crash analysis" on the remains. It was pretty brutal: there wasn't a single "ok" piece of the rocket. Surface mount components we stripped off boards, circuit boards chipped into tiny pieces and conformally smushed into crevaces in the Aluminum... and there was almost zero evidence of the compact flash card we hoped to find. We think we found the carier board, but the CF card itself seems to have disintigrated.

Oh well.

Here's a movie with Brian and Andrew giving a walk through of the remains.

A few things we know now:

lv2_postmortem_with_labels.sized.jpg

Composite image of LV2 spread out on the FAB 155 oval conference table, complete with white sheets.
For the original composite without the labels,
click here.

Update September 6th, 2005

Tim's analysis of the line cutters is complete. Here's his email, which pretty much sums up our best guess as to what happened:

At the impact site several incongruities presented themselves. All
four electric matches were expected to have been fired, but the nose
failed to detach and the linecutters appeared not to have actuated.

Clearly the black powder nose separation charge had fired, but was the
ignition the result of impact with the ground?

Had the recovery electronics really fired the matches at the right
time?

A quick examination of the linecutters revealed that the cutter knives
had broken free of the plastic shear pins that hold them in place.
This is normal if the cutters have been actuated, but the primary
linecutter still had the un-cut 4mm line running through it. Had the
linecutter been initiated on time and subsequently failed to cut?

There was a disturbing lack of burnt powder smell coming from the
linecutters. What was going on here?

Both linecutters were partially crushed and so could not be
disassembled normally. On Friday i cut them apart, and have since
inspected both of them.

-------------

Observations:

  The primary cutter did not cut the 4mm line. The backup cutter had
  the line pulled out during impact, but i suspect it also failed to cut.

  Both linecutter's electric matches have been fired

  Both linecutters contain un-burnt smokeless (mostly nitrocellulose)
  powder grains.

  The 4mm line protruding from the primary cutter shows evidence of
  melting, indicating moderately high temperatures, but only on one
  side of the cutter. The line sticking out the other end of the same
  cutter is un-melted.

  The primary linecutter's knife, which is hardened steel, shows no
  evident damage and may actually be re-usable. The secondary cutter's
  knife still needs to be cut free of the body, so its condition is
  not completely known.

--------------

Thoughts

The quantity of un-burnt powder in both cutters is similar. The
primary has 42 un-burnt grains by my count, the backup 43 grains. Both
were loaded with 55(+/-)1 grains so about a dozen grains are missing
from each cutter.

I suspect that when the electric match fired, the grains adjacent to
the match also ignited, but the grains further removed from the match
failed to ignite.

This scenario is consistent with several observations. When found
after impact, the cutter knives were out of place, their shear pins
were broken but there was no smell of powder and the line was not
cut. An incomplete combustion of the powder might generate sufficient
pressure to break the shear pins and move the cutter knife, but not
enough pressure to cut the line. If the line doesn't cut the knife
can't move forward far enough in the body to reach the vent holes and
the combustion gases can't easily escape, so the characteristic smell
would be absent or muted.

Why would the smokeless powder fail to ignite if the electric match
fired?

The most convenient explanation is the altitude at which the match
fired.

Telemetry indicates that the linecutters fired approximately 21,000 ft
above MSL. At that altitude atmospheric density is about 44% of sea
level density. Although it is somewhat surprising to me, there may be
insufficient gas at that altitude to effectively transfer heat between
adjacent powder grains. We have been worried about this before in
connection with the nose ejection charge, but never with the
linecutters since their operational design altitude is only ~1000 ft
AGL.

I speculate the top of the fault tree looks like this:

1) Software Succeeds in commanding recovery node to fire pyrotechnics
2) Recovery node connects charged HV capacitor to terminal block
3) Ignition wires conduct current to electric match
4) Electric match fires
5) Powder adjacent to match burns
6) Combustion gases transfer heat to surrounding powder
7) Powder charge completes combustion
8) Ejection charge gas pressure pushes nose cone forward breaking
   shear pins
9) Nose cone clears PVC ejection charge holder
10) Recovery module pressurizes
11) Aluminum sleeve on nose cone clears recovery module under the
    combined influence of momentum and the pressurized recovery module

Logs indicate that (1) took place on schedule. (2) relies on the
testing history (zero failures) and the fact that the recovery node
firmware checks for a charged HV capacitor before attempting to
fire. The log indicates that all fire attempts completed
successfully. The recovery node hardware design is capable of detecting
most actual failures up to (4) but the firmware has not been
implemented.

Therefore the fault tree branches at (3) and (4). (3) relies on
a pre-flight Ohmmeter check and an otherwise successful history. If
(4) failed the fault would probably not be at (3) because of the
redundant wires and matches to the ejection charge. The observed
firing of the cutter matches would be evidence of success at (4)
unless the cutter matches were triggered on impact. This seems
plausible since the core of the match contains a small amount of
shock-sensitive lead picrate compound, but impact triggering should
have burnt all the powder (zero failures in all previous tests). Also
the quantity of picrate is quite small and well protected in an epoxy
dip, further protected inside the linecutter body. However, striking an
electric match between hammer and anvil sets it off with a bang
(hearing protection recommended).

So, for the failure to be at (4) we must have the electric matches
fire on impact (plausible buy not certain) and the linecutter powder
charge burn incompletely near ground level (unlikely), therefore
failure at (4) is unlikely.

Based on all previous tests (5) is assumed. Early electric match tests
explored the possibility of barely-initiated matches exhibiting slow
burning and low peak temperatures, and some effect was seen, but it
was slight. The recovery node is designed to deliver the firing energy
at power levels many hundreds of times greater than the minimum
required to overcome the slow-burn effect, so the match firings are
expected to be consistently hot which should assure (5).

(6) appears to me a probable failure. I have no other explanation for
the incomplete burn of the linecutter powder charge. We have
successfully used the present nose ejection charge system at about the
same altitude AGL attempted this time, but several things unique to
this launch make a complete powder burn less likely. The powder was
old. Prepared, i think, for the February launch attempt. It may have
been wet or partially oxidized. Also there is a slightly higher base
altitude in Brothers. I haven't compared actual pressure and
temperature at apogee from the two launches.

(7) likely did not take place. Although the sleeve fit of the nose
could have jammed and prevented the ejection of the nose cone, this
seems unlikely because the system has been tested and found to be
fairly reliable. Also the fit was hand tested just prior to launch and
seemed ok. The extent of burning seen on the nomex drogue cloth is
also consistent with incomplete combustion of the ejection charge. Had
the ejection charge burned completely at altitude the heat transfered
to the nomex cloth should have been on order what we have observed
before, where the cloth survives mostly intact and can be re-used. If,
however, the charge had gone off on impact with the cloth crushed
against the burning charge in the heated wreckage, it makes sense that
the cloth would burn through. I don't have any black powder. Anyone
know for sure that black powder ignites when struck? Anyone got some
powder and a hammer? (Just a pinch, head clear of recoil please ;)

Given success at stage (7), (8) is virtually assured, the volume to
which the ejection charge is confined before breaking the shear pins
is quite small, and the ejection charge pressure in the confined
volume large. The shear pins are four weak ~3/32 inch PVC rods. I
don't think we have ever observed a problem breaking the shear pins.

Once the nose cone moves forward the pressure separating the PVC
ejection charge container is probably un-stoppable. Even if the nose
cone were to jam at this point the combination of high force and
well-centered point-of-action would probably move the nose forward and
separate the charge container. Even if the nose fully jammed the
pressure is probably sufficient to break the PVC pipe and pressurize
the recovery module.

Once the recovery module is pressurized, there is yet another probable
failure mode. The pressurized gas in the module leaks out around the
drouge deployment bag and subsequently through the module cutouts and
down the aeroshell. Therefore if the nose cone hangs up, or the
deployment bag or drouge cloth is not tightly packed, or possibly if
the altitude is high enough that insufficient air is pre-existing in
the module, the module pressure may neither peak high enough nor
maintain long enough to fully eject the nose cone.

This is probably bad design, and we have said previously it ought to
be changed. My feeling is that this particular problem probably did
not cause the ejection failure we saw this time, but i have no way in
hand to know that for sure.

--------------

Wrap Up

The most likely causes of failure in order are:

  Incomplete combustion of the ejection powder charge

  Jamming of the nose cone combined with pressure loss in the recovery
  module

  Recovery node failure or associated wiring

To increase the system reliability in future do these things
(in priority order):

  Use only fresh or properly stored powder charges for flight

  Test ejection system under vacuum

  Re-design ejection system to avoid relying on whole-module pressurization

  Test ejection more often

  Finish recovery node firmware

Note that some people have noted that we probably should move away from gunpowder gas generators and move to cold gas systems; we may very well do that.