Continued working with Nathan to get smokeping data successfully into the
event detection system. I generated some random data to fill the
historical buffers and then continued to run it over live data, which
generated a small number of plausible looking events. I'm now looking into
the scalability and resource usage of this as it seems a little higher
than it should be. Also polished the dashboard graphs slightly, changing
them to use more sensible axis and better resolution data.

Spent some time with Richard, Tony and Shane thinking about the future
direction of AMP. We've got some good ideas and have a whiteboard full of
initial planning for the work that needs to be done.

Read draft introductions to a number of 520 reports and gave some
hopefully useful feedback. Everyone seems to be on the right track so far,
looking forward to reading more.

Fixed the bug in the tput test that would sometimes cause it to refuse
connections for a minute when it was meant to be re-establishing a new
test connection. It was erroneously waiting for more data when there was
no more to follow, so wouldn't continue until select timed out. Updated
the NZ mesh with all the fixes from the last couple of weeks.

Worked on the backend for the event detection web interface to use a more
flexible and secure database abstraction. Made a few small changes to the
web interface to try to hide information that wasn't always needed, but
still make it available if required.

Investigated all the historical event groupings and found a few rare cases
where it wasn't doing the right thing due to the order in which events
arrived or due to some missing common attributes. Came up with an approach
to sometimes rebuild groups as needed to minimise their number and get
better matches.

Found what looks to be a threading bug in AMP measured that has been
troubling me for a while. Test threads check that the nametable is up to
date before running but it was possible for them to deadlock on accessing
the file. I've had a fixed version of the code running for a couple of
days on some of the machines that were most often affected and have yet to
see the problem again. Hopefully that's fixed!

Fixed a couple of small bugs in the AMP matrix tooltips that were
triggering events on child elements without the appropriate attributes, so
weren't displaying information.

Wrote a program to insert common path information into the event database
to use for grouping events. Testing so far with this data shows that
fewer, larger groups of events are being created. Some of the membership
is a little bit questionable, so am know in the process of having it
describe the reasoning behind creating each of the groups.

Short week this week due to being in Wellington for Thursday and Friday.
While I was there I caught up with Jamie and Sam Russell at REANNZ for a
chat about AMP and perfSONAR deployments on the network. There should be a
lot of new monitors going in shortly and it would be great if we could run
both measurement platforms.

Spent some time investigating error messages that have been showing up
lately in amplet logs. It appears there is some weirdness happening with
raw icmp6 sockets receiving packets that should have been filtered out by
a socket option. Reading through the kernel source it looks like filters
are doing exactly what they should be doing and I know believe it's due to
packets arriving and being buffered in the time between the socket being
created and the filters being set.

Changed the tooltips in the matrix display to all be fetched via ajax
calls, so none of that data is sent to the client initially. This should
speed up page generation (no need to fetch data for the last week) and
shrink the raw page size futher. Will hopefully deploy and test this on
the NLNOG RING matrix shortly.

Spent some time improving the performance of the NLNOG RING AMP matrix
page - with tens of thousands of cells the page got rather large and slow.
I've culled the individual tooltips down to one reusable one, drastically
reducing the number of DOM elements on the page as well as reducing the
size of the raw HTML. It's still a monster but is almost becoming
manageable. Next step will likely be to move all the tooltip data to an
AJAX call rather than embedding it in the page.

Fixed up the getCommonPath function in AMPcentral to better fetch data
from the desired time period. The ending condition for the time period was
using an incorrect value which resulted in using much longer periods. This
now gives me correct path data through the web interface which I can use
for event detection and hopefully smarter grouping of events. Added
database support for dealing with common attributes between sources and
destinations, now need to collect the data.

Rewrote my AMP data sampling program to properly sort all data by time
rather than by source/destination pair, and to deal properly with fetching
multiple data types in a single run.

Spent some more time working on building useful groups of events for
RTT/loss data. I'm trying to find a compromise between including all
events that happen about the same time and grouping only those events that
are obviously related, while allowing events to be in multiple groups
where that makes sense. Some of these issues are coming about because my
sample data extraction program doesn't guarantee strictly increasing
timestamps in the warm-up phase while fetching historical data.

Tidied up some error messages in the icmp test in AMP where non-echoreply
responses were being incorrectly examined for the embedded triggering
packet. It should now properly index into those packets and record the
correct error type codes.

Noticed that sometimes the AMP tput test was failing to run in both
directions on some nodes and tried to investigate why. Running the tests
manually works, but scheduling them through AMP often fails to get the
return path test to run. Looks like there is some sort of timing issue
where the connection takes a long time to close and this prevents it from
being re-established in the other direction (the single threaded server is
still waiting for close() to return). Have yet to figure out an answer to
this.

Spent some time with Shane, Brad and Jamie poking at the Network
Diagnostic Tool (NDT) used by perfsonar, mlab and as part of the
nzbt.org.nz broadband test. Some of the results we were getting weren't of
the quality we were expecting, so we put together our own little test lab
to see how it works. Our initial tests using a virtualised server couldn't
sustain gigabit speeds across the network bridge in one direction, despite
working fine in the other (and NDT performed less than half as good as
iperf). With two physical, directly connected machines we finally managed
to get the expected TCP performance but the extra analysis that NDT
performed was still bogus - it reports network limits that are much lower
than they actually are (and lower than what the test just observed!), RTT
values that are 500 times larger than they actually are, etc.

Worked on some ldap scripts to add new voodoo users into appropriate
groups. Tested them successfully on a machine configured very nearly the
same as voodoo. Will hopefully give them a run early next week and see how
well they work in the real environment.

Expanded on some of the install instructions for ampcentral in response to
some new users having issues. There was lots missing about new database
configuration that needs to be done to make sites and data available via
the web interface. Also built new Ubuntu and Debian amplet packages for
them to use. Had a few issues with the SSL API changing slightly and new
gcc being much more pedantic, but it builds and runs now.

Updated the display of event groups to show all events from other sites
that are related to help give a feel for what is going on. Found and fixed
a few issues with recording the timing of event groups when starting with
a fresh database and importing historical data - they should now start at
the time of the first event in the group and end at the time of the last
event.

Had more trouble with the emulation network than expected. Built a custom
ramdisk using proper tools (mkinitramfs etc) and a default Debian kernel
rather than using entirely custom ones. This worked fine, but after
installing an old image the machines would refuse to boot. Turns out that
the disks used to be part of a RAID and didn't have a useful MBR (and
frisbee didn't fix this). Built a master machine to create images from,
built a new Squeeze image and am now trying to convince frisbee to send
the entire image. I'm thinking a new version of frisbee may be required.

Tried to track down why the http2 test was performing DNS queries multiple
times for the same hostnames. Even with all the DNS cache sharing options
set in libcurl it will repeat requests, unless I force IPv4 only. A vague
line in a changelog looks like this might be fixed, but too recently for
the change to make it into Debian.

Finished putting together basic historical usage data for KAREN, it shows
a nice up and to the right trend.

Got the new graphs for latency and traceroute data rolled out onto the
live ampcentral site. Updated the install scripts to properly create the
new databases for API keys etc required to access the data in the new
style. Put this and the most recent amplet tarballs online on the WAND
research software site so they are now available, should write a quick
blog post advertising them now.

Tried further to track down the cause of incorrect data to be reported
sometimes in the new combined dns/icmp test. In the process fixed a couple
of small bugs in the options parsing, but the main one refuses to show up
while being observed.

Watched a couple of streamed sessions of the Future Broadband conference
that seemed interesting. A lot of businesses at the conference seem quite
cautious and risk averse with UFB, not wanting to invest in it till it has
proven itself in some way, while user advocacy groups can't wait to get
online. Could be some interesting developments here in the next while.