Exporting from NNTSC is now back to a functional state and the whole event detection chain is back online. Added table and view descriptions for more complicated AMP tests; traceroute, http2 and udpstream are now all present. Hopefully we can get new AMP collecting and reporting data for these tests soon so we can test whether it actually works!

Had some user-sourced libtrace patches come in, so spent a bit of time integrating these into the source tree and testing the results. One simply cleans up the libpacketdump install directory to not create as many useless or unused files (e.g. static libraries and versioned library symlinks). The other adds support for the OpenBSD loopback DLT, which is actually a real nuisance because OpenBSD isn't entirely consistent with other OS's as to the values of some DLTs.

Helped Nathan with some TCP issues that Lightwire were seeing on a link. Was nice to have an excuse to bust out tcptrace again...

Looks like my L7 Filter paper is going to be rejected. Started thinking about ways in which it can be reworked to be more palatable, maybe present it as a comparative evaluation of open-source traffic classifiers instead.

Short week this week, as I spent Thursday and Friday in Wellington at the cricket.

Wrote an assignment on libtrace for 513, along with some "model" answers.

Continued reading and editing Meenakshee's report.

Had a vigorous discussion with Brendon about what he needs the NNTSC export protocol to do to support his AMP graphing needs. Turns out the protocol needs a couple of new features, namely binning/aggregation and a no-more-data indicator, which I started working on adding. So far, this has mostly involved taking some of the working code from my anomaly detector feeder program, which is an NNTSC client, and turning it into a NNTSC client API.

Put out a request to our past students for their Honours reports so that they can be published on the website. Thanks to those who have responded.

Added a data parser module to NNTSC to process the tunnel user count data that we got from Lightwire. Managed to get the data going all the way through to the event detection program which spat out a ton of events. Spent a bit of time combing through them manually to see whether the reported events were actually worth reporting -- in a lot of cases they weren't, so I've refined the old Plateau and Mode algorithms a bit to hopefully resolve the issues. I also employed the Plunge detector on all time series types, rather than just libprotoident data, and this was useful in reporting the most interesting behaviours in the tunnel user data (i.e. all the users disappearing).

Added a couple of new features to the libtrace API. The first was the ability to ask libtrace to give you the source or destination IP address as a string. This is quite handy because normally processing IP addresses in libtrace involves messing around with sockaddrs which is not particularly n00b-friendly. The second API feature was the ability to ask libtrace to calculate the checksum at either layer 3 or 4 based on the current packet contents. This was already done (poorly) inside the tracereplay tool, but is now part of the libtrace API. This is quite useful for checksum validation or if you've modified the packet somehow (e.g. modified the IP addresses) and want to recalculate the checksum to match.

Also spent a decent bit of time reading over chapters from Meenakshee's report and offering plenty of constructive criticism.

The new NNTSC now supports the LPI collector and installs nicely. Still waiting on Brendon to get his AMP message decoding code finished to his satisfaction and that will be the next thing to get working on the data collection side.

Also started developing a new data query / export mechanism that allows clients to connect and register their interest in particular data streams to receive ongoing live data as it is collected by NNTSC. The old approach for this involved the client explicitly stating the ID numbers for the streams they wanted data for, which was pretty suboptimal because it required knowledge that should really be internal to the database.

The other problem is that now we have different data table layouts for each different type of data stream, we need to inform clients about that structure and how to interpret the data stream.

All of this meant that I've had to design and implement an entirely new protocol for NNTSC data export. It's a request/response based protocol - the client can request the list of collections (i.e. the different data stream types), details about a specific collection (i.e. how the stream and data tables are laid out) and the list of streams associated with a given collection. It can then subscribe to a given stream, giving a start and end time for the time period required. If the time period includes historical data, that is immediately extracted from the database and sent. If the time period includes future data, the client and stream id is remembered so that new data can be sent to the client as it rolls into NNTSC.

Currently at the stage where we're getting historical data out of the database and preparing to send it to the client, with live data being the next main task to do.

Been working pretty much constantly on my report for the past few weeks now. At this point I have a bit of tidying up before I write my conclusion and hand it in on tuesday. Overall I'm pretty happy with the report. I've received really good feedback from Brendon and Scott regarding how to improve and clean up my writing. At this point I'm on schedule to finish on time.

Another busy week with Assignments. It's getting to that crunch time of the year, and what with thhe 520 final write up I suspect it's going to be a pretty hard slog.

Attended the HPC Roadshow on Thursday along with Brad, Brendon, and Shane. We met up with Jamie there too. Some interesting talks and it's good to build knowledge about subjects outside my usual domain.

Spent last week preparing for my 520 presentation. Practice talks went well.

Didn't get any 520 work done last week. Pretty flat out playing catch up at the moment after my time off. Slowly getting back on top of assignments and lectures.

Started pulling things together over the past few weeks. The modules are now all integrated into the main application. Some of the forking and threading code needs some TLC but things are at least functional at this point. The main problem is scalability. I'm starting a thread for each data source which falls over when you follow a few hundred data sources. I need to be more clever about identifying resources and ensuring I'm not reading from them more than once for multiple streams. I may also require some changes/additions to the AMP api to allow me to request multiple data items in a single request.

Next Brendon wants me to stream the data (from multiple sources) in order. I'm just restructuring code so I can get at the time codes together, however I've been sick on and off this week which hasn't helped progress. After that I'll try and get the database implementation going.