Niagra and network stacks, TCP and talloc: LCA Presentations Day 1 Morning

Overnight interlude: I spent all evening installing WordPress 2.0, and fixing up a few old posts for XHTML compliance. The new WYSIWYG editor is neat, but will lose chunks of unparsable markup (ie. missed quotes and brackets). New posts will prolly be fine to use it for, but for the moment I’m sticking with writing straight HTML.

The whole AJAX interface thing is cool. I’m looking forward to the PHP5 talk this afternoon.

Of course, once I had that done, I decided to grab a new theme. This one’s pretty cool, although the whole lense thing is a bit weird…

And I’m now appearing on Planet linux.conf.au 2006, although because I use the excerpt in all my posts to produce a clarification (or declarification) in Chinese Kung Fu Novel Chapter Synopsis Style, my posts end up being quite short on the site, while long on my page.

This morning’s keynote by David Miller was interesting. He maintains the Linux networking stack, and also is the sole porter of the Sparc64 port. So he actually gave three presentations, an overview of the recent changes in the Linux networking stack, a presentation about the Linux port to the new Sun Niagra CPU line, and a brief talk about how to actually deal with kernel maintainers. Lack of wireless there meant I didn’t get my laptop out, so don’t have much more to say about it.

Well, I’ll talk about the new Sun chip, known as Niagra, UltraSPARC T1 or CoolThreads depending on who’s marketing department you ask. It’s a 8-core CPU, each core actually runs four threads in a round-robin fashion when they are able to be scheduled, and leaving them out when they’re waiting on main memory or the FPU or otherwise. This means that any task which can actually _use_ 32 threads for integer-only code will be able to run fast. Kernel compiles are a prime example (looking forward to the kernels-per-second numbers for comparison to the 128 CPU PowerPC G5 box talked about at LCA05. This would also be very nice for video encoding, I suspect. Mind you, the small Sun Fire T1000 Server (shipping March 2006) lists at US$3495, so I doubt I’ll have an array of these to play with anytime soon…. Imagine a Beowulf cluster of these things. ^_^

Morning tea interlude: Posters have gone up. There’s Thousand Parsec, WorldForge and FAI. I’ve looked at WorldForge and Thousand Parsec before, at LCA05, but if I have time tonight (Ha!) I might see where they’re up to these days. FAI on the other hand I’ve only been vaugely aware of, since I never seem to deploy more than one box at a time… But now it’s in my blog, so I’ll be able to find the link when I do want it.

Congestion Advancements with Ian McDonald. A technically-oriented delve into the new congestion control algorithm module structure for TCP, as touched upon by Dave Miller.

Ian presented both the work done recently to generalise and modularise the congestion control algorithms for TCP in the Linux kernel, which had originally been kind of ad-hoc and wide-raning in their touching. The interface they use is fairly simple (if you know TCP backwards, that is ^_^) and they turn out to be per-socket switchable. This will allow much easier use of different algorithms, which are optimised for various combinations of high and low bandwidth, high and low latency, and timeout vs loss vs congestion vs drop situations.

He then presented his current research project, which is a TCP-like protocol (I think… Or was it a congestion-control algorithm?) called TCP-Nice, which is designed to back off from congestion so that the rest of the network functions as if it wasn’t there, while it uses all the left-over bandwidth… I like this, I’d love to see BitTorrent ported to use it. Then I could give free TCP-Nice traffic, and lower my TCP quotas significantly. ^_^ A vast improvement over my previous Second-Class Traffic plan.

He then presented a futher, already live use of the modularised congestion control code in Linux, DCCP. This is a session-based congestion controlled (like TCP) unreliable (like UDP) protocol, mainly intended for multimedia traffic, where you want as much as possible to get through, to back off (somewhat) under congestion, while not doing retransmits and re-ordering since retransmitting live data is a pain.

It’s in the final call for the RFC, and he’s already gotten it working. It’s in the 2.6.14 Linux kernel, with a NAT fix to come in 2.6.16. However, they still haven’t gotten the perfect congestion control algorithm for multimedia streams… The TCP-like CCID2 isn’t very good, the smoothed and slower-falling version CCID3/TFRC didn’t help much, and the latest attempt, MFRC is currently too agressive, and needs tuning to avoid killing other traffic under congestion conditions. But it’s getting there, and shows a lot of promise.

Netem: A last-five-minutes gem… Introduces loss, delay, reorder and duplicate packets on an intermediate box. Can only currently work on output queues.

Finally for the morning, Rusty Russell presented Talloc. Talloc was touched upon by Tridge in his “non-junk” code tour at LCA05, but he didn’t spend too much time on it, looking mainly instead at tdb and ldb…

Basically, talloc is a hierachial pool allocator, which gives destructors, pools and hierachy to your memory allocation calls. This mean that managing your memory usage in C becomes sensible. It’s mainly been driven by Samba, which in fact produces huge whacks of memory allocation… Rusty showed a graph of it, I’ve no idea where to find it. (There was also a URL to the program to make such graphs, I missed that too. >_<) Anyway, it’s pretty impressive.

Andrew Bartlett pointed out last week that he’s using talloc in Samba 4 to trivially wrap the krb5-allocated blobs coming out of the kerberos libraries. This basically gets him free destructors, solving the nasty lifetime problems kerberos’s allocation and free activities otherwise bring.

nfsim uses talloc to simulate kmalloc, providing simple and easy kernel memory leak detection in the netfilter modules being tested. Also has a very neat graphical live talloc allocation tree display. I think that is really neat!

Now to find myself a project to use talloc on… That’s also what I said last year about tdb, as it happens. I actually have one for the latter…. I want to unbone FreeRADIUS‘s IP Pool module, specifically so I don’t have to kill FreeRADIUS to make changes to the pools. I just didn’t get it done in the last 9 months. Gah.

In the more general programming talk at the beginning of the talloc presentation, Rusty suggested that interfaces should be hard to misuse first, easy to use second. He also suggested the following list of tools as being of great importance:

  • distcc
  • ccache
  • ccontrol – This one’s new to me. In fact, I’m still not clear what it does…
  • Mercurial – Source control tool. I’ve not tried it, but Alan DeKok from FreeRADIUS uses it for his own development, and then breaks up the patches for shoving into CVS for the rest of us…

Interview with a Sambpire: LCA Tutorials Afternoon

Samba4 status presented by Andrew Tridgell, Andrew Bartlett, and Jelmer Vernooij.

Samba 4 Tech Preview 1 has been released. In the tradition of named released, it’s known as “Rigged Demo”. ^_^ It’s been packaged in Debian experimental for months, but you of course knew that…

The list of cool cool things is kinda long, but here’s some very quick notes: Javascript-based scripting API. Finally, actually scriptable remote AD administration. In fact, there’s Javascript embedded in the server-side of the new, AJAJ-enabled SWAT allowing remote web users to remotely administer other machines in the AD domain…

Which of course leads to the fact that Samba4 TP1 can already function as a AD Domain Controller. Tridge gave a live demo of Active Directory Vampire! Point it at a Win2k3 AD DC, and hit “exsanguinate”. Less than a minute later, a windows XP machine which was on the domain with the old DC can login without missing a beat.

Andrew B? was telling me on Friday night that Samba4 is a monolithic server, but is completely non-threaded… When a part needs to wait for a response, it runs the event loop again. If a message comes in for another part of the system, that part gets to run, simply growing the stack of paused subsystems. This follows on sensibly from Tridge’s talk last year about how bad multithreading is, how bad IPC is, and why can’t the various Samba4 components all just get along in one address space…

The Samba4 core team took a brief vacation earlier in the year to look at the new Windows Vista tech-preview, specifically to pull apart the new and completely undocumented SMB2 protocol MS has slipped underneath. Consistent with their drive to make everything a database, it apparently has some kind of support for database-like transactions and rollbacks. The good news is that Samba4 has client-side support for it, and will probably release a full SMB2 product before Microsoft does. Just another way in which Open Source stifles innovation™, I guess.

Also this afternoon, a presentation on L2TPNS by Brenden O’Dea, who presented the same software at LCA05… I don’t remember much about the presentation last year, but I had at the time just started playing with it for IPSec over 802.11, since I didn’t have any WPA-capable hardware and was sick of cables running along the roof at Shane’s place.

Since then, I’ve moved, I’ve got WPA-supported gear, and I’d forgotten who Brenden worked for. I told someone “a small ISP in Mildura” a couple of weeks ago. As it happens, he’s from Optus, and his software is supporting about 170k concurrent ADSL connections. So it’s good good stuff. As it happens, I’m back in the L2TP market just now, for both ADSL and hopefully a VPN solution, so I was glad this particular presentation was very very featureful, and talked frankly about the challenges involved. Also, Optus’s ISP serverside is totally Linux-based, and as other departments hand servers over to them to operate, they get Linuxed too. It’s a wonderful thing… Sounds like a great place to work.

Linux Australia AGM… Nothing to see here, move along people.

Keysigning… Not really entertaining, although I am equal 11th in the MSD, for those registered this year. This puts me out at the end of the unfashionable western spiral of the central cluster.

Well, it was faster than last year. Done in about 45 minutes. ^_^

So, dinner hunting time. WordPress 2.0 upgrade to do tonight. And LPI study. And more S.O.S to watch… And gotta get more sleep than last night. (No last-minute libmysqlclient15 bug hunting this time…)

(Edit: &tm; is not the HTML entity for the tradmark symbol. &trade; is)

The power (or lack thereof) of QT4: LCA Tutorials Morning

Overnight interlude: Talked to rene, he didn’t seem enthused about a dscape package, so I’ll do it myself (based on his code. ^_^). Also, halfway through S.O.S. I really just want to hit Irie-san with something… Gah!

LCA2006 conference opening this morning. As I’d gleaned from the Rough Guide to New Zealand, it is unexpectedly hot. This place looks more and more tempting, although the Internet link (NZ – Spiritual home of IP over Carrier Pigeon according to one wag) scares me.

QT4 tutorial this morning. Not a lot to say, lots of concentrating on slides. However, I did observe a serious problem that I was not expecting… They’ve not supplied a single power board, and the powerpoints in Castle 1 are even more dearth than at the ANU. Castle 2 (where the Debian miniconf was) is excellently appointed for powerpoints, on the other hand. On the gripping hand, this was my main contribution to the pre-LCA network survery they undertook… Oh well, I’ll see if I can go make noise on IRC or something…

Well, maybe I’ll say something about QT4. I’ve not done any GUI programming before, so it’s both interesting, and looks like a lot of work. Much as I can read and understand C++, I think I’d be more comfortable doing it in perl or similar. Also, the presenter moved quite quickly, so I suspect this was aimed slightly over my head (although I could follow what was going on, I certainly couldn’t be aping him as we went. I guess I was spoilt by Rusty and Robert’s kernel module tutorial last year… Maybe it’s like The Princess Bride in that it’s a beloved movie, but for many people watching it years later pales in comparison to their fond memories… Although neither the kernel module tutorial nor The Princess Bride have paled in my memory yet.

Oh yeah. And I had to keep stopping to apt-get install things during the tutorial. We were told we’d need qt4 w/sqlite support (qt4-dev and qt4-sql), nothing was mentioned of the Assistant (qt4-doc qt4-dev-tools) Designer (qt4-designer) nor the SQLite client (sqlite3) and admittedly the first and last are optional. I guess the presenter assumed we’d be fetching qt4.1 from source and installing it…

Morning tea: Rene’s online, and _is_ working on a dscape packageset. ^_^ And in completely unrelated seen-on-IRC news: OpenJazz Jazz Jackrabbit 1 reimplementation (uses original tilesets etc)…Someone’s started porting it to the Nintendo DS. More on this after I find cookies or something.

Also during morning tea, I was roped into coming to the Perl BOF on Thursday afternoon. Although there’s several BOFs on I’m interested in, the perl6 talk from Monday has me all Perl-enthused right now. Especially if it means I can sensibly resurrect my SOAP server for CBIT, and send out my auto-emails without the evils of perl5’s format code. (This modules implements Exegesis 7, although I think what Damian described on Monday was slightly different. There’s no Synopsis 7 yet either. -_-

More QT4. I18n support’s there and easy to use programatically, although they apparently expect literals in UTF-16… The suggestion is to work in latin1 and just provide a translation for the target language to start with. This is prolly a quite sensible idea in general, as it allows your documentation writers to rewrite the text in the interface as needed, and lets the programming team leave the messages in the untranslated version in a format and phrasing useful to them, which as everybody knows is rather different from the way users work. Who is General Protection and why did his mistake crash my program?

This reminds me of a neat thing an IRC friend of mine once showed me that she was working on for a games company. She’d managed to independently re-implement po (in fact, it looked more like the QT4 i18n does, now I’ve seen it) for Visual Studio programs, while extending it to not just text, but all kinds of resources, drop-in-able with DLLs. So graphics with embedded text, video, audio and country-specific non-language things were all trivially handleable by their l10n teams. I bet it was doing encryption too, from my experience with Japanese games’ text resources.

Wow, haven’t seen or heard from her since the fall of #pgsm to TVNihon… One of the sadly few IRC friends I have who get my programming stuff, my linux stuff, my anime stuff, my random Japanese stuff, and was still genki in the morning. ^_^

News flash! Trolltech has announced that US spelling is indeed wrong… In some cases internationalization is simple, for example, making a US application accessible to Australian or British users may require little more than a few spelling corrections. ^_^

Lunch: Seen on Planet linux.conf.au 2006: Splashpower, an induction-charger for mobile devices. According to Arjen Lentz they’re in discussions with distributors. Amusingly, each submenu on their site has a picture at the top of someone who stuck the little metal strip to their foot and then walked across the pad. Possibly this could supplant firewalking as a harmless but dangerous-looking power-activity (or would that be extreme walking?) of the future.

Also lunch: I finally understand why the power-point cable on my PowerBook’s adaptor comes off. You can replace it with a plug, which is almost as neat a solution as the retractable version I was talking to Jez about last week. I wonder if that’s something I need to poke my boss about, or if it’s an optional extra…

Sockmonkeys of fire: LCA Miniconfs Day 2 Afternoon

Horm’s followup talk on Debian kernel packaging from LCA2005 was great. At LCA2005, Horms gave us a rundown on the new kernel-team plan they had developed and not yet implemented (as they were waiting for the Sarge release, amongst other challenges). Since then, Sarge has released, they started the process with 2.6.12, and they’ve brought all 11 Linux architectures’ kernel packages together as of 2.6.15. Which is a hell of an achievement.

They’re using a SVN directory to track the security and sarge-affecting patches, to prevent losing track of problems in either people’s inboxes or the security team’s enormous pile of issues.

A brief interruption due to a firealarm put us somewhat behind schedule… Turned out to be faulty equipment overheating in the roof.

Horms mentioned that he spent a year doing bug responding for the linux kernel, and burnt out… I must remember to talk to him about that, I like answering bugs. ^_^

Bdale’s presentation on “HP and Linux” was interesting, showing how a corporation and a community can cooperate in a way that suits both. I’d love to say more, but I’ve had wireless problems for two hours and it’s sliped my mind…

I might change back to dscape, actually… This wireless has been fairly dodgey all day. I’ll see if I can massage Rene’s packages into building devicescape, he’s done the hard yards of dpkg-divert etc.

Keith Packard returned to tell us how the launch of their LV2 rocket which he spoke about last year went. It was spectacular. Go read their site, but long-story-short, they now have a 3000G acceration-tested sockmonkey.

Bdale again took the stage, for another State Of The Project-type thing. He briefly talked about how things such as the Debian Project Leader Team “SCUD” had gone in the nine months since it came together, and how that related to what he’d said last year at the other end of the past nine months. He then moved on to talking about some of the more general issues facing Debian, some new and some quite old.

As well as the onsite mirror server (which we can’t add to…) we now have an onsite proxy server.

Hmm, wireless seems to have settled down now… Still gonna poke Rene about dscape packaging. ^_^

For those who care about perpetuating memes: LCA Miniconfs Day 2 morning

In overnight news, I’m now on both the softmac bcm43xx drivers. Thanks to Rene Engelhard for letting me at his upload-ready builds while they idle in the Debian NEW queue. They worked OK in the unicol lounge, but I’ve not yet had them in the real world (“lecture theatres”) so I still have the dscape drivers lying around as backup.

I also (as of 2am) have now upgrade to the new, 9base-based wmii 2.5.1 and already hit my first 9base bug. (Date doesn’t handle “NZDT”, unclear if it’s 9base’s fault yet). It’s close enough to the 2.2 I was using before that I don’t envision too many issues. ^_^ The main problem apps (rdesktop and mplayer) appear to be working fine. I had to build both from source, as wmii’s not been uploaded from the autobuilder, and 9base appears to have had three from three buildd failures although not on major architectures (ie PPC) yet. The main improvement is that it no longer warps my mouse to the top-left corner on frame change… Now if I could get the bottom-bar to show the multiple windows open in a frame, I could dispense with that line entirely and get an extra line visible on my xterms. ^_^

More Debian miniconf today. Like all really good technical people, it didn’t start until 10am. On the other hand I was here at 8:45 to give myself time to sort out any bcm43xx issues. (Apart from having to manually put either the channel or AP address into iwconfig sometimes… I think it doesn’t failover properly.) So I spent an hour on email, and IRC discussions of the awfulness of MS Exchange and the wonders of Japanese schoolgirl porn DVDs.

First up, Matthew Palmer presented a delve into the mechanics of the .deb format and dpkg’s handling of it. This was quite interesting, and faster than reading all the docs (Policy, dpkg manual, lord knows what else!) and I hope sets the tone for today to “technical”.

Next, Russel Coker presents SE Linux, and where Debian stands with it. It seems that Debian was his initial target, but it proved too resistant at the time and he’s gotten it into Fedora Core instead. As I recall, SE Linux support has been slowly working its way into Debian, so we might be seeing the “targetted” policy, which is what FC installs by default, working its way into Debian installs sooner or later.

Targetted policy is a policy designed to lock-down and affect only daemons that are either large and complex, or have a history of security issues (former eg. Apache, latter eg. bind, dhcp-server). This way it doesn’t interfere with normal usage or workflows, although it therefore doesn’t protect user login sessions.

SE Linux is hard… I think I get it, and at the same time I think I don’t. I maybe ought to do the hands-on tutorial on Thursday, but it clashes with too many things I want to attend.

Oh, MLS is both fairly clear and interesting. Basically, on top of the DT model, this allows levels (eg Top Secret, Secret, Classified and Unclassified, but it’s arbitrary) to be assigned to both programs and objects. Apart from certain exceptions (specially authorised users/programs) this means that a less-secure program cannot read a more-secure object, but can write to it, and a more-secure program cannot write to a less-secure object but can read it. This prevents accidentally declassifying information, and is something the military already does, apparently.

It’s slightly more complicated, because programs have more than one level at once… If a program has only a level matching an object then it’s read-write. If a program has a matching level and a higher level than the object, it’s read-only, preventing “write-down”. If the program has no levels greater than or equal to the object’s level, then it’s not readable, preventing “read-up”. I presume at no point does it become write-only, because that’s kinda daft… *flicks hair* — Security Barbie

And now, lunch… Well, I just took a penicilin capsule, so I’ve gotta wait half an hour… Just got bdale to explain his method of doing keysignings without needing a trusted printout of the list. ^_^ While I’m bumming around, might see if I can get any joy out of this atheros card with the driver from http://ath-driver.org

Edit: That’s perpetuating, not pepetuating. WordPress needs a spellchecker.

The bear dances: LCA Miniconfs Day 1 Afternoon

Even before the presentation, useful tidbits came flying out. Damien and an audience member had a discussion about practical aspects of presentations (ie. looking at code and things during the presentation). Damien pointed out that he structures his presentations around the assumption that people have nothing, and that the audience doesn’t really want to be interactively compiling and examing code and whatnot. As it happens, I’m hiding up the back anyway, trying not to be noisy with my keyboard. I’m actually back here because that’s where the powerpoint is, but I guess it’s all for the best.

I don’t think I can do justice to this talk. So trust me, it’s fascinating. He’s given it before and gets rave reviews. ^_^

It’s certainly an amazing and useful and complements my recent reading of the Perl 6 Synopses, (I’m only half-way through 2, as I was reading them during a series of outages at work, which solved itself right as I got into the swing of it…

OK, one thing that is so glorious I have to write it:

my $π = 355/113;

And that’s not even the coolest thing. Well, I think it’s cool, ASCII people will think I’m dumb (To whom I say “«»”, “¥” and with a bit of luck “≅” although that is currently only ~~ and in fact is rendered differently in different fonts. For me, by hand it’s two ~s on top of each other…), but there’s just so much good good stuff here.

Actually, the coolest thing might be that the Quantum::Superpositions module (by Damien Conway, as it happens) is part of the basic perl6 syntax. He’s calling them “Junctions” or “Junctive Operators” now.

Side note: The men’s toilets near Burns 4,5,6,7 appear to be designed such that, no matter how carefully you aim, you reflect your urine back onto your own shoes. I highly recommend a look… Women might want to knock first or something, and possibly also visit the urinals in the Link area to get an idea of what you’re comparing to.


OK, 10 minutes before the (scheduled) end, the coolest neat thing appears!

...

compiles but does not run. That’s actualy its job. It came up in the “roles” discussion. Roles are like “interfaces” in Java, but more.

I guess the final summary is that perl6 is all the DWIM of perl5, plus more, plus real Object-Oriented, plus Aspect-Oriented, plus a real and good VM, plus literate programming, plus functional programming… But wait, there’s more! And no XS!

And in good news, a very very very provisional release suggestion is in the next twelve to eighteen months. ^_^ Want it sooner? What perl6 needs: Money for the people they’ve got to put more time into it (including hiring Damian to train perl), developers to put time into it, and people to champion the cause in their organisations… Hmm. Isn’t that the Debian wish-list? (And the Ubuntu got-list…)

Gonna be a hell of a keynote on Friday.

Debian bunfights, AJ sells his soul, and Battlestar Galactica: LCA Miniconfs Day 1 Morning

Happily, wifi works in the theatres. ^_^

First up in the Debian minconf was just enough of S2 Ep1 of Battlestar Galactica to remind me of where the Australian TV season’s up to, and probably frustrate those who haven’t already seen it.

Next was a presentation by a Ubuntu MOTU member regarding Debian/Ubuntu collaberation. A bit of interesting information about how Ubuntu’s Debian-collaberation stuff works, a few buns thrown, and a fair bit of jovial back-and-forth. Just what I was hoping for. Interesting link: http://revu.tauware.de/~lucas/versions/ – Version comparisons between Ubuntu and Debian.

Interlude: Coffee, chat, and the next 20 minutes of the BSG episode.

Anthony Towns and his way of trying to get people to pay him to work on Open Source projects. It’s kind of an inverse bounty where he posts what he wants people to work on, and people put up money to what they want done. It involves a Supply and Demand curve pair apparently, but he didn’t post slides… So here’s the link to it: AJ Market. The discussion evolved into a look at “betting markets” as a replacement for bounty systems. The idea is, you bet $x against something being implemented. If it’s implemented, you got your feature. If it’s not implemented, you get your money back, plus money that was bet _for_ it being done. People working on implementing it bet on it being implemented, and share the winnings if it is implemented. Obviously this places a bit of risk on the side of the implementors, but it’s a fascinating idea.

Overhearing a conversation about this while I was packing up, apparently part of this also involves the buy and selling of “bets”, which actually turn out to be more like shares… The idea being that a programmer can pick up cheap shares (eg. because the project’s been ignored for ages, and is seeming very unlikely) and then complete it, realising the full value of the shares. This requires that any initial investments are made on both sides of the bet. This situation would encourage people to ignore things they know they _could_ do until it’s worth it to do it, but at the same time if you want something done, you can buy your shares on both sides, and give away the “will be done” shares to a likely candidate, which gives them an incentive to get it done, as the shares are only on paper until the project is completed.

I’m sure AJ’s blog has more useful information, or will do really soon now. I’m interested, myself. Anyone wanna bet me that I can’t do Linux WMB for things other than ralink cards? ^_^

An interlude in the history of PHPRasmus Lerdorf is actually presenting a tutorial onPHP5 behind Web2.0 (AJAX etc) stuff which I’m probably going to go to on Thursday afternoon, much as it clashes with the tutorial on programming Asterisk PABX at serveral levels and the latter is highly relevant to what I’m supposed to be doing for a living.

This afternoon, however, it’ll be the Perl 6 session at the Damien Conway miniconf in Burns 7. ^_^