The night Gmail stopped talking to me

I used to be on a lot of mailing lists. Back in the 90’s and 00’s, I was doing a lot of Open Source contribution.

Debian, Wine, Thousand Parsec, OpenJPEG, Second Life Viewer, rt2x00, dwm, X.Org, uim, FreeRADIUS; to name-drop a few. My tendency to show up and contribute a single patch in those days meant having to subscribe to a -devel list (and possibly a -patch list), email your patch to the list, and keep monitoring your thread to respond to feedback, iterate on the patch, and eventually enjoy the warm glow of commit.

And once I was on the list, I tended to never leave. I also had a tendency to become the Debian maintainer if one was needed. And in one case, the Release Manager for a release.

Tonight, it ended in blood and tears for many such lists…

Actually, that’s not true. Many of those lists had ended themselves since I was subscribed. And it looks like some time in 2009, my university email address expired and so my subscriptions through that went away. (Apologies to the bounce-bots….)

Still, some of the larger projects stuck around, and had my now-20-year-old email address. Even my mobile phone number is slightly younger than that. Actually, some of my colleagues are younger than that, I suspect.

That means I’ve had the same email address for more than half my life. Or longer than Google has been around.

Speaking of Google, at some point I realised that maintaining my own email server was a dumb idea, and migrated my email to Gmail. I remember hunting down a script to upload all my own email via IMAP, and leaving it running for days.

Being a compulsive hoarder, I couldn’t leave anything behind. That gives me full email history to 1998 (according to my sorted folders) except a year where a typo erased my inbox instead of archiving it in the early 2000’s.

One thing about mailing lists, though, is they’re archived elsewhere. So it’s much easier to delete old email from a mailing list, even for a hoarder. However, I realised this much too late, and so email continued to accumulate.

Gmail’s UI isn’t great at unsubscribing from Mailman mailling lists, which is what everyone serious in Open Source used, “back in the day”. Mailman’s own development email list was hosted on Sourceforce, because… reasons.

There’s a header in each email indicating how to unsubscribe, but you have to “show original” to see it. Usually it’s a http: or mailto: link, which is not clickable in that view.

What the Gmail UI does do well, that the Inbox UI (what I normally use these days) does not, is let you multi-select in any way other than “Click one by one on the emails you want”. They have an ‘All’ button, which selects all the email on-screen, and then gives you a “Really, All” link to let you select everything that matches the current filter/label.

This will happily let you select and delete over 77k emails in one click. That was my largest folder, wine-patches, but only because I unsubscribed from debian-devel last time I tried to clean up my email.

What it will not happily do is delete 77k emails (move them to Trash) quickly. In fact, while I was waiting and occasionally clicking the “Something timed out, try again” links, I decided I should write a blog post.

This blog post.

I was hoping to end with “And I’m still waiting”, but it did finish while I was working on this. So despite the rocky start, Gmail and I are once again on speaking-terms.

Never go to bed angry with your email provider, as the proverb should go.

Instead, I get to explore how long it takes to delete 131,585 emails from “Bin”. That should give me time to follow this up with some thoughts on how Open Source project communication has changed. Expect a post about that some time in the coming week, since it overlaps with something I need to do in the coming week as well.

Update: It finished in under half an hour, while I was editing and adding links to this post. And relived some of my finest wall-of-text emails where I completely miss the recent release of Git when considering distributed version control.

Only 3gB of email deleted… It looked larger in the rear-view mirror.

Advertisements

TBBle Scarry’s Busy, Busy Weekend

Often my weekends start out with grandiose plans of what I might try and get done.

This weekend (and the preceding evenings I guess) saw me produce a Wine patch I was only playing with out of interest but which turns out to affect Warhammer Online, although I didn’t know it until after I implemented the patch, and a WIne patch I’ve been meaning to prototype for a while using XInput 2 to fix a long-standing Wine bug which also affects Warhammer Online.

I also got back to watching Life On Mars, although I’ve only managed one episode and a bit. It’s pretty damned good.

I also decided to make gyoza, as I have fond, alcohol-supported memories of the last time I made them.

I managed to lazy my cooking even more than usual. I’m using a recipe I picked up last time I made them off a site called The Food Palate by Deborah Rodrigo, whom Google has since informed me is from Sydney but both that site and her personal blog appear to have fallen off the Internet, sadly. However, I distilled (with the help of Kirky at work) the ingredients down to this:

Ginger, chives, chili flakes, coriander, garlic, sesame seed oil, soy sauce for dumplings, and gyoza skins

Ginger, chives, chili flakes, coriander, garlic, sesame seed oil, soy sauce for dumplings, and gyoza skins

Adding half a kilo of lean pork mince, and about a half-hour, you get:

30 gyoza, freezer-bound

So not as bad as the ugly cake I made recently, but still not spectacular. And unlike the cake, I don’t yet know if these turn out to be poison or not.

I expect that they’ll be delicious, and not even slightly poisonous. And unlike my cake, I’m not going to try to share them with anyone. ^_^

It could be worse, at least I seem to have not managed to poison my housemate’s lizards, Prime and Grimlock, whom I’ve been feeding while he’s away this weekend. I’m not sure how I could get “put grasshoppers into the box” wrong, but I don’t think I did. I think they’re pretty neat names for lizards, reflecting Mick’s inner geek, and his outer geek, although Prime seems to be larger than Grimlock which is to the best of my knowledge the wrong way ’round.

I was going to try and leverage in a rant about characters in children’s books with alliterative names at this point, and observe that one of my favorite authors as a young child, Richard Scarry happened to avoid that, but upon actually looking him up, I realise the characters whose names I’d forgotten quite often had alliterative names. The characters I remembered still had non-alliterative names, so it’s not as bad as some authors I can’t be bothered remembering, but I’ll chalk that one up as being disappointed by a childhood memory.

A less disappointing childhood memory turns out to be Piers Anthony‘s Incarnations of Immortality series. I read the series when I was quite young, and I’m only re-reading the first one at the moment, but it reminds me how good a writer he is, and why I loved his books so much as a child. Also because he’s alphabetically early on the shelves. I don’t know why I seem to do that. I think when I’m picking a new series, I start at the beginning and go until I’ve chosen one. So that favours the alphabetically early.

I’ve managed to get a whole bunch of reading done recently, which is good. Sadly, Borders now wants me to pay $7 on a $14 book to order it in from overseas, and it turns out most of the series I’m following keenly enough to actually order books are on that list, so I may end up having to do an Amazon order. Which is annoying, because I’m also looking for some DS games: Ace Attorney: Trials and Tribulations appears to be discontinued in Australia and the US, and Impossible Mission never seems to have been released here at all. Along with wanting Race on DVD, I have a fair bit of overseas shopping to do, and the local financial climate is not exactly conducive to that. -_-

Anyway, the above is my documentation supporting why I should not be left alone for days at a time. ^_^

Edit: Fix images after changing hosting.

GLUTton for wxPynishment

A followup to last year’s foray into Python and OpenGL.

I noticed today that wxPython includes a GLCanvas class which uses PyOpenGL. However, as of 2.8.9.2, wxPython’s demo crashes due to wxWidgets bug 10203 which is fixed in 2.8 wxWidgets post 2.8.9, and in 2.10.

Until a wxPython release comes out based on either of those, there is a workaround. The script update_manifest.py, which wxPython includes to change the manifest in your python.exe and pythonw.exe to use the Windows XP comctl32.dll, also fixes this problem, so even though I’ve been aware of this bug for ages, I’ve only learned about this workaround tonight by reading the wxpython mailing list archive.

Now that I’ve got that patched, and PyOpenGL installed, the GLCanvas demo in wxPython runs, and the cube demo works. The cone demo however comes back with this:

OpenGL.error.NullFunctionError: Attempt to call an undefined function __glutInitWithExit, check for bool(__glutInitWithExit) before calling

This turns out to not be a surprise, as I don’t have GLUT (glut32.dll) installed. Sadly, the wxPython demo code doesn’t test the result of the OpenGL.GLUT.glutInit method in PyOpenGL, so this exception is simply output without causing the cone window to abort.

Since the draw code for the cone calls glPushMatrix before any of its glut calls, and the glut calls throw an exception so you never call glPopMatrix, you end up filling your matrix stack, and getting a lot of error spam in your output window, where the later errors can easily push the older errors out the top of your scrollback buffer.

I turned out to be too lazy to build my own glut (it’s anecdotaly possible) but a lucky hit with Google informed me that Nvidia’s Cg Toolkit includes both a win32 and x64 version of glut32.dll. You wouldn’t be able to distribute it as there’s no license indication for glut apart from the license for the whole Cg Toolkit. The glut.h file included however is the one from normal Glut (or so it appears) so I doubt it’s anything except the win32 version of upstream glut.

On this point, it’s not obvious to me if freeglut is supposed to be a drop-in replacement for glut32.dll, or a souped-up alternative. It doesn’t help that the freeglut configure file includes an option to switch that mode on or off (producing libglut.so or libfreeglut.so) while the .mak file (for NMake) only produces freeglut.dll not glut32.dll. So I guess it’s intended to be both. The next step would be to see if freeglut can build from configure using mingw64 and produce a drop-in glut32.dll.

However, I don’t care that much. I only wanted to see the wxPython GLCanvas demo run. I won’t be using GLUT (or event GLCanvas, to be honest) myself so this has had plenty of time devoted to it anway.

I guess I hope that the main benefit of this blog posting is to allow those very occasional forum posters who go looking for glut32.dll for Vista x64 or XP x64, or even Vista 32-bit, to find it in the NVidia Cg Toolkit. So far I’ve seen several such questions when searching Google for a copy myself, but no one ever finds one for 64-bit. (There’s a 32-bit one in the bullet physics SVN repository, if you don’t want NVidia’s one.)

And for those same forum posters, a quick note. The x64 build of glut32.dll goes in %WINDIR%\system32 on x64 machines. The win32 build of glut32.dll goes in %WINDIR%\SysWOW64. If you get this wrong, you’ll get the same error messages as not having the file present at all. It’s prolly both easier and a better idea to actually drop the glut32.dll next to the program you’re running, unless you have both 32-bit and x64 versions in the same directory for some reason…

Grinding code in Warhammer Online

My original plan was to only use the Windows XP 64 installed on my laptop for video games (and then only when necessary due to a Wine disfeature) and Linux for everything else. My World Of Warcraft days actually worked quite well for this, as it played very nicely under Wine. However, as interesting things (ala my previous blog post) sometimes crop up while I’m in Windows, and also as I’m now playing games that aren’t so nice under Linux, I’ve ended up being in Windows more than Linux. And now I’ve found myself distracted from games playing by, of all things, MMO UI Addon programming, keeping me in Windows even more.

You’d think with my strong awareness of the commercial nature of grind, I’d prolly be trying to get all the playtime I can out of my monthly subscription to Warhammer Online. Instead, I seem to be burrowing my head down into some UI programming in Lua. Like WoW, WAR (or WHO as a friend of mine calls it) uses LUA to implement its user interface and provides a way of adding modules in to modify, adjust or just plain futz with the interface. The big site for WAR addons (like WoW addons, in fact) is Curse Gaming and they even provide a Sourceforge-like site for addon development called CurseForge.

Anyway, why am I doing this, given I managed to avoid WoW addon programming for my entire playing time? Apart from external reasons I’m not going to post here, WAR being brand new is missing a fair few addons. None I can’t live without, but one it does lack is DrDamage, which enhances your ability tooltips with the actual effective values of the ability once gear and stats are taken into account.

Part of the issue is that WAR’s combat calculations are not fully understood yet. An excellent primer is available at Disquette’s Weblog and Warhammer Alliance has a Mechanic Analysis forum as well. I’ve posted some comments at the former, but the latter requires you to be a “WAR Soldier” before you can post, and I seem to still be a “WAR Recruit”, which means I haven’t contributed enough to the Warhammer Alliance forums. Ah well.

So anyway, my addon. LibCombatCalcs is my first MMO addon, basically supposed to encapsulate the various combat number mechanics of WAR so that I or someone else can write tools like DrDamage (or RatingsBuster) which magically continue working when they change the mechanics, and which don’t need large hard-coded tables of information duplicated across each addon.

It also intends to tie together the seperate sources of combat information into a single coherent stream for other addons to listen to.

Anyway, we’re not there yet. What it does do right now is record hits against monsters, and give you a little window with /lcc mobinfo which shows the calculated toughness of the monster (from an unambigous non critical autoattack) and the calculated values for all the subsequent abilities you used, letting you see if my calculations (and therefore my transcriptions of the community’s understanding) are correct, and/or where things need tweaks. I’ll be using this (and I hope others do too, I don’t want to build a level 40 of each class to do this…) to identify the sources of DPS that contribute to each ability.

Anyway, there it is. I’d love to hear feedback about it, preferably at Curse/CurseForge but here is fine too if you hate those sort of sites. You can clone the git repo from CurseForge, and it currently autopackages every commit I push so you can also grab and install the zips.

By the by, this is my first time using mysgit although I did contribute some work to a different msys git effort, and it combined with Console and an updated Vim with some nice colour schemes (I’m using xterm16 at home and work now) makes me a much happier Windows programmer on my laptop.

On other fronts, I’ve recently been playing with Python-Ogre, hoping to knock out a 3D physics-based tech demo of some kind with it in the middle-term future. (May end up being a Christmas break project…). After my disappointments with 64-bit Python and Pyglet under Windows, I may end up doing it under Linux. Ideally it’s cross-platform of course. I’ve also done some more serious work on my book cataloging software using Elixir, SQLAlchemy and SQLite to turn my collection of text files into a real database. However, there’s not a particularly good way of dealing with schema changes that I can wrap my head around, so I’ve put that on hold while I think about how the data’s going to have to look in the long run. And then I got distracted, so it’s on the Christmas break pile too.

In order to put sprites on screen, you must first compile the universe

I’ve just spent my weekend attempting to get pyglet or pygame going on Python for Windows AMD64 (2.6b2).

The process uncovered bugs in pygame and python, including one which is fixed in 2.6b3, although there’s no Windows builds of that version yet. Also, you can’t currently rebuild Python under mingw. Still some patches to go: pygame tries to build a safe version number for bdist_msi however it gets it wrong in release, I suspect the rules have shifted slightly in 2.6…; and whatever else I’ve modified in my build trees that needs to be sent upstream.

Mingw64 was able to build stuff now (thanks to @NightStrike on the #mingw-w64 IRC channel) but Python’s pretty insistent that I build against msvcr90 and mingw64 doesn’t have an import library for that yet (mingw32 does, but mingw64’s runtime collection is only up to 3.11 or so). @NightStrike informed me that pexports has been ported to amd64 on the mailing list, but I haven’t dug it up yet, but that should allow me to link to msvcr90 from mingw.

Either way, I now have a modified win32/Makefile.gcc for zlib which doesn’t use dllwrap (deprecated since 2002) and works with mingw64 nicely.

After mingw64 worked but I hit the requirement of msvcr90, I grabbed the Windows SDK and the DirectX SDK. Once you know to run dx_setenv.cmd in the Windows SDK cmd window and to use vcbuild /useenv (otherwise it’ll ignore the results of dx_setenv.cmd) things seem to just work. You need to set DISTUTILS_USE_SDK in your environment for Python to trust your compiler version choice, too.

Converting SDL‘s Visual C projects and solutions was easy. Change platform from win32 to x64, change any /MACHINE: entries from I386 to AMD64. I should submit that change to SDL, once I convert and test the examples as well. SDL also builds with mingw64, but I suspect it doesn’t produce a DLL at the end right now.

The Windows SDK 6.1 includes msvcr90 and the Visual Studio 2008 beta 2 toolchain, so that works for Python extensions.

However, after building everything fine, initialising pygame fails:

Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

C:\Documents and Settings\root>python
Python 2.6b2 (r26b2:65106, Jul 18 2008, 18:24:10) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygame
>>> pygame.init()
(3, 1)
>>> pygame.get_error()
'DirectInputDevice::SetDataFormat: Invalid parameters'
>>>

The Internet knows nothing about that particular error in reference to pygame. Since I had to install the DirectX SDK to get this far anyway, I’ll see what that produces in terms of debugging output, when I get back to it.

As for pyglet, which started me on this whole mess? Upstream says it’s not supported under x64, don’t bother submitting bugs. Before I found that, I tried to work through their hello_world.py but pyglet.window.Window() came back from OpenGL with “invalid operation” calling glGetString(GL_VENDOR). The whole exercise with pygame came about from me trying to find a way to test if PyOpenGL was having similar problems as both PyOpenGL and pyglet use opengl via ctypes.

OpenGL on windows is hard. There’s a huge amount of setup needed to get a context to be able to call openGL commands, and PyOpenGL doesn’t provide that code. PyOpenGL’s sample code relies on pygame to do this… -_-

And to think this all started because I wanted to prototype a game idea I had rather than just telling my work colleagues about it and hoping for the best. Oh well, maybe next weekend.

It does warm my heart to know that despite being a professional video games programmer I can still come home and spend a weekend doodling around programming.

I started watching Cheap Love, only two episodes in but I’m pleased to see that already the main love pairing have already met and realised they’ve feelings for each other. Most J-dramas I watch make us wait until half way through the season to let the characters know what’s been painfully obvious to the audience since the opening titles rolled.

And in further good news, I’m down below 136kg. ^_^

My task for this week is to turn my barcode scans of my book collection into something I can access while I’m in Melbourne next weekend so I don’t have to call home from Melbourne book stores again to get Mick to look at my shelves for me. I’ve also given my mother the book-barcode-scanning bug, but I imagine she’ll buy some software off the Internet to manage the collection. Recommendations appreciated. ^_^ (Obviously, if you want to recommend such software for me, go ahead as well. But my requirements are somewhat pathological)

Oh, and I’m in Melbourne next weekend for Medleys. Taking younger sister (they’re both younger, I’m talking about younger younger. Elder younger will be on stage) and hopefully catching up with Phil.

Been eating “Instant Stew” this week. Finally refined recipe down to:

  • ~1.2kg of mixed heart-smart meat, diced
  • 1kg bag of Home Brand mixed frozen vegetables
  • 1 tin (400g) diced tomatoes
  • Random spices as I find them in the cupboard

Combine all ingredients in crock pot. Put on before work, arrive home after work. Makes 6 lunch servings or 3 dinner servings, ~350 calories per lunch serving. I blame recent weight-loss success on this stuff.

Turns out that making stew, unlike Python games under Windows x64, doesn’t involve first creating the universe.

Things that happen when my brain gets full

I recently was linked to CCG Workshop which is a site where you can play collectable card games (CCGs) online. It’s interesting because they have this gatlingEngine software, which apparently runs the game for you using a set of rules in a gatlingML file.

I thought this would be a wonderful chance to document the rules for the Love Hina CCG, which I never finished translating as you can see, but the gatlingDevKit and all the developer documentation requires that you sign an NDA and suchlike.

Discussions on the forum (the developers talk openly on the public forum, so I have an idea what’s not under NDA ^_^) indicated the gatlingML files were XML, but when I got one while trying to play a game, it was quite clearly binary.

The first four bytes are !HZL which I thought looked really familiar, but it took a fair while before I clicked that that was “LZH!” backwards, LZH being the compression algorithm used in the LHA family of archivers. Of course, research indicated that none of the LHA family of archivers actually wrote a file with !HZL at the front.

Poking about some more, I noticed that the gatlingEngine is written in Delphi (and is legacy code anyway) and went looking for Delphi compression libraries. Thankfully, the vast majority do PKZIP-compatible compression, and the first one I tried that supported LZH compression was Tlzrw1. (Apologies for the quality of the link, the 1998 link in the read file is dead, and the Wayback machine record for it indicates that the author’s page didn’t mention the library anyway) So I note that the library in question attributes its LZH code to LZHUF.C which Google duly turns up for me. I change the code a bit to stop assuming a 16-bit word, handle the header at the front, and suddenly I have a utility which can encode and decode files compressed with the LZH mode of Tlzrw1. (Which has been ported to C# and Delphi.NET, Google tells me.)

Now of course someone needs the interest, gumption and skills needed to produce an open-source program that can process gatlingML files and run games from them. ^_^

Oh, and a cool thing: progress bar for cp, courtesy of Chris Lamb via Planet Debian.

Edit: Missing quote put a whole whack of text inside an <a>-tag.

Things to do in Sydney while the wireless is dead

Well, now that I’m back on the ‘net fairly reliably, I can post on what I’ve been doing for the past few days.

Firstly, I was off the Internet because I was flat-out busy on Saturday, in transit on Sunday, and wireless did not arrive at International House until about 11pm Monday night. That time I did spend on the ‘net today, at the conference, was spent in a combination of processing CBIT emails since Friday, and wrestling with my wireless network card.

My local build of the d80211 version of the bcm43xx driver got signal, would even get traffic through, but when it tried to reassociate to a different AP (all the APs here are running on channel 11… Although I was sitting next to someone who saw one on channel 1, which I’m guessing was rouge… I also saw some IBSS networks on the same SSID….) it would corrupt something nasty, kick the screen brightness up to full and oops with slab errors in short order.

The 2.6.18 (2.6.18-3-powerpc Debian build) bcm43xx softmac driver didn’t crash or anything, but generally performed worse, and when the Debian miniconf’s theatre (Mathews A) was full, my connection suffered or would completely fail to dhcp. >_<

On the plus side, the presentations were great. AJ gave us a rundown of debian-devel (ie 12 months of flamewars) and other significant Debian going-ons. Keith Packard produced a whole bunch of neat X things slated for X.org 7.3 (input hotplugging, dynamic output selection and modesetting, which is exactly what I need to get the projectors I keep plugging into to work better than 640×480…). Russell Coker talked about the various security gaps still remaining in Linux.

In non-conference goings on, I was talking to someone on IRC who’s gotten Second Life Viewer building under Linux/PowerPC (a previously unsupported platform) and I’m going to see if we can get a .deb built. I’ve already created an ELFIO package, and have the OpenJPEG source to try packaging tomorrow. I’ve also sent off an email to the person who ITP’d secondlife-client for Debian already, to see if he wants to co-operate, or if I’m just tooling about.

Speaking of tooling about, I decided it’d be a good idea to upgrade my bcm43xx-d80211 build to something more recent than mid December, but it seems the 2.6.20 workqueue changes mean I can’t compile it against 2.6.19 anymore. The rt2x00 d80211 stack has backwards compatibility macros for the workqueue stuff, but I don’t really feel like hacking those into bcm43xx, it’s already a large and unsteady beast.

BTW, cogito’s update could handle resuming better. Although it happily detected it was resuming a failed update, it had to keep refetching the packs. I eventually realised it would eventually time-out a fetch if I didn’t ^c it and happily try again, presuming I had in the meantime walked outside or reloaded the driver.

Anyway, so I’ve decided tonight (while I was still off the wireless) that I’d finally bite the bullet and build myself a custom dscape.git kernel, to see if the pain I keep suffering from the bcm43xx-d80211 driver is just my cheap-ass backport. That was still building when the wireless came up, and then barfed because KConfig happily let me include both the PCI and SoC versions of the OHCI USB host driver, which provide the same symbols. I must remember to file a bug report about that, or at least check linus’s git tree in case it’s already fixed. (Both drivers recommended yes, but are patently incompatible as they require different endianness of the host interface). I’ve restarted the make-kpkg, hopefully that’ll build overnight and I can try it in the morning.

I also put some time into my Remembering the Kanji book. I was going to do an hour, but after about a half-hour (with a break to configure and fire the kernel build off) I was yawning, and figured I’d prolly left the imaginative-memory zone. I was going to watch some Gokusen but thought I’d take a last wander over to the IH whiteboard to see if the wireless was up. Bizarrely, it was.

So I wandered onto the ‘net, checked email, volunteered myself to package Thousand Parsec for Debian, added the Kanji I studied to Reviewing the Kanji (a web site for reviewing the stuff you learn in Remembering the Kanji) and updated my blog.

Which funnily enough, is where we came in

ごくせん Vol.1ごくせん Vol.2ごくせん Vol.3