Yann’s Blog

Just an update on the GMail delivery problems from eve-central.com. Of the 30-odd addresses subscribed from gmail, we were happily delivering mail to all of them with no bounces and other service interruptions. All it took was setting “Precedence: bulk” in the mail headers and GMail took the e-mail.

Now it seems any amount of volume we push to GMail causes the server to rate-limit messages. And rate-limiting messages means we have an ever growing backlog of e-mail which never gets delivered, making the whole feed less than useful. At this time there is no real solution to this problem.

When subscribing to the e-mail feed, make sure your account can handle THOUSANDS of messages per day. Many ISP accounts just don’t like this type of volume and aren’t good choices. GMail used to be an acceptable choice, but isn’t at this point. The best choice is still your own hosting account or dedicated server, where you can do real-time processing.

We’ll monitor the situation and turn delivery to a few gmail accounts on peridoically, but no guarantees.

What if you had a high-level library in Python for driving revision control systems and their working copies?

Well, anyvcs to the rescue! Or, as it stands today, not yet. There isn’t much code to go around with anyvcs yet - it only works with Git and can’t do much beyond tell you the version or what the latest commit ID is on a branch. But thats exciting right?

You can look at the source for anyvcs here and clone it via

git clone git://stackworks.net/anyvcs.git

The high level goals for this project include:

1. Simple backend for most distributed VCS operations, such as checkout, branch, apply patch, reset, etc
2. Targets primarily git and mercurial, or other VCSes which operate in a similar manner. Subversion isn’t a likely first target for instance, nor is bzr (though we would like to support both eventually).
3. Provide a comprehensive wrapper for Git commands via a Python library. If some of the functionality doesn’t end up in the core anyvcs abstraction, anyvcs.git will provide some VCS specific operations.

There is a great reason for this project - but I won’t reveal everything just yet.

I’m pushing out a new update to the site today. Several new features are included and bugs are fixed:

• The API has bug fixes related to the minimum quantity filter being misapplied and overridden in certain cases. The behavior should be consistent now.
• Top bar is now left aligned - looks a bit better?
• The logout button works again (yay!)
• Added a new Trade section, which includes the new Comparison tool I’ve blogged about earlier, in BETA form. There is no cost (ISK wise) to use at this moment (haven’t gotten about getting the subscription interface written yet). You need to be logged in to use this feature though. If you have any feedback and improvements, send me an e-mail or comment here in the blog.

Expect more updates soon

Just a quick one, showing off a new screenshot of the trade compare tool. See it here. Also implemented is a nifty little autocomplete AJAX feature for the system and region names.

I’ll be re-using this YUI component for the market type searching as well, with a fallback to the current page. Note that the autocomplete is not IGB compatible, but the trade comparison tool as a whole is not either (due to large page sizes and lots of presented information), making it a non-issue.

Also, I’m working on a market sell movement application to try to track volumes moved. The data to back this isn’t perfect by any chance, but perhaps will provide an overinflated view of the actual item movement.

I’ve been quiet about my IRM 2 activities recently, and I apologize. This article is not really a status update, but a reflection on mapping programs such as IRM 2 to a traditional RDBMS.

I’ve determined that I just built a database… inside another database. The IRM 2 core currently has a core table, which is quite literally an ID, Column, Value mapping table. The data simply looked liked this:

A100, 1, Name, Computer
A100, 1, Color, Blue
A100, 1, Owner, Bob

Every property simply gets dumped into this table, and requires extensive remapping on the application logic. The RDBMS doesn’t have much insight into schemas, as it subverts its notion of schemas. Searching for explicit property values requires joining huge amounts of data together. Data storage is inefficient, with records scattered throughout the datastore (and in many places on the disk), especially if records were updated after creation.

More and more I’m starting to feel that approaches such as BigTable, SimpleDB, and CouchDB are taking the correct approach to the large data storage problem. The data stored is as a series of documents with arbitrary and per-document schema. Each node is a replicated element, and data storage is not centralized but rather distributed.

The advent of Google’s AppEngine supporting this model of database reaffirms that this is the correct direction to move in for large scale online applications.

Now, what does all of this have to do with IRM 2 and gIRM? Stay tuned - I’m trying to find out myself

I’ve taken the chance now to upgrade some of our APIs, make them more consistent, cleaner, and better documented.

The first primary change is moving from the old /home namespace to the new /api namespace. This allows for some cleaner separation of the logic and not the old mismash of stuff. I’ve also cleaned up the marketstat API, allowing multiple type IDs to be passed in a single query.

The new API has a version header now as well <evec_api version=”2.0″>…</evec_api>. You can distinguish responses, and possibly in the future request specific versions of formatting.

As before, the API is open, not requiring a key to use. I’ve contemplated a key based approach but haven’t deemed in necessary yet. If you suddenly find you can’t connect to the server - you’ve probably been doing something naughty (or it broke).

I have a few IPs blocked for bone-headed use of the system - such as querying every typeID from 1 to 100000000, and in string order at that. Don’t do that

The old APIs based off of /home will continue to work for now. It would be a perfect world if everything would upgrade, but thats not always possible. I’ll monitor usage of both and cut off the old one once its no longer popular.

Many nifty things are afoot for EVE-Central.com. For one, I migrated away from mod_python, which was hanging up the Apache process several times per day for unknown reasons. There was even a monitor script to kill -9, ipcrm, and restart Apache if it wasn’t responding - this horrible restart was happening several times a day. The new EVE-Central.com is running CherryPy as the backend server now - the conversion was quick and painless, plus it performs better (barring any new introduced bugs).

On the new feature front, I’m working on a trade comparison tool for sell orders, trying to find price differences between systems or regions. Its in an alpha prototype stage, so not for public consumption yet. I expect it to launch in a week or two. Here is a link to a sample image.

I’m also mulling over a possible Premium subscription to EVE-Central. Paid in ISK, it would allow access to the aforementioned trade comparison tool, plus other options, such as ad-free view, and the ability to upload “premium” content which would not get pushed to normal users until a small time window had elapsed. Details still sketchy at this point

Just a quick update highlighting what is going on in girm.it, the Git repository for IRM 2 (on Grails).

1. It is now possible to create, view and browse Assets based upon Schemas, which are user defined.
2. You can create Schemas and Fields in a Schema
3. The functionality for linking CRI (Cross Reference Identifiers) is in place - needs more work on the GUI front.

While gIRM isn’t even into alpha stages, its showing a lot of promising functionality already. Next up is a first hash of the Ticketing support system (Tracking mk 2), as well as “personalities”.

Personalities are an abstraction to Schemas which allow you to associate sub-schemas with a master schema. For instance, you can create a “Network Connectivity”personality and associate it with both a printer and a computer. Personalities can also impart custom logic to the application - such as link graphs in the case of a Network Connectivity personality. The mechanism for this isn’t quite designed yet

Feature suggestions are always welcome!

In addition to trying out new Kool-aid with Groovy and Grails, I’ve decided to move from tracking in Subversion to tracking in Git the gIRM code base.

The gitweb is available here: http://stackworks.net/cgi-bin/gitweb.cgi?p=girm.git;a=summary

and cloning is available at

git://stackworks.net/girm.git/

Just a quickie.  A few screenshots presenting IRM on Grails. Developed much faster than my previous solution, and in general making me feel a lot better about its direction

.