26
Jun

Import / Export API: Progress Report #2

The mid-program mentor evaluation (a.k.a. crunch time) for the Summer of Code is almost here, and as such, I've been working round-the-clock to get my project looking as presentable as possible. The Import / Export API module has made significant progress since my last report, but there's still plenty of work left to be done.

It gives me great pride to assert that if you download the module right now, you'll find that it actually does something. :-) I know, I know - it doesn't do much (in fact, I may simply be going delusional and crazy from all the coding I've been doing, and it may actually do nothing) - but what it does do is pretty cool. The XML export engine is now functional, which means that you can already use the module to export any entities that currently have a definition available (which is only users and roles, at present), as plain-text XML:

XML export screenshot

XML export screenshot

The import system isn't quite ready to go as yet, but the XML-to-array engine is pretty much done, and with a little more work, the array-to-DB engine will be done as well.

The really exciting stuff, however, has been happening under the hood, in the dark and mysterious depths of the API's code. Alright, alright - exciting if you're the kind of twisted individual that believes recursive array building and refactored function abstraction are hotter than Angelina's Tomb Raiders™. But hey, who doesn't?

Some of the stuff that's been keeping me busy lately:

  • The new Get and Put API has been implemented. All engines now implement either a 'get' or a 'put' operation (e.g. get data from the database, put data into an XML string). All exports and imports are now distinctly broken up into a 'get' and a 'put' component, with the data always being in a standard structured form in-between.
  • Nested arrays are now fully supported, even though I couldn't find any structures in Drupal that require a nested array definition (I wrote a test module that uses nested arrays, just to see that it works).
  • The definition building system has been made more powerful. Fields can now be modified at any time after the definition 'library' is first built. Specific engines can take the original definitions, and build them using their own custom callbacks, for adding and modifying custom attributes.
  • Support for Alternate key fields has been added. This means that you can export unique identifiers that are more human-friendly than database IDs (e.g. user names instead of user IDs), and these fields will automatically get generated whenever their ID equivalents are found in a definition. This will get really cool when the import system is ready - you will be able to import data that only has alternate keys, and not ID keys - and the system will be able to translate them for you.

Also, for those of you that want to get involved in this project, and to offer your feedback and opinions, head on over to the Import / Export API group, which is part of the new groups.drupal.org community site, and which is open for anyone to join. I'd love to hear whatever you may have to say about the module - anything from questions about how it can help you in your quest for the holy grail (sorry, only African Swallows can be exported at this time, support for European Swallows is not yet ready), to complaints about it killing your parrot (be sure that it isn't just resting) - all this and more is welcome.

I hope to have more code, and another report, ready in the near future. Thanks for reading this far!

Comments are closed

Comments

26
Jun
2006

This sounds like a really grat piece of work, Jeremy. Really looking forward to seeing it in action!

29
Jun
2006

Hi Jaza - I've just been reading all the docs and trying out the download. In your original spec you thought the 20th would be your date for beta. What's the most likely date now?

The subtext: I started to upgrade the import_export module to 4.7 before I found your project and I'm wondering whether there's any point in continuing seeing as what you have is so much better designed and built.

Thanks,

smoothly

30
Jun
2006
Jeremy Epstein

I just had a look over my original proposal, and yes, indeed, I did put Jun 20 as my target date for quite a lot of things. But you know what they say about software development time estimates: add a few weeks to them, multiply them by 2, etc, and you get the real estimate. ;-)

The data definition API is certainly beta-quality already, as are the XML 'get' and 'put' engines, and the DB 'get' engine. But: the DB 'put' engine is still under development; the CSV engines haven't yet been started; and the UI is now out of scope, and will only be developed time-permitting (see the http://basement.greenash.ne...">official success criteria for more details). At this point, I'm not prepared to give any dates, except to say that I'm committed to finishing everything in the official success criteria by the SoC final deadline (Aug 20).

As for the import_export module, I think that it has a lot of great functionality; and I think that if you or others need that functionality for 4.7, then by all means go ahead and upgrade it. I would certainly encourage you to consider joining forces with the import / export API instead, or at least re-architecting the import_export module to be built on top of the API; but that decision is up to you.

03
Jul
2006

Thanks; I wouldn't give dates either. As far as Import-Export is concerned: I put a day in on the convert and it's 95% done. The UI is horrid, as you indicated in your notes when you were trying out the various modules. If I get time I may do something about that. Clearly using your api is a much better approach, and I think you are right, at the least it should be built on top of your api, and I'll look at that.

Are you interested in bug reports yet? I think it's premature, myself, but I did get a crash when I downloaded your code and tried it out. (Unsupported operand types in importexportapi.module on line 236). If you want I'll file a proper bug report with details.