Account Importer after the merger - hacks on hacks with postgres_fwd
Once the merger was done we immediately had a business need to start moving accounts from our old product to the new one.
As most of the data structure was rather similar I have decided to absolutely abuse
postgres_fdw module basically allows you to define remote servers and access them in your queries transparently.
So you can have
server_2 (remote), and
server_3 (other remote) and construct an insert query by selecting a bunch of stuff from
server_3 - and even do that in a single transaction - which is crazy powerful.
Leveraging that and a low memory footprint of
sequel gem we were able to create a migration app that managed to move the account of any size without too much of a problem. Most of the heavy lifting was done directly on the database level and we simply were doing some simple mappings / small AWS S3 migrations in ruby.
The biggest downside of that is that such tool is really hard to test - as you do most of the stuff in the database you have a lot of dependencies straight away. Think about existing schemas on remote servers - now you have to bring all that complexity and structure to your tests as well. But as a tool used once per account in the system I suppose it was a pretty good bargain.
I think it was one of most out of the box approach I have ever come up with that paid off. Using Postgres capabilities to transparently move data between different servers was kinda a lucky shot - I did not run any benchmarks beforehand (time pressure) and just trusted that people behind Postgres are way smarter than me and they know what they are doing (and they do).