There’s no better feeling in the world than coming into the office from a long weekend away to find that your Build Server has disappeared without trace.
Okay, so maybe there are better feelings in the world, but they don’t come close to the joy of finding that the database used by TeamCity saves very little other than user profiles and related settings, and that nearly all the really important stuff is saved elsewhere.
So, here’s how it all happened. I figured something was up after turning on the Build Radiator in my office to find a nice PHP error message staring at me, informing me it couldn’t get connection to the build server and therefore couldn’t load the data from TeamCity.
OpeningĀ up a browser window confirmed the fears that TeamCity was no more. A fantastically brilliant error (if you call your errors fantastically brilliant they seem less painful!) of “java.sql.SQLException: error in script file line: 172 out of memory” was there, staring us in the face. A cracking start to a Monday morning, and an even better start to the week.
Move time on a touch and after a few tweets, an offer of help from JetBrains own Hadi Hariri and an email to the JetBrains community support team, we confirmed the worst… the database had gone. For those of you who didn’t know, and until about five hours earlier neither did I; JetBrains uses an HyperSQL DB by default. Written away in the documentation is a plea to all those installing TeamCity, that once you’re using TC in a production environment you need to move away from HyperSQL and onto something else.
So, with this in mind we studied our options and a quick glance through the files stored within TeamCity’s directory structure we began to establish that TC stores a lot of information in XML files and less data in the DB than we originally thought.
We quickly set up a connection to our internal Oracle DB (easily the simplest setup for any of the DB options in TeamCity) and restarted the TC Web Service on the machine. It spotted the new connection and created all the tables it needed. Once it started up we realised just what we had lost… and what we still had.
Things we lost:
- User profiles
- User groups
- Audit logs
- Build status knowledge (was the last build successful or not)
However, it’s worth pointing out that some of our TeamCity settings are still on default, such as when to perform Build History clean-up processes so we can’t be sure whether what we use as default is stored in the DB.
Everything else was still there. All the projects, VCS profiles, build configurations, artifact links, build numbers, build agents knowledge – all there as if nothing had happened.
So my advice to anyone who uses TeamCity – especially those who’ve installed it recently – just double check where your DB is stored (Go to “Administration > Server Configuration> General”, and check the “Database” value). Possible corruption of the data in HyperSQL is apparently the exact reason why TeamCity recommend using an external DB after installation and whilst it’s not quite as bad as it seems, in that it’s only about 25% of the data you lose (depending on project numbers and team size), it’s still something you can prevent happening.