Planned Website Upgrades (Part 2)

An update on our plans for improving the performance of spotlight.com. We’ve now launched the first set of improvements to our site, and we’re seeing a small but noticeable improvement in site availability and response time. Pages that don’t rely heavily on Spotlight’s database appear to be significantly quicker, but we’re still seeing slowdowns and delays on database-heavy pages such as performer searches and the Spotlight Link. We have three improvements that we’re planning to launch during the week starting August 8th.

Firstly, we’re installing another database server – just like the one above. We’ve done an in-depth review of our website code and come up with a way to split our databases across two separate servers, so there’s one box dedicated to search cards, CVs and suggestions, and another completely separate box dedicated to the Spotlight Link breakdowns and e-mails for agents and performers. This means busy breakdowns will no longer cause delays for casting directors using the search and suggestions areas of our site, and gives us enough capacity to continue developing and launching new features without affecting other areas of our website.

Second, we’re introducing something called a “web application accelerator” – a specialist software component that’s designed to dramatically improve the performance of particular pages. We’re using a system called “Varnish”, which is the same system that Facebook uses to cope with their 750 million active users. This isn’t quite as simple as just plugging it in and switching it on – we’re having to modify quite a lot of our code so it’ll work properly with the Varnish accelerator in place – but we’ve identified a couple of key areas where these improvements should be relatively quick, so we’ll be rolling this out as soon as we can.

Finally, the improvements to Spotlight’s photograph database is nearly completed, but we’ve made some pretty drastic changes to the way this works behind the scenes and since photographs are such a key part of the casting experience, we’re going to hold off launching this until we’ve had a chance to test it thoroughly and we’re confident it’s not going to cause any issues.

Improvements to the Spotlight Link e-mail system

We’ve just launched a major upgrade to the system that sends e-mail notifications about new jobs posted to the Spotlight Link.

We’ve had reported delays of over an hour between jobs appearing on the Spotlight Link website and those same jobs arriving in agents’ e-mail inboxes. When several large breakdowns were posted in quick succession, the volume of e-mail notifications was causing a backlog of outgoing mail, and it was taking up to two hours to clear this mail backlog.

Our solution is a purpose-built high-performance e-mail system, that’s integrated directly into the Spotlight Link database. This allows us to spin up multiple e-mail servers running simultaneously; in our benchmark tests, this new system has been sending 25 e-mails per second, which means even the largest breakdowns will be processed and sent within a few minutes. We’re also now delivering e-mail directly to your internet provider instead of sending it via an intermediate server; this means if your ISP is having problems with incoming mail or your mailbox is over quota, we’ll know about it immediately, and can keep trying until delivery succeeds.

This new system is now up and running, and we’ll keep a very close eye on it over the next few days to ensure everything’s running smoothly.

Planned Website Upgrades (Part 1)

The team are currently going through final testing of the “phase 1” improvements to spotlight.com. This is a relatively minor upgrade, but will affect all areas of our site – performers, casting and agencies.

All being well, we plan to roll this out overnight so it should be in place on Thursday morning. As I’ve explained, this is the first part of a much larger plan to improve the performance and reliability of the whole of our site, so watch out for more news and upgrades over the coming weeks.

As always, you can get a sneak preview of what we’re doing by pointing your browser at http://labs.spotlight.com/, and e-mail feedback@spotlight.com if you have any comments or suggestions.

Website Performance Update

In my last post, I outlined a "three-point plan" to address the speed and performance issues users have having on spotlight.com. The first phase of this is done, and is now available for testing on our experimental server at http://labs.spotlight.com/ - if you'd like to help us test it, go ahead; use the labs.spotlight.com site just as you would the normal site, and let us know if you find any problems with it (e-mail to helpdesk@spotlight.com and mention that you're using the Labs server)

Obviously, we can't know for sure how effective these improvements are until they're on the live site and everybody's using them - but we've been tracking progress using a set of tools from Mozilla and Yahoo designed for analysing website performance, and so far things are looking promising. By changing the way our site links to the images and script files that are used to render most of our pages, we've reduced the size of a typical casting page substantially:

We'll be testing this thoroughly ourselves over the next few days, and hope to get it rolled out onto our live site later this week. Meanwhile, we're working flat-out on the photo engine that our website uses, and next week we'll be reviewing the CV database and creating a plan for delivering the same kind of improvements to the CV system that we're working on for photographs.

Planned Performance Improvements to spotlight.com

Over the last few months, we’ve had lots of feedback from our users about the speed of our site – in a nutshell, it’s too slow. Searches take too long; CVs take too long, the site is generally slow to respond, and you’re often finding that pages will freeze or fail to load.

Why is it happening?

The short answer – we’ve grown. We have more casting professionals, more performers and more breakdowns and jobs on Spotlight than ever before, and our main database server can’t always keep up with the number of searches, CVs and breakdowns that are going through Spotlight when things get busy.

The servers that power Spotlight’s website are set up like this:

 

The web servers do all the processing – they turn raw data into formatted Web pages – and the database server stores the information (CVs, search results, breakdowns, suggestions) that’s used by the web servers.

Over the last year, we’ve upgraded from a single web server, to two, to three – and we’ve now hit a point where, particularly for casting users, the web servers are no longer the problem. We’ve set up all sorts of statistics and monitoring on our servers so we can identify the areas that are actually causing problems, and here’s how we’re going to fix them.

What are we doing about it?

Here’s what’s actually causing the load on our website during the course of a busy afternoon – this shows the total time taken to process requests for each of these groups, so it’s directly indicative of how long you’re waiting for pages to load when you’re using our site.

We’re going to address those areas in the following order:

1. Scripts and Stylesheets

When you request a page from Spotlight’s site, there’s a lot of additional information that we send down along with it – you don’t see this extra data directly, but your internet browser uses it to format the page, enable interactive elements like forms and navigation, and generally enhance the experience of viewing web pages. This extra data is the big blue chunk on the pie chart.

The first thing we’re planning to do is to modify the way our site delivers this additional data, by allowing your web browser to store it and use the copy you’ve already got instead of requesting a fresh copy along with every page. That’ll reduce load on our servers by 24%, and will also reduce the amount of data that’s being sent backwards and forwards with each page whilst you’re using the site. We’re starting work on this right away and it’ll take 2-3 days to get the new code tested and launched, so this should be in place by the middle of next week.

2. Photos and Thumbnails

We know that out-of-date photographs are unacceptable for a directory like Spotlight, so our web photo engine currently checks that a photograph is up to date whenever anyone loads a photo from our website. This is slowing things down massively, and so we’re going to replace it with a “smart notification” system, whereby whenever a photograph is modified or replaced on our main image server, the image server will actually notify the website that that photograph has changed. This is something we’ve never been able to do before – it’s only possible because of upgrades we’ve made to our software in the last 3-4 months – but I’m confident that once this is in place it’ll make a big difference to the load time of photographs in particular, and to the overall speed of the site.

At the moment, we’re looking at 4-5 working days to get this up and running, which means it should be in place by the end of July.

3. Spotlight Web CVs

Third, we’re going to overhaul the way Spotlight CVs are stored and published. We’re not planning any changes to how the CVs look – this will be completely behind-the-scenes – but we’re going to upgrade the underlying data storage engine so that instead of CVs being formatted “on demand” based on raw data, we’ll capture the CV whenever any information changes, and then store those pre-formatted CVs in a high-speed indexing system so they can be searched, viewed and printed without lots of (relatively slow) database queries.

As you can appreciate, this is a much bigger job; we’re still working on the detailed plans for how we’re going to tackle this and how long it’s likely to take, and I’ll let you know when we have a plan in place for this.

…and after that?

We’ll keep monitoring the site; we’ll keep listening to your feedback, and we’ll keep working on it. These improvements are currently our absolute top priority, and they will remain so until our site is solid, stable and lightning-fast.

Welcome to the Spotlight Developers Blog

Hi. I’m Dylan, and I run the software team who look after Spotlight’s databases and websites.  We’ve set up this dedicated blog to give you a bit more insight into what’s happening behind the scenes here at Spotlight. We’ll let you know about problems we’re aware of, and share our plans for fixing them; we’ll talk about new products and services, and we’d love to know what you think – what works well for you, where you think we can improve things, and ideas or suggestions for things you’d like to see added to the site.

I know some of you are frustrated because whenever you ask for an update on something that’s causing you problems, all you hear is “we’re working on it”. I’m hoping that this blog will give you a much better idea of what’s causing the problems you’re experiencing, what exactly we’re doing to address them, and how soon you can expect to see changes and improvements to the main site.