So I wrote my first smallish web app in Crystal and it’s been running for over two months now without much of supervision (thanks to monit and systemd). It basically, um, collects some metadata from IMDB every week or so. Based on that data I’m trying to find underrated movies worth watching. Some general thoughts for future-me:

What about the app itself?

You can find it here. You need to use basic auth: <REDACTED> / <REDACTED> (update VII.2017): have to remove those for now). Because of various reasons I won’t be able to publicly release it as Amazon will probably try to shut it down. Sample reference.

But we can look at the data/fun facts (made easier thanks to metabase). Keep in mind the data is just a small portion of what IMDB is; also I will refer to reviews with ratings only.

  1. So far I have over 126.000 movies in 27 different categories and 2.230.000 reviews (metadata only) for those movies - and those are reviews only with rating included - most of pretty old reviews does not include rating, if I’m not mistaken that feature was added later on around year 1998.

  2. You can’t get past 200th page in categories listing, probably due to Elasticsearch pagination limitations (thanks @sztos for this precious knowledge).

  3. There is a movie written by John Malkovich scheduled to be released in 2115.

  4. Seems that IMDB users tend to write reviews when they really like given movie. A little over 20% of collected reviews have 10-stars rating (on the other hand 1-star reviews are 9% of total).

  5. In year 2006 around 180.000 new reviews were posted - that’s almost 500 reviews a day!

  6. The oldest review is for a movie Gummo, it was added on July 27, 1998.

  7. According to current formula in the app most overrated movie is The Oogieloves in the Big Balloon Adventure with a rating of 6.1 and weighted rating based on reviews of 1.8. And the second one is Star Wars: Episode VIII - The Last Jedi, which I actually liked ;).

  8. Most underrated movie is Superbabies: Baby Geniuses 2 probably due to troll reviews :P. The second one is The Promise.

  9. Most up-voted review (2974 of 2994 helpfulness score) is for a movie The Red Maple Leaf. Note: at least so far, had major hiccup regarding parsing reviews and data is still being updated.

  10. Movies have on average 35 reviews, 7 reviews median. Most reviewed movie is The Dark Knight with almost 5100 reviews!

That’s it for now, if you found that app somewhat useful let me know :).