ProRanks history video

Official forum of apterous.org, the website which allows you to play against other people over the Internet.
Post Reply
Thomas Cappleman
Series 72 Champion
Posts: 330
Joined: Fri Nov 07, 2008 9:42 pm

ProRanks history video

Post by Thomas Cappleman »

I made one of those moving bar charts showing ProRanks over time: https://www.youtube.com/watch?v=YfWpClM ... e=youtu.be

Only shows the top 20 as some weeks don't have a full top 25 because of deactivated accounts. Very low-effort recording technique as well.
User avatar
Ronan M Higginson
Enthusiast
Posts: 289
Joined: Fri Apr 19, 2019 5:11 pm

Re: ProRanks history video

Post by Ronan M Higginson »

Great vid!! Thanks very much :D
User avatar
Matt Morrison
Post-apocalypse
Posts: 7822
Joined: Wed Oct 22, 2008 2:27 pm
Location: London
Contact:

Re: ProRanks history video

Post by Matt Morrison »

nice work Tom!

What was Ned up to with his 2348? Who is Jake Thornhill?
Gavin Chipper
Post-apocalypse
Posts: 13215
Joined: Mon Jan 21, 2008 10:37 pm

Re: ProRanks history video

Post by Gavin Chipper »

Cool. Is this a "thing" then? I saw something like this relating to Formula One only a couple of days ago.
User avatar
Jon O'Neill
Ginger Ninja
Posts: 4545
Joined: Tue Jan 22, 2008 12:45 am
Location: London, UK

Re: ProRanks history video

Post by Jon O'Neill »

Yeah it's a thing, and this is a great example of such a thing.
I remember Pendleton's rise to dominance. Amazing part of apterous history.

Nice.
User avatar
Jon O'Neill
Ginger Ninja
Posts: 4545
Joined: Tue Jan 22, 2008 12:45 am
Location: London, UK

Re: ProRanks history video

Post by Jon O'Neill »

It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.
User avatar
Thomas Carey
Kiloposter
Posts: 1478
Joined: Sun Jan 09, 2011 4:17 pm
Location: North-West of Bradford
Contact:

Re: ProRanks history video

Post by Thomas Carey »

This is good
cheers maus
Thomas Cappleman
Series 72 Champion
Posts: 330
Joined: Fri Nov 07, 2008 9:42 pm

Re: ProRanks history video

Post by Thomas Cappleman »

Jon O'Neill wrote: Tue Nov 19, 2019 12:05 am It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.
Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.
User avatar
Ian Volante
Postmaster General
Posts: 3956
Joined: Wed Sep 03, 2008 8:15 pm
Location: Edinburgh
Contact:

Re: ProRanks history video

Post by Ian Volante »

Nice, but I must grumble about the excision of my week in 25th!
meles meles meles meles meles meles meles meles meles meles meles meles meles meles meles meles
Matt Bayfield
Devotee
Posts: 539
Joined: Thu May 14, 2009 8:39 am
Location: Seated at a computer

Re: ProRanks history video

Post by Matt Bayfield »

Thomas Cappleman wrote: Tue Nov 19, 2019 10:08 am Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.
Would this non-alpha-character gremlin also affect Tom Chafer-Cook, who had a peak ProRank of 3rd?

Also, I have no time for Ian's gripe... some of us round here have a peak ProRank of 21st.
Neil A Collins
Newbie
Posts: 24
Joined: Fri Jun 30, 2017 1:54 pm

Re: ProRanks history video

Post by Neil A Collins »

This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?
Thomas Cappleman
Series 72 Champion
Posts: 330
Joined: Fri Nov 07, 2008 9:42 pm

Re: ProRanks history video

Post by Thomas Cappleman »

Neil A Collins wrote: Sun Dec 01, 2019 2:09 pm This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?
My Python code's below. I used requests (a nice intuitive module) to just get each page in turn, then regexed the output to find usernames and rankings.

I'm sure there'd be a better way to query the actual database, especially as my method can't get anything beyond the top 25, as it's all the same URL.

Code: Select all

import requests
import re
import xlsxwriter
from datetime import timedelta, date

# Fill in your details here to be posted to the login form.
payload = {
		'inUserName': 'thomas@cappleman.co.uk',
		'inUserPass': <password>
}

# Use 'with' to ensure the session context is closed after use.
with requests.Session() as s:
		p = s.post('https://www.apterous.org/proranks.php?period=105392', data=payload)
		# print the html returned or something more intelligent to see if it's a successful login page.
		print p.text
		user_dict = {}

		# An authorised request.
		# 104804
		for period in range(104804, 105393):
				r = s.get('https://www.apterous.org/proranks.php?period=%s' % period)
				users = re.findall( r'href="viewuser\.php\?user=[0-9]*">([^<]*)</a><span class="rating">', r.text)
				ratings = re.findall(r'"rating">(?:<br />Pro Rank: )?([0-9]*) <span', r.text)
				proranks_date = re.findall(r'<h3>Published ([0-9a-zA-z ]*)</h3>', r.text)

				for x in range(min(len(users), len(ratings))):
						user = users[x]
						if user not in user_dict:
								user_dict[user] = {proranks_date[0]: ratings[x]}
						else:
								user_dict[user][proranks_date[0]] = ratings[x]

workbook = xlsxwriter.Workbook('proranks.xlsx')
worksheet = workbook.add_worksheet()

def daterange(start_date, end_date):
    for n in range(int ((end_date - start_date).days) / 7):
        yield start_date + timedelta(n) * 7

start_date = date(2008, 8, 11)
end_date = date(2019, 11, 25)
worksheet.write(0, 0, "Name")
column = 1
for single_date in daterange(start_date, end_date):
		worksheet.write(0, column, single_date.strftime("%-d %B %Y"))
		column += 1

row = 1
for user, user_info in user_dict.items():
		worksheet.write(row, 0, user)
		column = 1
		for single_date in daterange(start_date, end_date):
				formatted_date = single_date.strftime("%-d %B %Y")
				if formatted_date in user_info:
						worksheet.write(row, column, user_info[formatted_date])
				column += 1
		row += 1

workbook.close()
Post Reply