ProRanks history video

Official forum of apterous.org, the website which allows you to play against other people over the Internet.
Post Reply
Thomas Cappleman
Series 72 Champion
Posts: 200
Joined: Fri Nov 07, 2008 9:42 pm

ProRanks history video

Post by Thomas Cappleman » Mon Nov 18, 2019 7:06 pm

I made one of those moving bar charts showing ProRanks over time: https://www.youtube.com/watch?v=YfWpClM ... e=youtu.be

Only shows the top 20 as some weeks don't have a full top 25 because of deactivated accounts. Very low-effort recording technique as well.

User avatar
Ronan M Higginson
Rookie
Posts: 72
Joined: Fri Apr 19, 2019 5:11 pm

Re: ProRanks history video

Post by Ronan M Higginson » Mon Nov 18, 2019 8:07 pm

Great vid!! Thanks very much :D

User avatar
Matt Morrison
Post-apocalypse
Posts: 7512
Joined: Wed Oct 22, 2008 2:27 pm
Location: London
Contact:

Re: ProRanks history video

Post by Matt Morrison » Mon Nov 18, 2019 9:50 pm

nice work Tom!

What was Ned up to with his 2348? Who is Jake Thornhill?

Gavin Chipper
Post-apocalypse
Posts: 9034
Joined: Mon Jan 21, 2008 10:37 pm

Re: ProRanks history video

Post by Gavin Chipper » Mon Nov 18, 2019 11:43 pm

Cool. Is this a "thing" then? I saw something like this relating to Formula One only a couple of days ago.

User avatar
Jon O'Neill
Ginger Ninja
Posts: 4211
Joined: Tue Jan 22, 2008 12:45 am
Location: London, UK

Re: ProRanks history video

Post by Jon O'Neill » Mon Nov 18, 2019 11:55 pm

Yeah it's a thing, and this is a great example of such a thing.
I remember Pendleton's rise to dominance. Amazing part of apterous history.

Nice.

User avatar
Jon O'Neill
Ginger Ninja
Posts: 4211
Joined: Tue Jan 22, 2008 12:45 am
Location: London, UK

Re: ProRanks history video

Post by Jon O'Neill » Tue Nov 19, 2019 12:05 am

It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.

User avatar
Thomas Carey
Kiloposter
Posts: 1145
Joined: Sun Jan 09, 2011 4:17 pm
Location: North-West of Bradford
Contact:

Re: ProRanks history video

Post by Thomas Carey » Tue Nov 19, 2019 7:47 am

This is good
cheers maus

Thomas Cappleman
Series 72 Champion
Posts: 200
Joined: Fri Nov 07, 2008 9:42 pm

Re: ProRanks history video

Post by Thomas Cappleman » Tue Nov 19, 2019 10:08 am

Jon O'Neill wrote:
Tue Nov 19, 2019 12:05 am
It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.
Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.

User avatar
Ian Volante
Postmaster General
Posts: 3409
Joined: Wed Sep 03, 2008 8:15 pm
Location: Edinburgh
Contact:

Re: ProRanks history video

Post by Ian Volante » Tue Nov 19, 2019 1:08 pm

Nice, but I must grumble about the excision of my week in 25th!
meles meles meles meles meles meles meles meles meles meles meles meles meles meles meles meles

Matt Bayfield
Devotee
Posts: 538
Joined: Thu May 14, 2009 8:39 am
Location: Seated at a computer

Re: ProRanks history video

Post by Matt Bayfield » Tue Nov 19, 2019 6:30 pm

Thomas Cappleman wrote:
Tue Nov 19, 2019 10:08 am
Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.
Would this non-alpha-character gremlin also affect Tom Chafer-Cook, who had a peak ProRank of 3rd?

Also, I have no time for Ian's gripe... some of us round here have a peak ProRank of 21st.

Neil A Collins
Newbie
Posts: 15
Joined: Fri Jun 30, 2017 1:54 pm

Re: ProRanks history video

Post by Neil A Collins » Sun Dec 01, 2019 2:09 pm

This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?

Thomas Cappleman
Series 72 Champion
Posts: 200
Joined: Fri Nov 07, 2008 9:42 pm

Re: ProRanks history video

Post by Thomas Cappleman » Tue Dec 03, 2019 3:50 pm

Neil A Collins wrote:
Sun Dec 01, 2019 2:09 pm
This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?
My Python code's below. I used requests (a nice intuitive module) to just get each page in turn, then regexed the output to find usernames and rankings.

I'm sure there'd be a better way to query the actual database, especially as my method can't get anything beyond the top 25, as it's all the same URL.

Code: Select all

import requests
import re
import xlsxwriter
from datetime import timedelta, date

# Fill in your details here to be posted to the login form.
payload = {
		'inUserName': 'thomas@cappleman.co.uk',
		'inUserPass': <password>
}

# Use 'with' to ensure the session context is closed after use.
with requests.Session() as s:
		p = s.post('https://www.apterous.org/proranks.php?period=105392', data=payload)
		# print the html returned or something more intelligent to see if it's a successful login page.
		print p.text
		user_dict = {}

		# An authorised request.
		# 104804
		for period in range(104804, 105393):
				r = s.get('https://www.apterous.org/proranks.php?period=%s' % period)
				users = re.findall( r'href="viewuser\.php\?user=[0-9]*">([^<]*)</a><span class="rating">', r.text)
				ratings = re.findall(r'"rating">(?:<br />Pro Rank: )?([0-9]*) <span', r.text)
				proranks_date = re.findall(r'<h3>Published ([0-9a-zA-z ]*)</h3>', r.text)

				for x in range(min(len(users), len(ratings))):
						user = users[x]
						if user not in user_dict:
								user_dict[user] = {proranks_date[0]: ratings[x]}
						else:
								user_dict[user][proranks_date[0]] = ratings[x]

workbook = xlsxwriter.Workbook('proranks.xlsx')
worksheet = workbook.add_worksheet()

def daterange(start_date, end_date):
    for n in range(int ((end_date - start_date).days) / 7):
        yield start_date + timedelta(n) * 7

start_date = date(2008, 8, 11)
end_date = date(2019, 11, 25)
worksheet.write(0, 0, "Name")
column = 1
for single_date in daterange(start_date, end_date):
		worksheet.write(0, column, single_date.strftime("%-d %B %Y"))
		column += 1

row = 1
for user, user_info in user_dict.items():
		worksheet.write(row, 0, user)
		column = 1
		for single_date in daterange(start_date, end_date):
				formatted_date = single_date.strftime("%-d %B %Y")
				if formatted_date in user_info:
						worksheet.write(row, column, user_info[formatted_date])
				column += 1
		row += 1

workbook.close()

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests