A group for contestants and lovers of the Channel 4 game show 'Countdown'.
http://www.c4countdown.co.uk/
Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.Jon O'Neill wrote: ↑Tue Nov 19, 2019 12:05 am It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.
Would this non-alpha-character gremlin also affect Tom Chafer-Cook, who had a peak ProRank of 3rd?Thomas Cappleman wrote: ↑Tue Nov 19, 2019 10:08 am Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.
My Python code's below. I used requests (a nice intuitive module) to just get each page in turn, then regexed the output to find usernames and rankings.Neil A Collins wrote: ↑Sun Dec 01, 2019 2:09 pm This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?
Code: Select all
import requests
import re
import xlsxwriter
from datetime import timedelta, date
# Fill in your details here to be posted to the login form.
payload = {
'inUserName': 'thomas@cappleman.co.uk',
'inUserPass': <password>
}
# Use 'with' to ensure the session context is closed after use.
with requests.Session() as s:
p = s.post('https://www.apterous.org/proranks.php?period=105392', data=payload)
# print the html returned or something more intelligent to see if it's a successful login page.
print p.text
user_dict = {}
# An authorised request.
# 104804
for period in range(104804, 105393):
r = s.get('https://www.apterous.org/proranks.php?period=%s' % period)
users = re.findall( r'href="viewuser\.php\?user=[0-9]*">([^<]*)</a><span class="rating">', r.text)
ratings = re.findall(r'"rating">(?:<br />Pro Rank: )?([0-9]*) <span', r.text)
proranks_date = re.findall(r'<h3>Published ([0-9a-zA-z ]*)</h3>', r.text)
for x in range(min(len(users), len(ratings))):
user = users[x]
if user not in user_dict:
user_dict[user] = {proranks_date[0]: ratings[x]}
else:
user_dict[user][proranks_date[0]] = ratings[x]
workbook = xlsxwriter.Workbook('proranks.xlsx')
worksheet = workbook.add_worksheet()
def daterange(start_date, end_date):
for n in range(int ((end_date - start_date).days) / 7):
yield start_date + timedelta(n) * 7
start_date = date(2008, 8, 11)
end_date = date(2019, 11, 25)
worksheet.write(0, 0, "Name")
column = 1
for single_date in daterange(start_date, end_date):
worksheet.write(0, column, single_date.strftime("%-d %B %Y"))
column += 1
row = 1
for user, user_info in user_dict.items():
worksheet.write(row, 0, user)
column = 1
for single_date in daterange(start_date, end_date):
formatted_date = single_date.strftime("%-d %B %Y")
if formatted_date in user_info:
worksheet.write(row, column, user_info[formatted_date])
column += 1
row += 1
workbook.close()