Forum:Project Database Integration

Premise
The wiki's activity has declined severely in the last year; I believe this has to do with a significant reduction in the need for this wiki due to other resources which work "better". When the Clan updates were released by Jagex, they included a Clan Database which can be found on their main website.

Runescape Clan Database

This database gives an up to the minute stats listing, a roster of members, and provides basic "need to know" information on a clan. The inefficiency of this wiki when compared to such a system is astounding, and... quite honestly, we cannot compete with the database unless we can match it.

To that end, I have been discussing the possibility of a new project for this wiki. User:MinigameGod,User:The Excel, and User:The.draziw are working on a system that will allow this wiki to include the basic information found via the database, while also being a medium that is not constricted by character limits, severe censorship, and more adaptability.

The principal idea is to generate a script that will allow the wiki to update information found via the database. We would like to create a template, which creates a basic infobox on a page that provides all needed clan info from the database.

Runescape Clan users will opt to utilize the wiki because it is not restricted by a 2000 character limit. It allows for uploads of pictures. Basically, it works better...

If we can expand this wiki by utilizing the database content, we could easily generate a wiki with 10,000 pages or more (which would make it one of the largest wikia wikis in existence. More importantly, anyone who googles any clan in runescape would see our page as one (if not thee) top result, resulting in a very high traffic for the site.

Info Box Template
To start, a page would simple need a template that has the specific clan's runescape clan channel (cc) name. This would generate an info box which shows the basic information that is readily available from the database, CC name, owner's / deputy owners, total members, total level, citadel, etc. Like this:

Clan Declaration Template
To accompany the basic info box template, a second template will be made that copy's the text displayed on a clan's database page, the clan's deceleration.

Page Creating Bot
Our plan is to further this goal by generating a script or bot that goes to the database, finds clan pages that are not currently on our wiki, and then creates a page on our wiki without a person having to do anything. To do that, the bot creates a page, adds the previously mention templates, and also adds a template to let users know the page has not been edited by a person and needs expanded.

Human Editors
A human editor can visit the page after a bot has created it (or create a page and input the templates themselves) and remove the Pcbot template, and add any content they like.

Example Page
For the purposes of this demo, I have opted to utilize the cc: Clan Quest. Clan Quest Database Page

03:20, September 19, 2013 (UTC)

E-Mail to Jagex

 * I've prepared an email to send to Jagex ask them if they have any public plain-text database which is dynamic with the homepage clan database. I'm not sure of the terminology, so if Excel or anyone else with the knowledge would like to edit it below, feel free! Also, would anyone know the email address that would be most appropriate to send this to? 11:55, September 19, 2013 (UTC)

Hi,

I am emailing you on behalf of the admin team and community of the RuneScape Clans Wiki. My username there is MinigameGod and on RuneScape I am Minigame God.

The RuneScape Clans Wiki (http://runescapeclans.wikia.com/wiki/RuneScape_Clans_Wiki) has been much like it's sister wiki, The RuneScape Wiki, in that it has been around for a very long time and many players have benefited from it. Simply put, it allows players to create a free webpage for their clan that can display any information that relates to them, including logos, pictures, external links, etc. Many clans use it as a source of advertisement and as a central point where they can give-out and display information ,such as emails, IRCs, external links and more to prospective clan members! This is compared to the official RuneScape clan database which only allows plain-text, is limited to 2000 characters, and forum QFC.

This wiki was started before RuneScape even added support for clans, compared back when clans were limited to friends chats. As such, we have hundreds of over a thousand clan pages with more being added constantly.

The wiki has been inactive for a while, but with the talented and experienced team of admins we have decided to re-vamp the entire project. One of the many ideas that our admins have had includes having a bot to scan through the present clan database of the official RuneScape website to while analysing the clan pages we have already on the wiki and comparing them to the clans that are active on RuneScape. This would allow us to mark the pages accordingly, to incur and better decipher whether our pages should be archived or remain as an active clan. This gives new users the ability to not only recognize active clans but also the historical timeline of disbanded or inactive clans.

To make the lives of the admins easier, rather then manually scanning through the dynamic RuneScape database, we would like to be able to code a bot to do it for us automatically. To do this, We would require a plain text form for a bot to be able to recognize such as the GE price form:
 * http://services.runescape.com/m=itemdb_rs/api/graph/ GE ID HERE .json

Such a form would likely have clan name, amount of members, home world, citadel level, slogan and description. Maybe even with more detailed clan stats? As such, we would like to know if such a form exists and if the wiki would be able to take advantage of this data and/or what limitations there may be with copyright its user, etc.

Thanking you, Ben


 * I have made edits as I think they word best. Honestly, I have better contact with Jagex directly (Jmods are in my clan after all...) and perhaps we would be better off if I emailed them? I want to make sure that this file is not already publicly accessible before I send this in and look a fool. >.> 14:37, September 19, 2013 (UTC)
 * It's not nessarily even there, they may have to add a service to provide JSON files for the clans DB. 16:38, September 19, 2013 (UTC)
 * Write EXACTLY what we need and I will request it from Jagex. My offsite is an official fansite so I have access to fansite support contacts. I must point out, the support for fansites like to respond once, lol. 02:28, September 22, 2013 (UTC)
 * Sounds great! Basically, we're asking whether Jagex can create a plain text form for the Official Clan Database, like the above GE Price ID, which displays dynamic information for every clan so we can have bots accurately correct information for the clans we have listed here. I'm assuming the URL would have the clan name in it, which would allow the bot to cross-reference the clan page name on the wiki to the database so it can correct and update vital clan information, such as the amount of members, home world, citadel level, slogan, leader and description. Wait for Excel's final input first though.

We could start coding a bot to update the flag of the clan with the URL: http://services.runescape.com/m=avatar-rs/a=13/Maxed/clanmotif.png?cachebust=1379827918031 With "maxed" being replaced by any clan... If the clan doesn't exist it just displays the default flag. So the bot could recognise this URL as the clan being non-existent in RS.

Here are some links to have a look at:
 * 1) Player Highscores Update Bot Frequency: http://www.webcitation.org/5nQNHDweQ
 * 2) Player Highscores Normal: http://services.runescape.com/m=hiscore/a=13/compare?user1=Jake
 * 3) Player Highscores Lite: http://services.runescape.com/m=hiscore/index_lite.ws?player=Jake
 * 4) Clan Highscores Normal: http://services.runescape.com/m=clan-hiscores/compare.ws?clanName=Maxed
 * 5) Clan Highscores Rank: http://services.runescape.com/m=clan-hiscores/a=13/ranking?clanId=14541
 * 6) Clan Members: http://services.runescape.com/m=clan-hiscores/a=13/members_lite.ws?clanName=Maxed This downloads a .CSV file. Is shows all clan members with their RSNs, rank, total exp and total kills, in particular the clan leader.
 * 7) Clan Exists and/or Recruiting: http://services.runescape.com/m=clan-home/a=13/properties.js?clanName=Maxed

I just used Maxed and Jake as the examples. The 5th link showing Maxed having an ID of 14541. I've been reading through code for hours trying to find where there amount of members data comes from... I've tracked it to ... but I have no idea how to get into the class... I'll keep looking though! 07:45, September 22, 2013 (UTC)
 * I'm responding to show I have read everything, lol. Until excel responds and contributes with his infinte experience I will postpone any contributions on my part. :P  15:41, September 22, 2013 (UTC)

This is a good idea. Is there any way to possibly integrate the Grand Exchange data as well, as that could potentially be useful for a few clans (especially merching or commerce-related ones) to display? I know sites like www.grandexchangecentral.com do that. Triumvirate Department of State   20:08, September 22, 2013 (UTC)

Comments Unrelated to Execution
.