DevelopmentTeam < Priorities < WhoisRefresh (14, 10-9, 8-6, 4-2, 4)

This project actually brings together several smaller projects into one. Essentially, it involves a rewrite of the page creation bot to use several new data sources.

1. (10) WhoisParsing (Arif Iqbal) (7-10)
   
   • AddressParsing
2. (2) Rewrite PageCreationBot (Ghufran, Umar Sheikh)
   
3. MediaWiki:InitialDomainPage
4. (4) Rewrite PageScrapeBot (Laiq)
   
5. (4) WhoisRefreshRunRefresh (Jason and Ali)
   

edit Steps to DoneDone

• Branch whois-refresh
• Get it running on our local machines
• Assess where we currently are
• Ruthlessly prune ... spin off any tasks that aren't absolutely essential onto their own pages that will be considered later

edit Synopsis

A new domain-page is scraped, and populated

• When a red-link is clicked
• When a search for a domain doesn't return a page

edit AcceptanceTest

Look at 50 pages that we know don't have contact info and verify that the contact info is coming in.

edit Background

• rfc3912 the current protocol specification
• Domain Statistics by TLD
• 100 oldest dot com domains
• Registrar Stats

The ranking and percentages come from http://populicio.us/toptlds.html and are at least as stale as November 2006.

edit Information We Need

• Date of lookup
• Registrant Address
• Admin Address
• Phone
• Email

Question: do we need both the registrant and admin addresses or is one enough? In the past we've only used one. - Ray | talk

edit Next

• Get gpMakeImage working so that tests pass
• Send email
• Given an address get a lattitude/longitude
• Obfuscate address

 .
 .
 .

edit Latitude/Longitude

Need a table so that we can associate one or more lat/long pairs with a page.

edit Some neat regular expressions

These are from the book, "The Ruby Way"

• The following regex matches a phone number in the NANP format (North American Numbering Plan). It allows three common ways of writing such a phone number:phone = /^((\(\d{3}\) |\d{3}-)\d{3}-\d{4}|\d{3}\.\d{3}\.\d{4})$/

"(512) 555-1234" =~ phone # true "512.555.1234" =~ phone # true "512-555-1234" =~ phone # true "(512)-555-1234" =~ phone # false "512-555.1234" =~ phone # false

• Here is a regex to match a U.S. ZIP Code (which may be five or nine digits):

zip = /^\d{5}(-\d{4})?$/

• The following regex matches all the 51 usual codes (50 states and the District of Columbia):

state = /^A[LKZR] | C[AOT] | D[EC] | FL | GA | HI | I[DLNA] | K[SY] | LA | M[EDAINSOT] | N[EVHJMYCD] | O[HKR] | PA | RI | S[CD] | T[NX] | UT | V[TA] | W[AVIY]$/x

edit Whois Records We Need

We need 50 more whois records covering the range of formats for each of the following registrars:

• public domain
• ONLINENIC
• STRATO
• BASICFUSION.COM
• DOMAINNAMESALES | domain name sales
• core
• METAPREDICT.COM

We have a few but need 50 whois records covering the range of formats for each of the following registrars:

• ascio
• beijing Innovative
• belgium Domains
• capitol_domain
• discount_domain
• domaindiscover
• domain_doorman
• domainsite
• dotregistrar
• dotster
• fabulous.com
• gandi
• hichina
• innerwise
• joker.com
• Key systems
• Mark Monitor
• Melbourne IT
• Moniker
• NameKing
• Name.Net
• Names4ever
• namesdirect
• nameview
• nicline
• ovh
• psi-usa
• registerfly
• schlund
• srsplus
• wild west domains