SitePerformanceMonitoringTools
Contents
What (summary)
Instrumentation that provides a history of performance statistics for each part of the page load pipeline.
Why this is important
The responsiveness and performance of the site makes a big difference in how many pages visitors will view, and how often they will come back. A poorly performing site will also wear out our active members causing some of them to leave.
DoneDone
- A Dashboard with Red, Yellow or Green for each pipeline item.
- The history for each pipeline item stored in the database.
- Definitions of acceptable benchmarks for each pipeline item.
- Definitions of acceptable freshness for each pipeline item.
- All monitoring meets the definitions for minimum freshness.
Performance Priorities
- View normal page
- View random page
- Edit click until available
- Save click until rendered
- Render invalidated frontpage
Instrumentation Steps
=== # End to end on each
- Deploy instrumentation boxes in various locations
- Determine and instrument the pieces
- MediaWiki profiling
- Raw database queries
Steps to DoneDone
- Articulate the request pipeline
-
Identify each request in the pipeline - Friday ... How to perform / retrieve data for each request
-
- Aggregate pipeline benchmarks
- Tuesday ... Implement the probes
- Wednesday portland devs ... Articulate database schema
- Wednesday portland devs ... Push XML results to central HTTPS server
- Wednesday portland devs ... Remote location / benchmark results stored in database
- Integrate into monitoring
- Dashboard to identify overall health
- Notifications via email/paging critical problem arises
- Analyze pipeline benchmarks
- XML output for monitoring integration
- Graph performance for each location
- Detailed graph view on each request
- Define acceptable benchmarks for each request
Pipeline
DNS request - www.aboutus.org & images.aboutus.org [R]
- Local resolver / cache
- Queries against the local resolver at the remote location provides little insight into health of the www.aboutus.org site. If the record does not exist in the local resolver cache (or the TTL has expired), the DNS root servers will be contacted and the authoritative servers. If the record already exists in the cache then it will respond immediately. If the local resolver does not reply as expected, then the issue likely lies with the remote location or possibly the authoritative name servers (or somewhere between).
- Authoritative name server
ns1.dnscloud.com ns2.dnscloud.com
- Response time of the authoritative server is critical. This can also be measured from any location, though, network latency and connectivity will be a factor.
- dig www.aboutus.org @ns1.dnscloud.com
- dig www.aboutus.org @ns2.dnscloud.com
- dig images.aboutus.org @ns1.dnscloud.com
- dig images.aboutus.org @ns2.dnscloud.com
IP connectivity [R]
- Network connectivity and latency can be measured using ping and traceroute utilities. Most issues with connectivity will most likely be caused by network problems between the two locations which we have no control over. In some cases, the issues could be caused by router, switch, or load-balancer issues on the AboutUs side, but these items will affect all remote locations.
- ping www.aboutus.org
- traceroute -n www.aboutus.org
HTTP request - /index.php
- Response time of a single /index.php GET request is critical. Performance relies on a number of factors.
- Physical server load
- CPU
- Disk I/O
- Available memory - too little memory causes swapping thereby causing disk I/O performance degradation
- Network throughput
- Apache process performance
- CPU usage
- Available threads
- Memcached
- Down cache / timeout
- Database query
- ?Query for each /index.php request?
- physical DB (slave) server loads
- replication
- MySQL performance
- DNS request - images.aboutus.org
- Image GET request
- image size
- number of images per page
- NFS server load (disk I/O)
- network throughput
- curl --silent --write-out %{time_total} http://www.aboutus.org/index.php | tail -1
- Other potentially useful -w output variables for curl: http_code time_total time_namelookup size_download speed_download
Acceptable Benchmarks
Max cold-request to fully rendered time for front page
- 0
curl -L --form wpSave=Save\ page --form wpTextbox1=replacement\ text --form wpEditToken=\\ --form wpEdittime=$edittime http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=submit
Render invalidated frontpage
Potential Hurdles
- False positives
- Caching
Questions
Does the pipeline include all of these? Record a history of how long to
- lookup DNS for www.aboutus.org, images.aboutus.org, ... from different parts of the world
- Setup a port 80 TCP connection with each of the squal boxes from different parts of the world
- Load the frontpage without any client caching
- Retrieve a memcache item from each combination of two squal boxes (one client, one memcached server)
- Load the core css files
- Load the core js files
...