Difference between revisions of "SitePerformanceMonitoringTools"
(→Steps to DoneDone) |
|||
|
(43 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
| − | <noinclude><big>[[OurWork]] < [[DevelopmentTeam]] < [[DevelopmentTeamPriorities|Priorities]] < </noinclude>(''' | + | <noinclude><big>[[OurWork]] < [[DevelopmentTeam]] < [[DevelopmentTeamPriorities|Priorities]] < </noinclude><strike>('''[[PairDays|4]]''') [[SitePerformanceMonitoringTools]] ('''[[Ethan]]''') {{JustTinyEditIcon|SitePerformanceMonitoringTools}}</strike><noinclude></big> |
== What (summary) == | == What (summary) == | ||
Instrumentation that provides a history of performance statistics for each part of the page load pipeline. | Instrumentation that provides a history of performance statistics for each part of the page load pipeline. | ||
| + | |||
| + | Related pages: | ||
| + | * [[AboutUsPerformanceMonitoring]] | ||
| + | * [[SystemMonitoringDashboard]] | ||
== Why this is important == | == Why this is important == | ||
| Line 8: | Line 12: | ||
== [[DoneDone]] == | == [[DoneDone]] == | ||
| − | * A Dashboard with Red, Yellow or Green for each pipeline item. | + | * :A Dashboard with Red, Yellow or Green for each pipeline item. |
| − | * The history for each pipeline item stored in the database. | + | * :The history for each pipeline item stored in the database. |
| − | * Definitions of acceptable benchmarks for each pipeline item. | + | * :Definitions of acceptable benchmarks for each pipeline item. |
| − | * Definitions of acceptable freshness for each pipeline item. | + | * :Definitions of acceptable freshness for each pipeline item. |
| − | * All monitoring meets the definitions for minimum freshness. | + | * :All monitoring meets the definitions for minimum freshness. |
==Performance Priorities== | ==Performance Priorities== | ||
| Line 29: | Line 33: | ||
==Steps to DoneDone== | ==Steps to DoneDone== | ||
| − | * Articulate the request pipeline | + | * <strike>Articulate the request pipeline</strike> |
| − | ** Identify each request in the pipeline | + | ** <strike>Identify each request in the pipeline</strike> |
| − | ** How to perform / retrieve data for each request | + | ** <strike>How to perform / retrieve data for each request</strike> |
| − | * | + | * <strike>Aggregate pipeline benchmarks</strike> |
| − | + | ** <strike>'''Tuesday''' ... Implement the probes</strike> | |
| − | * | + | ** <strike>'''Wednesday portland devs''' ... Articulate database schema</strike> |
| − | + | ** <strike>'''Wednesday portland devs''' ... Push XML results to central HTTPS server</strike> | |
| − | * Push XML results to central HTTPS server | + | ** <strike>'''Wednesday portland devs''' ... Remote location / benchmark results stored in database</strike> |
| − | ** | + | ** Project revised for local monitoring only using Zabbix. Service on nimbus communicates with agents installed on each server and stored in MySQL. |
| − | + | * <strike>Integrate into monitoring</strike> | |
| − | + | ** <strike>Dashboard to identify overall health</strike> | |
| − | * Analyze pipeline benchmarks | + | ** <strike>Notifications via email/paging critical problem arises</strike> |
| − | ** Graph performance for each location | + | ** Available at https://admin.aboutus.org/zabbix/ |
| − | ** Detailed graph view on each request | + | * <strike>Analyze pipeline benchmarks</strike> |
| − | ** | + | ** <strike>XML output for monitoring integration</strike> |
| + | ** <strike>Graph performance for each location</strike> | ||
| + | ** <strike>Detailed graph view on each request</strike> | ||
| + | ** Information available at http://www.aboutus.org/AboutUsPerformanceMonitoring/ | ||
| + | * <strike>Define acceptable benchmarks for each request</strike> | ||
| − | * | + | == Repositories == |
| − | ** | + | * cd into the directory you want to check the client out into |
| − | ** | + | * git clone nimbus:/opt/git/geophone-client |
| + | * make a few changes in that directory | ||
| + | * git status ... to see what is different | ||
| + | * git diff ... to go through the change and make sure you aren't including something accidentally | ||
| + | * git add YYYY ... to include modified or new file YYYY in the commit | ||
| + | * git commit -m 'Here is why I made the changes I did for this commit' | ||
| + | * git push ... to make sure that the remote repository has your changes | ||
| + | * gitk ... from within the directory shows the tree of revisions | ||
==Pipeline== | ==Pipeline== | ||
| Line 54: | Line 69: | ||
* Local resolver / cache | * Local resolver / cache | ||
| − | + | Queries against the resolvers at the remote location provides little insight into health of the www.aboutus.org site. If the record does not exist in the local resolver cache (or the TTL has expired), the DNS root servers will be contacted and the authoritative servers. If the record already exists in the cache then it will respond immediately. If the local resolver does not reply as expected, then the issue likely lies with the remote location or possibly the authoritative name servers (or somewhere between). | |
* Authoritative name server | * Authoritative name server | ||
| Line 61: | Line 76: | ||
ns2.dnscloud.com | ns2.dnscloud.com | ||
| − | + | Response time of the authoritative server is critical. This can also be measured from any location, though, network latency and connectivity will be a factor. | |
| + | |||
| + | : dig www.aboutus.org @ns1.dnscloud.com | ||
| + | : dig www.aboutus.org @ns2.dnscloud.com | ||
| + | : dig images.aboutus.org @ns1.dnscloud.com | ||
| + | : dig images.aboutus.org @ns2.dnscloud.com | ||
| + | |||
| + | '''Results''' | ||
| + | * www.aboutus.org | ||
| + | : ns1.dnscloud.com | ||
| + | :: IP Address | ||
| + | :: Query time (ms) | ||
| + | : ns2.dnscloud.com | ||
| + | :: IP Address | ||
| + | :: Query time (ms) | ||
| + | |||
| + | * images.aboutus.org | ||
| + | : ns1.dnscloud.com | ||
| + | :: IP Address | ||
| + | :: Query time (ms) | ||
| + | : ns2.dnscloud.com | ||
| + | :: IP Address | ||
| + | :: Query time (ms) | ||
| − | === IP | + | === IP Connectivity === |
| + | Network connectivity and latency can be measured using ping and traceroute utilities. Most issues with connectivity will most likely be caused by network problems between the two locations which we have no control over. In some cases, the issues could be caused by router, switch, or load-balancer issues on the AboutUs side, but these items will affect all remote locations. | ||
| − | : | + | : ping -c 5 -i 0.2 -q www.aboutus.org |
| + | : traceroute -n www.aboutus.org | ||
| − | + | '''Results''' - <i>Five ICMP packets</i> | |
| − | + | * packet loss (%) | |
| + | * average response time (ms) | ||
| + | |||
| + | === HTTP Frontpage === | ||
| + | <i>/index.php and requisite pages</i> | ||
| + | |||
| + | Response time of the frontpage request is critical. Performance relies on a number of factors. | ||
* Physical server load | * Physical server load | ||
| Line 97: | Line 142: | ||
** NFS server load (disk I/O) | ** NFS server load (disk I/O) | ||
** network throughput | ** network throughput | ||
| + | |||
| + | : curl --silent --write-out %{time_total} --output > /dev/null http://www.aboutus.org/index.php | tail -1 | ||
| + | : <i>write-output variables: http_code time_total time_namelookup size_download speed_download</i> | ||
==== Acceptable Benchmarks ==== | ==== Acceptable Benchmarks ==== | ||
| Line 105: | Line 153: | ||
** 3 < t sec is unacceptable | ** 3 < t sec is unacceptable | ||
| − | === HTTP | + | '''Results''' |
| + | * Request time (ms) | ||
| + | |||
| + | === HTTP Search === | ||
| + | Response time for search results to be displayed displayed. | ||
| + | : curl --location --form auSearch=aboutus.org http://www.aboutus.org/Special:AboutUsSearch | ||
| − | + | '''Results''' | |
| + | * Request time (ms) | ||
| − | : curl - | + | === HTTP Random === |
| + | Response time for a random page to be displayed. | ||
| + | : curl --location --silent http://www.aboutus.org/Special:Random | ||
| − | + | '''Results''' | |
| + | * Request time (ms) | ||
| − | + | === HTTP/HTTPS Authentication === | |
| + | Measure the time it takes for a user to log into the www.aboutus.org site. | ||
| − | === | + | : curl --location --cookie-jar - --form wpName=user\ name --form wpPassword=password http://www.aboutus.org/index.php?title=Special:Userlogin\&action=submitlogin\&type=login\&returnto=Wiki |
| + | : curl --location --insecure --cookie-jar - --form wpName=user\ name --form wpPassword=password https://www.aboutus.org/index.php?title=Special:Userlogin\&action=submitlogin\&type=login\&returnto=Wiki | ||
| − | + | '''Results''' | |
| + | * Request time (ms) | ||
| − | + | === HTTP Edit === | |
| − | : | + | : http://www.aboutus.org/index.php?title=domain.com&action=edit |
| − | + | '''Results''' | |
| + | * Request time (ms) | ||
| − | === HTTP edit page | + | === HTTP Save === |
| + | : edittime=`curl --location http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=edit | grep wpEdittime | awk -F\" '{print $2}'`<p> | ||
| + | : curl --location --form wpSave=Save\ page --form wpTextbox1=replacement\ text --form wpEditToken=\\ --form wpEdittime=$edittime http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=submit | ||
| + | : <i>warning: will replace all article body with text in wpTextbox1</i> | ||
| − | === Render | + | '''Results''' |
| + | * Request time (ms) | ||
| + | |||
| + | === HTTP Render Invalidated === | ||
| + | : curl http://www.aboutus.org/Wiki -d action=purge | ||
| + | |||
| + | '''Results''' | ||
| + | * Request time (ms) | ||
==Potential Hurdles== | ==Potential Hurdles== | ||
| Line 145: | Line 216: | ||
| + | </noinclude> | ||
[[Category:DevelopmentTeamProject]] | [[Category:DevelopmentTeamProject]] | ||
| − | |||
Latest revision as of 05:44, 19 October 2007
Contents
What (summary)
Instrumentation that provides a history of performance statistics for each part of the page load pipeline.
Related pages:
Why this is important
The responsiveness and performance of the site makes a big difference in how many pages visitors will view, and how often they will come back. A poorly performing site will also wear out our active members causing some of them to leave.
DoneDone
- :A Dashboard with Red, Yellow or Green for each pipeline item.
- :The history for each pipeline item stored in the database.
- :Definitions of acceptable benchmarks for each pipeline item.
- :Definitions of acceptable freshness for each pipeline item.
- :All monitoring meets the definitions for minimum freshness.
Performance Priorities
- View normal page
- View random page
- Edit click until available
- Save click until rendered
- Render invalidated frontpage
Instrumentation Steps
=== # End to end on each
- Deploy instrumentation boxes in various locations
- Determine and instrument the pieces
- MediaWiki profiling
- Raw database queries
Steps to DoneDone
-
Articulate the request pipeline-
Identify each request in the pipeline -
How to perform / retrieve data for each request
-
-
Aggregate pipeline benchmarks-
Tuesday ... Implement the probes -
Wednesday portland devs ... Articulate database schema -
Wednesday portland devs ... Push XML results to central HTTPS server -
Wednesday portland devs ... Remote location / benchmark results stored in database - Project revised for local monitoring only using Zabbix. Service on nimbus communicates with agents installed on each server and stored in MySQL.
-
-
Integrate into monitoring-
Dashboard to identify overall health -
Notifications via email/paging critical problem arises - Available at https://admin.aboutus.org/zabbix/
-
-
Analyze pipeline benchmarks-
XML output for monitoring integration -
Graph performance for each location -
Detailed graph view on each request - Information available at http://www.aboutus.org/AboutUsPerformanceMonitoring/
-
-
Define acceptable benchmarks for each request
Repositories
- cd into the directory you want to check the client out into
- git clone nimbus:/opt/git/geophone-client
- make a few changes in that directory
- git status ... to see what is different
- git diff ... to go through the change and make sure you aren't including something accidentally
- git add YYYY ... to include modified or new file YYYY in the commit
- git commit -m 'Here is why I made the changes I did for this commit'
- git push ... to make sure that the remote repository has your changes
- gitk ... from within the directory shows the tree of revisions
Pipeline
DNS request - www.aboutus.org & images.aboutus.org
- Local resolver / cache
Queries against the resolvers at the remote location provides little insight into health of the www.aboutus.org site. If the record does not exist in the local resolver cache (or the TTL has expired), the DNS root servers will be contacted and the authoritative servers. If the record already exists in the cache then it will respond immediately. If the local resolver does not reply as expected, then the issue likely lies with the remote location or possibly the authoritative name servers (or somewhere between).
- Authoritative name server
ns1.dnscloud.com ns2.dnscloud.com
Response time of the authoritative server is critical. This can also be measured from any location, though, network latency and connectivity will be a factor.
- dig www.aboutus.org @ns1.dnscloud.com
- dig www.aboutus.org @ns2.dnscloud.com
- dig images.aboutus.org @ns1.dnscloud.com
- dig images.aboutus.org @ns2.dnscloud.com
Results
- www.aboutus.org
- ns1.dnscloud.com
- IP Address
- Query time (ms)
- ns2.dnscloud.com
- IP Address
- Query time (ms)
- images.aboutus.org
- ns1.dnscloud.com
- IP Address
- Query time (ms)
- ns2.dnscloud.com
- IP Address
- Query time (ms)
IP Connectivity
Network connectivity and latency can be measured using ping and traceroute utilities. Most issues with connectivity will most likely be caused by network problems between the two locations which we have no control over. In some cases, the issues could be caused by router, switch, or load-balancer issues on the AboutUs side, but these items will affect all remote locations.
- ping -c 5 -i 0.2 -q www.aboutus.org
- traceroute -n www.aboutus.org
Results - Five ICMP packets
- packet loss (%)
- average response time (ms)
HTTP Frontpage
/index.php and requisite pages
Response time of the frontpage request is critical. Performance relies on a number of factors.
- Physical server load
- CPU
- Disk I/O
- Available memory - too little memory causes swapping thereby causing disk I/O performance degradation
- Network throughput
- Apache process performance
- CPU usage
- Available threads
- Memcached
- Down cache / timeout
- Database query
- ?Query for each /index.php request?
- physical DB (slave) server loads
- replication
- MySQL performance
- DNS request - images.aboutus.org
- Image GET request
- image size
- number of images per page
- NFS server load (disk I/O)
- network throughput
- curl --silent --write-out %{time_total} --output > /dev/null http://www.aboutus.org/index.php | tail -1
- write-output variables: http_code time_total time_namelookup size_download speed_download
Acceptable Benchmarks
Max cold-request to fully rendered time for front page
- 0
- curl --location --form wpSave=Save\ page --form wpTextbox1=replacement\ text --form wpEditToken=\\ --form wpEdittime=$edittime http://www.aboutus.org/index.php?title=ObsidiansAnd.com\&action=submit
- warning: will replace all article body with text in wpTextbox1
Results
- Request time (ms)
HTTP Render Invalidated
- curl http://www.aboutus.org/Wiki -d action=purge
Results
- Request time (ms)
Potential Hurdles
- False positives
- Caching
Questions
Does the pipeline include all of these? Record a history of how long to
- lookup DNS for www.aboutus.org, images.aboutus.org, ... from different parts of the world
- Setup a port 80 TCP connection with each of the squal boxes from different parts of the world
- Load the frontpage without any client caching
- Retrieve a memcache item from each combination of two squal boxes (one client, one memcached server)
- Load the core css files
- Load the core js files
...