Economics of GIS
benchmarking and system evaluation
costs and benefits
return on investment
Benchmarking and system evaluation
functional requirements study (FRS) is complete
collection of product descriptions
master data list
all data needed to make products
indexed to products
prioritized
conceptual data model
geodatabase design
list of functions
specified in general terms
not specific to any software vendor
not specific to e.g. raster, vector
indexed to products
prioritized
Request for Proposals (RFP)
can your company provide the system needed to do this?
software
hardware
training
possible prior RFI, RFQ to identify potential suppliers
software vendors
system integrators
subcontracts to software, hardware vendors
perhaps 6-10 responses
evaluation of bids
must be open, transparent
must stand up to litigation
large contracts are commonly contested
is the bid over-configured?
or under-configured?
two issues
does the proposal live up to its claims?
how does it rate compared to other bids?
risks
planning horizon at least five years
long-term stability of company
technological change
worth spending significant proportion of contract value on evaluation
5%?
10%?
benchmark tests to reduce risks
pilot studies
cost/benefit analysis
benchmarking reduces risks in system selection
reassures the customer
builds confidence in vendor
costs borne by vendor
say $100k for an elaborate benchmark
possibly subsidized by customer
two types of benchmarking
qualitative
does the function exist?
is it easy to use?
quantitative
can the proposed system handle the volume of work?
benchmark script
lists a series of steps the vendor must execute
subjective evaluation by observer
objective evaluation of performance
stopwatch
allows the most important functions to be examined
perhaps all
realistic data volumes
real data sets, real products
Performance evaluation (PE)
allocation of scarce computing resources
subfield of computer science
break tasks into subtasks that can be evaluated
Gibson Mix
a standard set of instructions used to benchmark computers
need the equivalent for GIS
subtasks for GIS PE?
must be independent of vendor
e.g. of raster vs vector
based on FRS
list of 75 functions
generated by agency's staff
quantitative benchmark
needs a mathematical model
predicting resource utilization
computing time
staff time
storage volume
from inputs
numbers of point, line, area features
GIS PE is difficult
uncertainties about proposed approach
raster or vector?
algorithms used
high level at which tasks must be specified
difficulty of predicting workload
no chance of high accuracy
but limited accuracy is better than nothing
Performance evaluation model
library of subtasks L
the set of all GIS functions
overlay, buffer, measure area, digitize ...
set of information products identified by FRS
R1, R2, R3, ...
each requires a set of subtasks
each product required a known number of times per year
product i is required Yij times in year j
j runs up to the end of the planning horizon
e.g. 5 years
any task uses resources
Mak is the amount of resource k used by subtask a
k is computing time, staff time, storage volume
a is one of the subtasks in L
need to find a relationship between Mak and indicators of task size
number of features processed
determine the relationship from the quantitative benchmark
using examples in the benchmark script
compute the total demand on resources by the entire project
with useful though not perfect accuracy
Wkit
amount of resource k used by subtask t in a single generation of product i
Wki = sum of Wkit over tasks t
total amount of resource k used in generating product i once
Vkj = sum of (Wki Yij) for all products i
total amount of resource k used in year j
summary
1. define products and tasks (FRS)
2. evaluate each task from qualitative benchmark
3. analyze system's overall capability given (2)
4. obtain performance measures from quantitative benchmark
5. build performance models
6. determine future workloads
7. predict future resource utilization, compare to resources proposed
Example benchmark
forest management agency
responsible for millions of hectares of land
33 information products identified
50 different GIS functions required
out of 75 in the library
acquisition anticipated to exceed $2 million
three phases of benchmark
data input
conversion, digitizing
tests of functions
observed by team
generation of 4 selected products
at least one test of every function required
several tests for the most heavily used functions
12 different tests of digitizing
ranging from <10 to >700 polygons
qualitative benchmark
score every function
0 = best in industry
9 = impossible to implement without major modification
maximum score for any task in a product is measure of difficulty of making product
quantitative benchmark
emphasis on staff time as resource measure
predictors are number of polygons, number of polylines in task
no distinction between straight and wiggly lines
sample results:
polygons polylines time (mins) 766 0 930 129 0 136 0 95 120 digitizing was done by vendor's staff
well trained, familiar with software
speeds are optimistic
analysis
m = k1p1 + k2p2
m = staff time
p1 = number of polygons
p2 = number of polylines
k1, k2 staff time per feature to be determined
best fit equation
m = 1.21 p1 + 0.97 p2
1.21 minutes per polygon, 0.97 minutes per polyline
computing time resource
m = 2.36 p1 + 2.63 p2
in seconds
uncertainties in prediction
34% for staff time
44% for computing time
computing time less predictable than staff time?
totals:
year staff time (mins) 1 185,962 2 302,859 3 472,035 4 567,823 5 571,880 6 760,395 one person year = 2,000 hours = 120,000 minutes
by year 6 need will be for 6 digitizing stations
or 3 working 2 shifts each
or 2 working 3 shifts each
summary
difficult to predict performance even under ideal circumstances
particularly for GIS with functions defined at a high level
useful to compare performance against vendor's own claims
assumes that the configuration will be the one used in the benchmark
expert digitizers
any prediction however uncertain is better than none
reducing risk in system acquisition
Cost/benefit analysis
what is it, why do it?
system is large investment
need to ensure benefits exceed costs
need to assess return on investment
basis for political, managerial support
accrual
who pays the costs?
agency
taxpayer
users through fees
system vendor
demonstration project
marketing value
who do the benefits accrue to?
benefits that accrue to the organization are not necessarily all
society at large may benefit
Defining costs
initial feasibility study
hardware and software
maintenance contracts
training
personnel
supplies
overheads
Cost issues
data acquisition
acquired for other purposes as well?
how to share costs
one-time versus recurring costs
apples and oranges?
sum over planning horizon?
Defining benefits
more difficult to quantify
tied to products
and improved decision making
tangible benefits
decreased costs
savings in staff timie
avoided costs
increased revenue
intangible benefits
improved decisions
decreased uncertainty
improved image
Examples of benefits
GIS reduced the previous costs of making maps manually
tangible, quantitative
garbage collection company reduced staff costs through better scheduling
tangible, quantitative
average response time by emergency vehicles reduced
intangible?
less fire damage
better recovery from heart attack
reduced costs of logging
through better siting of roads, better planning of cutting
implies bad decisions would otherwise have been made
avoided costly litigation in land ownership case
what would have happened without GIS?
Forest Service finds better location for a campsite
controlled experiments on decision-making?
take two managers, give one the information...
Example
Washington State Department of Natural Resources
7 regional offices, 1 central office in Olympia
3 million acres of state land
2 million forested
charged with producing revenue
managing natural resoruces
public service
examples of GIS products
base maps of land use, land cover
land lease and land exchange maps
road and bridge maintenance maps
environmental impact analysis
potential debris flow hazard maps
fire hazard maps
timber harvest tracking
spatial allocation of crew workloads
costs
hardware and software 33%
maintenance contracts 9%
staff 43%
travel 1%
supplies and services 14%
benefits
compared to previous arrangements
based on extrapolating those costs
comparing to new system
1. increased revenue
2. decreased costs
3. staff savings
4. cost reductions
summary:
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Total cost reductions 230,500 771,000 796,000 830,000 864,000 991,000 1,091,000 1,124,000 New tangible benefits 0 0 0 10,000 25,000 342,000 1,057,000 2,093,100 Total benefits 230,500 771,000 796,000 840,000 889,000 1,333,000 2,148,000 3,217,100 Total system costs 630,000 310,000 445,000 796,000 1,454,000 1,091,000 1,213,000 1,256,000 Benefits less costs -399,500 461,000 351,000 44,000 -565,000 242,000 935,000 1,961,100 intangible benefits: orphan roads
historically, road construction unregulated
orphan roads from old logging
poorly located
steep gradients
potential for debris flow
two disasters
loss of life
cost in millions
GIS used to locate orphan roads on steep slopes
at stream crossings
potential trigger points
inspection and mitigation