Everyone agrees that charitable giving is generally a good thing, but few actually take the time to figure out where best donations should go. Thankfully, some organizations fulfill this basic societal need.
Giving What We Can is a non-governmental organization that offers objective comparisons between charities in order to determine which groups are more cost-effective in terms of creating good. This is not an idle distinction; even among aid programs that focus on extreme poverty, the difference between charities' effectiveness is not a factor of ten or a hundred—it can reach a factor of ten thousand to one.
This extreme deviation between truly cost-effective charities is what makes Giving What We Can such a useful tool. Their detailed analysis and reporting of results helps to give direction to people who wish to ensure their charitable donations accomplish as much as possible.
However, their data collection doesn't stop there.
It is through in-depth analysis like this that Giving What We Can establishes itself as the best charity evaluation organization available today.
Recently, I wanted to spend some time thinking about how I might be able to estimate the cost-effectiveness of a large and broad charity like Oxfam. What follows are some ideas I had while brainstorming. Keep in mind that most of my ideas never got parsed into properly grammatical English, but I've included my notes here anyway in case others might find it useful.
We want to underestimate the total expected VPD, because there is less certainty in a continuance of funding from a broad organization than a group that has a singular mission whose funding ratio we can have confidence in. Therefore, all estimations must be under, not over.
A first run through would start by:
- itemize each way they spend money last year.
- determine value per dollar for each individual item.
- assign these value per dollar as weights for each item.
- Sum over the product of each vpd weight by percent allocated to get total vpd.
100% = $100
25% funds * 4vpd = 4(25%)vpd = 1 vpd
is
$25 * 4vpd = 25*4 v = 100 v
However, broad charities like Oxfam sometimes move in and out of different areas, and might use varying percentages, since they have no clearly stated specific mission. (look this up to verify previous sentence). So we need to find confidence levels of the percentages. Some will have low variance, meaning we can be confident in the level of funding for that area. Some will have high variance, meaning we cannot depend on the percentage level remaining the same in future years.
So let's clarify the equation with new info. (look up proper way to express sum in document form)
SUM sub i ( Ci * Fi * Vi ) + Fr(Rr)
Ci = ith item confidence level. range [0,1].
Fi = ith item funding percentage. range [0,1].
Vi = ith item vpd. Possibly in DALY units?
Fr = funding percentage that goes toward revolving projects.
Vr = average vpd of all revolving projects oxfam so far funded.
Ci will be 1 for all projects they continually fund year in/year out at the same rate. If rate drops for ith item each year, underestimate appropriate number. Zero, maybe? If ith item revolves from project to project, set Ci to zero as it is included in special term R.
Fi will be funding percentage of ith item last year.
Vi is vpd (in DALY) of ith item.
Fr refers to percent of funding that tends to go toward revolving projects, and is not consistent year to year.
Vr is expected vpd of a new project. We can use average vpd of all revolving projects oxfam so far funded.
n order to estimate the cost-effectiveness of a broad charity, we must find a way to assign a value per dollar (VPD) consideration that uses the same units of welfare already being used with more focused charities.
We start by itemizing each way the organization spent money last year, and determining the VPD of each focused effort. Then we assign these VPD calculations as weights for each item, and multiply them by the amount of funding allocated for that item per donated dollar (FPDD). Summing over these will give is a total VPD for 2010-11.
However, broad charities sometimes modify their percent level of funding for each item from year to year, unlike focused charities who stick to the same mission. Some items might be dropped or added unexpectedly, and there may even be a portion of the funding allocation which regularly funds new projects in the hopes of finding something promising. So we must modify our formula to include our confidence levels in funding percentages by looking to see which items have high funding percentage variance from year to year:
SUM sub i ( Ci * Fi * Vi ) + Fr(Vr)
The above formula uses five distinct terms. Vi is the VPD for the ith item. Fi is the 2010-11 FPDD for the ith item, ranging from (0-1]. Ci is our confidence level in Fi, ranging [0-1]; 1 for consistently funded projects, 0 for items that regularly funds new projects each year, and appropriate underestimated values for all inconsistently funded projects. Fr is the FPDD that goes toward new projects each year. Vr is the expected VPD of a newly funded project, which we estimate by averaging the VPD of all past revolving projects. The r terms effectively replace all Ci terms that received an automatic 0.
Drawbacks to this method include a much more lengthy process of determining VPD, since broad charities would require finding VPD for several separate efforts. Also, this formula does not take into account the propensity for a broadly funding charity to find and retain new projects which increase overall VPD over time. Nevertheless, the decision to make the formula underestimate expected VPD is deliberate; the benefit a broad charity...
...broad charity has R&D benefit that focused charities do not. This allows them to find new funding opportunities that may have potentially larger VPDs. This benefit is not simulated in the given formula, but this is a deliberate choice as it does not affect expected VPD....
While there are several issues involved in estimating the cost-effectiveness of charities in general, three apparent major issues must be considered for broad charities such as Oxfam.
- Projects are funded inconsistently. One benefit of broadly funding organizations is the capability to fund new projects continuously in the hope of finding a particularly efficient program. However, this results in a lack of confidence in continued funding levels for projects.
- Analysis takes time. Whereas focused groups require a single investigation, broad charities require scrutiny of each project they fund. In Oxfam's case, it might result in two orders of magnitude of additional work.
- Data is scarce. Unlike other organizations, broad charities tend to not be as specific in how they fund particular projects, resulting in large error bars around any expected utility calculation.
The goal is to create an algorithm for broad charities which will return a value per dollar (VPD) expected utility in the same units of welfare used for focused organizations (such as modified DALYs that consider more than health). One such formula is described below. It uses weighted VPDs, confidence levels in projected funding per dollar (FPD), and expected VPD in newly funded programs to determine an overall VPD for the charity.
∑ ( Ci ⋅ Fi ⋅ B(Vi)) + (Fr ⋅ B(Vr))
where
C = confidence level in F
F = FPD
V = VPD
B = Bayesian function
r = revolving projects
The algorithm takes the sum of all Bayesian-corrected VPDs of each consistently funded project weighted by the proportion each is funded and our confidence level in its continued funding (ascertained through historical funding trends), then adds a funding-weighted term of expected VPD for newly funded projects (established via average VPD of revolving projects). The Bayesian function used should be equivalent to whatever is already used in modeling more focused charities.
By utilizing confidence levels and the concept of revolving program funding, the algorithm is able to deal with (1). Unfortunately, the suggested formula only serves to highlight problem (2). Summing over every individual funded project means one must learn the VPD of everything the organization funds. Even if (3) were not an issue, this would take a great deal of time—perhaps so much time that one would be better served by analyzing VPDs of focused organizations instead.
However, (3) is indeed still an issue. A cursory glance over Oxfam's 2010-11 annual report shows not enough detail, although more information is available on Oxfam affiliate websites, such as http://www.oxfam.org.hk/filemgr/1326/ARprojectlist2010-11.pdf. Still, the lack of data when you compare it to Against Malaria's individualized reports is stunning. With no mention of how much was spent on each project, nor details on how effective all funded projects were, (3) would seem to be a roadblock that severely limits the confidence one can have in assigning a VPD to any non-featured project. Hopefully, this issue can be resolved through requesting information that is not publicly available.
While there are several issues involved in estimating the cost-effectiveness of charities in general, four apparent major issues must be considered for broad charities such as Oxfam.
- Projects are funded inconsistently. One benefit of broadly funding organizations is the capability to fund new projects continuously in the hope of finding a particularly efficient program. However, this results in a lack of confidence in continued funding levels for projects.
- Analysis takes time. Whereas focused groups require a single investigation, broad charities might require scrutiny of each project they fund.
- Data is scarce. Unlike other organizations, broad charities tend to not be as specific in how they fund particular projects, nor in quantifying all accomplishments, resulting in large error bars around any expected utility calculation.
- Welfare unit choice is unclear. Funding multiple projects means incorporating units that can somehow compare DALYs to beneficial outcomes that are irrelevant to health.
A possible solution to (1) is to construct a mathematical model that incorporates confidence levels in continued funding (C) and an extra term to estimate the value of newly funded projects (n), such as:
∑ ( Ci ⋅ Fi ⋅ B(Vi)) + (Fn ⋅ B(Vn))
The above algorithm also incorporates the current funding per donated dollar level (F), the value per donated dollar (V), and whatever Bayesian function (B) on V that is currently being used in evaluating non-broad charities. Confidence levels can be ascertained through historical funding trends while an estimate of the value per dollar (VPD) of newly funded programs can be considered as the average VPD of past newly funded projects.
Unfortunately, the algorithm fails once we consider (2). Summing over every individual project which requires independent VPD appraisal would take a debilitating amount of time. The only way to efficiently solve this issue is to collapse the entire summation to a snapshot of total expenditure and total effects. Thankfully, this resolution also deals handily with first part of (3). If we abandon attempts to distinguish between individually funded projects, the lack of data on that front becomes a non-issue.
However, the second part of (3) still looms large. Oxfam's 2010-11 annual report includes cherry-picked accomplishments, but a full listing seems to be missing even from Oxfam affiliate websites. The closest thing to a full description seems to be the New Projects report (http://www.oxfam.org.hk/filemgr/1326/ARprojectlist2010-11.pdf), which does not bother quantifying any of their listed items. This seems to make (3) a roadblock that severely limits the confidence one can have in assigning a VPD to any non-featured project. Hopefully, this issue can be resolved through requesting information that is not publicly available.
The final problem is not unique to broad organizations, but (4) definitely stands out as the most intractable part of dealing with broadly funding charities. Oxfam has several goals that do not impact DALYs in any way, and figuring out a way to do expected value calculations using non-health related welfare units is something that will take much more than a 500 word response can really accommodate.
No comments:
Post a Comment