Created page with "{{User:MCruz (WMF)/Sandbox/Program Reports header | The numbers | 2 = Some general text about the inputs, outputs and outcomes goes here }} {|cellpadding="15" style="margin:a..."
New page
{{User:MCruz (WMF)/Sandbox/Program Reports header | The numbers | 2 = Some general text about the inputs, outputs and outcomes goes here }}
{|cellpadding="15" style="margin:auto;max-width:70em"
|-valign="top"
|style="font-size:1.4em;line-height:normal;padding-top:.7em;text-align:center"|[[File:Plan icon BLACK-01.svg|60px|center|link=]] <br/>
<div id=inputs>
'''Inputs'''
|
<div style="background-color:#CCE9E2;padding:0.5em">
;In order to learn more about the inputs that went into planning workshops, we asked program leaders to report on:
* The budget that went into planning and implementing the workshop
* The hours that went into planning and implementing the workshop
* Any donations that they might have received for the event: a venue, equipment, food, drink, giveaways, etc.
</div>
=== Budget ===
<div style="background-color:white;border:1px solid color:#FFB870;padding:0.5em">
;Reported budgets were available for the majority of Wiki Loves Monuments contests. The average national contest costs just over {{Highlight|$3,600 US}} to implement.
</div>
Reported budgets were provided for {{Highlight|26}} Wiki Loves Monuments implementations ({{Highlight|96%}}). {{Highlight|One event, directly reported, indicated only volunteer hours.}} Budgeted costs for Wiki Loves Monuments ranged from {{Highlight|$392.05 to $29,101.59}} with an average<ref>Averages reported refer to the median response.</ref> of {{Highlight|$3,614.50}}. <ref>''Mean''= {{Highlight|6,340.68}}, ''SD''= {{Highlight|$6,516.49}}</ref>
=== Hours ===
<div style="background-color:white;border:1px solid color:#FFB870;padding:0.5em">
;{{Highlight|Too few program leaders and implementations reported staff and volunteer hours for us to provide an evaluation for this pilot report.}}</div>
Due to a low response rate by program leaders on this pilot survey, we are unable to report a solid evaluation on input hours for Wiki Loves Monuments at this time. We obtained data on staff input hours for only one, and volunteer hours for just two, directly reported WLM implementations.
=== Donated resources ===
<div style="background-color:white;border:1px solid color:#FFB870;padding:0.5em">
;{{Highlight|The majority}} of Wiki Loves Monuments implementations used donated resources, with meeting space and prizes/give-aways being the most common. </div>
The majority of Wiki Loves Monuments implementations ({{Highlight|78%}}) reported whether they had used any donated resources. More than half of the implementations reported using donated meeting space ({{Highlight|62%}}) and donated prizes/give-aways ({{Highlight|52%}}). In addition, {{Highlight|19 percent}} reported using materials/equipment that were donated and {{Highlight|14 percent}} used food donations (see ''Graph 1'').<Br/>
[[File:WLMDonated Resources Bar.png|thumb|left|500px|'''Graph 1: Use of donated resources.'''<br></br>
The donated resources most often reported for Wiki Loves Monuments were meeting space ({{Highlight|62%}}) and prizes or give-aways ({{Highlight|52%}}). Wiki Loves Monuments events appear to benefit often from the use of donated resources to carry out the events.]]
{{clear}}
{{Cot|bg=white|title=References}}
<references/>
{{Cob}}
</div>
|-
|colspan="3"|<hr />
|-valign="top"
|style="font-size:1.4em;line-height:normal;padding-top:.7em;text-align:center"| [[File:Handshake icon BLACK and WHITE.svg|60px|center|link=]] <br/><div id=outputs>'''Outputs'''
|
<div style="background-color:#CCE9E2;padding:0.5em">
;We also asked about two outputs in this section:
* How many participants contributed to Wiki Loves Monuments?
* How many of those participants were new users?
We used this output and budget inputs to learn:
* What was the dollar cost for each participant
</div>
=== Participation ===
<div style="background-color:white;border:1px solid color:#FFB870;padding:0.5em">
;The average Wiki Loves Monuments implementation has {{Highlight|190 participants}}. For 2012, the majority of participants were new users, making up {{Highlight|87%}} of the participant base.
We were able to collect participant numbers for all {{Highlight|27}} Wiki Loves Monuments implementations. The number of participants in the national contests ranged from {{Highlight|67 to 2,005 people.}} The average national contest has {{Highlight|190 participants}}.<ref>''Mean''= {{Highlight|394}}, ''SD''= {{Highlight|456}}</ref> The majority ({{Highlight|87%}}) of users who participated in {{Highlight|Wiki Loves Monuments 2012}} were new users. <s>We were unable to gather this information for Wiki Loves Monuments 2013, since the contest was just ending as we were analyzing the data.</s>
</div>
;On average, Wiki Loves Monuments implementations cost about {{Highlight|$14.50 USD per participant.}}
Budget and participant count was available for 26 of the 27 implementations used in this evaluation. Looking at programming costs in terms of the number of participants, we learned that programming costs ranged from $1.20 USD to $326.98 USD and averaged $14.58 USD per participant.<ref>''Mean''=$40.20, ''SD''=$66.77</ref><br/>
{{Cot|bg=white|title=References}}
<references/>
{{Cob}}
</div>
|-
|colspan="3"|<hr />
|-valign="top"
|style="font-size:1.4em;line-height:normal;padding-top:.7em;text-align:center"| [[File:Handshake icon BLACK and WHITE.svg|60px|center|link=]] <br/> <div id=outcomes>'''Outcomes'''
|
=== Content Production and Quality Improvement<ref>Note: Although "content production" is a direct product of the program event itself and technically a program [[m:Programs:Evaluation_portal/Library/Glossary#O|''output'' rather than ''outcome'']] most of the program leaders who participated in the logic modeling session felt this direct product was the target outcome for their programming. To honor this community perspective, we include it as an outcome along with quality improvement and retention of "active" editors.</ref> ===
<div style="background-color:white;border:1px solid color:#FFB870;padding:0.5em">
;The average national Wiki Loves Monuments contest leads to just {{Highlight|over 5,600 photos}} uploaded to Commons during the month of September. {{Highlight|For Wiki Loves Monuments 2012, the average number of photos placed in Wikimedia projects was 885.}} </div>
We collected photo upload counts for all 27 WLM implementations. Upload counts ranged from {{Highlight|1,572 to 32,987}} (''Graphs 2 and 3''). The average event uploaded {{Highlight|5,635}}<ref>(''Mean'' = {{Highlight|8,864}}, ''SD''={{Highlight|8,846}})</ref> photos during the month of September {{Highlight|2012 or 2013}}.
Regarding image use:
*Wiki Loves Monuments <s>2012</s> 2013 data for image use ranged from {{Highlight|387 to 6,103 unique images used.}} The average number of images used from a <s>2012</s> 2013 Wiki Loves Monuments event was {{Highlight|885}}.<ref>(Mean=1929, SD=1948)</ref> {{Highlight|2012}} also shows us that the more images uploaded, the more images are used (see ''Graph 4'').
*Wiki Loves Monuments <s>2013</s> 2014 data for image use ranged from {{Highlight|10 to 1,687 unique imaged used.}} The average number of unique images used from a <s>2013</s> 2014 Wiki Loves Monuments event was {{Highlight|262}}<ref>{{Highlight|As of November 4, 2013.}}</ref><ref>(Mean={{Highlight|413}}, SD={{Highlight|478}})</ref> (see ''Graph 5'').
<s>Do note that the numbers reported for 2013 are not final. Since the event ended around the time that we were collecting data, we expect image use in Wikimedia projects to rise.</s>
;The average Wiki Loves Monuments photo upload costs almost {{Highlight|50 cents USD.}} However, more money spent on a national Wiki Loves Monuments event does not necessarily relate to more participants or more image uploads.
Based on reported budgets for Wiki Loves Monuments implementations and the actual photo uploads, the amount of dollars spent for each photo uploaded ranged from {{Highlight|$0.09 USD to $9.17 USD each.}} The average implementation cost {{Highlight|$0.48 USD}} <ref>(Mean={{Highlight|$1.1}}, SD={{Highlight|$2.00}})</ref> per photo. We also looked at how much it cost in terms of Wiki Loves Monuments photos which were used on Wikimedia projects. For <s>2012</s> implementations, the cost per used photo ranged from {{Highlight|$0.51 USD to $26.51 USD}}, averaging {{Highlight|$2.43 USD}}<ref>(Mean={{Highlight|$6.12}}, SD={{Highlight|$8.22}})</ref> per image. There does not seem to be a direct relationship between budget, participant count, or images uploaded (see ''Graphs 2-5'').
<gallery widths="500px" heights="300px" perrow="3">
File:Budget_Participation_Photos_Added_2012.png|'''Graph 2: Budget, participation, and photos added.'''<br></br> As illustrated in the graph, budget inputs ranging from less than {{Highlight|$1,000 USD to almost $17,000 USD.}} The number of participants ranged from {{Highlight|75 to 2,005,}} while the number of images added ranged from nearly {{Highlight|2,000 to more than 30,000.}} The varying sizes of the bubbles — with larger bubbles representing more images uploaded — show that the number of photos increase significantly when events have over 500 participants. There does not seem to be a direct relationship between budget, participant count, or images uploaded. The bubble size doesn't get larger or smaller — meaning when more money is invested in an Wiki Loves Monuments implementation, that doesn't mean the event will have a higher participant count or a higher upload count.
File:Budget_Participation_Photos_Added_2013.png|'''Graph 3: Budget, participation, and photos added.'''<br></br> As illustrated in the graph, budget inputs ranged from less than {{Highlight|$500 (USD) to nearly $30,000 (USD)}}<ref>Note: This implementer also had the largest budget in <s>2012</s> and did not use the full amount so they applied the remaining budget to their <s>2013</s> event budget. It is unknown whether the full budget will be allocated to event expenses. </ref> and number of participants ranged from as few as 67 and as many as 1485. The number of photos/media added (illustrated by bubble size and label in the graph) ranged from more than {{Highlight|15,00 to nearly 25,000.}} It appears from the varying sizes of the bubbles on the graph (larger bubbles representing more photos added) that the number of photos added increases along with the number of participants. We also see that event participant counts had a much more stable range of participant numbers for the majority of events, with the median {{Highlight|50%}} reporting between {{Highlight|156 to 208 participants.}} <s>Unlike the graph for 2012, we see that the largest bubbles, representing the greatest number of photos added, seem appearing at lower budget levels.</s>
</gallery>
<gallery widths="500px" heights="350px" perrow="3">
File:Budget_PhotosAdded_UniqueUsed_2012.png|'''Graph 4: Budget, photos added, and unique photos used for 2012.''' <br></br> This bubble graph compares the number of photos added with the budget, using bubble size to show how many unique photos were added to Wikipedia pages for 2012 competitions. We generally see that as the number of total photos added increased, the number of photos used on Wikipedia pages also increases.
File:Budget_PhotosAdded_UniqueUsed_2013.png|'''Graph 5: Budget, photos added, unique photos used for 2013.'''<br></br> This bubble graph compares the number of photos added with the budget, using bubble size to show how many photos were added to Wikipedia pages for 2013 competitions. There has not been sufficient time after the end of the competitions to allow photos to be uploaded to Wikipedia pages, showing that the largest number of photos used is only {{Highlight|1,687, }}even though total photos uploaded is around {{Highlight|25,000.}}
</gallery>
[[File:Dollars_Per_Photo_Box.png|thumb|left|500x350px|'''Graph 6: Cost per photos added.''' <br></br> This boxplot shows the distribution of 2012 and 2013 Wiki Loves Monuments competitions. In comparing between the two years, we see that the dollar cost per {{Highlight|photo range was greater in 2012 than in 2013}}; however, the median cost was lower for {{Highlight|2012 at $0.37 per photo added.}} This suggests that spending per photo between both years may not be very different. More data would be needed to confirm. ]]{{clear}}
=== Photo quality ratings ===
<div style="background-color:#FFB870;padding:0.5em">
;Only a very small percentage ({{Highlight|0.8%}}) of Wiki Loves Monuments {{Highlight|2012}} images are rated as Quality, Valued and/or Featured Images on Wikimedia Commons.
<s>This section focuses only on images uploaded for Wiki Loves Monuments 2012, since the 2013 competition had was just ending as we finished of our data collection; not enough time had passed for ratings to have been awarded at that time.</s> </div>
By {{Highlight|November 2013, about 13 months after the 2012 contests ended}}, out of the 12 contests examined, five ({{Highlight|42%}}) had Quality images, three ({{Highlight|25%}}) had Valued images, and most ({{Highlight|7 contests, 58%}}) had Featured Pictures associated with them. Out of the 126,424 photo uploads we examined for Wiki Loves Monuments 2012, 994 (0.8%) were rated as a Quality Image, Valued Image, or Featured Picture (see ''Graph 8''). Out of those {{Highlight|994 rated photos}}<ref>Some photos may have been rated in multiple categories.</ref>:
*'''{{Highlight|932}} ({{Highlight|94%}}) were rated as Quality Images.''' Reports of images rated as Quality ranged from {{Highlight|0 to 484}} with the majority ({{Highlight|91%}}) of Quality images generated by only two contests, one which contributed {{Highlight|484 (52%)}}, and another which contributed {{Highlight|364 (39%)}}.
*'''{{Highlight|30 (3%)}} were rated as Valued Images.''' Reports of images rated as Valued ranged from {{Highlight|0 to 27 with the majority (90%)}} of Valued images generated by one contest.
*'''{{Highlight|32 (3%)}} were rated as Featured Pictures.''' Reports of Featured Pictures ranged from {{Highlight|0 to 17}} and were associated with several of the contests.
Importantly, that the majority of images rated Quality and Valued generating from only one or two contests, indicating that those program leaders are likely doing something that others are not in order to achieve these ratings.
<gallery widths="500px" heights="350px" perrow="3">
File:Quality_Rated_vs_Unrated_Pie.png| '''Graph 7: Total rated photos vs. unrated photos.''' <br></br>This pie graph compares how many photos were rated to how many were unrated out of all the photos that were uploaded during Wiki Loves Monuments events. The number of rated photos is about {{Highlight|0.08%}} of the total number of photos uploaded for all Wiki Loves Monuments in 2012.
File:Three_Ratings.png| '''Graph 8: Quality ratings broken down by category.'''<br></br> This pie graph compares only the photos that received a rating of Quality, Valued or Featured image for WLM 2012. Of all the photos that received a quality rating, {{Highlight|94%}} received a Quality rating, {{Highlight|3%}} were rated as Valued images, and 3% were rated as Featured images. Due to the method for which ratings are granted to images, each photo may be included in more than one quality category.
</gallery>
=== Recruitment and Retention of New Editors ===
As of reporting time, we were only able to evaluate recruitment and retention of active users for <s>2012</s> {{Highlight|2014?}}. A few things to remember with this section:
* An active retained user is considered one that makes 5 or more edits a month. <ref>https://www.mediawiki.org/wiki/Analytics/Metric_definitions#Active_editor</ref> We examined both the "pages created" (a new page gets created when someone uploads a file) and "edit count" using Wikimetrics to measure user activity on Wikimedia Commons.
* A "survived" editor is defined as one that made at least one edit in a given window of time after the event ended.
* User retention was measured for both ''survival'' and ''active'' user retention. Retention was assessed for the first three months after the event (Quarter 1) and the second set of three months (Quarter 2: months four, five, and six) following the event. Here we considered users as ''surviving'' if they had made a single contribution, and "active" if they had averaged <ref>by counting those who had made 15 or more edits for the three month quarter. However, users could make 15 or more edits during any period of time, not five or more ''each'' month necessarily. This solution allows us to even much of the month-to-month variation of users to get at those users who are more or less staying active on the project. For example someone who made five edits in the first month, three edits in the second month, and six edits in the third month following the event would be counted as active for the full three month quarter. At the same time, someone who made all their contributions in the first month following the contest, and nothing further, would also be counted as surviving or active for the quarter. </ref>5 or more edits a month, during the follow-up window.
* We couldn't isolate only Wiki Loves Monuments uploads so user contributions and retention numbers are based on any contributions made by participants during September 2012, and the three and six month retention dates.
* Three month follow-up represents the first quarter following the event, from October to December 2012.
* Six month follow-up represents the second quarter following the event, from January to March 2013.
==== New users ====
<div style="background-color:#FFB870;padding:0.5em">
;Six months after the end of Wiki Loves Monuments 2012, {{Highlight|0.4%}} of new users were retained as active contributors to Wikimedia Commons. </div>
Due to Wiki Loves Monuments 2013 ending recently, recruitment and retention data for only 2012 competitions is reported here. Our retention period of 3 months examines the number of uploads over the 3-month period from October to December 2012 while the 6-month retention period lasts through March 2013. The count of uploads per user is pooled; that is, the number of uploads from October through December of 2012 are counted together. Thus, in order to measure 5+ edits per month, we actually use a limit of 15 edits over three months for the 3- and 6-month follow-up point.
For these data, we used Wikimetrics to calculate the number of pages created in namespace 6 for files of Wikimedia Commons. The ''pages created'' option in Wikimetrics includes not only photos uploaded, but may also count videos and audio files added. We are unable to isolate just photos uploaded from these other possible media types, so user activity may include these as well. We also could not isolate photos uploaded for Wiki Loves Monuments, versus for other reasons, during the month of September so that the counts associated with the month September are not necessarily photos for the contest. For instance, in 2012, this made for the observation of {{Highlight|399,123 pages created}} by the full cohort of new and existing users while the actual number of photos uploaded specifically for Wiki Loves Monuments 2012 was {{Highlight|365,322, 8.4%}} of the pages created by the cohort of contributors were not photos for Wiki Loves Monuments. In addition to pages created, we also examined Wiki Loves Monuments participant behavior in terms of their edit count on Commons. Edit counts may include creation of a new file page, edits to category or other information on an existing file, or new versions of the image that may be uploaded.
{{Highlight|13,179 new users}}<ref>New users were defined as WLM 2012 uploader usernames which had not contributed to commons in the previous year.</ref> participated in Wiki Loves Monuments 2012. {{Highlight|5,399 (41%)}} of these uploaded five or more files and {{Highlight|5,552 (42%)}} made five or more edits in September 2012. In regard to retention (see ''Graph 9''):
* '''Three-month follow-up''' (First quarter: October–December 2012)
::Survival (Any contributions)
:::*{{Highlight|797 (6%)}} of new users from Wiki Loves Monuments 2012 had made at least one edit and 627 (4.8%) had uploaded at least one new file to Wikimedia Commons.
::Active (Average 5+ contributions a month)
:::*{{Highlight|152 (1.2%)}} of new users from Wiki Loves Monuments 2012 uploaded 15+ files during the first quarter of retention follow-up.
:::*{{Highlight|221 (1.7%)}} of new users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the first quarter of retention follow-up.
* '''Six-month follow-up''' (Second quarter: January–March 2013)
::Survival (Any contributions)
:::*{{Highlight|220 (1.7%)}} of new users from Wiki Loves Monuments 2012 had made at least one edit and 187 (1.4%) had uploaded at least one new file to Wikimedia Commons.
::Active (Average 5+ contributions a month)
:::*{{Highlight|47 (0.4%)}} of new users from Wiki Loves Monuments 2012 uploaded an average of 15+ files during the second quarter of retention follow-up.
:::*{{Highlight|59 (0.4%)}} of new users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the second quarter of retention follow-up.
Although new user activity declined over the three and six month follow-up periods, during the month of Wiki Love Monuments in 2013, there was an increase in activity for the new users from Wiki Loves Monuments 2012. In September 2013, {{Highlight|451}} of the previous new user cohort ({{Highlight|3.4%}}) contributed photos/media to Commons. Of those previously new users which were active that month, {{Highlight|306 (68%)}} added five or more files during the 2013 contest time.
<gallery widths="500px" heights="350px" perrow="3">
File:New user retention.png| '''Graph 9: New user retention, 2012.''' <br></br>This bar graph shows the number of new users retained for periods of three months at three and six months following the event, as well as during Wiki Loves Monuments 2013. A total of {{Highlight|13,179}} new users participated in Wiki Loves Monuments 2012. Interestingly, measuring activity by edit count captured significantly more active users in the initial months following the contest, however, the added value of the edit count metric diminishes over time for new users.
</gallery>
==== Existing users ====
<div style="background-color:#FFB870;padding:0.5em">
;Not surprisingly, existing editor retention is higher than new user retention for Wiki Loves Monuments, with {{Highlight|29%}} actively uploading files and {{Highlight|57%}} actively editing on Wikimedia Commons six months after the event ended. Half of the existing users who participated in Wiki Loves Monuments 2012 ''also'' participated in Wiki Loves Monuments 2013. The number of returning existing contributors seen for Wiki Loves Monuments 2013 was, for the most part, replenished by those Wiki Loves Monuments 2012 contributors who were retained as near the same number of existing users who participated in 2012 did not return for 2013's competition. </div>
{{Highlight|1,632 existing users}} participated in Wiki Loves Monuments <s>2012</s> 2013. {{Highlight|1,236 (76%)}} of these uploaded five or more files and {{Highlight|1,307 (80%)}} made five or more edits during the contest in September. In regards to retention (see ''Graph 10''):
*'''Three-month follow-up (First quarter: October–December 2012)'''
::Survival (Any contributions)
:::*{{Highlight|1,073 (66%)}} of existing users from Wiki Loves Monuments 2012 had made at least one edit and 975 (60%) had uploaded at least one new file to Wikimedia Commons.
::Active (Average 5+ contributions a month)
:::*{{Highlight|566 (35%)}} of existing users from Wiki Loves Monuments 2012 uploaded 15+ files during the first quarter of retention follow-up.
:::*{{Highlight|759 (47%)}} of existing users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the first quarter of retention follow-up.
*'''Six-month follow-up (Second quarter: January–March 2013'''
::Survival (Any contributions)
:::*{{Highlight|934 (57%)}} of existing users from Wiki Loves Monuments 2012 had made at least one edit and 845 (52%) had uploaded at least one new file to Wikimedia Commons.
::Active (Average 5+ contributions a month)
:::*{{Highlight|477 (29%)}} of existing users from Wiki Loves Monuments 2012 uploaded an average of 15+ files during the second quarter of retention follow-up.
:::*{{Highlight|643 (39%)}} of existing users from Wiki Loves Monuments 2012 succeeded at making 15+ edits during the second quarter of retention follow-up.
Half of the cohort of existing users (811 users) contributing to Wiki Loves Monuments 2012 also uploaded media during Wiki Loves Monuments 2013. Of those 652 users (80%) uploaded five or more new files to Wikimedia Commons during that time. While new editor retention appears low, compared to existing editor retention, the number of new users that returned for Wiki Loves Monuments 2013 (457 previously new users) helps to replenish the pool of returnees for 2013.
<gallery widths="500px" heights="350px" perrow="3">
File:Existing user retention.png| '''Graph 10: Existing user retention, 2012''' <br></br> This bar graph shows the number of existing users retained for periods of three and six months as well as during the following Wiki Loves Monuments. There were a total of 1,632 existing users of Wiki Loves Monuments in 2012. Measuring activity by edit count captured significantly more active users than did the "pages created" metric, capturing 4 to 12% more active users at each measurement point.
</gallery>
{{Cot|bg=white|title=References}}
<references/>
{{Cob}}
</div>
|}
__NOTOC__