Output / Report
From Technical Implementation to Measurable Impact: Evaluating Smart City Projects with a Focus on Impact
The Urban Digitainability Lab at the TUM Think Tank has published a new research report on how German municipalities can systematically measure the impacts of digital and sustainability-oriented urban development projects.
The report “Drivers and Barriers to Impact-Oriented Evaluation Processes in Smart Cities” (in German) was written by Andreas Marx, Helene von Schwichow, and Nina Faecks. It presents the results of a collaborative research project with the cities of Freiburg im Breisgau, Gütersloh, Kempten im Allgäu, and Regensburg, which are participating in the federal funding program “Smart Cities Model Projects.” The research was funded by the Mercator Foundation.
Looking Beyond Successful Implementation
Municipal digitalization projects are often evaluated based on their immediate implementation: How many sensors were installed? Was a digital platform developed? How many people used the service?
Such indicators provide information on whether a measure has been implemented. However, they do not necessarily show whether this has led to actual social, environmental, organizational, or economic changes.
An impact-oriented evaluation therefore goes a step further. It examines, for example, whether digital projects improve administrative processes, promote social participation, use resources more efficiently, strengthen climate resilience, or enhance quality of life.
The research report therefore asks not only whether smart city projects have been successfully implemented, but also what changes they bring about among their target groups and within local government.
Four Cities and Eight Projects
As part of the nine-month research process, eight different municipal projects were examined. These included, among others, parking space sensors, soil moisture measurements, pedestrian traffic counts, digital youth work, digital services for older adults, barrier-free access to UNESCO World Heritage sites, and data-driven climate adaptation measures.
The Input-Output-Outcome-Impact model—or IOOI model for short—served as the conceptual framework. It distinguishes between four interrelated levels:
- the resources invested in a project and the activities carried out,
- the outputs and products resulting directly from these,
- the medium-term changes among the respective target groups,
- and the long-term societal impacts.
The study combined 27 expert-based, project-specific impact matrices, two rounds of data collection, and a transdisciplinary evaluation of the evaluation process itself. This made it possible to examine both the initial impacts of the projects and the practical experiences with impact evaluation in day-to-day administrative work.
Key Findings
A key finding is that impact-oriented evaluation has not yet become firmly established in municipal practice. Many stakeholders were familiar with traditional performance metrics and output-oriented reporting, but had little experience defining and measuring outcomes and long-term impacts.
In many of the projects examined, technical implementation was already well advanced. Digital infrastructures, platforms, workshops, and data systems had often been successfully established. Evidence of medium- and long-term effects, however, varied greatly depending on the project’s stage of development.
Measuring long-term societal impacts at the impact level proved particularly difficult. Such changes often become visible only after several years and are additionally influenced by political, demographic, institutional, and environmental factors. Within the study period, only one impact indicator could be assessed as having been met.
This highlights both the significant amount of time required and the difficulty of clearly attributing long-term societal changes to a single municipal project.
Structural Barriers in Municipal Practice
The study identifies several recurring barriers to successful impact evaluation. These include:
- limited personnel and time resources,
- a lack of knowledge about impact measurement,
- fragmented or unavailable data,
- unclear responsibilities,
- difficulties in collaboration between different departments,
- a lack of internal budgets,
- low political prioritization of evaluation.
The Data collection represented the greatest expenditure of time and personnel throughout the entire process. Relevant data was often not centrally available, was not collected regularly, or was tied to specific individuals.
An indicator assessed as “not met” therefore does not necessarily mean that a project had no impact. It may also indicate that the necessary data was not yet available, that the data collection had not yet been carried out, or that the corresponding impact may only become apparent at a later date.
What Supports a Successful Impact Evaluation
Despite the existing challenges, the report identifies several key success factors.
The IOOI model provided stakeholders with a clear framework and a common language for discussing project impacts. External scientific support was also considered particularly valuable—and in some cases indispensable—especially when impact-oriented approaches were being applied for the first time.
Other important factors include:
- a clearly designated point person for the evaluation process,
- the early involvement of relevant departments,
- improved communication between different administrative units,
- realistic and clearly defined indicators,
- the actual use of evaluation results for project management and strategic decisions.
The report emphasizes that impact evaluation should begin as early as the project planning phase. Impact goals, indicators, data sources, and responsibilities should not be determined only after implementation.
Long-term impact goals can serve as strategic guidance even if they are not yet fully measurable within the original project timeframe.
Recommendations for Local Governments, Policymakers, and Funding Agencies
Local governments should establish impact evaluation as an integral part of project management and allocate dedicated time, personnel, and technical resources for this purpose. Clear lines of responsibility, interdisciplinary collaboration, and partnerships with academic institutions can further enhance the quality and credibility of the evaluation.
Funding agencies and policymakers should systematically integrate impact evaluation into funding programs. At the same time, earmarked funding, practical guidelines, advisory services, and standardized templates must be made available.
Evaluation should not be viewed solely as an additional reporting and accountability requirement. Rather, it can help improve public decision-making, allocate resources more effectively, and increase the transparency and legitimacy of municipal actions.
For research and academia, the report underscores the need to further develop practical, project-specific indicators, expand methods for measuring long-term impacts, and establish lasting partnerships between universities and local governments.
Impact evaluation is thus not merely a tool for measuring project outcomes. It can also trigger organizational learning processes, improve collaboration between departments, and help municipalities define their goals more clearly.
By asking not only whether a digital solution has been implemented, but also what actual changes it brings about, cities can better understand how their projects contribute to sustainable and socially equitable urban development.
