United Nations - The Idea Bureau

Introduction

Sparrow is a one-of-a-kind internal social media reporting platform built for the United Nations’ Department of Political and Peacebuilding Affairs (DPPA) to assist in providing up to date content analysis on global issues of conflict and humanitarian importance.

The United Nations Department of Political and Peacebuilding Affairs (DPPA), situated in New York City, is responsible for monitoring and assessing global political developments, as well as advising and assisting the UN Secretary General and his envoys in the peaceful prevention and resolution of conflict around the world.

We began working with the UN Department of Political and Peacebuilding Affairs (DPPA) team in 2020, having previously collaborated with one of their design consultants on another project of ours. The team approached us to create a platform that would assist in building bespoke reports using data from social media on various topics of interest that could then be shared; these include subjects such as war, conflicts and humanitarian issues taking place around the world, e.g. the conflict in Ukraine.

The Problem

The project of creating the platform was a very complicated one. They required an internal social media reporting tool that allows the users (UN officers) to take Twitter lists and compile data in a way that’s easily readable, customisable, and ultimately could be sent to any UN member across the globe in a specific format.

Previously, the DDPA team were manually creating reports in Microsoft Word, adding screenshots of content to include as graphics and analysing data gathered by hand. This was incredibly time consuming and took a lot of work to produce. These reports are important because they demonstrate details like public perception and feeling via the keywords and hashtags they use online.

“Social media remains a key channel for enabling collective political expression for youth and is also being used more prominently for diplomatic signalling in international affairs”

Martin Waehlisch

Senior Political Affairs Officer at the United Nations

Our Solution

We created the Sparrow platform to give the UN DPPA team the ability to report on a number of social media content types, including:

‘Top Tweets’ (posts form X, formerly known as Twitter)
Influencers ranked by likes and retweets, or ‘number of followers’
Connectivity analysis (a graph to show if users follow each other)
Top hashtags
Top keywords
Hashtags and keywords used over time (for the duration of the reporting timeline)
Hashtags and keywords comparison across lists (comparing the use of hashtags between lists if the report has two or more)

Sparrow gives users the ability to create reports out of this information in either a simple or more advanced format. Both of these formats are customisable, as are many of the features within the report – for example: it provides users the ability to add titles, and start and end dates to reports; drop in new pages; and select preset modules to demonstrate complex data visualisations and graphics. These graphics can then be saved out as individual files for users to populate other documents, as required.

Client:: United Nations
Project Type:: Bespoke Platform
Tags:: Charity, Social Media, Reporting

“Sparrow directs you towards the people with the most followers, to the tweets that are getting most noticed. It complements our own manual monitoring and reporting.”

Pascale El Kassis

Head of Public Information UNSCOL

How the platform works

Before the Sparrow platform existed, the team had a very basic prototype put together by a Designer in Adobe InDesign; other than this, they were only able to create reports manually. We built the platform from scratch, and have been iterating on its design with new features over the course of the last 2 years. It’s currently on version 2 – in the first version, it wasn’t possible to view the output until the report itself was complete. Since then, we’ve implemented updates users can see in real-time as they build their reports.

Sparrow uses Laravel – a PHP framework that provides pre-built common tools, such as login/registration etc, leaving us to concentrate on specific requests from the team – as its’ application platform, with Laravel Nova providing admin panel functionality and features to make the experience more user friendly.

By implementing a ‘headless’ version of Google Chrome (i.e. Chrome running on a server), we were able to convert the reports from a HTML format into a PDF. The DPPA works with people globally, so we had to ensure the reports could also cater to international audiences: to do this, we implemented the Google Translate API, which converts Tweets into users’ preferred languages.

As a development consultant, we’re responsible for commissioning, configuring and maintaining the server infrastructure for Sparrow and the wider UN DDPA team. To fulfil the requirement, we manage infrastructure using Amazon Web Services (AWS): this allows us to scale Sparrow’s resources up and down as demand fluctuates, as well as helping to load balance traffic to maintain good performance during high demand.

Working together

The Sparrow project was especially exciting, as working with the DPPA meant we had to travel to New York to collaborate with the team directly in the second phase. We went to the UN headquarters in midtown Manhattan to participate in workshops with UN desk officers, an activity that assisted in devising wireframe concepts for the platform to discover what could be achievable within the time and budget.

We undertook user testing for the reports with a number of Sparrow users globally – this was all done remotely, so users could access our pre-written tasks online. This was especially useful to us, as it provided an insight into how a variety of users operated the platform; it also gave opportunities for those involved in creating the reports to request functionalities directly. We’re very careful to not only keep these testing sessions consistent, but to reassure participants that it’s the product being tested and not them as individuals.

Our user testing approach uses Lookback, an app that enables us to record participants’ screens, audio and video (all with their permission); we request that they describe what they’re doing aloud and also take written notes as they perform the tasks. Following the sessions, we then review this feedback and determine any improvements we can make, both in this phase and in future work, to make the platform more user friendly, and ultimately, improve user experiences.

“Sparrow allows me to identify relevant keywords and prominent social media stakeholders in countries under my purview. It supports targeted research and contributes to separating valuable information from information noise.”

Jean Charar

Political Affairs Officer at UN DPPA

Overcoming complexities

One of the main challenges of the project for us was that the reports contain varying amounts of data – some of which is very graphic heavy – that needed to be represented within an A4 PDF format. This was extremely important, as the reports require the ability to be easily downloaded and shared within the wider team, while maintaining brand consistency and a professional look. They’re also used by UN officers from around the world; as such, the reports may need to be printed using very old equipment or accessed via unreliable internet connections.

Because the content varies so much in length, can contain multiple images, or require translation, we’ve set a maximum number of tweets (posts from X, formerly known as Twitter), hashtags, keywords and influencers that can be included within any given report. The reports can also be downloaded as plain text, if needed due to lack of modern equipment.

6.5m

posts have been fetched and processed from X
3000+

reports have been created since launch
3.4m

hashtags have been processed on Sparrow

Data retrieval and visualisation

The data is best represented visually, particularly via the use of graphs, to show the results in an engaging way. To begin with, we created a proof of concept using Storybook – a front-end tool that allowed us to mock up prototypes of the report graphs – in order to demonstrate how these visualisations would work to the DPPA team. Once they were on board, we then built them out in the development phase. This required much testing before we even started designing what they would look like to ensure they’d work as expected.

This project gave us an opportunity to use the D3 data visualisation library for the first time. It’s a JavaScript library that produces dynamic, interactive data visualisations within the browser, allowing us to output highly visual and interactive charts that respond to report changes in real-time. Although it was a big learning curve for us, it ultimately enabled us to deliver much higher quality reports.

The visualisation charts can show things like the relationships between Twitter users and give them ‘Community Scores.’ In order to maintain readability for anyone using older equipment, and to improve accessibility, we used shapes and line variations to distinguish data on the graphs.

Capturing data in real-time

The Sparrow platform is required to fetch and analyse large amounts of data in order to run reports. As such, it’s important this data retrieval is performed in a timely manner, so it doesn’t have a negative effect on the user experience. We realised how much of a challenge it would be during the discovery phase of the project, and to overcome it, we developed a number of concepts to determine what would be possible in reducing user waiting times.

By starting the data retrieval process early on in the creation of the reports, we were able to achieve the goal of reducing time impacts on users. This means the capture of data can take place in the background while users are still defining report details.

We also run the data capture in batches – and in parallel – further reducing the time needed for it to complete.

Caching and re-using data collected for previous reports enabled us to reduce the amount of data that’s fetched and analysed, as it can be stored with complex logic. The result of this is the Sparrow platform can process up to tens of thousands of Tweets in a very short space of time, sometimes generating a document within 30 seconds of users completing the report creation process.

Conclusion

A big win for this project was being able to create something bespoke for our clients that doesn’t exist anywhere else. The complex technical processes we devised and implemented allow users to create reports easily and in a way that provides their coworkers great value, wherever they’re situated.

Sparrow’s functionality makes it easier for users to consume data, allows for customisation and analyses the data collected automatically.

It required a number of phases for us to learn and improve the system incrementally; however, we’re extremely proud of the end result. We’re incredibly pleased to continue working with the UN DPPA team, providing them with ongoing platform and infrastructure support.