Socio-spatial secondary data source for all over the world

In this post I want to echo this fantastic source of socio-spatial data, perfect for your GIS analysis in most of the countries all over the world

The files have been created from OpenStreetMap data
and are licensed under the Open Database 1.0 License. See for details about the project.

This file contains data as of 2020-10-12T20:42:02Z and every day
a new version of this file is being made available at:

A documentation of the layers is available here:

Geofabrik also makes extended shapefiles to order; please see for details and example

Socio-economic analysis with nighttime light series. Step 1: downloading shapefile of the study area

Context: we analyze economic changes in Colombian municipalities between 1993 and 2020.

We need to download ArcGis, a geographic information system (GIS) for working with maps and geographic information maintained by the Environmental Systems Research Institute (Esri). It is used for creating and using maps, compiling geographic data, analyzing mapped information, sharing and discovering geographic information, using maps and geographic information in a range of applications, and managing geographic information in a database. Particularly important for the purpose of this post is ArcMap, one of the applications, which is primarily used to view, edit, create, and analyze geospatial data. In the absence of license for this software, you can use free software such as GVSig.

Once installed either of the GIS software you use, it is time to upload the cartography of your study area. Normally, that cartography is in a geospatial vector data format called shapefile. VDS Technologies provides shapefile of administrative division across the globe. We download the administrative division of Colombia. The file includes 23 documents, including the dataset of data and the proper geospatial vector data (.shp). COL_adm0 corresponds to the boundaries of the whole country, COL_adm1 the department and COL_adm2 the municipalities, our basic analysis unit.

In ArcMap, we “add data”, look in our folder and select COL_adm2

The boundaries of the municipalities will show up in the visor Clicking in Windows, Table of Content, another window shows up in the left side with a list of the layers you can visualize.

Clicking right bottom above COL_adm2 layer, you can see the table properly, with all the values associated to each of the municipalities showed in the visor. This table of content often contains demographic and other data and we can do some basic analysis. For instance, visualize population density using a scale of colors where the intensity works as an indicator of the number of inhabitants in that particular region.

Now that we have the cartographic basis for our analysis, next post is about “downloading nighttime images”, that is the “raw data” we are considering to analyze economic performance in each of the Colombian municipalities between 1993 and 2020, something that, using more official statistics would be more difficult.

Socio-economic analysis with nighttime light series: downloading nighttime images

Context: we analyze economic changes in Colombian municipalities between 1993 and 2020.

In this post, we focus on how to download our “raw data”, i.e. the nighttime light data for Colombia.

Step 1: Download nighttime light series from NOAA National Centers for Environmental Information (NCEI). Particularly Version 4 DMSP-OLS Nighttime Lights Time Series (DMSP) – The DMSP annual composite data contain average radiance values of cloud-free coverages, reflecting the persistent lights from cities, villages, and roads, with a spatial resolution of about 900m, and a temporal coverage of 1992 to 2013 – and VIIRS data which is available from 2013 on and with is a finer spatial resolution of 450m approximately. We will later do a post on the particularities of VIIRS, here we focus on DMSP.

Once we are in NOAA website, we will see two different tables. We use “Average Visible, Stable Lights, & Cloud Free Coverages” table. Why? because clouds is a major constraint in night light analysis and this table contains cloud free images.

We click on every year and wait until the file is downloaded. We save all the files in a folder “Nightlight Analysis” and within that folder, we create one new sub-folder for each of the years.

Each of the year contains eight files compressed in a format .tar

Once we place each of the .tar files in their respective year folder, we extract the file and obtain the following files. The most important for our analysis is F162008.v4b_web.stable_lights.avg_vis.tif At the end of this exercise you should have as many avg_vis.tif as years you want to analyse. Then you will be ready to perform the next step: uploading nighttime light file to a GIS software, and convert them from raster to points before we build our dataset.

Socio-economic analysis with nighttime light series: uploading nighttime light file to a GIS software

Context: we analyze economic changes in Colombian municipalities between 1993 and 2020.

After having downloaded nighttime images from NOAA National Centers for Environmental Information (NCEI), and install ArcGIS or any other GIS software in our computer, we proceed now to upload nighttime files for our analysis.

NOAA provide images covering the whole world which can be difficult to manage in terms of size. For that reason, we will only upload the images corresponding to our study are, i.e. Colombia. To do that, make sure you have uploaded first the map of the study are, here Colombia (see previous post)

In ArcMap, we go to Windows—Search and one window with search engine will show up in the right side.

In order to extract only the images corresponding to Colombia, we search the tool “extract by mask”. There you will find further details on this tool. Extracting all the nighttime light files we should have stored previously in our folder is the first step to build our dataset of light intensity, i.e. our economic indicator.

Once we click on “extract by Mask” this window will appear asking for input raster, which is the night light file stored somewhere in your computer, in my case, in the folder I stored all the years “nightime light time series DMSP-OLS”. Then, input raster or feature mask data. This is the “mask” properly, and the way to say the software you only want one part of the whole image, in this case, the boundaries of Colombia, so we choose “COL_adm2”, the shapefile we previous open in ArcMap. Finally, in output raster, this is the file that will store all the data from the nighttime light series, the file that will contain our dataset for our analysis.

After clicking OK, we will be able to visualize Colombia image at night

We will extract as many files as years we want to analyse. In our case, we uploaded from 1993 to 2013, which as visible in the “table of content window”.

But before making that dataset look like a classic dataset we use in Spss or similar statistical softwares, we need to convert all the nighttime light files into a different format. These files are in format raster. In computer graphics and digital photography, a raster graphics or bitmap image is a dot matrix data structure that represents a generally rectangular grid of pixels, viewable via a bitmapped display, paper, or other display medium.

Importantly, NOAA assign a value from 0 to 63 to each of the pixels, where 63 means the maximum light intensity. However, in practice, we cannot see the value of each of the pixels we need for our analysis, so that we need to convert this raster into “Points”, meaning that every single pixel you see in the above image will be converted in a point with a given value, this time associated to a table of content.

We go again to window—-search and look for “raster to point”. We have two options, selecting “raster to point” and convert all the raster one by one into points or simply “extract value to points”, which will allow us to extract all the values from all the raster and then convert them into points.

Option 1

Option 2

Following the option 2, we select the file where we previously created “dataset” (in the image basedatosrecurse) and add all the rasters by clicking +. We click ok and wait until the program is able to perform the transformation.

The software will now show a new layer in the Table of content window “basedatosrecurse” (or dataset as labelled above) Clicking right bottom with the cursor on this layer, we “open attribute table” and see all the data with a classic dataset used in Spss. Every record correspond with one tiny point in the map (former pixel in raster format) and have assign a value from 0-63 for each of the years, where 63 indicates greater economic growth. You can see the dataset I created for Colombia here in my ResearchGate profile

We are almost ready to perform our advance economic analysis. Before, we need to link each of the points with the municipalities. See our next posts.