Benito Zaragozí

Group zipping GIS Files made easy

2023-04-03T00:00:00+00:00

If you work with Geographic Information System (GIS) data, you’re likely familiar with multi-file formats such as ESRI Shapefiles or SAGA GIS raster files. These file formats consist of several individual files with different extensions that need to be kept together to maintain their spatial information. If you’re sharing or storing these files, it can be cumbersome to handle each file-group individually, especially if the files need to be compressed. This is where a bash script called Group Zipping comes in handy.

In this post, I propose a Bash script that helps compress GIS files that come in multiple files. Here, I explain how the script works and highlight some of its features, such as parallelization, software and folder checks, filters, and a variety of available compression formats.

Understanding multi-file GIS formats

There are many well-known multi-file formats outside of the GIS domain. Here are some examples:

Microsoft Office documents (.docx, .xlsx, .pptx) are actually ZIP archives that contain multiple files, including XML files and media assets.
Java JAR files are ZIP archives that contain compiled Java classes and resources.
Adobe PDF files can contain multiple components, such as fonts, images, and metadata.
EPUB files, used for e-books, are also ZIP archives that contain multiple HTML, CSS, and media files.
Git repositories consist of multiple files that represent the codebase, including code files, configuration files, and metadata.

These are just a few examples, but there are many other multi-file formats in various domains, such as multimedia, software development, and scientific research.

As you can see, multi-file formats are a really common aproach for structuring (text based) data but they are usually combined into single containter, mostly for practical reasons. However, in other ocasions the formats are used without such combination.

Before we dive into the script, let’s take a moment to review some examples of GIS formats that come in multiple separate files.

ESRI Shapefiles

ESRI Shapefiles (or just Shapefiles or shapes) are a popular GIS format that store vector data, such as points, lines, and polygons. As you probably know, a complete shapefile consists of at least three files:

.shp: the file that contains the geometry of the features (points, lines, polygons) in the shapefile.
.dbf: the file that contains the attribute data (e.g. names, population, etc.) for the features in the shapefile.
.shx: the file that contains the index of the features in the .shp file.

Optionally, a shapefile may also have other files associated:

.prj: the file that contains the projection information for the shapefile
.sbn and .sbx: files used for spatial indexing

Maybe, shapefiles are being replaced by other more modern formats but they are still widely used in many contexts. The structure of a shapefile is stil one of the first explanations in many “Intro to GIS” courses.

SAGA Raster Files

SAGA raster files are another common GIS format that store raster data, such as elevation, temperature, rainfall, etc… A SAGA raster file consists of at least two files:

.sdat: the file that contains the actual data
.sgrd: the file that contains metadata about the data, such as the extent, resolution, and projection information

Optionally, a SAGA raster file may also have:

.sdat.aux.xml: an auxiliary XML file that can store additional metadata or coordinate system information
.sdat.prj: a file containing the projection information for the raster dataset
.sdat.cpg: a file that specifies the encoding of the raster dataset
.sdat.vat.dbf: a dBase file containing a value attribute table (VAT) for the raster dataset

Other GIS formats, such as GeoTIFF, NetCDF, and or KML/KMZ, also can store data in multiple files, maybe containing an external projection file or multimedia associated files that are referenced in the attribute table. Regardless of the format, it’s important to properly organize and compress the files to avoid losing data or metadata, and to make it easier to share or store the data.

Introducing the Bash Script

To help manage and compress multi-file GIS formats, I propose a Bash script that compresses files sharing a common prefix before the first dot in their filenames. Sometimes this pattern (before the first dot) is not the same as a 3-4 characters extension (e.g. in the case of .sdat.aux.xml, which is a composed extension). The script can work with many other formats but I have tested it with ESRI Shapefiles and SAGA rasters. Of course, it may also be useful in non GIS applications.

The main features of this script include:

In terms of compression formats, the script supports several commonly used formats, including gzip, bzip2, xz, zip, lzma, lzip, lrzip, and 7zip. By default, the script uses gzip, but you can choose a different format using the -f or --format option.
The script uses GNU Parallel to compress files in parallel, which can significantly reduce the compression time for large datasets. By default, the script uses only 1 of the CPU cores available in the system as the maximum number of parallel jobs. However, you can specify a lower or higher value using the -j or --jobs option. It’s worth noting that running too many parallel jobs can overload the system and lead to slower processing times, so it’s recommended to use a conservative number of jobs, especially if you’re compressing large files or working on a system with limited resources.
The script also performs a series of checks to ensure that the required software and folders are available before starting the compression process. If any issues are detected, the script will print an error message and exit with a non-zero status code.
Additionally, the script allows you to filter the input files by specifying a pattern using the -p or --pattern option. The pattern can be any string that matches the common prefix of the files you want to compress.

The script provides some flexibility but to further optimize the compression process, here are some additional tips:

Read and write to different disks: If possible, read the files from one disk and write the compressed files to another. This can help reduce disk contention and speed up the overall process.
Limit the number of parallel jobs: Depending on the available resources, it may be more efficient to limit the number of parallel jobs to a smaller number. Experiment with different values to find the optimal number of jobs for your system.

The script could be adapted for different requirements and situations, but now it assumes that compressing is the main task of the computer when it is executed.

Usage

The script supports the following parameters:

-h, --help: Display the help message and exit.
-i, --input-folder: The folder containing the files to compress (default: current directory).
-o, --output-folder: The folder where compressed files will be saved (default: input folder).
-f, --format: The compression format to use (supported: gzip, bzip2, xz; default: gzip).
-p, --pattern: The pattern to filter file names in the input folder (e.g., ‘pattern’; default: all files).
-j, --jobs: The number of parallel jobs to run (default: number of CPU cores).

Maybe a couple of examples will clarify what the script does.

Example 1: Compressing an ESRI Shapefile

Let’s say you have the following files in a folder:

some-streets-file.dbf
some-streets-file.prj
some-streets-file.shp
some-streets-file.shx

You can compress them using the following command:

./group-zipping.sh -i /path/to/files -o /path/to/output -f gzip -p 'streets'

This will create a file called some-streets-file.tar.gz in the output folder. The same with other shapefiles stored in the input folder.

Example 2: Compressing a SAGA raster file group

Let’s say you have the following SAGA raster file in a folder:

some-elevation-file.sdat
some-elevation-file.sgrd
some-elevation-file.sprj
some-elevation-file.mgrd
some-elevation-file.sdat.aux.xml

You can compress the file-group using the following command:

./group-zipping.sh -i /path/to/files -o /path/to/output -f gzip -p 'elevation'

This will create a file called some-elevation-file.tar.gz in the output folder. Again, this will do the same for the other SAGA raster files present in the input folder.

Some last thoughts

The group-zipping.sh script can be a useful tool for compressing such groups of files. It supports several compression formats and includes several parameters that can be customized to fit your specific needs. With its support for parallelization, software and folder checks, the pattern filter… this script can help you to save time and disk space when working with large groups of GIS files.

In my case, I had 36 medium sized (3GB) SAGA raster files to compress. Nowadays, I’m working on Ubuntu 22.04 and I have all the dependencies installed (tar, gzip, GNU Parallel, etc). I achieved reducing the file sizes to 1/3 while keeping all associated files toghether (this means storing 30 GB instead of 100 GB). Using a -j 8 parameter this was done in about 20 minutes in my personal computer.

It took some time to write the script but I feel it won’t be the last time I will be compressing my results like this.

I think there is still room for improvements. Maybe I will try to:

Implement other compression formats.
Develop a progress bar and more feedback for the user (but not affecting much the main tasks).
Create unit tests to ensure there are no undesired effects.

The Script

Read the code and try the script taking some precautions (e.g. backup copies, etc). I have not tested all the compression formats (zip and gzip work for me) but I think that they work in a similar way, so it should work correctly. Here I share the README file and the Bash script. They are stored as a GitHub gist.

DNI electrónico y autofirma en 2020: cómo NO perder 2 horas para firmar una solicitud

2021-07-02T00:00:00+00:00

Estas últimas semanas, junto con otros compañeros y compañeras, hemos empezado a prepararnos para obtener la formación necesaria para pilotar drones para nuestros proyectos. El primer paso es realizar del CURSO DE FORMACIÓN Y EXAMEN DE PILOTO A DISTANCIA EN CATEGORÍA ABIERTA, SUBCATEGORÍAS A1/A3. Para saber más podéis revisar la información sobre la Formación de pilotos UAS/drones en categoría ‘abierta’. Hay mucha gente hablando de estas cosas y no hace falta que me ponga pesado. Recomiendo el canal de YouTube de Rafa Ocón para oir hablar a un profesional sobre la actualidad del mundo de los drones.

Lo primero es registrarse en la página web de la AESA para realizar una Nueva solicitud. Una vez firmada y enviada la solicitud se tarda un tiempo en recibir acceso al la plataforma Moodle de la AESA en la que se realiza este primer curso (A1/A3) y el examen para obtener el certificado. El trámite resulta sencillo a no ser que no tengas un certificado electrónico actualizado y todo el software necesario para firmar electrónicamente. En fin, esto no será problemático para la mayoría pero desde Linux todo cuesta un poquito más :)

Bueno, también se dice que se puede enviar por correo postal y esperar la respuesta…

Firmar la solicitud

El DNIe va cambiando de versiones y resulta que yo tenía uno (de 3 años de antiguedad) con el que ya no se me permitia actualizar los certificados electrónicos. Una vez con el DNIe actualizado y teniendo un lector de tarjetas inteligentes compatible con Linux. El que yo tengo ya no está disponible pero supongo que habrá nuevos modelos (Lector de tarjetas descatalogado).

Lo primero sería leerse las instrucciones e instalar el software que facilitan en el Portal del DNI electrónico del Cuerpo Nacional de Policía (Cómo utilizar el DNI»Qué hace falta para utilizarlo). En Ubuntu 20.04 hice lo siguiente:

sudo apt install pcscd pcsc-tools pinentry-gtk2 libccid

#https://www.dnielectronico.es/PortalDNIe/PRF1_Cons02.action?pag=REF_3000
wget https://www.dnielectronico.es/descargas/distribuciones_linux/Ubuntu_libpkcs11-dnie_1.5.3_amd64.deb
sudo dpkg -i libpkcs11-dnie_x.x.x_i386.deb

Autofirma

A continuación hace falta software para firmar PDFs u otros documentos. Nuevamente, esto és más sencillo en Windows con el Adobe Acrobat PDF Pro. En Ubuntu se puede instalar Autofirma.

Lo primero és que hay que instalar la versión de Java necesaria para que funcione, unas librerias y después ya se puede instalar AutoFirma:

sudo apt install openjdk-8-jre
sudo apt install libnss3-tools
mkdir temporal_autofirma
cd temporal_autofirma
# Ir a https://firmaelectronica.gob.es/Home/Descargas.html a comprobar si `1.6.5` es la última versión
wget https://estaticos.redsara.es/comunes/autofirma/1/6/5/AutoFirma_Linux.zip
unzip AutoFirma_Linux.zip
sudo dpkg -i *.deb
cd ..
rm -rf temporal_autofirma

Autofirma se puede utilizar desde el escritorio para firmar un PDF normal o también se puede utilizar desde el navegador, lo que “permite la firma en páginas de Administración Electrónica cuando se requiere la firma en un procedimiento administrativo”. No obstante, al intentar firmar la petición a la AESA y cerrar mi solicitud, le daba a utilizar AutoFirma pero después no sucedía nada :-S

Una hora sin poder avanzar, desinstalar todo y volver a empezar, pero AutoFirma no puede ser desinstalado porque estaba en uso… :-? Al repasar este post de UbuntuLeon.com, me di cuenta de que no le había prestado atención a la versión de Java que estaba utilizando. Hacía falta utilizar Java8!!!!

Así que se puede configurar haciendo

sudo update-alternatives --config java

y seleccionar la versión 8 de java, si és que se había instalado en los pasos previos.

Al final todo funciona correctamente y pude realizar mi solicitud. Desde entonces he utilizado más amenudo la firma con el DNIe cuando estoy en casa. No obstante, me sigue sorprendiendo que una aplicación de la que tanto dependemos no esté más actualizada con las últimas dependéncias y/o haya un cierto versionado para las configuraciones habituales.

Una reflexión sobre todo esto

Estamos ya metidos en pleno siglo veintiuno pero parece que hay cosas que no cambian tan rápido como esperabamos hace unos años. Los trámites con la administración són cada vez más sencillos pero hace falta una reflexión. ¿Cuanto tiempo falta hasta que la administración apueste de verdad por el software libre y abierto?—y nos liberemos de tener que pagar el doble (primero nuestros impuestos y después el software para poder realizar nuestros trámites). Sí, aunque no lo pensemos habitualmente y muchas veces los tengamos ya instalados, Windows y los programas de Adobe son de pago.

Desde que me decidí a utilizar principalmente software libre, puedo clasificar mis dias entre:

Aquellos en los que me siento orgulloso y feliz por ser más independiente, cuando todo resulta más fácil (80% del tiempo).
Los dias en que sufro para hacer algo pero aprendo bastante por el camino (18% del tiempo).
Otros en que lamento el exceso de tiempo que pierdo haciendo cosas que son más sencillas utilizando software propietario (2% del tiempo).

Hoy sería un dia del tercer tipo pero, afortunadamente, mi fe en el software libre es fuerte.

Referencias

How to cite geospatial standards and other GIS things

2020-06-10T00:00:00+00:00

Nowadays, there are tons of tools for managing and formating your references, some of the most useful options are to use Google Scholar for reference management and Mendeley (desktop+web) for document management (mostly for PDFs). I use the GS for getting Bibtex or APA7 formated references when I don’t need to reread the document or Mendeley when it’s a document I would like to keep.

These tools were practical enough since this year I started using Zotero and Cablibre for managing my collections of papers and digital books. I will share a few toughts of this in a future post.

Right now, I’m mostly interested in having a complete Bibtex database for my references and this can be done by using any of the previous solutions (FOSS or not). However, there are certain specific ocasions where you can’t find an already formated reference in GS or there is no DOI code that can be used by Mendeley or Zotero to compile the Bibtex entry. In those ocasions you need to do it the old way…

Can’t found a Bibtex entry in the previous options?

This was the case when I was trying to cite ISO and OGC standards for a paper (these don’t have DOI or ISBN). As usual in life, many people share the same problem you have. Ismael Olea shares a solution in this post. This is how the SQL standard looks like following his suggestions:

@techreport{iso_sql_2016,
  address = {Geneva, CH},
  type = {Standard},
  title = {Information technology -- {Database} languages -- SQL -- {Part} 1: {Framework}},
  shorttitle = ISO/IEC 9075-1:2016,
  url = {https://www.iso.org/standard/63555.html},
  language = {en},
  number = ISO/IEC TR 9075-1:2016,
  institution = {International Organization for Standardization},
  author = {ISO Central Secretary},
  year = {2016}
}

I think it would be useful to keep a repository of those documents and items that are not usually cited. I appreciate VERY MUCH the way GS, Mendeley, Calibre and Zotero help us with these arduous tasks.

A small tip: Did you know that Calibre helps you in finding metadata for PDF books? It even downloads the covers from Amazon!!! You should try it ;)

Choosing a desktop marker app for Ubuntu 18.04 LTS

2020-05-15T00:00:00+00:00

Introduction

In the last few months many professionals have stopped leaving home to go to work and, among these, there are a good number of people who, like me, are trying to find the best way to work from home.

In my office I have to work with different operating systems but at home there is only Ubuntu. This is why in the last few weeks I have had to go back to review a good number of apps that I haven’t used on a daily basis for a long time.

In this post I am going to talk about those applications that we use when we are making a presentation with the computer and we want to point, write or draw something in a spontaneous way. This is about the desktop annotation tools.

My needs

First or second thing I made two months ago, when the coronavirus changed our lives, was to setup my working environment. I’m still on it but I feel that I’m very close to solve all the minor problems I experienced in the begining. I had to upgrade my personal computers from Ubuntu 16.04 to Ubuntu 18.04. Many new programs have dependencies that make them very awful to install.

In a streaming web conference, I saw one presentation where the researcher was using epicpen for pointing out things on his slides and figures. That was much more better than using the mouse pointer. Epicpen is still in development but it’s pretty cool and has many features. The main problem with this app right now is that it’s not designed to be multiplatform and it’s not FOSS. I’m not a designer, I just want to sketch a couple of messy lines and arrows, so I’m not even considering a comercial software.

Actually, I have a wish list of the desirable characteristics these programs should have. This apps should be capable of:

Drawing points, lines and polygons of at least 4 or 5 colours.
Writing text using any of both options, using the keyboard or writing with a digitizing tablet.
Draw on top of my documents and figures without being too anoying for the people watching (e.g. too many buttons and controls arround).
Speeding up my presentation (e.g. shortcuts for switching pens, colours, etc)
Working in a two monitor configuration (e.g. when recording video I don’t want the app occuping my space).
When preparing software tutorials, it would be awesome to draw over expanded app menus, but I think I’m asking for too much :)

Ardesia

A few years ago, I used to use Ardesia for sketching over the screen. A GPL licensed software that was able to do almost all I needed. However, this project is no longer maintained by its creator. It was working with difficulties in Ubuntu 16.04 but it was still distributed from official packages. It’s a pitty because it was very useful.

In case you want to try it (under your own responsibility), Ardesia’s code is available from different repositories and it still works in Ubuntu 18.04. I have installed it following the project README with a few small changes:

# Install Ardesia on Ubuntu 18.04
# Use code from non trusted repositories at your on risk!
# libpng12-dev is no longer available, use libpng-dev
sudo apt-get install gcc make automake autoconf intltool libtool libtool-bin libxml2-dev libgsf-1-dev libgtk-3-dev libatk1.0-dev libx11-dev libpng-dev libglib2.0-dev libgconf2-dev libfontconfig1-dev libfreetype6-dev libgsl0-dev libc6-dev xdg-utils librsvg2-dev

# Download sources from github. I just made a fork.
git clone https://github.com/benizar/ardesia.git
cd ardesia/

# Compile as a *.deb package
sudo apt-get install devscripts
./autogen.sh
make deb
sudo dpkg -i ardesia_1.2-1_amd64.deb

Now, run it with parameters so you can control its behaviour

ardesia --help
ardesia -g south -d -t 50

Pylote

There is a nice Python based tool called Pylote, just download and double click on the launcher (or python3 pylote.pyw). Pylote is actively mantained and has lots of options.

Pylote is the perfect tool if I wanted to create detailed presentations but I think it lacks the interactivity I want. Once you start Pylote you will no longer be able to modify anything on the desktop. Also, during a videoconference, when you need to explain something, you can’t spend time recreating the screan. I would need a program more independent from what’s happening in the screen and I was unable to find any keyboard shortcuts that could facilitate the work.

Gromit-MPX

This is a pretty nice app and it’s easy to install from APT. The thing I most liked about Gromit is its simplicity. Once Gromit is started it can be used through keyboard shortcuts:

F9: toggle painting
SHIFT-F9: clear screen
CTRL-F9: toggle visibility
ALT-F9: quit Gromit-MPX

An it can be configured to switch colours and shapes using different mouse+keyboard combinations. This is the configuration I prefer:

As you can see, I only need 4 colours (3 mouse buttons+left click while pressing ALT), [Shift] switches from pen to highlighter, control is for drawing arrows… I think that you got the idea.

I proved myself that the productivity when creating a slideshow of desktop captures is much higher than using other methods. However, I only liked to use the yellow highlighter for everything :)

Introducció a Git de Benito Zaragozí

Conclusions

There are many software programs for highlighting things on the screen. The three I’ve tested had nice options and they are really helpful in my tasks but, for simplicity, I can only use one of them. Let’s see:

Ardesia feels too old and is no longer mantained. It’s a pitty!!
Pylote has a lot of tools and configuration is not easy (for what I need). The tool I need should be used through shortcuts and in combination with screen captures.
Gromit covers all my requirements and it’s very simple. All the configuration can be automated.

I think I will stay with Gromit for a while. Only, if I had to ask for an additional feature, I just would like to be able to draw over unfolded and context menus (e.g. right button). When that’s the case you must draw over the screen capture and do it the old way.

Using OBS Studio with V4L2 for making my videoconferences more appealing

2020-05-08T00:00:00+00:00

Introduction

Nowadays, I’m doing more videoconferences, presentations and recording classes for my students using OBS (+ other software). When I find some time, I’m using a chroma background for my videos, so I considered to use the chroma for videoconferences and other online meetings.

There are many apps for doing this dinamically through the use of virtual webcams. In windows, we even could use Snapcam for adding special effects and in Ubuntu we could use Webcamoid for this. However, I wanted to do this using OBS because I would like to have more control over the scene, for example, I could show parts of my desktop or some slides without the need of sharing the screen.

My working environment includes and old laptop with Ubuntu 16.04 LTS, a PC with 18.04 LTS, and I use the following videoconferencing platforms.

For particular use:
- Skype for Linux (ver. 8.59.0.77).
- Hangout, working from firefox and chrome (I can’t share screen from firefox, but it’s my favourite web browser).
- I recently met Jitsi Meet which works quite well too.
And for a more professional use:
- Teams for Linux, with a paid account from my University. Webcam was not working with the desktop app, so now I’m working from Chrome web browser.
- Zoom client for Linux version 3.5 (specifically 3.5.361976.0301), with a paid account from a conference I’m attending.

In this post I’m explaining how I’m working with OBS-studio as a virtual webcam. This way I’m able to use the same configurations, scenes and shortcuts I have created for my videotutorials.

Setting up the computer

Installing all the necessary packages for using this approach is not as straightforward as I would like but there is a lot of information arround to make it done.

This excelent post from srcco.de explains how to use this approach for using it with Google Meet, while Justin Warren makes a more detailed post explaining How To Use OBS Studio With Zoom.

After reading this and other references that I attach at the end of this post, I installed everything using the following commands:

# APT install basics
sudo apt-get install obs-studio
sudo apt install v4l2loopback-dkms
sudo apt install qtbase5-dev

sudo apt-get install libobs-dev      # In Ubuntu Bionic Beaver (18.04)
#sudo apt-get install liboobs-1-dev   # In Ubuntu Xenial (16.04)

# Build plugin
git clone --recursive https://github.com/obsproject/obs-studio.git
git clone https://github.com/CatxFish/obs-v4l2sink.git
cd obs-v4l2sink
mkdir build && cd build
cmake -DLIBOBS_INCLUDE_DIR="../../obs-studio/libobs" -DCMAKE_INSTALL_PREFIX=/usr ..
make -j4
sudo make install

I’m not sure if we still need the sources for libobs after installing the repository from APT. Another problem is that the plugins are not installed in the correct directory. Probably, we could provide a parameter to the make install command but I ended up doing as most of people do things:

cp /usr/lib/obs-plugins/v4l2sink.so /usr/lib/x86_64-linux-gnu/obs-plugins/

If you are installing obs-v4l2sink for the first time, maybe you can answer my doubts and remove unnecessary installation steps.

Starting a session with a virtual camera

# Removing previous devices
sudo modprobe -r v4l2loopback

# Creating a new device with #10 id
sudo modprobe v4l2loopback video_nr=10 card_label="OBS Video Source" exclusive_caps=1

Now, I’m ready to start OBS and connect the webcam. Sometimes it will be necessary to check the obs-v4l2sink plugin. Go to Tools > V42L Video Output

I let the autostart off because… This is necessary when changing videoconferencing platform, because each platform may accept different video input formats. For example, I have tested these in my computer:

YUV420: Hangout, Zoom

YUV2: Skype, Yitsi

Are there any problems?

I found the answer here or at least I can start the process from zero.

It can be done by removing the v4l2loopback module:

sudo modprobe -r v4l2loopback

and then we can check that all the virtual devices are gone

ls -l /dev/video*

Final thoughts

This is an old draft I prepared during the first lockdown of the COVID pandemics (almost 1,5 years old). Problably there are better solutions for this now, because configuring OBS and virtual cameras before every videoconference was quite painful and I ended up showing my very tidy office when I work from home.

If I had an oportunity I would like to show you the usage in a short video. Another problem for using this approach is that you need a chroma screen in every location you need to do a videoconference, and that’s not very practical.

References

Extending Geopackages via SQLite’s Run-Time Loadable extensions

2018-07-27T00:00:00+00:00

Introduction

SQLite is expected to be very important for GIS users. It’s non-trivial that two of the most promising GIS formats are based on SQLite: Spatialite and Geopackage. This way, understanding the basics of SQLite can be very useful for making the most of these data formats.

Particularly, geopackages are good for many different reasons and one can expect to find all of our favorite features in this modern GIS data format. In this sense, I’m a huge fan of Postgres extensibility. When using Postgres, you can create new operators, aggregates, functions, types, etc… and pack everything in a custom extension. PostGIS itself is a Postgres extension. Every-time we need to create a complex parametrized query, we can use Postgres CREATE FUNCTION and encapsulate it. This makes it easier to read/write complex queries (or Postgis geoprocesses) and improves code (re)usability. I would like to have such useful features in other databases and it would be amazing to have this features when working with Geopackages.

After my very first searches, I have learnt that SQLite is not intended to have stored functions, so CREATE FUNCTION does not work. However, in SQLite we can map functions from several programming languages (C, PHP, Python, Perl, .NET, among others) to SQL functions, namely user-defined functions and pack them in Run-Time Loadable extensions. It would be amazing if this approach could work also for Spatialite and Geopackages.

There are various ways for extending SQLite, but using its C API seems more straightforward and portable. I guess that everything done this way should work in the most used GIS (QGIS, ArcGIS, etc). Let’s do some tests…

Experimenting

I’m working with Ubuntu 16.04, SQLite 3.11.0 and QGIS (2.8 and 3.0). First, install sqlite3 and the libsqlite3-dev.

sudo apt-get install sqlite3 libsqlite3-dev

I’m not installing QGIS directly on my Ubuntu. Instead, I’m using dockers for testing on different QGIS versions without complications (check the kartoza QGIS docker lib for mor info).

Sample extension

There are some official SQLite extensions and source code to read and get some ideas. After exploring some resources, I’ve created a simple ifelse function based on this example.

/* ifelse.c */

#include "sqlite3ext.h"
SQLITE_EXTENSION_INIT1;

#include <stdlib.h>

typedef sqlite3_int64 i64;

static void ifelseFunc(sqlite3_context *context, int argc, sqlite3_value **argv) {
        int flag = sqlite3_value_int(argv[0]);
        char *v1 = (char *)sqlite3_value_text(argv[1]);
        char *v2 = (char *)sqlite3_value_text(argv[2]);

        if (flag != 0)
                sqlite3_result_text(context, v1, -1, SQLITE_TRANSIENT);
        else
                sqlite3_result_text(context, v2, -1, SQLITE_TRANSIENT);
}


int ifelse_init( sqlite3 *db, char **error, const sqlite3_api_routines *api )
{
    SQLITE_EXTENSION_INIT2(api);

    sqlite3_create_function( db, "ifelse", 3, SQLITE_ANY, 
            NULL, ifelseFunc, NULL, NULL);

    return SQLITE_OK;
}

Then, compile it to get the binaries.

gcc -shared -fPIC -o ifelse.so ifelse.c -lsqlite3

SQLite command-line testing

We need to be sure that this will work on a raw SQLite database, from the command-line, so I’m creating a dummy Landuse/Landcover (LULC) database.

sqlite3 lulc

After inserting some test data, we will use the ifelse function to print a description for two LULC codes (FOR -> Forest; URB -> Urban).

CREATE TABLE lulc (cover text NOT NULL);
INSERT INTO lulc VALUES('FOR');
INSERT INTO lulc VALUES('FOR');
INSERT INTO lulc VALUES('URB');
INSERT INTO lulc VALUES('FOR');
INSERT INTO lulc VALUES('URB');

Now, load the extension via the SQL function and the command-line, which should print an empty line.

SELECT load_extension('/path/to/extension/ifelse.so', 'ifelse_init');

The ifelse function can be tested and it works as expected.

SELECT ifelse(cover='FOR', 'Forest', 'Urban') FROM lulc;
--Forest
--Forest
--Urban
--Forest
--Urban

Finally, it can be checked that it is necessary to load this extension every-time we connect to the database. So .quit from sqlite3 and try to use the function again.

sqlite3 lulc
SELECT ifelse(cover='FOR', 'Forest', 'Urban') FROM lulc;
Error: no such function: ifelse

As you can see, it is necessary to load the extension every-time we connect to the lulc database.

SELECT load_extension('./ifelse.so', 'ifelse_init');
SELECT ifelse(cover='FOR', 'Forest', 'Urban') FROM lulc;
--Forest
--Forest
--Urban
--Forest
--Urban

Working with geopackages in QGIS

Unfortunatelly, I have tested this approach via DBmanager in QGIS 3.0, but it didn’t work. The extension can’t be loaded using this approach. The load_extension() call returns a pretty clear message: Not authorized.

SQLite load_extension() doesn't work since QGIS 3.0.

We can do this test in QGIS 2.8 and it will work correctly for Spatialite. However, Geopackages were not so well supported before QGIS 3.0. This difference between QGIS versions is due to a change in the Python version used since QGIS 3.x (see this answer on gis.stackexchange).

Conclusions

We have revised how to use the SQLite C API and it seems a very powerful approach for distributing data with some useful logic. For example, imagine that you want to distribute some data to GIS users, avoiding to describe complex parametrized queries. This could be achieved through Run-Time Loadable extensions…

I have discovered that the most recent Python distributions are blocking this mechanism for some security reasons and this is directly incorporated into QGIS. However, this functionality seems very useful and I think that there are other ways for loading extensions without compiling QGIS from sources.

Resources

These are some resources I have revised for writing this post.

Geopackages

SQLite

Pull multiple git repositories when working with that not-so-usual PC

2018-07-24T00:00:00+00:00

At least, every few days I change my workplace and I work on a different computer (home-laptop-work). For this reason, I usually need to ensure that I’m using the latest version of each repository.

After writing a script to facilitate the task of clonning several Git repositories, it is logical to consider the same when I need to pull them. In this post I want to share the script I use to automatically pull all my git repositories in one step, no matter if they are stored in GitHub, GitLab or Bitbucket. Practically, its the same script proposed by douglas, but simpler and following my personal naming convention…

The multiple-git-pull.sh script is stored in a Gist and allows me to pull several repositories stored in the same working folder (e.g. /home/user/git).

Avoid some headaches and innecesary pulls just by executing this script in your git working folder (if you have one :)

bash multiple-git-pull.sh

I hope you like it ;)

Clone multiple git repositories when setting up your Ubuntu

2018-04-02T00:00:00+00:00

This post is to share a bash script I always wanted to write. I have been searching for a while for a simple script for automatically clonning all my git repositories, no matter if they are stored in GitHub or Bitbucket. I have found many interesting approaches but I finally decided to write my own solution.

The multiple-git-clone.sh script is stored in a Gist and allows me to create several lists of repositories (e.g. grouped by owner, project, platform, etc) and clone all in a single step.

Of course there are many improvements I could use (e.g. error management, ask for pull instead of clone if necessary, etc) but for now it is enough when setting up a new machine.

I hope you like it ;)

Cómo obtener las alturas de los edificios a partir de la cartografía catastral

2017-01-14T00:00:00+00:00

Introducción

Hace varios meses una compañera me preguntó si existían datos descargables o había algún modo rápido obtener el número de plantas y las alturas de todos los edificios de Benidorm.

El modelo de urbanización en altura de Benidorm suele ser estudiado por ser una alternativa a la urbanización dispersa que predomina en la región. Este modelo tiene interés en cuanto a la ocupación del suelo o la gestión de los recursos hídricos, entre otros aspectos. En este caso, el número de plantas y las alturas de los edificios, podría utilizarse para calcular el coste energético de bombear el agua potable a lo alto de esas torres… La verdad es que no sé si al final se utilizaron los datos :-?

Pues sí, hay muchas maneras de obtener esos datos… En primer lugar, en España es posible modelizar una ciudad en 3D utilizando la estupenda cartografía catastral de que disponemos. Esta idea ha sido recurrente en cursos, congresos y también lo he leído en alguna ocasión (JIDEE08).

Por otro lado, con la llegada de los datos LIDAR no se ha hablado tanto de los datos catastrales para este fin. Sin embargo, los análisis con LIDAR, y de otras fuentes, pueden ser relativamente costosos en cuanto a procesado, filtrado, agregación, etc. Por este motivo, yo siempre prefiero consultar primero una base de datos que ya tenga preparados los datos necesarios. Si no fuese suficiente, ya trabajaremos con LIDAR o lo que sea…

Utilizando un DNI electrónico es posible descargar los ESRI Shapefiles de los con los datos volumétricos de Benidorm o de cualquier otro municipio. Es posible consultar toda la información necesaria sobre la descarga en la Sede Electrónica de Catastro.

Números romanos y algo más

En este enlace se puede consultar la especificación técnica donde se describe el ESRI Shapefile que nos podemos descargar. Merece la pena detenerse en el “4 ANEXO I”.

En estas imágenes vemos la capa WMS de catastro a la izquierda y el shapefile descargado a la derecha.

En el shapefile, hay una columna denominada CONSTRU que contiene los volúmenes construidos, pero en números romanos y en combinación con varios códigos y signos. Se trata de un montón de información dentro de cada celda. Por ejemplo, -II+IV+TZA describe una parte de un edificio que tiene dos plantas subterráneas, cuatro plantas sobre rasante y, en lo más alto, una terraza (TZA).

Esta cartografía también contiene información sobre las piscinas o las pistas deportivas, entre otros datos valiosísimos. El número de plantas se puede obtener, pero también hay que saber interpretar los códigos a partir del conocimiento de la zona de estudio. Por ejemplo, puede haber plantas en las que no haya suministro de agua potable como un vestíbulo muy elevado, una bóveda, etc.

Finalmente, le dediqué un tiempo a tratar de entender mejor estos códigos. A pesar de que habría que saber interpretar mejor la información, creo que tiene sentido sumar todos los números romános positivos de cada código y considerar aparte algunos códigos que podrían alterar nuestros cálculos. Por ejemplo, terrazas (TZA) o semisótanos (SS) podrían alterar un cálculo de alturas, ya que podrían significar alturas de 1,5 o 2 metros, en vez de los 3 metros que hemos visto en las referencias de la introducción (JIDEE08).

Función para obtener las plantas/alturas de los edificios

Me consta que hay scripts, más o menos de contrabando que circulan de disquete en disquete :-P

No lo he buscado a fondo, pero los ejemplos que he visto de momento eran para extraer el número de plantas de la columna CONSTRU desde la calculadora de campos de Arcmap. Sin embargo, no eran funciones muy elegantes y no servían para edificios de más de 40 alturas (en Benidorm esto se nota mucho) Todo ello sin llegar a hablar de software libre.

Mis primeros proyectos de programación GIS también fueron VBScript para la calculadora de campos de ArcGIS, después probé con Python, pero desde entonces no había vuelto a trabajar con la calculadora de campos de un SIG de escritorio. Particularmente, me gusta la productividad de este tipo de scripts para tareas sencillas.

Una vez insertada la función desde el editor de funciones, queda guardada y ya la podemos utilizar.

En este caso me pareció apropiado probar con Python y la calculadora de campos de QGIS, ya que tiene una documentación excelente (qgistutorials). Quien sabe, quizás más adelante me anime con los plugins con PyQGIS :)

El script (al final del post) consiste en una función que hace lo siguiente:

Elimina las plantas bajo rasante y los caracteres innecesarios
Trocea el código y extrae los números romanos
Utiliza una función de Mark Pilgrim’s para convertir los números romanos a enteros
Suma todos los valores de cada parte de edificio.
Guarda el resultado en una columna nueva

El número de plantas o las alturas se obtendrían ejecutando la función así:

get_cadastre_building_floors("CONSTRU")
get_cadastre_building_floors("CONSTRU") * 3

Resultados y discusión

Los resultados no se pueden entender sin recordar el hecho de que los elementos representados en la capa GIS son partes de edificios, no edificios enteros (ver documentación). Entonces, un único edificio puede estar representado por varias partes, cada una con un número distinto de plantas…

De un modo ágil se pueden repasar los resultados:

Comprobando que la lectura de los números romanos da el resultado apropiado.
Revisando esta página web sobre los rascacielos de Benidorm se puede valorar cómo de precisos son los cálculos que acabamos de realizar.

CONSTRU	plantas	altura (3m)
-I+L	50	150
-I+XLVIII	48	144
-I+XLV	45	135
-I+XLV	45	135
-V+SS+XLIV	44	132
-V+SS+XLIV	44	132
-IV+SS+XLIV	44	132
-IV+SS+XLIV	44	132
-V+SS+XLIV	44	132
-V+SS+XLIII	43	129

En esta tabla se muestran los resultados para los 10 edificios (partes de edificios) más altos de Benidorm. Al comparar estos datos con los de la web anterior se puede ver que:

El edificio Intempo (192 metros / 47 plantas) no estaba construido cuando se digitalizó esta capa.
La altura media de las plantas es mayor de 3 metros, quizás por los elementos de diseño que buscan competir por ser los edificios más altos.
Los primeros 4 valores de plantas se corresponden con partes del Gran Hotel Bali (186 metros / 52 plantas).

El Gran Hotel Bali de Benidorm. Ortofoto del WMS del PNOA y, en superposición, la capa SHP de catastro con las alturas calculadas.

Independientemente de que se puedan extraer las alturas utilizando la codificación actual, entiendo que es cuestión de no mucho tiempo en que esta información deba ser publicada mediante otro tipo de formatos más flexibles (GML, GEOJSON, etc).

Código a mejorar…

Aquí os dejo el gist con la función. Es un buen punto de partida para lo que queráis hacer ;)

Introducción a la geodatabase del SIOSE

2016-12-14T00:00:00+00:00

Hola a todos, en este post quiero compartir dos nuevas presentaciones de un seminario temático que he impartido dentro de la asignatura “SIG aplicado a la Ordenación del Territorio” en el Grado de Geografía de la Universidad de Alicante.

Recomiendo leer el post anterior sobre este tema y toda la documentación disponible. En esta ocasión no se profundiza tanto en la modelización de datos y se estudia la base de datos del SIOSE con SQL y la funcionalidad de PostGIS.

SQL para geógrafos.

En esta presentación se asume que el público conoce el uso de un SIG y ha digitalizado usos del suelo. A continuación, se presenta SQL en un contexto profesional y se describe la base de datos del SIOSE.

Introducción a la geodatabase del SIOSE (I) de Benito M Zaragozí

Introducción a SQL.

En esta segunda presentación se trabaja con Postgres/Postgis por primera vez. El objetivo es dar un rápido vistazo a SQL y entender que para consultar el SIOSE con libertad es necesario conocer algún lenguaje de consulta.

Introducción a la geodatabase del SIOSE (II) de Benito M Zaragozí

Datos utilizados

Los datos del SIOSE utilizados en estas presentaciones están disponibles en varios enlaces. Por ejemplo, en el Centro de Descargas del CNIG (España) o en la sección de descargas del Terrasit (Comunidad Valenciana).

No obstante, los alumnos que asistieron a dichas sesiones pueden encontrar interesante trabajar directamente con los backups creados y restaurados varias veces en clase:

Las versiones del software son las que vienen con el OSGeo Live GIS Disc 9.5. Salvo alguna modificación, esta version de OSGeo es la instalada en el Aula de Geomática del Instituto Interuniversitario de Geografía,

Una vez más, os animo a conocer el SIOSE ;)