Creating an ArcGIS Online map using Seismic data

Intro

For quite long time I wanted to combine seismic data and produce a map for the Greek region. Last week I had the opportunity to work on it and I will explain the steps I followed bellow.

In order for you to be able to create this map you should create an ArcGIS Online (AGOL) account in this link.

Data

The data used in this map were downloaded from two different resources. The first data set was the Earthquakes data and they were downloaded from the Faculty of Geology of the National and Kapodistrian University of Athens and the data portal that they provide.

It is possible to select the desired parameters and download the file in .csv format. I selected to have earthquakes with a magnitude greater than 4 (Richter magnitude scale) and depth of 0-200km. The region was left as the default.

The second dataset used was downloaded from the Greek Database of Seismogenic Sources, which is a repository of geological, tectonic and active-fault data for the Greek territory and its surroundings. The dataset includes composite and individual seismogenic sources. Unfortunately the dataset hasn’t been updated for the last 5 years, but it gives a very good indication of the Seismic sources in the region.

The later dataset was in .kmz format which I converted to GeoJSON in this converting website. Easy to use and multi-format availability.

ArcGIS Online

There is this very nice tutorial on how to upload data to AGOL and it explains in detail on the section “Add the CSV file as a layer” how to do so with a .csv file. Same procedure can be followed for a GeoJSON file.

Overview of the Greek Region

I used the ESRI topographic map as a basemap. The symbology of the earthquakes is based on the magnitude (size) and depth (colour) attributes.

The map can be found in this link.

The overview might be a bit hectic for the untrained eye, but as you zoom in to specific regions you can see the details and check all the attributes by clicking to all the Features.

Details of a smaller region with Pop-up

ArcMap vs. ArcGIS Pro

In this post I will compare the two ArcGIS Desktop applications: ArcMap & ArcGIS Pro. There is a big discussion around this for the last 2-3 years, so I will try to give a comparison matrix to be very fast and straightforward.

The most important comment I believe is that according to ESRI documentation, ArcGIS Pro will eventually replace ArcMap. ESRI will continue to support and maintain ArcMap. There are still some capabilities of ArcMap that are not yet in ArcGIS Pro, so some ArcGIS Desktop users will be using both ArcGIS Pro and ArcMap till Pro has everything you need to fully support your work. 

While Esri has moved most of its development efforts to ArcGIS Pro, ESRI will continue to provide support and minor bug fixes in ArcMap for a long time (you can find the ArcMap Product Life Cycle on the Esri website).

Comparison Matrix

• 

  ArcMap vs ArcGIS Pro Matrix

FUTURE IN ARCGIS DESKTOP TECHNOLOGY

It is worth it to spend some time and read the Esri User Conference 2018 Q&A (check here). Find below what it is mentioned in the sub-section titled as “What is the future of the ArcGIS Desktop technology?”

[…] Over the last several years, we have been migrating the capabilities of ArcMap into ArcGIS Pro, and with the recent 2.2 release, most of the commonly used capabilities have been implemented in ArcGIS Pro. Over the next year, we expect ArcGIS Pro to provide virtually all the commonly used capabilities found in ArcMap.

[…] ArcGIS Pro represents a major enhancement in capabilities for our users because it is fully integrated into the ArcGIS platform and the Web GIS pattern of working on and with web layers, web maps, and web scenes. This means that in common workflows where a user generates a high-quality map or 3D visualization, he or she can author a 2D or 3D map and share it to ArcGIS Online or ArcGIS Enterprise to make it immediately available as web maps for other users in their organization. 

ArcGIS Pro also has a new and modern user interface, 2D and 3D views, multi-threaded geoprocessing, support for multiple layouts, and many of the features and functions requested by users for many years. ArcGIS Pro can also be easily virtualized in cloud environments. Currently, over 12% of ArcGIS Pro users have deployed it in this manner. Esri is working on new innovations in ArcGIS Pro in the “connected desktop” and the services architecture, including many apps that complement and extend the desktop environment and capabilities of our users.

JMeter Generate report

In the previous post, we went through basic steps on how to create a simple JMeter performance test for an ArcGIS Server Map Service. In this post I will explain how to to use the JMeter dashboard generator, in order to visualise the results.

JMeter supports dashboard report generation to get graphs and statistics from a test plan (JMeter documentation). The JMeter dashboard generator is a modular extension of JMeter. Its default behaviour is to read and process samples from CSV files to generate HTML files containing graph views. It can generate the report at end of a load test or on demand.

Configuration

By default the you are able to generate a dashboard report in JMeter. If not, Check if your JMeter configuration in jmeter.properties file under JMETER_HOME/bin directory, follows these settings:

jmeter.save.saveservice.bytes=true
# Only available with HttpClient4
#jmeter.save.saveservice.sent_bytes=true
jmeter.save.saveservice.label=true
jmeter.save.saveservice.latency=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.time=true
jmeter.save.saveservice.connect_time=true
jmeter.save.saveservice.assertion_results_failure_message=true
# the timestamp format must include the time and should include the date.
# For example the default, which is milliseconds since the epoch: 
jmeter.save.saveservice.timestamp_format=ms

Report Generation

You can combine the report generation with the jmeter command of running the test or you can run them separately. While learning how to use the generate report module, I suggest to I run the generate report command separately. But I will detail both of them.

All at once

You can run them all at once using the following command:

jmeter -n -t [path to test JMX file] -l [path to test result file] -e -o [Path to output folder]

In this way you cover all the JMeter functionality as required in the ArcGIS Server performance testing. It is a very useful tool to automate the procedure but not for starters.

From an existing sample CSV/JTL result

If you followed the previous post correctly you should have a .csv file created at the end of the test. You can use this file to generate the report with the use of the Generate report module of JMeter, as follows:

jmeter -g [path to result file] -o [path to report output folder]

In my case were the file was named Performance test.csv I used the following command:

jmeter -g /<<PATH>>/"Performance test.csv" -o /<<PATH>>/HTML_Reports

Please note the “” on the file name. Also, at the end of the created file I added the “HTML_Reports” which is a folder created to save all the required files within.

Report

You can now navigate to the folder you defined and check the results. If you open the “index.html” you will see directly the report.

Response Time

Report Overview

Details on the tables and figures created can be found on the JMeter documentation. Another detailed documentation is this one.

Report Configuration

As you can see the report is nothing but an HTML file including css, js and pages. Obviously this is highly configurable as a result but also in the basis of the creation of the file. More details on the jmetervn post.

ArcGIS REST API query tips for beginners

Lately I checked gis.stackexchange and there are several easy questions regarding ArcGIS REST API and how to query it. I would like to share a few simple tips in order to do so. Details on the query operation functionality on the ESRI documentation.

First of all I find very helpful the Chrome extension “Map Services Enhanced”. It is an amazing tool that can help even advanced users/developers to understand the status of an ArcGIS REST Service.

An extension for analyzing, interrogating, and troubleshooting ArcGIS REST Services. This extension works with ArcGIS Server REST pages to collect and present additional data related to the services you are seeing. You can search for relevant data, and find the corresponding map service. It also reviews data from links to services

Besides the obvious help of the details in the HTML interface of the REST API, it gives a Query Helper on the Query operation of the service.

Lets use an example in this post to make it easier to experiment. The example can be this, an open service by ESRI. If you open the link and you have installed the Map Services Enhanced extension, you will see that there are 134 (at this moment) features. You can click the Query operation in the lowest point of the website.

The absolute tip for a beginner when he first opens the Query page of a service is to  add in the Where: field the value 1=1. In this way you will get all the records but not all the attributes. It is a good way to have an overview. PLEASE NOTE that if the service has a lot of records, this query will delay a lot and possibly could cause some issues on the server.

If you want to have all the available fields as an output you can add in the Output Fields: the * value. In this way you will have a very good overview of the service and you can easily adjust the query to reflect your needs. For example instead of setting 1=1, we can select all the records Where:DAY=’SUN’ which will return all the records where the day is set as Sunday.

The Query Helper can really assist you on that. For example if you didn’t check the amount of Features in the previous webpage, you can press the Get CountOnly *. In this way you will just get the number of the available records.

Those were just some very simple tips. You can dig into the ESRI documentation to get more details on the functionality, but I would suggest you to work on the Query operation and try to find out yourself the possibilities of the tool.

In a latter post I will give more details, especially on the way that you can automate the query operation as a Python procedure.

 

ArcSDE licensing

I was really curious about the ArcSDE licensing and tried to find some information in the internet about it. In my opinion it is very weird that the Enterprise GeoDB creation requires keycodes (Authorisation file) from ArcGIS Server  software and that there should be another way to do so. But after digging and surfing, there is not much information about that topic.

On this gis.stackexchange Question (answered on August of 2015) it is mentioned that:

“Esri incorporated ArcSDE into ArcGIS Server with the release of ArcGIS 9.2. In order to deploy ArcSDE technology as an enterprise geodatabase you must have, or be prepared to purchase, an ArcGIS Server license. Since the introduction of ArcGIS 10.0, the actual enterprise geodatabase installer lives within ArcPy, and can be invoked either as an ArcToolbox tool or directly from Python.”

Checkin on the Create Enterprise Geodatabase tool documentation for the authorization_file option, it is explained as:

“Provide the path and file name of the keycodes file that was created when you authorized ArcGIS Server. If you have not already done so, authorize ArcGIS Server to create this file.”

Also, in this GeoNet post (from 2011), it is mentioned that:

“You cannot purchase a license for ArcSDE. Since 9.2, the ArcSDE application server is a *part* of the Enterprise ArcGIS license.”

Finally, in this ESRI community post, it is explained on details what to do with the authorisation file (in case you miss it) and it notes that:

“NOTE: You must have authorized ArcGIS Server to create the keycodes file ahead of time!”

So, it is clear that there is no separate installation any more for ArcSDE and that this could be done via ArcGIS Desktop software (ArcMap or ArcGIS Pro). Nevertheless it is required to have ArcGIS Server licence in order to be able to create an Enterprise Geodatabase.

Theory & creation of GDB using PostgreSQL

ArcSDE

ArcSDE (Spatial Database Engine) is a server-software sub-system (produced and marketed by Esri) that aims to enable the usage of Relational Database Management Systems for spatial data. The spatial data may then be used as part of a geodatabase (wiki source).

While traditional RDBMS software keeps track of the tables and records contained in the database, ArcSDE pushes the relational model higher so that client software can manage geographic data – which comprise several tables – seamlessly. The user need have no awareness of nor dealings with the particulars of the RDBMS. The GIS environment routes all connections to the database through the ArcSDE middleware, which manages the storing and retrieval of data (wiki source).

Geodatabase

The geodatabase is a “container” used to hold a collection of datasets. There are three types:

1. File geodatabases—Stored as folders in a file system. Each dataset is held as a file that can scale up to 1 TB in size. The file geodatabase is recommended over personal geodatabases.
2. Personal geodatabases—All datasets are stored within a Microsoft Access data file, which is limited in size to 2 GB.
3. Enterprise geodatabases—Also known as multiuser geodatabases, they can be unlimited in size and numbers of users. Stored in a relational database using Oracle, Microsoft SQL Server, IBM DB2, IBM Informix, or PostgreSQL. This one requires the use of ArcSDE technology.

There is a comparison matrix on the ESRI documentation, where you can see all the details and differences between them.

Essential readings about the geodatabase can be found in this ESRI documentation. We will skip a big part of the documentation to go to the actual creation of the Enterprise geodatabase by connecting to the PostgreSQL.

Get started with geodatabases in PostgreSQL

The main tutorial to follow is this from ESRI documentation. It is very important to have a look on the PostgreSQL database requirements for ArcGIS 10.6.x in order to check the compatibility between the versions. In my case I do not have the minimum supported (PostgreSQL 10.3 & PostGIS 2.4) but I have PostgreSQL 10.5 & PostGIS 2.5. It says in the documentation “Newer minor versions are supported but are not certified”. So lets test it.

During creation of the geodatabase there is a more detailed description here. While running the “Create Enterprise Geodatabase” geoprocessing tool I entered the following options:

• DB platform: PostgreSQL
• Instance: localhost
• Database: myfirstgdb
• Tablespace Name: <empty>
• Authorization file: C:\Program Files\ESRI\License10.4\sysgen\keycodes

After finishing I can check in pgAdmin that the myfirstgdb geodatabase exists! The next step is to create a user to own data as described in ESRI documentation. Data stored in your geodatabase should be owned by users other than the sde user. Use the Create Database User geoprocessing tool to create a user in the PostgreSQL database cluster and a schema in your new database.

First step is to create a database connection to the newly created database. In the add Database connection menu select:

• DB platform: PostgreSQL
• Instance: localhost
• Database: myfirstgdb

Now run the Create Database User tool. Very straightforward procedure as well. Once again you can check this via pgAdmin.

Now that you have a user who can add data to your new geodatabase, connect to the database as that user. The easiest way to do that is to alter the connection file you created earlier. A small test is to try and import some data to “myfirstgdb”.

 

 

 

Privileges for using ArcGIS with a Db2 database issue

In one of the projects aI was involved in the past, we encounter an issue which I would like to share. During deployment of map services, I was not able to amend the query layer and therefore not able to publish a new service.

The first message I got was:

The second message I got was:

The third message I got was:

Working on the database directly and granting access to the arcgisro user, we managed to make it work.  Since we fixed it, I got to find what exactly was the issue. Luckily enough the ArcGIS docs has a detailed article on the privileges for using ArcGIS with a Db2 database issue.

The problem started when the read-only users in the database got more restricted privileges excluding the:

1. SYSCAT.ROLEAUTH
2. SYSCAT.DBAUTH
3. SYSCAT.TABAUTH