Skip to content
Snippets Groups Projects
Commit 6bdfd0cf authored by lcalisto's avatar lcalisto
Browse files

Scripts and latest updates

parent 45262a0b
Branches
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
README.md
+ 293
− 70
View file @ 6bdfd0cf
......@@ -580,10 +580,90 @@ query MyQuery {
}
}
```
## Using variables
In GraphQL we are able to use variables in our queries. This variables are important for:
- Scripting, in order to be able to interact with our script variables
- Ingest complex JSON objects into our query
- Make sure the query is easy to read
When using GraphiQL we have a query variables box. Inside this box we can add our variables in JSON format.
Lets demonstrate the usage of variable in the following queries:
- Get __first 10 profiles__ from continent __Europe__
Inside `Query variables` add `first` and `continent` variables:
```json
{
"first": 10,
"continent": "Europe"
}
``````
The GraphQL query will be:
```graphql
query MyQuery($first:Int, $continent:String) {
wosisLatestProfiles(
first: $first
filter: { continent: { likeInsensitive: $continent } }
) {
continent
countryName
region
datasetCode
latitude
longitude
profileId
}
}
```
In your GraphiQL you should have something as bellow image:
![variables](./images/variables.jpg "wosisLatestProfiles variables")
__Using arrays [ ]__
- Using an __array [ ]__ get __first 10 profiles__ from continent __Europe or Africa__
Inside `Query variables` box:
```json
{
"first": 10,
"continent": ["Europe","Africa"]
}
``````
The GraphQL query will be:
```graphql
query MyQuery($first:Int, $continent:[String!]) {
wosisLatestProfiles(
first: $first
filter: { continent: { in: $continent } }
) {
continent
countryName
region
datasetCode
latitude
longitude
profileId
}
}
```
In the next chapter we'll make use of variables to better provide JSON components to our queries.
## Spatial queries
In order to use spatial queries, for this examples lets use 2 geometries of Gelderland in GeoJSON format.
This API has spatial capabilities. Its possible to perform several __spatial queries__ and apply __spatial filters__. Spatial components are GeoJSON based.
In order to use spatial queries, we'll lets use 2 geometries of Gelderland in GeoJSON format.
You can use https://geojson.io to visualize, create and update GeoJSON geometries.
......@@ -612,36 +692,14 @@ You can use https://geojson.io to visualize, create and update GeoJSON geometrie
"features": [
{
"type": "Feature",
"properties": {},
"geometry": {
"coordinates": [
6.025363925650851,
52.501157816882994
],
"type": "Point"
}
},
{
"type": "Feature",
"properties": {},
"geometry": {
"type": "MultiPoint",
"coordinates": [
5.158391536033605,
51.775118267397204
],
"type": "Point"
}
},
{
"type": "Feature",
"properties": {},
"geometry": {
"coordinates": [
6.742439219867151,
51.96023476075487
],
"type": "Point"
}
[6.025363925650851,52.501157816882994],
[5.158391536033605,51.775118267397204],
[6.742439219867151,51.96023476075487]
]
}
}
]
}
......@@ -650,7 +708,9 @@ You can use https://geojson.io to visualize, create and update GeoJSON geometrie
![gelderlandPoints](./images/gelderlandpoints.jpg "gelderland points")
- Get __first 3 profiles__ that fall inside Gelderland using the MultiPolygon geometry.
In order to simplify and make a more easy-to-read query we'll make use of `variables` in our spatial queries.
- Get __first 3 profiles__ that fall inside Gelderland using the MultiPolygon geometry. In this query we also make sure all __profiles have at least one layer__.
```graphql
query MyQuery($geomGelderland: GeoJSON!) {
......@@ -662,10 +722,17 @@ query MyQuery($geomGelderland: GeoJSON!) {
region
profileId
datasetCode
latitude
longitude
geom{
geojson
x
y
}
}
}
```
Inside `Query variables` add the following:
Inside `Query variables` add the `geomGelderland` variable:
```json
{
......@@ -679,8 +746,71 @@ Example on what you should see in GraphiQL:
![geomGelderlandExample1](./images/geomGelderland_example1.jpg "geomGelderlandExample1")
## Using variables
The __GEOM object__ corresponds to the geometry. Please spend some time exploring this object in GraphiQL interface. Make sure you explore the `Filter` capabilities.
- Using the previous query change the `query variables` to the points geometry:
```json
{
"geomGelderland": {
"type": "MultiPoint",
"coordinates": [
[6.025363925650851,52.501157816882994],
[5.158391536033605,51.775118267397204],
[6.742439219867151,51.96023476075487]
]
}
}
```
You will see that the same query now produces __no results__. This is because we are searching for WoSIS Profiles that `intersect` the provided geometry. In this case we must use a different spatial filter.
- Get __first 3 profiles__ that fall inside the BBOX of the points in our MultiPoint geometry. In this query we also make sure all __profiles have at least one layer__.
```graphql
query MyQuery($geomGelderland: GeoJSON!) {
wosisLatestProfiles(
first: 3
filter: {layersExist: true, geom: {bboxIntersects2D: $geomGelderland}}
) {
continent
region
profileId
datasetCode
latitude
longitude
geom {
geojson
x
y
}
}
}
```
## Pagination concepts
Depending on the way you create your query it can evolve hight computational resources. Besides, if not using pagination you could easily create a query that returned a huge records with all the problems that brings.
To solve this problem __we enforce pagination in this GraphQL API__.
For the moment, in order to make things easier we offer a simpler list interface for the connections based on __Offset-based Pagination__. This means we *temporary disabled* [Relay Cursor Connections](https://relay.dev/graphql/connections.htm).
__If you are an advanced user and you would like to have `Relay Cursor Connections` please contact us.__
__The `First:` argument__
All queries must have a `first` argument in the connections. So far we used this in all our queries. This argument indicates the *maximum number of items* to return.
__The `Offset:` argument__
`Offset` is an optional argument that indicates *where in the list the server should start when returning items* for a particular query.
This arguments `first` and `offset` are extremely important when you need to extract and download data.
We'll make use of pagination on our scripts. We'll show how to use pagination and extract a considerable amount of data from WoSIS using this GraphQL API.
## Scripting
### Python examples
......@@ -688,16 +818,17 @@ Perhaps also on Google Colab.
The simplest way to perform a graphQL request in python is to use requests.
- Get the fist 10 profiles:
- Get the fist 10 profiles and add it to a Pandas dataframe:
```python
import requests
import json
import pandas as pd
# GraphQL query
query = """
query MyQuery {
wosisLatestProfiles(first: 10) {
wosisLatestProfiles(first: 5) {
continent
region
countryName
......@@ -713,60 +844,152 @@ query MyQuery {
url='https://graphql.isric.org/wosis/graphql'
# Send POST request
r = requests.post(url, json={'query': query})
# Print status_code if needed
# print(r.status_code)
# Print status_code
print(r.status_code)
# Parse JSON
parsed = json.loads(r.text)
# Print JSON
json=json.dumps(parsed, indent=4, sort_keys=True)
print(json)
# Convert to pandas dataframe
df = pd.json_normalize(parsed['data']['wosisLatestProfiles'])
# print dataframe
print(df)
```
The response will be:
The result will be:
```json
{
"data": {
"wosisLatestProfiles": [
{
"continent": "Africa",
"countryName": "Zambia",
"datasetCode": "AF-AfSIS-I",
"geomAccuracy": 1e-06,
"latitude": -16.044876098632812,
"longitude": 28.257427215576172,
"profileCode": "icr056260",
"region": "Eastern Africa"
},
{
"continent": "Africa",
"countryName": "Zambia",
"datasetCode": "AF-AfSIS-I",
"geomAccuracy": 1e-06,
"latitude": -16.044876098632812,
"longitude": 28.257427215576172,
"profileCode": "icr056261",
"region": "Eastern Africa"
}
]
},
"meta": {
"graphqlQueryCost": 2
![python_q1_result](./images/python_q1_result.jpg "python_q1_result")
Using variables in our script:
- Get the __first 3 profiles__ that are __inside Gelderland region__ and add it to a Pandas dataframe:
```python
import requests
import json
import pandas as pd
# GeoJSON geometry
geomGelderland = {
"type": "MultiPolygon",
"coordinates": [ [ [ [ 5.177260142422514, 51.74291774914947 ], [ 5.126747881386732, 51.737828850498403 ], [ 5.137580867932065, 51.772905259431077 ], [ 5.014540023249575, 51.808984680959583 ], [ 5.031415073146523, 51.841084802107702 ], [ 4.993967909252922, 51.861222725420994 ], [ 5.062358224116345, 51.859362053527242 ], [ 5.180226727863164, 51.96744832651509 ], [ 5.236867149255078, 51.978757478459428 ], [ 5.321611332014112, 51.954919171164796 ], [ 5.486214078473083, 51.98382644510454 ], [ 5.627223829712356, 51.952386168324438 ], [ 5.550342661060417, 52.10541954546126 ], [ 5.459242995490565, 52.080225755481266 ], [ 5.514079463312799, 52.135923065932062 ], [ 5.439875615559026, 52.171197458274222 ], [ 5.44103943147957, 52.205693438951691 ], [ 5.393219147822698, 52.220626892173925 ], [ 5.404643399611359, 52.249630480909225 ], [ 5.533281176545358, 52.27274084169683 ], [ 5.587707385036856, 52.361454261431376 ], [ 5.787257137970521, 52.422573287061603 ], [ 5.876205471530124, 52.522025026941051 ], [ 5.925559518063968, 52.474057592745915 ], [ 6.027857569808684, 52.509606205409327 ], [ 6.099483437203417, 52.469970896552461 ], [ 6.130552948323514, 52.399978162269164 ], [ 6.078506385563601, 52.369523051161245 ], [ 6.066224466859907, 52.318839289847247 ], [ 6.163909067147507, 52.21749619292715 ], [ 6.38185154627214, 52.246112812566473 ], [ 6.492401220236633, 52.177371870181403 ], [ 6.671338986248984, 52.165683203635673 ], [ 6.662399005672591, 52.130167439615931 ], [ 6.760572413121598, 52.118779940206082 ], [ 6.687853003658449, 52.039856158091141 ], [ 6.832754328999235, 51.972938087693585 ], [ 6.721969582522561, 51.89606334135938 ], [ 6.683993990179909, 51.91757645733221 ], [ 6.472507886098918, 51.853823023864017 ], [ 6.390566170881016, 51.87396806966867 ], [ 6.401818441765064, 51.827262656663407 ], [ 6.117889496603739, 51.901659142837225 ], [ 6.166559884993931, 51.840721643435401 ], [ 6.063485632339608, 51.86545122678897 ], [ 5.962978284523374, 51.836913960582471 ], [ 5.946569966406273, 51.813479919592751 ], [ 5.992067051189349, 51.770245909123908 ], [ 5.943962150919553, 51.741816814422592 ], [ 5.893409336802974, 51.777852926426895 ], [ 5.765188291802036, 51.752789880063702 ], [ 5.638112608999517, 51.819025176083443 ], [ 5.493105254357093, 51.830750957327069 ], [ 5.403157084105017, 51.821611677731141 ], [ 5.357568231054432, 51.757890339715857 ], [ 5.300338754648935, 51.737287437014395 ], [ 5.177260142422514, 51.74291774914947 ] ] ] ]
}
# GraphQL query
query = """
query MyQuery($geomGelderland: GeoJSON!) {
wosisLatestProfiles(
first: 3
filter: {layersExist: true, geom: {intersects: $geomGelderland}}
) {
continent
region
profileId
datasetCode
latitude
longitude
geom{
geojson
x
y
}
}
}
"""
# GraphQL endpoint
url='https://graphql.isric.org/wosis/graphql'
# Send POST request
r = requests.post(url, json={'query': query, 'variables': {'geomGelderland': geomGelderland}})
# Print status_code
print(r.status_code)
# Parse JSON
parsed = json.loads(r.text)
# Convert to pandas dataframe
df = pd.json_normalize(parsed['data']['wosisLatestProfiles'])
# print dataframe
print(df)
```
Using variables in our script:
The result will be:
![python_q2_result](./images/python_q2_result.png "python_q2_result")
- Get __all WoSIS profiles with layers that exist in Gelderland__ and also __export it to CSV__.
- Get all profiles from Portugal with Organic Carbon measurements
```python
import requests
import json
import pandas as pd
# GeoJSON geometry
geomGelderland = {
"type": "MultiPolygon",
"coordinates": [ [ [ [ 5.177260142422514, 51.74291774914947 ], [ 5.126747881386732, 51.737828850498403 ], [ 5.137580867932065, 51.772905259431077 ], [ 5.014540023249575, 51.808984680959583 ], [ 5.031415073146523, 51.841084802107702 ], [ 4.993967909252922, 51.861222725420994 ], [ 5.062358224116345, 51.859362053527242 ], [ 5.180226727863164, 51.96744832651509 ], [ 5.236867149255078, 51.978757478459428 ], [ 5.321611332014112, 51.954919171164796 ], [ 5.486214078473083, 51.98382644510454 ], [ 5.627223829712356, 51.952386168324438 ], [ 5.550342661060417, 52.10541954546126 ], [ 5.459242995490565, 52.080225755481266 ], [ 5.514079463312799, 52.135923065932062 ], [ 5.439875615559026, 52.171197458274222 ], [ 5.44103943147957, 52.205693438951691 ], [ 5.393219147822698, 52.220626892173925 ], [ 5.404643399611359, 52.249630480909225 ], [ 5.533281176545358, 52.27274084169683 ], [ 5.587707385036856, 52.361454261431376 ], [ 5.787257137970521, 52.422573287061603 ], [ 5.876205471530124, 52.522025026941051 ], [ 5.925559518063968, 52.474057592745915 ], [ 6.027857569808684, 52.509606205409327 ], [ 6.099483437203417, 52.469970896552461 ], [ 6.130552948323514, 52.399978162269164 ], [ 6.078506385563601, 52.369523051161245 ], [ 6.066224466859907, 52.318839289847247 ], [ 6.163909067147507, 52.21749619292715 ], [ 6.38185154627214, 52.246112812566473 ], [ 6.492401220236633, 52.177371870181403 ], [ 6.671338986248984, 52.165683203635673 ], [ 6.662399005672591, 52.130167439615931 ], [ 6.760572413121598, 52.118779940206082 ], [ 6.687853003658449, 52.039856158091141 ], [ 6.832754328999235, 51.972938087693585 ], [ 6.721969582522561, 51.89606334135938 ], [ 6.683993990179909, 51.91757645733221 ], [ 6.472507886098918, 51.853823023864017 ], [ 6.390566170881016, 51.87396806966867 ], [ 6.401818441765064, 51.827262656663407 ], [ 6.117889496603739, 51.901659142837225 ], [ 6.166559884993931, 51.840721643435401 ], [ 6.063485632339608, 51.86545122678897 ], [ 5.962978284523374, 51.836913960582471 ], [ 5.946569966406273, 51.813479919592751 ], [ 5.992067051189349, 51.770245909123908 ], [ 5.943962150919553, 51.741816814422592 ], [ 5.893409336802974, 51.777852926426895 ], [ 5.765188291802036, 51.752789880063702 ], [ 5.638112608999517, 51.819025176083443 ], [ 5.493105254357093, 51.830750957327069 ], [ 5.403157084105017, 51.821611677731141 ], [ 5.357568231054432, 51.757890339715857 ], [ 5.300338754648935, 51.737287437014395 ], [ 5.177260142422514, 51.74291774914947 ] ] ] ]
}
# GraphQL query
query = """
query MyQuery($first: Int, $offset: Int, $geomGelderland: GeoJSON!) {
wosisLatestProfiles(
first: $first,
offset: $offset,
filter: {layersExist: true, geom: {intersects: $geomGelderland}}
) {
continent
region
profileId
datasetCode
latitude
longitude
}
}
"""
# GraphQL endpoint
url='https://graphql.isric.org/wosis/graphql'
new_results = True
first = 100
offset = 0
all_results = []
while new_results:
# Send POST request
r = requests.post(url, json={'query': query, 'variables': {'first': first, 'offset': offset, 'geomGelderland': geomGelderland}})
# Parse JSON
parsed = json.loads(r.text)
# Add results to all_results object
all_results.extend(parsed['data']['wosisLatestProfiles'])
# for debugging
# print(json.dumps(parsed, indent=4, sort_keys=True))
# print(len(parsed['data']['wosisLatestProfiles']))
if not 'wosisLatestProfiles' in parsed['data'] or len(parsed['data']['wosisLatestProfiles']) == 0:
print('No more results')
# update new_results
new_results = False
else:
print('We have more results')
# update offset
offset = offset+first
df = pd.json_normalize(all_results)
# print dataframe
print('There are {} WoSIS profiles with layers inside Gelderland region'.format(df.shape[0]))
# Export dataframe to CSV
df.to_csv('wosis_gelderland.csv', index=False)
```
The result will be:
`There are 136 WoSIS profiles with layers inside Gelderland region`
CSV result file can be found [here](./scripts/python/wosis_gelderland.csv)
### R examples
## Conclusions
Filtering capabilities
Scripting capabilities
Its not to download all the data, for that we already have the OGC services. Its also not to replace OGC services but an extra service based on recent technology.
----------
images/python_q1_result.jpg 0 → 100644
+ 0
− 0
View file @ 6bdfd0cf
images/python_q1_result.jpg

33.2 KiB

images/python_q2_result.png 0 → 100644
+ 0
− 0
View file @ 6bdfd0cf
images/python_q2_result.png

17.1 KiB

images/variables.jpg 0 → 100644
+ 0
− 0
View file @ 6bdfd0cf
images/variables.jpg

37.3 KiB

scripts/python/wosis_gelderland.csv 0 → 100644
+ 137
− 0
View file @ 6bdfd0cf
continent,region,profileId,datasetCode,latitude,longitude
Europe,Western Europe,1043374,NL-Alterra,51.98697542967124,6.665127751403251
Europe,Western Europe,1043487,NL-Alterra,51.931844611547206,6.724923833764855
Europe,Western Europe,1043649,NL-Alterra,52.079851588741306,6.660048778683933
Europe,Western Europe,1043659,NL-Alterra,51.99366275331206,6.662697317751144
Europe,Western Europe,1043660,NL-Alterra,51.98215296783606,6.646356431984319
Europe,Western Europe,1043730,NL-Alterra,52.037119170174954,6.5774295292486435
Europe,Western Europe,1043780,NL-Alterra,52.0991175500654,6.584334798532214
Europe,Western Europe,1043672,NL-Alterra,51.84682506875407,5.043257305355638
Europe,Western Europe,1043726,NL-Alterra,51.8838260352328,5.098457949493585
Europe,Western Europe,1043996,NL-Alterra,51.870326759517646,5.091720086565306
Europe,Western Europe,934557,EU-FOREGS,52.173553466796875,6.122528076171875
Europe,Western Europe,936545,EU-FOREGS,52.174461364746094,6.1210808753967285
Europe,Western Europe,937026,EU-GEMAS,51.83639907836914,5.739999771118164
Europe,Western Europe,939307,EU-GEMAS,52.25299835205078,5.9720001220703125
Europe,Western Europe,940825,EU-GEMAS,52.24919891357422,5.962200164794922
Europe,Western Europe,1004638,EU-SPADE,52.28,5.672777777777778
Europe,Western Europe,1004950,EU-SPADE,51.834999084472656,6.394400119781494
Europe,Western Europe,1004960,EU-SPADE,52.14970016479492,6.0
Europe,Western Europe,1004998,EU-SPADE,51.87919998168945,5.724999904632568
Europe,Western Europe,1043359,NL-Alterra,51.84178267944147,5.763033086199862
Europe,Western Europe,1043414,NL-Alterra,51.84872593500576,5.48473003000238
Europe,Western Europe,1043437,NL-Alterra,51.95179702379335,6.2799998917712365
Europe,Western Europe,1043438,NL-Alterra,52.02127378968951,6.315550290398462
Europe,Western Europe,1043439,NL-Alterra,51.88610965746257,6.409792306850038
Europe,Western Europe,1043440,NL-Alterra,51.92130549918883,6.1692750325682395
Europe,Western Europe,1043442,NL-Alterra,52.25832382345286,6.105766840872823
Europe,Western Europe,1043443,NL-Alterra,52.03728303759915,6.505723015891941
Europe,Western Europe,1043444,NL-Alterra,52.11863727629245,5.978168456684717
Europe,Western Europe,1043446,NL-Alterra,52.20076299676038,6.034477778698324
Europe,Western Europe,1043452,NL-Alterra,51.888199716958894,5.792472877943635
Europe,Western Europe,1043453,NL-Alterra,52.25877941271585,6.119396011922248
Europe,Western Europe,1043465,NL-Alterra,52.378407507979595,5.974652930690879
Europe,Western Europe,1043471,NL-Alterra,51.900585080127975,6.215411726869212
Europe,Western Europe,1043473,NL-Alterra,52.40521938028812,6.00439923935644
Europe,Western Europe,1043492,NL-Alterra,52.244687487390784,5.6800436461661175
Europe,Western Europe,1043493,NL-Alterra,52.28018373346947,5.826815321162722
Europe,Western Europe,1043494,NL-Alterra,52.28007105513121,5.856121827941348
Europe,Western Europe,1043495,NL-Alterra,52.4498331472039,6.063862580416263
Europe,Western Europe,1043497,NL-Alterra,52.03497804815366,6.073614888303684
Europe,Western Europe,1043508,NL-Alterra,51.987466611723036,5.739505227093674
Europe,Western Europe,1043509,NL-Alterra,51.99022231789239,5.719144409739948
Europe,Western Europe,1043510,NL-Alterra,51.970230719160206,5.788848151133059
Europe,Western Europe,1043512,NL-Alterra,52.17526430836403,5.757069589710335
Europe,Western Europe,1043519,NL-Alterra,52.20482343488615,5.788035922737804
Europe,Western Europe,1043520,NL-Alterra,51.95188508618982,6.206175273395131
Europe,Western Europe,1043521,NL-Alterra,52.10432759786435,5.963747804540005
Europe,Western Europe,1043522,NL-Alterra,52.06868602565271,5.896199113204347
Europe,Western Europe,1043532,NL-Alterra,52.046005746082464,5.942588221387407
Europe,Western Europe,1043533,NL-Alterra,52.043936959064645,5.788051812613437
Europe,Western Europe,1043566,NL-Alterra,52.47105762154436,6.092371168306506
Europe,Western Europe,1043587,NL-Alterra,51.89160083384448,6.044113555216171
Europe,Western Europe,1043620,NL-Alterra,52.03994743239311,6.209826381610751
Europe,Western Europe,1043621,NL-Alterra,51.83393287134585,6.3937354398722
Europe,Western Europe,1043623,NL-Alterra,52.05661846867336,6.0773004393514105
Europe,Western Europe,1043638,NL-Alterra,52.117610990115374,6.384060344520573
Europe,Western Europe,1043644,NL-Alterra,52.00004075633502,5.868916034043594
Europe,Western Europe,1043669,NL-Alterra,51.93276197170273,5.695455058943515
Europe,Western Europe,1043676,NL-Alterra,51.82709423825635,5.436522987160779
Europe,Western Europe,1043677,NL-Alterra,51.99217435907258,6.228751338156893
Europe,Western Europe,1043678,NL-Alterra,52.04847991873956,6.143797528670743
Europe,Western Europe,1043713,NL-Alterra,51.94872298048873,5.8675651416097665
Europe,Western Europe,1043718,NL-Alterra,51.919463185820746,6.091037141423828
Europe,Western Europe,1043721,NL-Alterra,51.967561542028776,5.652476178687103
Europe,Western Europe,1043723,NL-Alterra,51.89163547263586,5.3516117726214665
Europe,Western Europe,1043743,NL-Alterra,52.16324078761394,6.5172114574599505
Europe,Western Europe,1043744,NL-Alterra,51.96038379304289,5.894679033957757
Europe,Western Europe,1043755,NL-Alterra,52.47483940984441,6.027816982476475
Europe,Western Europe,1043791,NL-Alterra,51.93403917432385,6.4581438693886115
Europe,Western Europe,1043795,NL-Alterra,51.90200151569097,5.9814933777062445
Europe,Western Europe,1043796,NL-Alterra,51.87728151034946,5.840879352427591
Europe,Western Europe,1043797,NL-Alterra,51.88530427207969,5.616695154056289
Europe,Western Europe,1043798,NL-Alterra,52.074578953736285,6.1386960484576365
Europe,Western Europe,1043815,NL-Alterra,52.36117635260774,5.798256948589491
Europe,Western Europe,1043831,NL-Alterra,52.07871436682097,6.422947030479737
Europe,Western Europe,1043833,NL-Alterra,51.794942564650114,5.793083826669608
Europe,Western Europe,1043841,NL-Alterra,52.235800550679436,5.636067834598172
Europe,Western Europe,1043844,NL-Alterra,52.14513769387515,5.88392230216291
Europe,Western Europe,1043845,NL-Alterra,52.11596455510221,6.263203600867022
Europe,Western Europe,1043847,NL-Alterra,52.15122870187573,6.351568474676469
Europe,Western Europe,1043855,NL-Alterra,52.00271527776022,6.131810973164188
Europe,Western Europe,1043863,NL-Alterra,51.81146569600635,5.930269436572976
Europe,Western Europe,1043887,NL-Alterra,52.00011733902099,5.7163052866684145
Europe,Western Europe,1043951,NL-Alterra,51.89118481019441,5.347689910672931
Europe,Western Europe,1043953,NL-Alterra,51.93651285653109,6.016850295952505
Europe,Western Europe,1043995,NL-Alterra,51.98141438476208,6.159405719681172
Europe,Western Europe,1043998,NL-Alterra,51.77478275456077,5.774066205363471
Europe,Western Europe,1107921,WD-ISCN,52.0,5.75
Europe,Western Europe,1108244,WD-ISCN,51.91669845581055,5.4166998863220215
Europe,Western Europe,1114660,WD-ISCN,52.13330078125,5.75
Europe,Western Europe,1115602,WD-ISIS,52.016666666666666,5.966666666666667
Europe,Western Europe,1115651,WD-ISIS,51.95,5.516666666666667
Europe,Western Europe,1115744,WD-ISIS,52.032222222222224,5.763055555555556
Europe,Western Europe,1115758,WD-ISIS,52.03861111111111,5.741944444444444
Europe,Western Europe,1115811,WD-ISIS,51.916666666666664,5.65
Europe,Western Europe,1115824,WD-ISIS,52.18333333333333,5.716666666666667
Europe,Western Europe,1115949,WD-ISIS,52.016666666666666,5.966666666666667
Europe,Western Europe,1115994,WD-ISIS,52.17583333333334,5.859166666666667
Europe,Western Europe,1116107,WD-ISIS,51.966906944444446,5.689726111111111
Europe,Western Europe,1116108,WD-ISIS,51.990613055555556,5.6509261111111115
Europe,Western Europe,1116109,WD-ISIS,51.971765,5.638358055555555
Europe,Western Europe,1124744,WD-WISE,52.03333333333333,5.683333333333334
Europe,Western Europe,1043795,NL-Alterra,51.90200151569097,5.9814933777062445
Europe,Western Europe,1108242,WD-ISCN,51.86669921875,5.25
Europe,Western Europe,1043831,NL-Alterra,52.07871436682097,6.422947030479737
Europe,Western Europe,1043493,NL-Alterra,52.28018373346947,5.826815321162722
Europe,Western Europe,1043995,NL-Alterra,51.98141438476208,6.159405719681172
Europe,Western Europe,1043951,NL-Alterra,51.89118481019441,5.347689910672931
Europe,Western Europe,1043844,NL-Alterra,52.14513769387515,5.88392230216291
Europe,Western Europe,1004950,EU-SPADE,51.834999084472656,6.394400119781494
Europe,Western Europe,1043649,NL-Alterra,52.079851588741306,6.660048778683933
Europe,Western Europe,1043716,NL-Alterra,51.935107337987105,5.559512922988639
Europe,Western Europe,1043674,NL-Alterra,51.93220446703495,5.265066791996666
Europe,Western Europe,1043659,NL-Alterra,51.99366275331206,6.662697317751144
Europe,Western Europe,1043791,NL-Alterra,51.93403917432385,6.4581438693886115
Europe,Western Europe,1108244,WD-ISCN,51.91669845581055,5.4166998863220215
Europe,Western Europe,1124638,WD-WISE,52.032222222222224,5.763055555555556
Europe,Western Europe,1043374,NL-Alterra,51.98697542967124,6.665127751403251
Europe,Western Europe,1043660,NL-Alterra,51.98215296783606,6.646356431984319
Europe,Western Europe,1043942,NL-Alterra,51.946393535796815,5.30211697110787
Europe,Western Europe,1124705,WD-WISE,51.916666666666664,5.65
Europe,Western Europe,1043508,NL-Alterra,51.987466611723036,5.739505227093674
Europe,Western Europe,940825,EU-GEMAS,52.24919891357422,5.962200164794922
Europe,Western Europe,1113886,WD-ISCN,52.08330154418945,5.466700077056885
Europe,Western Europe,1043521,NL-Alterra,52.10432759786435,5.963747804540005
Europe,Western Europe,934557,EU-FOREGS,52.173553466796875,6.122528076171875
Europe,Western Europe,1043497,NL-Alterra,52.03497804815366,6.073614888303684
Europe,Western Europe,1043793,NL-Alterra,51.808644752869455,5.298026536886261
Europe,Western Europe,1124844,WD-WISE,52.016666666666666,5.966666666666667
Europe,Western Europe,1043495,NL-Alterra,52.4498331472039,6.063862580416263
Europe,Western Europe,937026,EU-GEMAS,51.83639907836914,5.739999771118164
Europe,Western Europe,1043713,NL-Alterra,51.94872298048873,5.8675651416097665
Europe,Western Europe,1043359,NL-Alterra,51.84178267944147,5.763033086199862
Europe,Western Europe,1043533,NL-Alterra,52.043936959064645,5.788051812613437
Europe,Western Europe,1043998,NL-Alterra,51.77478275456077,5.774066205363471
Europe,Western Europe,1116110,WD-ISIS,51.935555,5.549381944444445
Europe,Western Europe,1043887,NL-Alterra,52.00011733902099,5.7163052866684145
scripts/python/wosis_graphql_q1.py
+ 6
− 6
View file @ 6bdfd0cf
import requests
import json
import pandas as pd
# GraphQL query
query = """
query MyQuery {
wosisLatestProfiles(first: 2) {
wosisLatestProfiles(first: 5) {
continent
region
countryName
......@@ -24,8 +25,7 @@ r = requests.post(url, json={'query': query})
print(r.status_code)
# Parse JSON
parsed = json.loads(r.text)
# Print JSON
json=json.dumps(parsed, indent=4, sort_keys=True)
print(json)
# Convert to pandas dataframe
df = pd.json_normalize(parsed['data']['wosisLatestProfiles'])
# print dataframe
print(df)
scripts/python/wosis_graphql_q2.py
+ 28
− 33
View file @ 6bdfd0cf
import requests
import json
import pandas as pd
# GeoJSON geometry
geomGelderland = {
"type": "MultiPolygon",
"coordinates": [ [ [ [ 5.177260142422514, 51.74291774914947 ], [ 5.126747881386732, 51.737828850498403 ], [ 5.137580867932065, 51.772905259431077 ], [ 5.014540023249575, 51.808984680959583 ], [ 5.031415073146523, 51.841084802107702 ], [ 4.993967909252922, 51.861222725420994 ], [ 5.062358224116345, 51.859362053527242 ], [ 5.180226727863164, 51.96744832651509 ], [ 5.236867149255078, 51.978757478459428 ], [ 5.321611332014112, 51.954919171164796 ], [ 5.486214078473083, 51.98382644510454 ], [ 5.627223829712356, 51.952386168324438 ], [ 5.550342661060417, 52.10541954546126 ], [ 5.459242995490565, 52.080225755481266 ], [ 5.514079463312799, 52.135923065932062 ], [ 5.439875615559026, 52.171197458274222 ], [ 5.44103943147957, 52.205693438951691 ], [ 5.393219147822698, 52.220626892173925 ], [ 5.404643399611359, 52.249630480909225 ], [ 5.533281176545358, 52.27274084169683 ], [ 5.587707385036856, 52.361454261431376 ], [ 5.787257137970521, 52.422573287061603 ], [ 5.876205471530124, 52.522025026941051 ], [ 5.925559518063968, 52.474057592745915 ], [ 6.027857569808684, 52.509606205409327 ], [ 6.099483437203417, 52.469970896552461 ], [ 6.130552948323514, 52.399978162269164 ], [ 6.078506385563601, 52.369523051161245 ], [ 6.066224466859907, 52.318839289847247 ], [ 6.163909067147507, 52.21749619292715 ], [ 6.38185154627214, 52.246112812566473 ], [ 6.492401220236633, 52.177371870181403 ], [ 6.671338986248984, 52.165683203635673 ], [ 6.662399005672591, 52.130167439615931 ], [ 6.760572413121598, 52.118779940206082 ], [ 6.687853003658449, 52.039856158091141 ], [ 6.832754328999235, 51.972938087693585 ], [ 6.721969582522561, 51.89606334135938 ], [ 6.683993990179909, 51.91757645733221 ], [ 6.472507886098918, 51.853823023864017 ], [ 6.390566170881016, 51.87396806966867 ], [ 6.401818441765064, 51.827262656663407 ], [ 6.117889496603739, 51.901659142837225 ], [ 6.166559884993931, 51.840721643435401 ], [ 6.063485632339608, 51.86545122678897 ], [ 5.962978284523374, 51.836913960582471 ], [ 5.946569966406273, 51.813479919592751 ], [ 5.992067051189349, 51.770245909123908 ], [ 5.943962150919553, 51.741816814422592 ], [ 5.893409336802974, 51.777852926426895 ], [ 5.765188291802036, 51.752789880063702 ], [ 5.638112608999517, 51.819025176083443 ], [ 5.493105254357093, 51.830750957327069 ], [ 5.403157084105017, 51.821611677731141 ], [ 5.357568231054432, 51.757890339715857 ], [ 5.300338754648935, 51.737287437014395 ], [ 5.177260142422514, 51.74291774914947 ] ] ] ]
}
# GraphQL query
query = """
query myQuery($first: Int, $offset: Int) {
wosisLatestProfiles(first: $first, offset: $offset) {
query MyQuery($geomGelderland: GeoJSON!) {
wosisLatestProfiles(
first: 3
filter: {layersExist: true, geom: {intersects: $geomGelderland}}
) {
continent
region
countryName
profileId
datasetCode
latitude
longitude
geomAccuracy
profileCode
geom{
geojson
x
y
}
}
}
"""
# GraphQL endpoint
url='https://graphql.isric.org/wosis/graphql'
url='http://localhost:8888/wosis/graphql'
new_results = True
first = 100
offset = 0
all_results = []
while new_results:
# Send POST request
r = requests.post(url, json={'query': query, 'variables': {'first': first, 'offset': offset}})
# Parse JSON
parsed = json.loads(r.text)
print(json.dumps(parsed, indent=4, sort_keys=True))
print(len(parsed['data']['wosisLatestProfiles']))
#exit()
if not 'wosisLatestProfiles' in parsed['data'] or len(parsed['data']['wosisLatestProfiles']) == 0:
new_results = False
print('-------------')
print('No more results')
else:
print('We have more results')
# update offset
offset = first+100
all_results.extend(parsed['data']['wosisLatestProfiles'])
# Print JSON
print(json.dumps(all_results, indent=4, sort_keys=True))
# Send POST request
r = requests.post(url, json={'query': query, 'variables': {'geomGelderland': geomGelderland}})
# Print status_code
print(r.status_code)
# Parse JSON
parsed = json.loads(r.text)
# Convert to pandas dataframe
df = pd.json_normalize(parsed['data']['wosisLatestProfiles'])
# print dataframe
print(df)
scripts/python/wosis_graphql_q3.py 0 → 100644
+ 59
− 0
View file @ 6bdfd0cf
import requests
import json
import pandas as pd
# GeoJSON geometry
geomGelderland = {
"type": "MultiPolygon",
"coordinates": [ [ [ [ 5.177260142422514, 51.74291774914947 ], [ 5.126747881386732, 51.737828850498403 ], [ 5.137580867932065, 51.772905259431077 ], [ 5.014540023249575, 51.808984680959583 ], [ 5.031415073146523, 51.841084802107702 ], [ 4.993967909252922, 51.861222725420994 ], [ 5.062358224116345, 51.859362053527242 ], [ 5.180226727863164, 51.96744832651509 ], [ 5.236867149255078, 51.978757478459428 ], [ 5.321611332014112, 51.954919171164796 ], [ 5.486214078473083, 51.98382644510454 ], [ 5.627223829712356, 51.952386168324438 ], [ 5.550342661060417, 52.10541954546126 ], [ 5.459242995490565, 52.080225755481266 ], [ 5.514079463312799, 52.135923065932062 ], [ 5.439875615559026, 52.171197458274222 ], [ 5.44103943147957, 52.205693438951691 ], [ 5.393219147822698, 52.220626892173925 ], [ 5.404643399611359, 52.249630480909225 ], [ 5.533281176545358, 52.27274084169683 ], [ 5.587707385036856, 52.361454261431376 ], [ 5.787257137970521, 52.422573287061603 ], [ 5.876205471530124, 52.522025026941051 ], [ 5.925559518063968, 52.474057592745915 ], [ 6.027857569808684, 52.509606205409327 ], [ 6.099483437203417, 52.469970896552461 ], [ 6.130552948323514, 52.399978162269164 ], [ 6.078506385563601, 52.369523051161245 ], [ 6.066224466859907, 52.318839289847247 ], [ 6.163909067147507, 52.21749619292715 ], [ 6.38185154627214, 52.246112812566473 ], [ 6.492401220236633, 52.177371870181403 ], [ 6.671338986248984, 52.165683203635673 ], [ 6.662399005672591, 52.130167439615931 ], [ 6.760572413121598, 52.118779940206082 ], [ 6.687853003658449, 52.039856158091141 ], [ 6.832754328999235, 51.972938087693585 ], [ 6.721969582522561, 51.89606334135938 ], [ 6.683993990179909, 51.91757645733221 ], [ 6.472507886098918, 51.853823023864017 ], [ 6.390566170881016, 51.87396806966867 ], [ 6.401818441765064, 51.827262656663407 ], [ 6.117889496603739, 51.901659142837225 ], [ 6.166559884993931, 51.840721643435401 ], [ 6.063485632339608, 51.86545122678897 ], [ 5.962978284523374, 51.836913960582471 ], [ 5.946569966406273, 51.813479919592751 ], [ 5.992067051189349, 51.770245909123908 ], [ 5.943962150919553, 51.741816814422592 ], [ 5.893409336802974, 51.777852926426895 ], [ 5.765188291802036, 51.752789880063702 ], [ 5.638112608999517, 51.819025176083443 ], [ 5.493105254357093, 51.830750957327069 ], [ 5.403157084105017, 51.821611677731141 ], [ 5.357568231054432, 51.757890339715857 ], [ 5.300338754648935, 51.737287437014395 ], [ 5.177260142422514, 51.74291774914947 ] ] ] ]
}
# GraphQL query
query = """
query MyQuery($first: Int, $offset: Int, $geomGelderland: GeoJSON!) {
wosisLatestProfiles(
first: $first,
offset: $offset,
filter: {layersExist: true, geom: {intersects: $geomGelderland}}
) {
continent
region
profileId
datasetCode
latitude
longitude
}
}
"""
# GraphQL endpoint
url='https://graphql.isric.org/wosis/graphql'
new_results = True
first = 100
offset = 0
all_results = []
while new_results:
# Send POST request
r = requests.post(url, json={'query': query, 'variables': {'first': first, 'offset': offset, 'geomGelderland': geomGelderland}})
# Parse JSON
parsed = json.loads(r.text)
# Add results to all_results object
all_results.extend(parsed['data']['wosisLatestProfiles'])
# for debugging
# print(json.dumps(parsed, indent=4, sort_keys=True))
# print(len(parsed['data']['wosisLatestProfiles']))
if not 'wosisLatestProfiles' in parsed['data'] or len(parsed['data']['wosisLatestProfiles']) == 0:
print('No more results')
# update new_results
new_results = False
else:
print('We have more results')
# update offset
offset = offset+first
df = pd.json_normalize(all_results)
# print dataframe
print('There are {} WoSIS profiles with layers inside Gelderland region'.format(df.shape[0]))
# Export dataframe to CSV
df.to_csv('wosis_gelderland.csv', index=False)
\ No newline at end of file
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment