README.md 4.68 KB
Newer Older
1
# WSM
2

Luís de Sousa's avatar
Luís de Sousa committed
3
World Soil Museum - REST API
4

5
## App structure
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

REST APP is on [appWSM.py](./appWSM.py) and then calling modules like `resources`. This app is based on:

- Flask
- Flask_restfull

App requires access to ISRIC database to retrieve profile information.

This app was build before the openapi standard and generic request and information can be found at the URL root level: `https://rest-wsm.isric.org/` Please consult the URL for the following examples:

- GET monolith pictures
- GET content filter (WRB classes, country)
- GET limit number of requested pictures

## Database

Mendes de Jesus, Jorge's avatar
Mendes de Jesus, Jorge committed
22
23
24
App accesses ISRIC database on `scomp1270.wurnet.nl`.

**NOTE:** To copy files check section below  [Monolith picture location](#monolith-picture-location)
25

26
### DB access configuration / Data path
27

28
DB options and monolith path location is picked up from env variables on script: [common/secret.py](./common/secret.py).
29

30
31
32
33
34
35
36
37
```bash
DB_HOST=scomp1270.wurnet.nl
DB_NAME=isric
DB_USER=hass005
DB_PASSWORD=<removed>
DB_PORT=5479
MONOLITH_PATH=/data/monolith
```
38
39
40
41

### DB queries

 In the code we have the following queries:
42

43
44
45
46
| code path         | method/function        | SQL                          |
|:-----------------:|:---------------------:|:-----------------------------:|
| common/filters.py | filter_classification()|select * from web_isis.vw_isis_profiles_with_classification_geo_img_rest where classification ilike %s"|
| common/filters.py | filter_classification() |select * from web_isis.vw_isis_profiles_with_classification_geo_img_rest where classification ilike %s|
Luís de Sousa's avatar
Luís de Sousa committed
47

Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
48
**NOTE:** Views need to be refreshed when new data is added to the upstream tables
Luís de Sousa's avatar
Luís de Sousa committed
49

50
### Monolith picture location
51
52
53
54
55

Monoliths used by REST are stored on ISRIC data volume, the access to this volume is very restricted and based on IP. Rserver (scomp1457.wurnet.nl), on this server users haas005, turku001 and ruipe001 have a symbolic link to the monoliths.

There the monolith pictures can be updated or modified

56
### Monolith file copy
Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
57
58
59

On the Rserver is it better to copy the monoliths from workspace scratch into `ISRIC_Monoliths`. On the home folder of user the command would look like:

60
```bash
Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
61
62
63
turdu001@scomp1457:~$ cp -rv ./ISRIC_Workspace/scratch/Upload_ToRest/combined_processed/* ./ISRIC_Monoliths/ 
```

Mendes de Jesus, Jorge's avatar
Mendes de Jesus, Jorge committed
64
65
The FB-IT mounts are not reliable, check if "ISRIC_Monolith" is accessible, it not mount it:

66
```bash
Mendes de Jesus, Jorge's avatar
Mendes de Jesus, Jorge committed
67
68
69
70
# mount all volumes from /etc/fstab
sudo mount -a
```

Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
71
72
In case of batch file permission it is advisable (NOTE: Only sudo can do it):

73
```bash
Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
74
75
76
77
find /mnt/ISRIC_Monoliths -type d -exec chmod 777 {} \; #for directories
find /mnt/ISRIC_Monoliths -type f -exec chmod 666 {} \; #for files
```

78
## Monolith image update and implementation
79
80
81
82
83
84
85
86
87
88
89

To update monolith images, it is necessary:

- Process images to have the different sizes
- Add them to the ISRIC data volume
- Update DB (web_isis.profiles_with_rest_images?) and views

It is advisable to use a rsync software when moving files around.

REST API doesn't make any assumptions on files and will crawl the fold structure for the pictures. It maybe necessary some code changes if there are to many changes.

90
## Local Deployment
91
92

This API can be deployed using a docker container. The code resides in the
Luís de Sousa's avatar
Luís de Sousa committed
93
host system, together with the monoliths images to be served. These images
94
must be accessible in a folder named `monoliths` at the folder root (same level
95
as this file). This code folder is then mapped into the container at startup.
Luís de Sousa's avatar
Luís de Sousa committed
96
97
98
99

The Dockerfile present in this project sets up the API with Gunicorn and nginx.
To build the container image change to the project folder and run:

100
101
102
`docker build -t wsm.rest .`

The resulting image is named `wsm/rest`.
Luís de Sousa's avatar
Luís de Sousa committed
103

104
To run the image it is necessary to map the right environment variables to connect to the database as well as mount the source code folder and bind port 8000 (where gunicorn listens):
Luís de Sousa's avatar
Luís de Sousa committed
105

106
Docker run command with envfile:
107

108
```bash
109
110
111
112
docker run -dit \
   --env-file /path/to/envfile \
   -v /data/monoliths:/data/monoliths \
   --name wsm.rest -p 8000:8000 wsm.rest
113
```
114

115
Docker run command with variables inline:
116

117
```bash
118
119
120
121
122
123
124
125
126
docker run -dit \
    -e DB_HOST=scomp1270.wurnet.nl \
    -e DB_NAME=isric \ 
    -e DB_USER=haas005 \
    -e DB_PASSWORD=<REMOVED>
    -e MONOLITH_PATH=/data/monolith 
    -e DB_PORT=5479 \
    -v /data/monoliths:/data/monoliths 
    --name wsm.rest -p 8000:8000 wsm.rest
127
```
Luís de Sousa's avatar
Luís de Sousa committed
128

129
Your container will now be available on [localhost:8000](http://localhost:8000)
130

131
## K8s deployment
132

133
- `ENV` variables are set as k8s secrets on the [ISRIC deployment repository](https://git.wur.nl/isric/ict/k8s-deployments)
134

135
- Application follows the standard k8s deployment.
136

137
- TODO: Write about semanatic tagging
138

139
## Image processing and metadata
140

Jorge S. Mendes de Jesus's avatar
Jorge S. Mendes de Jesus committed
141
Please consult the specific folder **[README](imageProcessing/README.md)**