Server side API

The next part in the equation. A little AHA! moment. After building the API dash for Covid for NZ I found that it was SLOW! I tried cleaning the code up a bit and that did help but it was still SLOW. I then thought about how plases like John Hopkins and how they were uploading heaps of data, their sites were slow but not that slow and they were pulling up volumes of data and rendering it.

The reason is they are using Server -side JavaScipt and a few tricks there to pull the data into a database and hold it there until it is called, so not using Client -side, where you are running and rendering all oin your browser on your PC.

So a bit like a wordpress site that has a database sitting as a back end, but it also makes API calls and pulls that data into the Database to make it ready for when you make a call to that website, so you are freshening the data in the database at regular intervals.

Node.js server

To make this magic happen you need to be running a server that is active and listening for calls for the website and also fetching API data regularly.

This makes sense to me, I now realise how they can get speed by running the server which will gather the data, thereby making server side JS API Fetch() functions and storing that data in a database, on that same server, ready to send it to the browser when requuested. So you can pre-process the data so that its in an appropriate form so you do not need to do much JS processing on the client side web browser, you just send static, pre-processed data to the webpage.

So all the mgic and heavy lifting happening in the background.

Following the tutorial 2.1 Server-side with Node.js – Working with Data and APIs in JavaScript

I installed Node.js and also choclety as well, that seemed to install the latest Python as well.

I then started up Node.JS is a subdirectory where I setup a directory with index.js in (copied from Github here) which is Code Train tutorial, and also a simple index.html file in a sub folder called “public”.

I use Explorer to go to directory and started command prompt:

Now I type in: node(ENTER) to start node.js, next I need to create the package.json file. This is the file that remembers all the dependancy libraries that you will be using with the node server. (A bit like tohe PIP and import that you use in Python to get and use packages.

We will want to load ” Express” package but first, seetup the package.json which is a configuration file ofor the project, can use npm to create for you. (NPM stands for Node Package Manager) :

>npm init

To run init you must be outside node, so to exit from that use >process.exit() [ENTER]

So I followed the video, basically selected all defaults and created package.json. Now I can use >npm install to install some packages, in this case Express which is a Node Framework for Webserver

>npm install express

This installs the package as subdirectory to folder (so only accessed by this programme) and any of its (Express’s) dependancy libraries, and also updates package.json. So the next time you start the project it knows what packages to load to make it run.

So we can now type the following to see if its running, and Node server is now listening, but to what?- Note , to get out of server running use Control “C” to exit.

const express = require(‘express’); const app = express(); app.listen(3000, () => console.log(‘listening at 3000’));

So we need it to find the html page to render. so we add:

app.use(express.static(‘public’));

After closing node and restarting and going to web browser and typing localhost:3000 this is what we get:

Video 2. Getting Geolocation

The Mozilla navigator object. The Navigator interface represents the state and the identity of the user agent. It allows scripts to query it and to register themselves to carry on some activities. A Navigator object can be retrieved using the read-only window.navigator property.

There are a few different properties that can be sent back, but this tutorial is using Geolocation of the device, in this case my PC, which is here!!

Navigator.geolocation Read only Returns a Geolocation object allowing accessing the location of the device.

Note- This only works on HTTPS websites, in this test, as we are not going over internet, (Server on laptop in tutorial) then its not an issue for example, but it means that whenb deploying it must have HTTPS.

So geolocation is available , so paste in some code from the docs page for geolocation that will call for that (as per vid above). But it happens asycronously, so has to get before it can return something, ( I think that is why there is a function inside as a “callback function”)

navigator.geolocation.getCurrentPosition((position) => {
  doSomething(position.coords.latitude, position.coords.longitude);
});

My actual location is 41°17’56.47″S, 174°47’50.75″E so a bit out. But a geolocation returned after first giving it permission to alow access to geolocation. You can get some other information too in the position object:

It was also quite slow in returning the information, in seconds rather than milliseconds. So, have ended vid 2 tut.

Vid 3 send data to Server from client

First off, I’m loading nodemon, this refires up the server if you change the code, so its always running, so a stop/start to refresh with update. Nodemon short for nodemonitor. the -g is for global install soit works in otherfolders/servers.

>npm install -g nodemon

This vid about Routing/ JSON parsing & Post with Fetch(). With a post request you need an endpoint, somewhere for that post to be stored and also a callback function to be able to look at the data you recieved and send back a promise/acceptance that it works.

The index.js (server side)The request holds all the information about that request, the body (what you want returned) and headers (what type of file structure) and other such stuff. The response is what I send back to the client, aknowledging that the request recieved, otherwise it could be sending stuff that does not get there.

The index.html (client side) needs to do a post fetch() to send something . The info for what to put in fetch() can be found here. Result below, its sent the Lat/long across.

So its coming across but now the server needs to understand it so it needs to parse the data using app.use(express.json({ limit: ‘1mb’ })); so using Expresses json parser. So we’ve received something. Now to aknowledge that its beeen recieved, we’ll send back what was sent.

He recodes some of it by using async and then await on client side where you can control how data presented, I just followed along and got the same result:

The handshake is complete and data is exchanged.

Adding a database

Why do you need a database? For persistance, to store data over a long period, even if server goes down.

He is using NeDB database. Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, 100% JavaScript, no binary dependency. Also it is a subset of mongoDB so noSQL, uses documents , so could store JSON. Install just >npm install nedb

Then need to load a datastore. Basically just followed along. All seems ok.

Querying a database

I sort of petered out at this point, later he uses Web Cam and takes image and stores it in Database (as a text string). I wasn’t that interested in that part. I downloaded the code he has on GitHub here.

I then started to watch the 3rd part of the series which allows server to make the API call and also a method to conceaql your API key from external sources. And then the deployment of a node.js server.

End comment

I really like his style and he has interesting projects, more focuissed on concepts and functionality rather than pretty content. Also reasonably short and intense and also shares the code.

I found a few issues as I was following but on going bck over what he’d done I pretty much got what he got. A little confused on the post(‘/api’, opetions) does the ‘/api’ mean you need a directory for that and you need to make one? I didn’t really fgure that out.

On reflecting on the tutorials what excited me was the api calls from the server ( series 3 vids) and the linking to the database. I’d need to run the node.js as a server on an instance somewhere and then make it an https site to then use it as a tracker for the mobile phone. If I was to try and use this as tracking log of where I’d got to.

Servers node.js or windows iis

As I started to think about getting node.js to run somewhere I watched a couple of videos on YT where they spun up an instance on a cloud and ran it from that. Then I thought, well can I spin up an instance of node.js this on my Windows server? Then I thought, surely thats what the Windows server is doing, so why cant I make API calls from my windows server? Then I thought about WordPress and what its doing with php and querying the database, it can also presumably make an external call to an api too.

I found this video of an example of a wordpress website calling external data:

The front image is of code from part 3 that I haven’t done yet but its interesting that it has got the location accurately onto leaflet map. Very impressive.

I need to do more research on server side querying of API, I can see this is an important feature for getting and receiving data from IoT devices. So getting info, storing info and rendering info is definitely something useful and powerful that I need to learn more of.

The API journey so far has been interesting. The issue is JS, Python, PHP or what?