Esteban Herrera has more than twelve years of experience in the software development industry, and having worked in many roles and projects, he has found his passion in programming with Java and JavaScript. Nowadays, he spends most of his time as an independent software developer, learning new things, writing articles and books, and teaching programming.
In this tutorial, we’ll create a web application that verifies the existence of computer files via time stampled transactions in the bitcoin blockchain (eg. Proof of Existence and Origin Stamp). We’ll use Node.js, Tierion, RethinkDB, PubNub, and jsreport-core.
In a few words, these sites allow you to store the hash of a file in Bitcoin’s Blockchain so that anyone can certify that the file existed at a specific time.
This has the following advantages:
It demonstrates data ownership without revealing actual data.
It checks for document integrity.
Using the blockchain, you can certify the existence of the document without the need of a central authority.
In practice, this decentralized proof can’t be erased or modified by anyone.
It’s anonymous, so others cannot identify you or your data.
Proof of Existence
For more information, check out a thorough explanation of proof of existence that covers more sites dedicated to enabling it.
Using our application, we will be able to upload a file by dragging and dropping it to a marked area. Then, its SHA256 digest will be calculated and sent to the Blockchain using Tierion Hash API.
Notice how the list of last documents registered is updated in realtime; we’ll be using PubNub’s Storage and Playback to gain realtime syncing.
The hash of the file and the Blockchain receipt given by Tierion is stored in RethinkDB, a NoSQL database with real-time capabilities. When Tierion alerts us that a block of hashes has been processed, we update the verification page (and the list of verified documents) in real-time, using RethinkDB’s changefeed:
As you can see, once the hash is anchored to the Blockchain, we can check it using sites like Block Explorer or Blockchain.info and get the receipt in PDF format using jsreport-core.
You can also verify that a file has been certified in the Blockchain by selecting the Verify option:
In case you stumble or want to skip ahead, the tutorial source code is available on Github.
The Blockchain
Before diving into the code, let’s explain what is the Bitcoin’s Blockchain (referred to as the Blockchain).
Wayne Vaughan, founder and CEO of Tierion, launched a challenge during the 2016 Consensus Hackathon: What is the Blockchain? The winner was Lilia Vershinina, who answered:
“The Blockchain is a trust layer for the Internet”.
Interesting. Let’s see if we can clarify this statement. By the way, you can see the whole story behind this (as well as other answers) here.
One of the main philosophies behind Bitcoin is decentralization. Rather than trusting a single party (like a government or an institution) to manage and produce currency, Bitcoin is a peer-to-peer system with no central point of failure. As such, the blockchain is a public database of Bitcoin transactions distributed across a global network of nodes.
The blockchain comprises all Bitcoin transactions, which are represented as a chain of blocks linked together, since 2009, the year of its inception. These transactions span from the genesis block, or the first block of the blockchain, to the latest block.
When Bitcoin transactions are broadcasted, miners pick them up locally to form a Bitcoin block. Miners verify this block using a designated mathematical formula.
A hash is used to represent the verification of the block. This hash is computed using this block and the hash of the previous block, thereby forming a chain of hashes. You can see the concept of trust here; if any transaction is modified or voided, the hash, as well as the rest of the chain, becomes invalid.
For this reason, as the chain grows and more copies of it are kept by miners, the more difficult it becomes to modify the blockchain. Moreover, the increasing mining difficulty causes blocks to be generated about every ten minutes, a rather slow pace.
This is a (very) high-level overview of the blockchain. There are countles articles and books that explain and discuss it. If you want to know more, try this search.
Now that we understand blocks, blockchain, Bitcoin, and mining, let’s dive into our web application.
Requisites
Tierion
We’ll need a Tierion free account. You can signup here.
In addition to its Hash API, Tierion offers datastores to record any data in the blockchain. Depending on your plan, you can use a certain number of datastores and records per month. However, we’ll be using the Hash API (that just record hashes in the blockchain), which at the time of this writing, it’s free to use with some limitations.
When your account is ready, go to your Dashboard and in the API tab, copy your client secret, we’ll need it later:
RethinkDB
You’ll need to install RethinkDB server. The easier way to install it is with one of the official packages.
This tutorial won’t cover the basics of RethinkDB, but you can consult RethinkDB’s ten-minute guide or my getting started guide if you haven’t work with it.
PubNub
Go to https://admin.pubnub.com/#/register and sign in. Create a new account if necessary. When you log in, you will be presented with this screen:
You can either create a new app or use the demo app. When you click on it, you’ll something like the following:
Save your publish and subscribe keys because we’ll need them later. Now click on the keyset panel, and go to the Application add-ons section to enable the Storage and Playback. PubNub’s Storage and Playback will let us determine the time period we want the messages to be stored. Enable Stream Controller add-ons as well. Finally, save the changes:
We’ll be using the brand-new PubNub SDK for Javascript version 4, which is a little different than version 3. Here’s the Node.js documentation and the Javascript (Web) documentation.
Ngrok
Tierion will trigger a webhook (similar to a callback) when a block of hashes has been anchored in the blockchain. An HTTP request will be made to our server, so we’ll need to deploy our application on the cloud or keep it locally and use a service like ngrok.
Ngrok proxies external requests to your local machine by creating a secure tunnel and giving you a public URL.
ngrok is a Go program, distributed as a single executable file (no additional dependencies needed). For now, just download it from https://ngrok.com/download and unzip the compressed file.
Node.js
You’ll also need Node.js and npm installed. You can download an installer for your platform here.
Requirements met, we can start on the app.
Setting up the project
Directories and Dependencies
Create a new directory and cd into it:
Now create a package.json file either with:
Or to accept the default values:
We will use the next dependencies:
hashapi-lib-node (to use the Tierion Hash API)
rethinkdbdash (to use RethinkDB)
pubnub (to use the PubNub API)
jsreport-core (to create the receipt)
jsreport-jsrender (as the template library to create the receipt)
jsreport-wkhtmltopdf (to create the receipt in PDF)
express (as the web framework)
ejs (as the template library)
body-parser (to parse incoming request bodies)
Add them with the following npm command:
The project will have the following directory structure:
The config.js file stores all the important configuration (use your own keys in place of the Xs):
Start up RethinkDB
Start RethinkDB with this command:
Go to the Data Explorer section of your RethinkDB’s dashboard and execute the following commands:
These commands will create the database and tables of the application. You can leave the server running in the background while we work on the rest of the application.
Start up ngrok
Now, in a new terminal window, navigate to the directory where you unzipped ngrok.
We’ll start ngrok by telling it which port we want to expose to the Internet. In our example, the port 3000:
Alternatively, if you’re on Windows:
Now, you should see something like this:
See that URL in the Forwarding row(s) with the ngrok.io domain? That’s your public URL. Yours will be different; ngrok generates a random URL every time you run it.
Change the url property on the config.js file to that URL.
This URL is not permanent. If you restart ngrok, it will give you another URL.
You can specify a subdomain. For example, to get the URL http://app.ngrok.io use the command:
However, this requires a paid plan. You can get more info and acquire the paid version on this page.
Nevertheless, as long as you don’t stop or restart ngrok, your public URL won’t change. Let’s leave ngrok running for now.
Next, let’s review the server-side code starting with the Tierion Hash API.
Tierion Hash API
Tierion Hash API is simple. First, you send your email and password to get a token that you must include in all requests. Next, you submit the token to get a receipt ID, which you’ll use to get the Blockchain Receipt about ten minutes later.
With Block Subscriptions there’s no need to check constantly for the receipt, a callback will alert you when a block of hashes has been processed and the corresponding receipts have been generated.
This is a REST API, or a Representational State Transfer Application Program Interface. Luckily, Tierion offers a client library for Node.js that makes things easier.
There are only two tricky parts when using this Block Subscription:
To authenticate the Block Subscription callback, we have to access the request raw body.
Block Subscriptions are not automatically deleted after a certain number of failed requests; they live indefinitely, so we have to manage their creation and update them on our side.
Let’s start by creating the server.js file with a standard Express configuration with EJS as the template engine:
To make the raw body available to in our request objects, we can configure body-parser like this:
This way, rawBody will contain the raw body of the request, for example:
For the Block Subscriptions, we’ll store the subscription in the subscription table so we can do something like this:
Check if the table has a subscription ID (from previous requests).
If there’s an ID stored, try to get a reference to the subscription:
If the reference is invalid, create a new subscription and update the ID for future requests.
If the reference is valid, check if the URL where the callback request is received is still the same:
If it’s the same, we’re done!
If it’s different, update it.
If there’s no ID stored, create a new subscription and save the ID for future requests.
Let’s translate this into code using the following approach:
First, we setup the Hash API client and the block subscription.
Then we configure the routes of the application by passing a reference to the Hash API client.
Then, we start the server.
Finally, we set up RethinkDB’s changefeed to listen for changes in the status of the receipts.
Create the models/subscription.js file. This will contain the code for the Block Subscription set up. The first part is:
It creates the variables required to do the work.
For this project, instead of using the official RethinkDB driver for Node.js, we’ll be using rethinkdbdash. Now, the connections are managed by the driver with a connection pool, so we don’t need to call r.connect or pass a connection to the run function.
Let’s create a function to create a block subscription. It looks like this:
The createBlockSubscription function configures a new Block Subscription callback to our server (with the URL variable). If it’s created successfully, we update the table in our database with the returned ID of the subscription. When this operation is completed, a callback will be executed.
Now, the function to setup the block subscription:
First, we authenticate to Tierion’s API. Then, we get the ID of the subscription from the database. If no ID exists, we create a subscription. Otherwise, the ID’s validated according to the plan we laid out before.
Now, if we update the last part of server.js, it should look like this:
But before starting the server, let’s configure the routes of the application. This will the topic of the next section.
The routes
Our application will have five routes:
/ will render the main page
/verify/:hash will render the verification page for the hash passed as a parameter
/hash to register a new hash
/tierion will receive the block subscription callback (you already saw it in the previous section)
/pdf/:id will render the receipt in PDF format
The code of these routes is simple, so it’s better to keep them all in one file. They’ll use the Hash API client, so let’s pass it as a parameter. Modify the last part of server.js as so:
In the routes/app.js file, enter these definitions:
First, we require() the modules that we’re going to use. We’ll see the code of the models/receipt.js file in the next section.
Then, we load the modules used by jsreport to render a PDF, (jsreport-wkhtmltopdf) and (jsreport-jsrender), so we can initialize the library.
The following line will read (in a synchronous way) the HTML of the file that will serve as the template for the PDF generated by jsreport:
Notice the use of __dirname to build the absolute path to the file. In Node.js, you should be very careful with relative paths. The only place where you can use relative paths reliably is in require() statements.
The next function will validate that the request received by Tierion’s block subscription really comes from Tierion:
Every callback includes a header containing the HMAC-SHA256 signature of the request. We have to calculate the signature using the raw body of the request and our account’s Client Secret. If the calculated and the provided signatures match, we have a valid request.
Now the routes. The first one simply shows the main page of the site:
The next one will get the hash from the database through the model object, and it will render the verification page on success
To register a new hash, we have to check if the hash the user is sending has already been registered. If not, we send it to Tierion and store a receipt object with the information we got at this point:
To keep things simple, the application will only support time in UTC (Coordinated Universal Time). The timestamp returned by Tierion is a UNIX timestamp, which represents the number of seconds since January 1, 1970, 00:00:00 UTC. Since the argument of the Date object in Javascript is the number of milliseconds since the Unix epoch, we have to multiply it by 1000.
The next route (for the Tierion callback) uses the validateRequest function showed before and simply marks the receipts once they’ve been sent to the blockchain. Marking will trigger RethinkDB’s changefeed to get the actual receipt objects and send the updates using PubNub (more on this in the next section):
Notice that we’re using rawBody to authenticate the request and a signature provided by Tierion. A request from Tierion looks like this:
Finally, the route to present the PDF receipt looks like this:
It gets the receipt object by its ID from the database and uses jsreport-core to generate the PDF file from the HTML template with jsrender.
In the next section, we’ll review the receipt model.
The core of the application
The core of the application is the models/receipt.js file. It contains the functions that interact with RethinkDB and PubNub to implement the functionality related to the blockchain receipts provided by Tierion.
The first part requires the modules used by the functions and creates the PubNub and RethinkDB objects:
The first function sets up RethinkDB’s changefeed, which will get the receipt object from Tierion, save it to the database, and publish the objects so the pages presented to the user can be updated in real-time:
If you’re curious, this is how a receipt object (version 2.0) from Tierion looks like:
Now that we have this function, let’s update the last part of server.js, so it can be called upon server initialization:
Back to receipt.js, this changefeed is triggered when sent_to_blockchain is set to true (and the record doesn’t have a blockchain_receipt). This is done by the markReceiptsAsSent function, called when Tierion’s callback is received:
Using RethinkDB’s time and date features, we mark all the records between the timestamps of processed hashes provided by Tierion.
To save a receipt object, we first clone the original object with the Object.assign() function, so we can safely change the timestamp property to use RethinkDB’s time type, insert it into the database, and publish an update to the registered document channel (more on this in the next section):
The other two methods on this file just get a receipt object by its hash and ID respectively:
Now let’s review the code of the HTML pages.
The index.html page
Here’s the complete code for the index page template:
It’s a simple HTML page. Before the <script> tags, we define a template that contains the HTML to add to the lists of the latest registered and confirmed documents:
About the Javascript files, we’re going to focus only on the essential parts. You can see the complete code for the Javascript (and CSS files) on the Github repository for this application.
For the public/js/index.js file, the important part is where the hash of the file is generated. We’re going to use the FileReader and cryptographic libraries of Javascript.
First, we test if the browser supports them. If that’s not the case, an alert is presented:
To read the file when it’s dragged or chosen, we have the following code:
One downside is that this approach won’t work with large files. Large generally means in the 200MB+ range, but the definition depends on your computer memory. A more robust implementation would read the file in blocks using workers. However, that’s a little complicated and beyond the scope of this tutorial.
Then, to calculate the hash (as a SHA256 digest), we use the digest function, which returns the hash as an ArrayBuffer. Then we convert to its hexadecimal (HEX) representation:
Now we have a HEX representation of the the original hash.
The next section will cover using PubNub’s Storage and Playback functionality to get the list of the latest registered and confirmed documents.
PubNub’s Storage and Playback and Multiplexing features
PubNub’s Storage and Playback feature allows you to store messages so they can be retrieved at a later time.
In the setup and save functions of the receive module, we saw how you can publish to channels using version 4 of PubNub SDK.
Notice how you can indicate not to storage a message with storeInHistory: false (when the storage add-on is enabled, by default, all messages in all channels are stored).
setup function:
save function:
This way, on the public/js/pubnub-index.js file, we create some variables and the PubNub object:
With multiplexing, we can subscribe to the channel for registered and confirmed documents:
Multiplexing means listening to multiple data streams on a single connection. Notice that our addListener function must come before our subscribe function.
Inside addListener, we use the subscribedChannel property to know from which channel the message comes from. The actual message is in the message property.
This is for listening to new documents after the page is fully loaded. For the initial list when the page just loads, we call the history function on both channels to retrieve the last five new documents on the channels (using reverse and count):
Next, we’ll move onto the verification page.
The verification page
The verification page is simple too:
After the .container div, you can find the templates to add the blockchain information. If a user comes to this page after the receipt is generated, the information is added on page load. Otherwise, PubNub is set up with the function setupPubNub, which allows it to listen for updates.
The code of this function can be found on public/js/pubnub-verify.js:
Here, we’re just subscribing to the channel with name equals to the hash we’re verifying to add the blockchain information when a message is received.
This way, the page of a verified hash looks like the following:
And the receipt in PDF format looks like this:
The template for the PDF receipt is similar to this page, the only significant difference is the QR code generated with QRCode.js. To create a QR code, use the JavaScript below:
And that’s it, the whole application. Remember that the entire code is on Github in case you want to review the files not covered here or if you missed something.
How do I verify the existence in the Blockchain?
I guess you may be thinking, but what are these merkle root, transaction id, and proof values that Tierion is returning? How can all this act as a proof of existence?.
Well, first of all, the file’s SHA256 digest is unique. You need to have the exact same file to generate the same hash. (Just try checking the hash of a file, and then adding a space to see how the new hash is completely different). This proves that the file existed at least as early as the time the transaction was confirmed. Otherwise, the hash we used would not have been generated.
About Tierion’s receipt:
The targetHash value is the hash of the record you wish to verify.
This and other recorded hashes are combined into a Merkle Tree.
With the Merkle Tree, a merkleRoot is calculated and inserted into the blockchain.
With a Merkle tree, you can prove that something belongs to a set, without having to store the whole set.
To validate a receipt, you must confirm that targetHash is part of a Merkle Tree, and the tree’s Merkle root has been published to a Bitcoin transaction.
Receipts generated from Tierion conform to the Chainpoint 2.0 standard. You can know more about this standard and how a receipt can be verified in this whitepaper, but in summary here’s the procedure:
Concatenate targetHash and the first hash in the proof array. The right or left designation specifies which side of the concatenation that the proof hash value should be on.
Hash the resulting value.
Concatenate the resulting hash with the next hash in the proof array, using the same left and right rules.
Hash that value and continue the process until you’ve gone through each item in the proof array.
The final hash value should equal the merkleRoot value if the proof is valid, otherwise, the proof is invalid.
If your targetHash was the only element in the Merkle Tree, it would be equal to merkleRoot.
Let’s see an example. To verify the hash of this receipt (691b6e0b3db43e22466734f7ec4b15d17687d5d8a92212fb99691e454cb9d9c4):
We concatenate the targetHash value with the right value of proof in the following way:
Then, we get th