Tag: cache

Caching large data with Redis

February 23rd, 2023

Reading Time: 2 minutes
Redis is an open-source, in-memory data structure store. It is commonly used as a database, cache, and message broker in web development. Redis stores data in key-value pairs and is known for its speed and performance. It can handle large workloads and support concurrent operations, making it a popular choice for caching large data. With its ability to store data in memory, support for various data structures, and built-in features like data expiration, Redis is a powerful tool for improving the performance and scalability of websites and applications.

Caching is a key component of any application’s architecture. It enables developers to create fast, responsive apps that are able to handle large workloads and support concurrent operations. Redis, an in-memory data structure store, is an increasingly popular choice for caching large data.

This post will explore how Redis works and why it is a great choice for caching large data. Redis stores data in key-value pairs and uses a data structure that is well-suited for caching.

Caching is an important aspect of web development that helps to speed up the performance of websites and applications. One of the most commonly used caching systems is Redis. This in-memory data structure store can be used as a database, cache, and message broker, making it an ideal choice for caching large amounts of data.

When it comes to large data sets, traditional databases can be slow, leading to poor performance for users. Caching this data in Redis can help to speed things up dramatically. Redis stores data in memory, making it incredibly fast compared to disk-based databases like MySQL. This means that when the same data is requested multiple times, Redis can quickly retrieve it from memory, rather than having to go through the slower process of retrieving it from disk.

Another advantage of using Redis for caching large data is its ability to support multiple data structures. Redis supports a variety of data structures such as strings, hashes, lists, sets, and sorted sets. This makes it an ideal choice for storing complex data structures that can be quickly retrieved and manipulated. For example, if you have a large product catalog, you can store it in Redis as a hash, with each product represented as a key-value pair. This makes it easy to quickly retrieve individual products, or to search the catalog for specific products.

Redis also has built-in support for data expiration, which means you can set a time limit for how long data should be stored in the cache. This is especially useful for data that changes frequently or becomes stale over time. By setting an expiration time for this data, you can ensure that it is automatically removed from the cache and updated from the database when necessary.

In terms of scalability, Redis is highly scalable and can handle large amounts of data without slowing down. It can be easily set up as a distributed cache, which allows you to split the data across multiple nodes, providing greater scalability and reliability.

In conclusion, caching large data with Redis can have a significant impact on the performance and scalability of your website or application. With its in-memory storage, support for multiple data structures, and built-in support for data expiration, Redis is an ideal choice for caching large amounts of data. Whether you’re looking to speed up your website, improve scalability, or store complex data structures, Redis is definitely worth considering as your caching solution.

November 10, 2023How to publish Salesforce platform events from Salesforce API from Node.js
November 8, 2023Advantages of using map over for loops in JavaScript
November 8, 2023Different ways to use for loops in Node.js
October 13, 2023Switch between different Node versions
October 7, 2023How to connect to a Redshift Database in Node.js
February 24, 2023Node.js with Express and MySQL scaffold

How to use node-cache to accelerate queries

January 29th, 2023

Reading Time: 2 minutes

Node-cache is an easy to use module that provides an in-memory caching system for Node.js applications. It helps speed up query operations by caching the results of often used queries, making it faster for subsequent queries.

Using node-cache is easy. To get started, install it with npm.

npm install node-cache

Database access is usually expensive so you should optimize roundtrips as much as possible.

You can avoid queries by caching at multiple levels:

Using a CDN
Caching at browser level

Another option it to cache right before running queries. It can be easy as “ok, I’m about to run this query for these parameters, did I execute it recently?”

How to use node-cache module to cache queries

We are going to build a small piece of code to show how to use node-cache module.
It will be used to cache a SQL query result but it can be used to cache any other result.

      
const { Pool } = require('pg');
const NodeCache = require('node-cache');
const crypto = require('crypto');

const log4js = require('log4js');

const queryCache = new NodeCache();
const logger = log4js.getLogger('db_helper');
logger.level = 'info';

let rejectUnauthorized = false;
if (process.env.NODE_ENV === 'development') {
  rejectUnauthorized = false;
}

// more options: https://node-postgres.com/api/client
const timeout = process.env.DB_TIMEOUT || 1000 * 10;
const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  statement_timeout: timeout,
  query_timeout: timeout,
  connectionTimeoutMillis: timeout,
  ssl: {
    rejectUnauthorized,
  },
});

/**
 *
 * @param {stirng} theQuery
 * @param {[]]} bindings
 * @param {boolean} withCache true to cache the result
 * @return {Promise<*>}
 */
module.exports.query = async function (theQuery, bindings = [], withCache = false) {
  if (withCache) {
    logger.info(`executing query with cache ${theQuery}`);
    const stringToHash = `${theQuery}${JSON.stringify(bindings)}`;
    logger.info(`string to hash: ${stringToHash}`);
    const hash = crypto.createHash('sha256').update(stringToHash).digest('hex');
    logger.info(`hash: ${hash}`);
    const value = queryCache.get(hash);
    if (value === undefined) {
      try {
        logger.info('no cache for this query, let go to the DB');
        const queryResult = await pool.query(theQuery, bindings);
        queryCache.set(hash, queryResult);
        logger.info(`cache set for ${hash}`);
        return queryResult;
      } catch (error) {
        throw new Error(`Error executing query with cache ${theQuery} error: ${error}`);
      }
    } else {
      logger.info(`returning query result from cache ${theQuery}`);
      log.info(queryCache.getStats());
      return value;
    }
  } else {
    try {
      logger.info(`executing query without cache ${theQuery}`);
      const result = await pool.query(theQuery, bindings);

      // delete all the cache content if we are inserting or updating data
      const auxQuery = theQuery.trim().toLowerCase();
      if (auxQuery.startsWith('insert') || auxQuery.startsWith('update') || auxQuery.startsWith('delete')) {
        queryCache.flushAll();
        queryCache.flushStats();
        logger.info(`the cache was flushed because of the query ${theQuery}`);
      }
      return result;
    } catch (error) {
      throw new Error(`Error executing query without cache  ${theQuery} error: ${error}`);
    }
  }
};

module.exports.execute = pool;

Photo by D.S. Chapman on Unsplash

Tag: cache

Caching large data with Redis

How to use node-cache to accelerate queries

How to use node-cache module to cache queries