Creating an Air Quality Monitoring Application with Arduino and ESP32 (2) - Data Communication UART

This article is the second part of a series:

1. Introduction to Sensors - DHT11 and MH-Z14A
2. Data Communication - UART (Implementation is done using UART, so we will only discuss UART)
3. Arduino Pitfalls
4. (Upcoming) WiFi Chapter: To save debugging time, I purchased an ESP32 development board, which already includes WiFi and Bluetooth functionality.
5. (Upcoming) MQTT Chapter: To transfer data to other devices, I will use the lightweight MQTT communication protocol.
6. (Upcoming) Grafana / Web Chapter: Since the data is stored in a database, we naturally want to display it in a flashy way! We will use Grafana + Prometheus and Svelte to visualize the data.

Communication is necessary between people, and similarly, communication is needed between hardware devices.

In the realm of networks, the most common protocol is likely TCP. To ensure the accuracy of data transmission, we need a series of mechanisms to guarantee data integrity and that both parties can receive messages. Similar measures are required in hardware communication.

There are three common hardware communication protocols:

• UART (Universal Asynchronous Receiver/Transmitter)
• SPI
• I2C

This article will focus solely on UART.

Data Transmission

Suppose we want to send 10010 to another hardware device. A straightforward way would be to connect 5 wires at once and send 1, 0, 0, 1, 0 simultaneously, as shown in the image below:

Although this seems intuitive, if we have 16 bits, we would need 16 pins; if we need 32 bits, that would require 32 pins. On a circuit, we generally prefer to use fewer pins, much like the popularity of wireless communication, as it simplifies the process and makes manufacturing easier.

So, is there a way to reduce the number of pins used? If both parties agree to send one bit every 100ms, then after 1600ms, we can receive 16 bits, as illustrated below:

Indeed, this method effectively solves the issue of having too many pins, but it converts what was originally synchronous data transmission into "asynchronous" transmission, sacrificing some speed in the process (it takes a total of 1600ms to send 16 bits). In hardware, this one-after-another data transmission method is called "Serial," while synchronous data transmission is referred to as "Parallel."

Today, we will introduce UART as a type of Serial Communication.

Returning to the previous image, if there's only one line and we only agree on how frequently to receive data, it seems insufficient, because under this agreement, we don’t know when the data will arrive or when it will finish. Therefore, we need a signal to let both sides know when to start and stop transmitting data.

UART Data Exchange

In UART, a start bit and a stop bit are defined to indicate to both parties when they will start sending data and when they will stop.

In an idle state, the line remains at a high level, but when data transmission begins, it shifts to a low level to signify the start bit, then the data is transmitted. Each data frame consists of 8 to 9 bits, with the last bit being an optional parity bit. After the data transmission is complete, the line returns to a high level to indicate the stop bit.

Unlike other communication protocols, UART does not have a clock signal to reference, so both parties must know each other's baud rate in advance to determine the speed of data transmission. Why is this number so important? Refer to the image below:

The red section indicates the correct baud rate, where data is correctly read. However, if the baud rate is doubled, the same bit may be read twice (the brown section).

To use UART, the following conditions must be met:

• Both hardware devices must share a common GND.
• Baud rate must be the same.

UART in Arduino

In Arduino, there is a built-in UART Serial Port, typically located on pins 0 and 1, marked as tx and rx. tx indicates transmit, sending data to another device; rx indicates read, receiving data from another device's tx. The RX pin connects to the other device's TX pin, while the TX pin connects to the other device's RX pin.

UART is provided by an independent IC or circuitry within the MCU. However, there is only one UART chip in Arduino. What if more than one device wants to communicate with the Arduino via UART? Or if we want to use the native Serial Port as a Debug Console for easier troubleshooting, without allowing other devices to use it? In our scenario, we want to communicate with the MH-Z14A via UART while also sending data to the ESP32.

This is where Arduino's built-in library — SoftwareSerial — comes into play. Software Serial allows any digital pin on the Arduino to be used as UART.

The term Software implies that this library simulates UART communication through programming, somewhat akin to hardware encoding and software encoding seen in videos.

In the MH-Z14A CO2 sensor we are using, the UART baud rate is 9600. Initially, we planned to implement our own SoftwareSerial to create a UART-compatible communication, but later discovered some issues to share:

• According to the datasheet, the transmission rate for this MH-Z14A sensor is 9600Hz. This means that the time to send each bit is 1/9600 = 1041.6666666µs. However, Arduino's delay can only go up to delayMicroseconds, so we must discard the decimal (due to floating-point considerations, the parameters for delay must always be integers), but such errors can already lead to incorrect data readings.
• The CPU's instruction cycle also takes time, and while this time is short, the cumulative errors during the UART communication process for each bit cannot be ignored.
• We suspect that simply using delay might be affected by other interrupts in Arduino, resulting in inaccurate timing.

The image below clearly illustrates the problems encountered with the delay function:

Ideally, the delay function should be quite accurate, but in reality, the CPU may experience offsets due to compiling and interrupts, as seen in the brownish blocks. We would either have to guess the time taken by the CPU and adjust the offsets or find ways to enhance timing precision. Later, we looked at the source code of SoftwareSerial... and they actually calculated it!

Furthermore, they directly calculated the number of CPU cycles! In hindsight, this approach makes sense; calculating cycles yields the most accurate results. However, this also means we need to write some assembly language to manipulate registers directly.

The MH-Z14A's baud rate is 9600, meaning each bit should take 1/9600 seconds. How many cycles does this translate to for the CPU? To simplify calculations, we set the CPU frequency to 9600Hz, resulting in exactly 1 cycle, calculated as: $\frac{CPU speed}{baud-rate}$.

In Software::begin, you can see the implementation:

void SoftwareSerial::begin(long speed)
{
  // ...

  // Precalculate the various delays, in number of 4-cycle delays
  uint16_t bit_delay = (F_CPU / speed) / 4;

  // 12 (gcc 4.8.2) or 13 (gcc 4.3.2) cycles from start bit to first bit,
  // 15 (gcc 4.8.2) or 16 (gcc 4.3.2) cycles between bits,
  // 12 (gcc 4.8.2) or 14 (gcc 4.3.2) cycles from last bit to stop bit
  // These are all close enough to just use 15 cycles, since the inter-bit
  // timings are the most critical (deviations stack 8 times)
  _tx_delay = subtract_cap(bit_delay, 15 / 4);

  // Only setup rx when we have a valid PCINT for this pin
  if (digitalPinToPCICR((int8_t)_receivePin)) {
    #if GCC_VERSION > 40800
    // Timings counted from gcc 4.8.2 output. This works up to 115200 on
    // 16Mhz and 57600 on 8Mhz.
    //
    // When the start bit occurs, there are 3 or 4 cycles before the
    // interrupt flag is set, 4 cycles before the PC is set to the right
    // interrupt vector address and the old PC is pushed on the stack,
    // and then 75 cycles of instructions (including the RJMP in the
    // ISR vector table) until the first delay. After the delay, there
    // are 17 more cycles until the pin value is read (excluding the
    // delay in the loop).
    // We want to have a total delay of 1.5 bit time. Inside the loop,
    // we already wait for 1 bit time - 23 cycles, so here we wait for
    // 0.5 bit time - (71 + 18 - 22) cycles.
    _rx_delay_centering = subtract_cap(bit_delay / 2, (4 + 4 + 75 + 17 - 23) / 4);

    // There are 23 cycles in each loop iteration (excluding the delay)
    _rx_delay_intrabit = subtract_cap(bit_delay, 23 / 4);

    // There are 37 cycles from the last bit read to the start of
    // stopbit delay and 11 cycles from the delay until the interrupt
    // mask is enabled again (which _must_ happen during the stopbit).
    // This delay aims at 3/4 of a bit time, meaning the end of the
    // delay will be at 1/4th of the stopbit. This allows some extra
    // time for ISR cleanup, which makes 115200 baud at 16Mhz work more
    // reliably.
    _rx_delay_stopbit = subtract_cap(bit_delay * 3 / 4, (37 + 11) / 4);
    #else // Timings counted from gcc 4.3.2 output
    // Note that this code is a _lot_ slower, mostly due to bad register
    // allocation choices of gcc. This works up to 57600 on 16Mhz and
    // 38400 on 8Mhz.
    _rx_delay_centering = subtract_cap(bit_delay / 2, (4 + 4 + 97 + 29 - 11) / 4);
    _rx_delay_intrabit = subtract_cap(bit_delay, 11 / 4);
    _rx_delay_stopbit = subtract_cap(bit_delay * 3 / 4, (44 + 17) / 4);
    #endif

    // Enable the PCINT for the entire port here, but never disable it
    // (others might also need it, so we disable the interrupt by using
    // the per-pin PCMSK register).
    *digitalPinToPCICR((int8_t)_receivePin) |= _BV(digitalPinToPCICRbit(_receivePin));
    // Precalculate the pcint mask register and value, so setRxIntMask
    // can be used inside the ISR without costing too much time.
    _pcint_maskreg = digitalPinToPCMSK(_receivePin);
    _pcint_maskvalue = _BV(digitalPinToPCMSKbit(_receivePin));

    tunedDelay(_tx_delay); // if we were low this establishes the end
  }
  ...
}

While the code is not extensive, the comments explain that they calculated the number of CPU cycles needed for each gcc version and adjusted the delay accordingly.

The implementation of tunedDelay is also interesting, as it is written in assembly language!

I am not familiar with the AVR assembly instruction set (or Intel for that matter XD), but it seems to function as a more precise delay function. Using volatile signals to the compiler that this section of code should not be optimized, as this variable may change, ensuring it is accessed directly from memory each time it is read.

void tunedDelay(uint16_t __count)
{
	asm volatile (
		"1: sbiw %0,1" "\n\t"
		"brne 1b"
		: "=w" (__count)
		: "0" (__count)
	);
}

Then in SoftwareSerial::write, this code:

size_t SoftwareSerial::write(uint8_t b)
{
  volatile uint8_t *reg = _transmitPortRegister;
  uint8_t reg_mask = _transmitBitMask;
  uint8_t inv_mask = ~_transmitBitMask;
  uint8_t oldSREG = SREG;
  bool inv = _inverse_logic;
  uint16_t delay = _tx_delay;

  if (inv)
    b = ~b;

  cli();  // turn off interrupts for a clean txmit

  // Write the start bit
  if (inv)
    *reg |= reg_mask;
  else
    *reg &= inv_mask;

  tunedDelay(delay);

  // Write each of the 8 bits
  for (uint8_t i = 8; i > 0; --i)
  {
    if (b & 1) // choose bit
      *reg |= reg_mask; // send 1
    else
      *reg &= inv_mask; // send 0

    tunedDelay(delay);
    b >>= 1;
  }

  // restore pin to natural state
  if (inv)
    *reg &= inv_mask;
  else
    *reg |= reg_mask;

  SREG = oldSREG; // turn interrupts back on
  tunedDelay(_tx_delay);
  
  return 1;
}

There are a few noteworthy code snippets here:

• *reg |= reg_mask directly manipulates the high and low levels through register operations rather than using the typical digitalWrite() method. I suspect this is to reduce unnecessary instruction cycles post-compilation.
• cli(): This disables the Arduino interrupt mechanism. It can be used when performing atomic operations or time-sensitive actions, but it means that during data transmission, other processes are temporarily halted. I find it puzzling why cli() is used instead of the official noInterrupts() documented function.
• SREG = oldSREG: This description suggests it re-enables interrupts. Why not simply call interrupts() instead?

I won’t explain every implementation detail, but through observing the source code, we discovered several things:

• Directly using delay lacks precision.
• To achieve accurate time control, cycle counts must be calculated directly.
• The interrupt mechanism cannot be used during the communication process.

Now that we understand the implementation details of SoftwareSerial, we can use it with greater confidence!

SoftwareSerial

The API of SoftwareSerial is the same as HardwareSerial (the pre-defined Serial), and based on the code implementation above, it has already handled UART communication for us. So we can simply use Serial.write to send command codes.

One thing to note is that on the Mega and Mega 2560, not every pin can serve as RX.

Not all pins on the Mega and Mega 2560 support change interrupts, so only the following can be used for RX: 10, 11, 12, 13, 14, 15, 50, 51, 52, 53, A8 (62), A9 (63), A10 (64), A11 (65), A12 (66), A13 (67), A14 (68), A15 (69).

Sending Commands to MH-Z14A Using SoftwareSerial

In the previous article, we also mentioned the command codes for the MH-Z14A; let's review them again.

Based on this command table, our primary interest lies in the command code 0x86. However, according to the datasheet, the entire command transmission must adhere to the following format:

Notably, there are a total of 9 bytes. The first byte will be 0xff, the second byte will be 0x01, the third byte is the command code, bytes 3 to 7 are 0x00, and the final value is the check value. The datasheet conveniently includes a sample code for calculating the checksum:

char getCheckSum(char *packet)
{
  char i, checksum;
  for( i = 1; i < 8; i++)
  {
    checksum += packet[i];
  }
  checksum = 0xff - checksum;
  checksum += 1;
  return checksum;
}

The calculation and use of checksum is a significant topic that will be covered in a separate article!

Using SoftwareSerial, we can write the following code:

#include <SoftwareSerial.h>
SoftwareSerial co2Serial(3, 4); // tx, rx pins; any digital pin will work

void sendCommand(byte command)
{
  byte commands[9] = {0xff, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
  commands[2] = command;
  commands[8] = getCheckSum(commands);
  co2Serial.write(commands, 9);
}

This will successfully send the command code to the MH-Z14A! According to the datasheet's return value description, we can use serial.readBytes to retrieve the response data.

byte response[9];
byte high = response[2];
byte low = response[3];
byte ppm = high << 8 + low;
Serial.println(ppm);

According to the datasheet, the ppm value is calculated as high * 256 (which means shifting high left by 8 bits) plus low. We can also check the correctness of the number using byte 8. However, we will skip implementing that here for brevity.

With this, we can successfully obtain the CO2 concentration value! Now let's take a look at the pin diagram of the MH-Z14A:

This sensor offers many pins with the same functionality, likely for debugging convenience. For example, the ground can be connected to pins 2, 3, 12, 16, or 22, giving you multiple options. Here, we will connect GND to the Arduino's GND, RXD to the SoftwareSerial's tx (the pin defined in serial rather than the actual tx pin), and TXD to the SoftwareSerial's rx.

Note: The MH-Z14A sensor does not come with pins soldered when purchased, so it cannot be directly connected to a breadboard. Initially, we used jumper wires, but found it extremely unstable. Please remember to solder the pins!

Related Links

• MH-Z14A Datasheet
• SoftwareSerial API Reference
• UART Wikipedia

Conclusion

In this article, we introduced UART and explained the principles behind SoftwareSerial, ultimately sending command codes to the MH-Z14A to obtain CO2 concentration levels. In the next article, we will share pitfalls encountered during the implementation! Stay tuned.