Wednesday, July 13, 2011

Fetching Large Amounts of Instrument Data with Matlab

Matlab is a great tool for handling and analyzing measurement data, but you can run into problems when trying to read or fetch large amounts of measurement data from an instrument. Problems range from timeouts, to memory errors, to only a portion of the measurement data being returned. In this post we will look at an easy to implement Matlab code example for fetching and storing large amounts of measurement data.

Let's start out by looking at a high level overview of the easy way to write a quick matlab script for making and fetching measurements with a remote instrument:

  1. Setup and open a connection to the instrument. One of the steps here is setting up the size of the input buffer for reading measurement data.
  2. Configure the instrument for the measurement.
  3. Trigger the measurement.
  4. Fetch the measurement data from the instrument using the query() method and store it in one huge string
  5. Convert the string of data to a number format array like double for post processing and analysis
This method is easy to implement but can lead to problems when dealing with large amounts of data. There are a couple of reasons for this. The first being it does not use memory very efficiently. Notice between the input buffer allocation, the long string that the data is read to, and the final numerical array there are three large chunks memory being allocated for one set of data. This can lead to "Out of memory" errors. Especially in Windows machines where RAM is fragmented between a number of processes and Matlab needs contiguous memory to store a numerical array. Another reason is the various protocol layers that are used in an IO operation can be unstable when reading a large contiguous chunk of data. 

A better and still easy to implement way to fetch large amounts of data is to format it as binary data (versus ASCII) over the IO and to break the data into smaller blocks and reassemble the blocks in Matlab. Lets demonstrate with example Matlab code. For this demonstration we will make 1M double format timing measurements on a 2 GHz oscillator using the Agilent 53230A universal counter. Below we will look at only the code portions that involve reading the measurement data. If you would like a full version of the "Lots_of_Data.m" script used here just email me. 

%The Lots_of_Data script makes 1M timestamp measurements on 
%a 2GHz oscillator using the 53230A. The measurements are read 
%from the instrument in binary form. The measurements are read
%and handled in 500 reading block sizes set the input buffer 
%size in bytes.We want to read in 500 readings per block each 
%reading is 23 bytes including comma. 'obj1' is the handle to the 
%53230A object
buffer = 23 * 500;
set(obj1, 'InputBufferSize', buffer);
%Perform measurement settings and trigger 
%Here we send readings to 53230A output buffer 
fprintf(obj1,'FETCH?');
%fread()will be used to read all measurements as binary data 
%instead of ASCII characters to cut down on overhead
%get prescale value first and convert to string then to double
prescale = fread(obj1,5);
str2double(char(prescale))'
%allocate double array for faster performance
arrayDouble = zeros(1,1e6);
%read blocks of 500 readings from the 53230A at a time. 
%Convert the binary values to doubles and add to the array 
%of doubles. This method conserves memory versus reading all 
%the data in at once
for i=1:2e3,
tempBin = fread(obj1,(23*500));
 %replace string commas with spaces so we can convert to double array
tempChar = char(tempBin)';
tempChar = strrep(tempChar, ',', ' ');
%replace array of string values to array of doubles
  if i ~= 1
        arrayDouble((((i-1)*500)+1):(i*500)) = strread(tempChar);
  else
      arrayDouble(1:500) = strread(tempChar);
  end
end

In the above example the 'fread' method was used to fetch the data in binary form instead of ASCII data. This reduces the size of the data being passed over the IO. Also the 1M readings were collected in blocks of 500 that were later reassembled in a numerical array for post processing and analysis. This means we only had to allocate one large chunk of memory instead of three like we did in the first example.

If you know of a better more efficient way to fetch large amounts of measurement data and store it in Matlab please share with a comment below or an email to me. If you are working with extremely large sets of data and get out memory errors check out Matlab's video on optimizing memory usage from the link below. You can also track your Matlab memory usage for optimization purposes using the free Memory Monitor program (link below).



6 comments:

  1. HI Neil,

    Great tip.
    Matlab is a fantastic software package but I find it to much complicated for this type of job. I prefer using a tool like DynaWorks -www.dynaworks.com - which is fully dedicated to test data management and processing. There is a large choice of interfaces with the most common instruments and the data are stored directly in a database so you can find them very easily later on. But more important you can gather data coming form different data acquisition systems and display them together, compare them with a click of a mouse, that's so sweet. The displaying capabilities are fully embed in the software package, so you don't need to take care of memory, don't need to write a line a code to display the collected data, a simple double do all of that for you. On the top of these awesome features, you have access a la large library of processing function so you can do what ever you need with your raw data, you can even plug a Matlab script if you need.

    This unique package is so cool that most of the Aeronautic and Space players are using it to gather, display, process and share their test and simulation data over there organization.

    Hopefully that can be useful to some of you guys

    ReplyDelete
  2. شركة نقل اثاث بالدمام التفاؤل شركة نقل اثاث بالخبر كما انها افضل شركة نقل اثاث بالجبيل نقل عفش واثاث بالجبيل والخبر والقطيف والدمام
    شركة نقل اثاث بالدمام
    شركة نقل اثاث بالجبيل
    شركة نقل اثاث بالقطيف

    ReplyDelete
  3. https://about.me/easteldmam easteldmam
    http://east-eldmam.skyrock.com/ east-eldmam
    http://east-eldmam.mywapblog.com/post-title.xhtml شركة نقل عفش بالدمام
    http://transferefurniture.zohosites.com/ شركة نقل عفش
    http://transferefurniture.hatenablog.com/ شركة نقل عفش
    http://khairyayman.eklablog.com/http-emc-mee-com-transfer-furniture-almadina-almonawara-html-a126376958 شركة نقل عفش بالمدينة المنورة

    ReplyDelete
  4. Nice blog. I would like to share it with my friends. I hope you will continue your works like this .Keep up the excellent work. You have a magical talent of holding readers mind.
    topcelebsjackets.com

    ReplyDelete
  5. Thank you for sharing them! I hope you will continue to have similar posts to share with everyone. Jacob Frye Black Coat

    ReplyDelete