Wednesday, July 13, 2011

Fetching Large Amounts of Instrument Data with Matlab

Matlab is a great tool for handling and analyzing measurement data, but you can run into problems when trying to read or fetch large amounts of measurement data from an instrument. Problems range from timeouts, to memory errors, to only a portion of the measurement data being returned. In this post we will look at an easy to implement Matlab code example for fetching and storing large amounts of measurement data.

Let's start out by looking at a high level overview of the easy way to write a quick matlab script for making and fetching measurements with a remote instrument:

  1. Setup and open a connection to the instrument. One of the steps here is setting up the size of the input buffer for reading measurement data.
  2. Configure the instrument for the measurement.
  3. Trigger the measurement.
  4. Fetch the measurement data from the instrument using the query() method and store it in one huge string
  5. Convert the string of data to a number format array like double for post processing and analysis
This method is easy to implement but can lead to problems when dealing with large amounts of data. There are a couple of reasons for this. The first being it does not use memory very efficiently. Notice between the input buffer allocation, the long string that the data is read to, and the final numerical array there are three large chunks memory being allocated for one set of data. This can lead to "Out of memory" errors. Especially in Windows machines where RAM is fragmented between a number of processes and Matlab needs contiguous memory to store a numerical array. Another reason is the various protocol layers that are used in an IO operation can be unstable when reading a large contiguous chunk of data. 

A better and still easy to implement way to fetch large amounts of data is to format it as binary data (versus ASCII) over the IO and to break the data into smaller blocks and reassemble the blocks in Matlab. Lets demonstrate with example Matlab code. For this demonstration we will make 1M double format timing measurements on a 2 GHz oscillator using the Agilent 53230A universal counter. Below we will look at only the code portions that involve reading the measurement data. If you would like a full version of the "Lots_of_Data.m" script used here just email me. 

%The Lots_of_Data script makes 1M timestamp measurements on 
%a 2GHz oscillator using the 53230A. The measurements are read 
%from the instrument in binary form. The measurements are read
%and handled in 500 reading block sizes set the input buffer 
%size in bytes.We want to read in 500 readings per block each 
%reading is 23 bytes including comma. 'obj1' is the handle to the 
%53230A object
buffer = 23 * 500;
set(obj1, 'InputBufferSize', buffer);
%Perform measurement settings and trigger 
%Here we send readings to 53230A output buffer 
%fread()will be used to read all measurements as binary data 
%instead of ASCII characters to cut down on overhead
%get prescale value first and convert to string then to double
prescale = fread(obj1,5);
%allocate double array for faster performance
arrayDouble = zeros(1,1e6);
%read blocks of 500 readings from the 53230A at a time. 
%Convert the binary values to doubles and add to the array 
%of doubles. This method conserves memory versus reading all 
%the data in at once
for i=1:2e3,
tempBin = fread(obj1,(23*500));
 %replace string commas with spaces so we can convert to double array
tempChar = char(tempBin)';
tempChar = strrep(tempChar, ',', ' ');
%replace array of string values to array of doubles
  if i ~= 1
        arrayDouble((((i-1)*500)+1):(i*500)) = strread(tempChar);
      arrayDouble(1:500) = strread(tempChar);

In the above example the 'fread' method was used to fetch the data in binary form instead of ASCII data. This reduces the size of the data being passed over the IO. Also the 1M readings were collected in blocks of 500 that were later reassembled in a numerical array for post processing and analysis. This means we only had to allocate one large chunk of memory instead of three like we did in the first example.

If you know of a better more efficient way to fetch large amounts of measurement data and store it in Matlab please share with a comment below or an email to me. If you are working with extremely large sets of data and get out memory errors check out Matlab's video on optimizing memory usage from the link below. You can also track your Matlab memory usage for optimization purposes using the free Memory Monitor program (link below).


  1. HI Neil,

    Great tip.
    Matlab is a fantastic software package but I find it to much complicated for this type of job. I prefer using a tool like DynaWorks - which is fully dedicated to test data management and processing. There is a large choice of interfaces with the most common instruments and the data are stored directly in a database so you can find them very easily later on. But more important you can gather data coming form different data acquisition systems and display them together, compare them with a click of a mouse, that's so sweet. The displaying capabilities are fully embed in the software package, so you don't need to take care of memory, don't need to write a line a code to display the collected data, a simple double do all of that for you. On the top of these awesome features, you have access a la large library of processing function so you can do what ever you need with your raw data, you can even plug a Matlab script if you need.

    This unique package is so cool that most of the Aeronautic and Space players are using it to gather, display, process and share their test and simulation data over there organization.

    Hopefully that can be useful to some of you guys

  2. شركة نقل اثاث بالدمام التفاؤل شركة نقل اثاث بالخبر كما انها افضل شركة نقل اثاث بالجبيل نقل عفش واثاث بالجبيل والخبر والقطيف والدمام
    شركة نقل اثاث بالدمام
    شركة نقل اثاث بالجبيل
    شركة نقل اثاث بالقطيف

  3. easteldmam east-eldmam شركة نقل عفش بالدمام شركة نقل عفش شركة نقل عفش شركة نقل عفش بالمدينة المنورة

  4. Nice blog. I would like to share it with my friends. I hope you will continue your works like this .Keep up the excellent work. You have a magical talent of holding readers mind.

  5. Thank you for sharing them! I hope you will continue to have similar posts to share with everyone. Jacob Frye Black Coat

  6. I was born in united states but grew up in New York.
    happy wheels, These are probably my two favorite games of all time, and I'm trying my best to make it in the top 1-3 of google.
    geometry dash

  7. If you are not interested in math and you need to write argumentative essay, this is something that you will need for your outline.

  8. I am very glad to find information on how to Fetch Large Amounts of Instrument Data with Mat-lab, information that is very exciting and informative. This is very informative, and I do wish that you can share more. When looking for a shop where to buy the most classy maroon wall to wall carpet, you can talk to us. We sell the most quality and beautiful wallpapers.

  9. Babel account & manag contracting companies
    The program includes 12 specialized systems in contracting field and providing explanation library
    of 145 videos with total of 6 hours of explanation and providing more than 200 questions in the
    field of contracting and group of essays which serve contracting companies.
    برنامج محاسبة مقاولات
    برنامج محاسبة شركات المقاولات
    برنامج مقاولات
    برنامج ادارة شركات المقاولات
    برنامج الاستثمار العقارى
    برنامج استثمار عقارى
    برنامج لشركة استثمار عقارى
    برنامج ادارة محطات الخرسانة
    برنامج محطات خرسانة

  10. Thanks for sharing this piece of information. I really enjoyed it. keep up the good work and all the very best of luck! Women Tan Leather Jacket

  11. Businesses usually execute maintenance at their facility which may cause inconvenience to customers and employees as well if they were not informed prior to the event. Sending out Text Blast for instant alerts can be extremely useful to notify customers and employees to avoid any inconvenience.

  12. I like the style of your writing content manor jessica lowndes trench coat. It just amazing and excellent. The blog post is very connective and attractive. All the information I found in post is very useful.keep it up,you are doing great..

  13. Great Blog! I should need to thank for the endeavors you have made in shaping this post. I am trusting in a practically identical best work from you later on also. I expected to thank you for these districts! A commitment of gratefulness is altogether to share. buy hayley atwell brown jacket online

  14. Very interesting post thanks for sharing Logo and website design will play the most important role in the brand establishment. So Get logo design and website design packages at very affordable prices.

  15. Nice blog thanks for sharing .I am really impressed with your writing abilities

    โปรโมชั่นGclub ของทางทีมงานตอนนี้แจกฟรีโบนัส 50%
    เพียงแค่คุณสมัคร สล็อตออนไลน์ กับทางทีมงานของเราเพียงเท่านั้น
    สมัครสล็อตออนไลน์ >>> Goldenslot
    สนใจร่วมสนุกกับ คาสิโนออนไลน์ คลิ๊กได้เลย
    มีทั้งคาสิโนออนไลน์ หวยออนไลน์ ฟุตบอลออนไลน์ สล็อตออนไลน์ และอื่นๆอีกมากมาย

  16. We value all of our customers and care about their Custom Research Paper Help Services thus, we ensure that we use the right content in writing Affordable Custom Writing Services that will see the client score good grades.